Memoirs of a DoD Red Teamer, Vol 2: The Smallest Team in DoD
Vol 1 ended with me walking into the engineering cell lead seat on the Marine Corps Red Team with a long list of things that needed work. This post is about what that work actually looked like and, more importantly, the conditions we were doing it under. The conditions matter, because they shaped everything about how the team operated, what we could and could not produce, and how I think about red team programs to this day.
What DoD Red Teaming Actually Is
Before I get into the Marine Corps Red Team specifically, it is worth saying what a DoD red team is, because the term is used loosely outside the building.
A DoD cyber red team is a unit accredited and certified to conduct offensive cybersecurity operations against friendly Department of Defense networks for the purpose of assessment and training. The accreditation is not optional. To stand up a red team, you go through a certification and accreditation process run by the National Security Agency with oversight from US Cyber Command. The process is rigorous looking at the paperwork and documentation, the infrastructure and its security, and how everything is run. Verification that you not only have a written policy for how you operate but that you actually follow your own written policy.
I have been through the C&A as the team chief, and it is not pleasant. I have also, since leaving the Marine Corps and while I worked at SpecterOps, done maturity assessments for red team programs in the private sector. The difference between the two is instructive, and I think worth spending a minute on, because I do not think the distinction is widely understood.
I hear it has been updated some but at least at the time the C&A was, in essence, a large compliance inspection and I have opinions on compliance. It was asking whether the team has the right paperwork, the right policies, the right documentation of its environment, and whether it can demonstrate that it operates within the rules it has written for itself. Those are real questions, and they matter, and a team that fails C&A should not be allowed to operate. But passing C&A does not tell you whether the team is actually any good at the work. It tells you that the team is allowed to do the work.
A real maturity assessment is asking a different question entirely. It is asking whether the team is skilled, prepared, repeatable, and continuously improving. It is looking at how the team plans and executes operations. How they handle the moments that did not go to plan and capture those lessons learned. How they develop new capabilities when the threat landscape moves. Whether the team’s work is reproducible by other members of the team or whether it lives in the head of one person. The skill of the individual operators is part of the picture, but it is not the whole picture, and that distinction is important. Talent is not the same as maturity. A talented team that cannot operate without its star is a brittle team. A mature team with average operators can outperform a talented team without process.
The C&A produces a team that is allowed to operate. The maturity assessment produces a team that operates well. Both have value but they are not interchangeable.
How Other Teams Looked
The other thing worth saying up front is that not all DoD red teams are the same. They differ in size, resources, manpower, support structures, and, most importantly, leadership buy-in.
Some teams in the DoD inventory at the time were big enough to have multiple distinct functions inside them. An operations cell. An engineering cell. A separate infrastructure team. Their own internal SOC watching their own operational network. A reporting team. A research and development arm. The kind of structure where the operators showed up to do the operations and the supporting functions were handled by people whose full-time job was to handle those supporting functions.
Some of those teams had real funding behind them. Hardware refresh cycles. Licensing budgets. Travel that did not require begging. Training pipelines that produced new operators rather than burning through the ones already on the team.
Some of them had leadership that understood what they did and provided the cover needed to do it well.
That was not us.
The Marine Corps Red Team
The Marine Corps Red Team, at full strength, was fourteen people. Most of the time we were not at full strength. We were, by a wide margin, the smallest accredited team in the DoD inventory at the time.
There was no separate infrastructure team. The engineering cell handled infrastructure, and the engineering cell was, in practice, a handful of people. There was no internal SOC. We watched our own operational environment because nobody else was going to do it. There was no dedicated research and development arm. R&D happened in the margins of whatever else we were doing, usually at night and usually because something broke during an operation and we needed to fix it before the next operation.
There was no funding to speak of. I want to be clear about this, because I think people from outside DoD often assume that working for the government means you have resources behind you. That is true in some pockets. It was not true for us. To say we did not have funding is an understatement. The hardware we were running our backend on was hardware the Army had decommissioned because it had aged out of their refresh cycle. We had picked it up, racked it, and put it to work. One service’s surplus was our entire operational environment.
The SIEM we used was free and open source. I spent a chunk of my first year as engineering cell lead upgrading that SIEM in place on the Army’s old hardware, because we needed the capability and we did not have the budget to license a commercial alternative. The infrastructure worked. We passed the next C&A inspection cleanly. There was real pride in that, and I am not going to pretend there was not.
I will also say, looking back, that I would have traded some of that pride for more nights at home with my wife and kids. The cost of running a red team on free software and hand-me-down hardware was that the engineering cell, including me, spent a lot of evenings and weekends keeping it running. The team produced. The team passed inspections. The team executed real operations on real networks. We did it because we were stubborn enough not to fail, and because nobody else was going to do it for us. But the math of where those hours came from is something I think about. They came from somewhere. They came from home.
Leadership
The other piece of the picture, and probably the most consequential, was where we sat in the org chart.
The Marine Corps Red Team fell under the Defensive Cyber Operations section. The defenders. The same people whose job it was to keep the network secure and who, organizationally, owned the responsibility for the security posture of the network we were attacking.
Think about what that arrangement does to the incentive structure for a minute.
Every finding the red team produced was, by the org chart, a finding against the boss of the team that produced it. We were sitting under the people whose performance our work reflected on. When we got into something we should not have gotten into, the natural read inside the building was that the red team had embarrassed the defenders. The natural read should have been that the red team had identified a problem with administration and operations, which is what we had actually done. The vulnerabilities were not the defenders’ fault. They were the responsibility of the people configuring and operating the systems, who sat in different sections entirely. The defenders were the ones who would have to respond if an adversary exploited the vulnerabilities, but they were not, in most cases, the ones who had created the vulnerabilities.
The analogy I have used for this, more than once, is borrowed from the fire service. If a house catches fire, the people who show up to put it out are the firefighters. You do not blame the firefighters for the fire. The firefighters did not cause it. The firefighters are the people who showed up to fix the problem that already existed. You blame the conditions in the house, and the person responsible for those conditions, and you give the firefighters credit for putting the fire out.
In that analogy, the blue team and the Cyber Incident Response Team are the firefighters. They are the people who show up when something is on fire and stop it from spreading. Red teams are not the firefighters. Red teams are the fire inspector. We are the people who walk through the house before there is a fire, look at the wiring, the storage of flammables, the placement of the smoke detectors, and the state of the exits, and tell the homeowner where the next fire is going to start if nothing changes. We are upstream of the fire. The firefighters are downstream of it. Both jobs are essential, they are not the same job, and neither of them are responsibly for starting the fire.
The org chart at the Marine Corps Red Team put the fire inspector under the fire chief. The reporting structure made it so that every finding the inspector produced reflected back, by the lines on the chart, on the people responsible for putting out fires. That is the wrong placement for both functions. The blue team’s performance should be measured by how well they respond to fires. The red team’s findings should be read as data about the people responsible for the wiring, which is to say administrators, operators, system owners, and the leadership that resources them. The fire inspector and the fire chief should sit at the same level, both reporting to the person responsible for the safety of the house, not stacked on top of each other where the inspection report looks like a complaint about the fire response.
This is a structural problem. It is not a personality problem. The defenders we worked with were good at what they did, and the friendships I built with them, including with the CIRT chief, were real and remain real. We went to bat for each other repeatedly. We both knew what the work actually was, and we both knew what the org chart was doing to how the work was perceived. The friendships made the situation workable. They did not fix it. The org chart was doing damage that no individual relationship could fully repair. The findings should have been routed somewhere that read them as upstream data about the state of the house. Instead they were routed to the people who would have to run into the burning building, and the reading was predictable.
What the Cell Did
In spite of all of that, the cell worked. I want to spend a small section on what we actually got done, because the conditions I just described could make it sound like nothing got produced, and that is not the case.
We got the phishing program up to a much better standard than where it had been. I will not get into the specifics of how, but the short version is that we modernized the tooling, automated a lot of what had been manual, and increased the volume of campaigns we could run by a significant margin without sacrificing the quality of the individual campaigns. We optimized the team’s own network, which had been more manual than it should have been. We worked on infrastructure consolidation, including the centralized logging capability I have mentioned in other posts. We did all of this while traveling and going on operations, which is the part nobody talks about when they describe a red team’s R&D efforts. The internal work happens in the margins of the external work. It does not stop when the operations start. The operations are the reason the internal work matters.
There was a stretch in there where I was the engineering cell lead, a normal team lead on operations when we were executing, and the operations chief for the broader DCO section at the same time. Three jobs. None of them part-time. The days were long, and the work got done.
The Throughline
The throughline of this post, and probably of the next several, is that small teams with limited resources and difficult organizational placement can still produce good work, but the cost of that work falls on the people doing it. It falls in hours. It falls in personal money spent on infrastructure the institution should have paid for. It falls in time at home, time with family, sleep, and energy for the things outside the building that should matter more than the things inside it.
The Marine Corps Red Team produced. We were proud of what we produced. I am still proud of it. I am also clear-eyed about what it cost, and I think anyone considering a similar role should be clear-eyed about it too.
Vol 3 is about the people who made the cost bearable. The friends in the foxhole. That is the part of this story I am actually looking forward to writing.
This was a stretch of my career working inside the Department of Defense, and the descriptions in this post have been generalized to keep the focus on the experience and the lessons rather than the tradecraft. No personnel beyond those who already publish under their own names are identifiable. No units, dates, locations, techniques, or outcomes are presented in a way that would identify specific systems or people. Where details have been generalized, they have been generalized on purpose.