Work / Homeland Security

Case study · CISA, U.S. Department of Homeland Security

A cyber training range that costs nothing when no one is using it

How a single-developer prototype replaced an expensive vendor platform, became the first software-as-a-service product to earn an Authority to Operate from DHS, and gave the agency something it had never had: the true cost of every class.

Short link: fredlackey.com/dhs
$0 while idle

Executive brief

The problem

CISA was paying a vendor roughly $50,000 a month for hands-on cyber training labs that were billed whether or not a class was running, took hours to prepare, and forced every course into the same network.

What was built

A cloud-native range, the Cybersecurity Virtual Learning Environment, that builds an isolated lab for each student on demand, delivers it in a browser tab, and tears it down when the work is done.

The outcome

Idle cost fell to zero. Labs became ready in under thirty seconds. Cost became visible by student, course, month, and quarter. The platform became the first SaaS product granted an ATO by DHS.

FirstSaaS product granted an ATO by DHS and CISA
$0cost while idle, replacing a ~$50,000 monthly bill
< 30 sto a ready lab, down from two to four hours
Per studentcost visibility, also by course, month, and quarter

The situation

The Cybersecurity and Infrastructure Security Agency trains the people who defend federal networks. Much of that training is hands-on: students work inside realistic environments where they configure firewalls, investigate intrusions, analyze logs, and handle live malicious code. Those environments, called cyber ranges, are expensive to run and difficult to get right.

CISA's ranges were supplied by a third-party vendor as a managed service. The agency paid approximately $50,000 every month, and it paid that amount whether a course was in session or the labs were sitting empty between classes. For a training program, that is the worst possible cost profile: demand comes in bursts around scheduled courses, but the bill never changes.

The cost was only part of the problem. Instructors were building their lesson plans around what the platform could do, rather than around what students needed to learn.

Why the existing approach failed

The vendor platform was built on large, pre-built virtual machine images. Every course received the same network: domain controllers, file servers, and other infrastructure, whether or not the lesson needed them. That design produced four problems that compounded each other.

  • Always-on billing. Fixed infrastructure was paid for continuously, regardless of use.
  • Slow preparation. Because the images were so large, preparing a single student's environment took two to four hours. Students waited, and exercises were deferred.
  • One network for every course. Instructors could not shape an environment around a specific lesson, tool, or threat scenario without an expensive reconfiguration.
  • No visibility and no exit. The agency could not see usage, attribute cost, or audit behavior, and the proprietary design offered no path to extend or migrate the platform.

Each problem pointed at the same root cause. The labs were treated as permanent machines to be rented, when they should have been temporary environments to be built.

The approach and the decisions behind it

I proposed replacing the rented machines with labs defined entirely as code, built for each student when needed and destroyed when finished. I wrote the first version of the platform myself, from the provisioning engine to the dashboards, to prove the idea before asking anyone to staff it. A handful of early decisions shaped everything that followed.

Build labs from code, not from snapshots.
Every environment is a version-controlled Terraform template. A lab is assembled from exactly the parts its course needs, which makes it small, fast to build, and identical every time.
Make idle free.
If labs exist only while someone is using them, an empty week costs nothing. Shutting services down became a routine, zero-cost action rather than a decision anyone had to weigh.
Give every course its own range.
Instead of one network for all courses, each course runs in its own isolated range: a Windows enterprise with Active Directory and Exchange for one class, industrial firewalls for another, a SIEM and log pipeline for a third.
Deliver everything through the browser.
Students reach Linux and Windows desktops through Apache Guacamole in a single browser tab. There is nothing to install, no VPN, and no credential for a student to mishandle.
Choose portability over convenience.
Several AWS-native services were used during development but deliberately kept out of the product, so the platform could be deployed to AWS GovCloud or Azure Government and operated by agencies with strict service restrictions.
Put instructors in control.
Self-service dashboards let instructors launch, monitor, and retire labs without filing a request with an engineering team or a vendor.

How it works

From a student's point of view, the experience is a single action: open the course and click to launch the lab. Behind that click, the platform does the following.

How the cyber range platform works Students launch labs from the student dashboard and instructors manage them from the admin dashboard. Both use the management API, which places requests on a provisioning queue. The range processor reads the queue and uses Terraform to build each student's lab inside an isolated course range, or hands over a lab from the warm standby pool. Each range contains a browser gateway, the student's machines, and controlled outbound access. Desktops are streamed back to the student's browser. Every allocation is recorded in a cost and usage ledger that feeds the admin dashboard. StudentsInstructors in the courseand operators Student dashboard Management APINode.js · PostgreSQL Admin dashboard Provisioning queue Range processorTerraform Warm standby poolready labs Cost and usage ledgerevery allocation Course range isolated network Browser gateway Student machines Controlled egress desktop streamed to the student's browser
Figure 1. The platform's moving parts. Every component is defined as code, from the cloud accounts down to each student's lab. Swipe the diagram sideways to see all of it.
  1. The student launches the lab from inside the course. The platform checks whether the student already has a lab and, if so, reconnects them to it where they left off.
  2. A lab is assigned or built. If a ready lab is waiting in the warm standby pool, it is assigned immediately. Otherwise the request goes on a queue, and the range processor assembles and applies the Terraform for that student, typically in under thirty seconds.
  3. The lab is isolated. Every student gets dedicated machines and a dedicated security boundary, named and tracked by a unique 32-character identifier. Outbound network access is decided at the infrastructure level, which is what makes it safe to study real malware.
  4. The desktop arrives in the browser. Randomized credentials are generated and injected directly into the gateway. The student never sees or handles them.
  5. The lab is released. After inactivity or at the end of the course, it is returned to the pool or destroyed, and its cost stops.

The system is organized in four layers, each deployed as code and each depending on the one beneath it: the cloud organization and identity controls, a central control plane, the application platform, and the individual course ranges. Every network, platform and range alike, follows the same five-tier subnet pattern, so security reviewers evaluated one design rather than dozens.

Making cost visible

The financial change was not a discount on the old bill. It was a different cost model. Under the vendor platform, cost was a fixed monthly figure with no breakdown. Under the new platform, nothing is billed while nothing is running, and every running lab is recorded from the moment it is provisioned to the moment it is released.

Figure 2. What the agency pays when no class is in session.
Figure 3. The new cost breakdown, from the quarter down to a single student. Before, only the monthly total existed. Proportions are illustrative.

For the first time, the agency could answer questions it had never been able to ask: what a particular course costs to deliver, what one student's lab cost over a multi-day class, how spending moved month to month and quarter to quarter. Because each allocation carries its own running cost, budget owners could forecast the next quarter from real usage and defend the figure with data.

The visibility changed behavior as well. Instructors began sizing labs deliberately, and because shutting a service down cost nothing to reverse, no one hesitated to do it.

The hard problems, and how they were solved

Earning an Authority to Operate

Nothing runs inside DHS without an Authority to Operate, and no SaaS product had ever been granted one. I wrote the security documentation, architecture diagrams, and compliance evidence myself. The design helped make the case: hardened container images from the Department of Defense's Iron Bank registry, no credentials exposed to users, strict isolation between students and courses, controlled outbound access, and central audit logging. The platform became the first SaaS product to receive an ATO from DHS and CISA.

Working within GovCloud's limits

AWS GovCloud is deliberately restrictive. It cannot host public DNS zones, and it limits access from ordinary tooling. Public DNS was placed in a commercial account and pointed into GovCloud, and all infrastructure changes run through a dedicated, access-restricted executor inside the GovCloud management account. DNS records are created in a second phase, only after the infrastructure they point to is confirmed to be running.

Many students starting at once

Classes start on a schedule, so demand arrives all at once. Provisioning requests are placed on a durable queue with a fast cache in front, which allows many labs to be built in parallel without blocking the interface. A warm standby pool absorbs the first wave so that most students never wait at all.

Keeping credentials away from students

Credentials are the usual weak point in training environments. Here they are randomized per lab, injected directly into the browser gateway, and never shown to the student or stored in the student dashboard. Access URLs are tied to the student's unique identifier and cannot be bookmarked or reused.

Course content without version control

The agency's learning system, Moodle, assumes every edit happens directly in production. It has no staging, no versioning, and no rollback. The management system added a content pipeline so course authors could develop, test, and promote material safely across environments.

Sizing the browser gateways

Remote-desktop gateways are memory-intensive, and courses differ widely in what they ask of them. Capacity was set per course rather than globally, and reduced for the heaviest ranges when monitoring showed memory pressure, so a demanding class could not degrade a lighter one.

Outcomes

The rented platform was retired. In its place, CISA had a system it controlled, whose cost it could see, and which it could shape around its courses.

  • The idle bill disappeared. The roughly $50,000 monthly charge was replaced by a model in which an idle range costs nothing.
  • Waiting disappeared. Lab preparation fell from two to four hours to under thirty seconds, and to effectively zero for students served from the standby pool.
  • Cost became a management tool. Spending could be traced by student, course, month, and quarter.
  • Courses became faster to build. New courses could be launched in days rather than weeks, each with an environment designed for its subject. The platform has hosted nine distinct ranges, from Windows enterprise defense to industrial control systems.
  • The platform supported hundreds of concurrent students across isolated environments.
  • It opened doors beyond the project. Its success contributed to the consulting firm that delivered it being awarded a follow-on federal contract worth more than $100 million.

My role

I invented the platform and built the original version single-handedly: the provisioning engine, the Terraform templates, the APIs, and the dashboards. Once it was proving itself, I brought in a DevOps engineer, then gradually grew a small team. Together we matured the invention into the platform the agency relies on.

Along the way I led the team's agile practice, trained engineers who were new to AWS, brought in a designer for the learning-system experience, and wrote the ATO package. I worked directly with instructors to turn learning objectives into lab designs, served as the primary technical contact for DHS, CISA, and FEMA stakeholders, and worked with the consulting firm's chief technology, information, and operating officers on the platform's direction.

This is everything that I’ve dreamt of. It’s perfect.
CISA leadership, on first seeing the platform

Next step

What this would mean for a similar organization

Most organizations that run hands-on training, labs, or sandboxes face the same pattern: environments that are expensive to keep running, slow to prepare, and impossible to cost accurately. The specific technology matters less than the principles behind it: environments defined as code, nothing billed while idle, isolation by default, access through the browser, and cost recorded at the level of the individual user.

If your organization is building something similar, or has an effort that has stalled, I am glad to compare notes on where it stands and what I would do next.

Start a conversation