Control Reliability Engineering for Security Controls
Run security controls like production systems: set coverage targets, catch drift early, test controls monthly and report control health to your board.
Loading...
Run security controls like production systems: set coverage targets, catch drift early, test controls monthly and report control health to your board.
What AI-driven security solutions really do: alert triage, behavior-based detection, risk ranking and automated response, and how to test the AI claims.
Most security programs assume their controls work. You buy the product, configure it, pass the audit and move on. Six months later it's still in the asset inventory and the license still renews, but nobody has checked whether it's doing anything.
Engineering teams keep production systems reliable with SLOs, error budgets and failure mode analysis, and most security programs have nothing like that for their controls. Once a control is in place, nobody asks whether the EDR agent is still running on 94% of endpoints or has slipped to 61%, or whether the SIEM correlation rules written in 2021 still fire after three cloud migrations. We assume they do, and that assumption is how programs fail quietly.
Control reliability engineering means borrowing the habits of site reliability engineering and using them on security controls. I don't count a control as working until someone has checked that it works, and I want each one to have an uptime target, an alert when it degrades and a known way to bring it back. Auditors may like that, but I care about it because the real risk sits there.
Browse the Full Cybersecurity Market. Every Category.
Every category on CybersecTools, from AI Security and Cloud Security to Threat Intelligence and Offensive Security. Filter by use case, industry, or company size.
Controls degrade for ordinary reasons. Agents get removed during OS upgrades, and firewall rules pick up exceptions until the original policy is mostly holes. A network change silences a log source and nobody updates the SIEM, while cloud workloads spin up outside the provisioning pipeline and never get enrolled in DLP or endpoint controls. Nobody does any of this on purpose, which is exactly why nobody notices.
Most programs also have no way to see it happening. Your quarterly access review tells you who has access, but it won't tell you whether PAM is enforcing session recording on 100% of privileged accounts or on 73%. That gap stays invisible until an incident exposes it.
SRE teams already solved a version of this for production systems. They never assumed reliability would stay put after launch, so they set SLOs, track error budgets, run chaos experiments and write runbooks for the ways things break. Each of those has a direct match on the security side:
Teams that work this way find their control failures during scheduled validation instead of in the middle of an incident.
None of this works without a baseline, and most programs don't have one. They have a list of products and a general sense that things are set up right, when what they need is a written, measurable statement of what "working" means for each control.
For each critical control, write down three things. The first is the coverage target, meaning the share of in-scope assets the control should cover. The second is how often you check it, and the third is the escalation threshold where drift becomes a risk executives need to hear about. For a program with 500 to 5,000 endpoints, this is a practical place to start:
| Control | Coverage target | How often you check |
|---|---|---|
| EDR coverage | 98% of managed endpoints | Weekly |
| MFA enforcement | 100% of privileged accounts | Daily, from IdP reports |
| Vulnerability scan coverage | 95% of in-scope assets scanned within 7 days | Monthly |
| Log ingestion completeness | 99% of defined log sources reporting within 24 hours | Daily |
| Backup integrity | 100% of critical systems with a tested restore in the last 90 days | Quarterly |
Most security dashboards show threat activity, like alerts fired, incidents opened and vulnerabilities found. That tells you what's happening now but says little about whether your controls are in a position to catch the next thing. A control health dashboard shows coverage instead of event counts. It answers what share of in-scope assets each critical control protects right now and how that has changed over the last 30 days. If EDR coverage drops from 97% to 89% over three weeks, you want to see that before an attacker does.
Build it in whatever BI or GRC product your team already uses. The data usually exists already in your endpoint management platform, your IdP, your SIEM and your vulnerability scanner, so the work is pulling it together and showing it. Give the dashboard one owner, one person who answers for it when a metric goes red, because a committee won't. If you'd rather buy than build, look at continuous controls monitoring products.
Compliance testing asks whether a control exists and is configured to policy. Validation asks whether it actually stops a realistic attack, and the two questions often have different answers.
Your annual penetration test is mostly compliance testing with extra steps. It runs once a year, it's scoped to avoid disruption, and the report gets filed. It tells you very little about whether your controls work on an ordinary Tuesday in March when nobody is watching.
Continuous validation means testing your controls on a regular schedule. Breach and attack simulation (BAS) products do this at scale, but you don't need one to start. Once a month you can send a test phishing email through your email security stack, try to run a known-bad file on an endpoint, and trigger a SIEM rule with synthetic log data. Write down the results and track the pass rate over time, and you have a validation program.
Compare Cybersecurity solutions side by side
Features, integrations, NIST coverage and deployment for every Cybersecurity solution we list, in one table.
Production systems have owners, so someone is on the hook when a system goes down. Security controls rarely work that way. The SOC "owns" the SIEM in a general sense, yet nobody personally answers for keeping log ingestion above 99%.
Fix that with a control ownership model. For each critical control, name one person as owner, never a team. That person owns the coverage target, the validation schedule and the escalation path when drift shows up, and they report control health at your monthly security operations review.
The conversation changes once you do this. Instead of "the SIEM is fine," you hear "SIEM log ingestion is at 96.4% this week, down from 98.1% last week, and here are the three log sources that went quiet after Tuesday's network change." That level of detail is what lets a program catch problems early instead of finding them during an incident.
Boards don't need to know how controls work. They need to know whether the organization's risk is getting better, staying the same or getting worse, and control reliability metrics answer that directly.
A board summary has three parts: current coverage across critical controls as a percentage, the trend over the last quarter, and any control below its threshold along with a date to fix it. In practice it can sound like this: "We have 12 critical security controls. 10 are at or above target. Two are below: DLP coverage dropped after the cloud migration, and backup testing slipped during the Q3 hiring freeze. Here's the plan and the timeline." That's specific and true, and it gives the board something real to ask about. For more on what lands with boards, see security metrics your board cares about.
Most security budgets pay to buy controls and nothing to keep them reliable. You buy the EDR platform but don't budget for quarterly agent audits, the yearly configuration review or monthly validation, so that work gets squeezed out by incidents and projects.
A model that works: for every major control, set aside 15 to 20% of its yearly license cost for reliability work. That pays for the staff time to watch coverage, run validation and maintain configuration. If you spend $200,000 a year on EDR, budget $30,000 to $40,000 of staff time to keep it at target coverage, which is simply what it costs for the control to actually work.
When you bring this to the CFO, frame it as insurance on money already spent. You paid $200,000 for a control, and the reliability budget makes sure it delivers the risk reduction you bought it for. If you're also under budget pressure, how micro-cuts wear down security programs covers how to defend this line.
Check a vendor before you sign
Mandos tracks funding, acquisitions, hiring and news for security companies. Look up any vendor on your shortlist before the renewal.
Control reliability engineering isn't a new category of spending. It's a habit you apply to the controls you already own. Most programs have the products and lack the routine to check that they work, and that gap is where incidents happen. Setting coverage targets, naming owners, running validation and reporting drift to leadership costs less than a single incident response engagement. Start with your three most critical controls, define what working looks like, then measure it and report it. For the opposite pattern, controls that exist mainly for show, read ceremonial security controls.
Frame it as protecting money already spent. If your board approved $500,000 in security products last year, the reliability budget is what makes sure that $500,000 actually reduces risk. Most CFOs already accept maintenance budgets for equipment, and security controls work the same way. Start small, with one FTE or 20% of an existing person's time on coverage monitoring and validation, and show the output in your next board report.
A basic program with coverage targets, a dashboard and named owners can run within 60 to 90 days for a team of 5 to 15 people. Spend the first 30 days defining baselines and assigning owners, and days 30 to 60 building the dashboard and running the first validation cycle. By day 90 you have a month of trend data, which is enough to show the board something real.
This is a team design question as much as a process one. The work needs someone who thinks operationally and can turn metrics into risk language. If your team is all technical specialists, consider giving coverage monitoring to a security operations analyst or someone close to GRC. The work isn't deeply technical, since most of it is steady measurement and escalation.
Use a traffic light table with three columns: control name, current coverage and trend. Red means below threshold with a fix planned, yellow means close to the threshold, and green means normal. A board can read it in 90 seconds. Add one short paragraph on each red item and you're done.
Rebuild baselines from scratch instead of carrying the old ones over. Within 30 days of finishing the migration, run a full coverage audit of every critical control against the new environment, and expect gaps. Cloud migrations nearly always break agent deployments, log source settings and network-based controls. Write down what you find, rank it by risk, and set new targets for the environment you actually have now.
BAS platforms automate the validation part of a reliability program. They keep testing whether your controls respond to known attack techniques, which manual validation can't do at scale. The tradeoff is cost, so get quotes before you plan around one. For teams under 20 people or security budgets under $2 million, a steady manual validation schedule often gives more value per dollar. You can compare options in the threat simulation category.
I wrote this guide from my own work leading security programs. Product and category links point to the CybersecTools catalog of 9,168 security products. Read the methodology for how we review products.