How the validation layer for Sentinel rules works.
A language model writes the detection rule; a deterministic check decides whether it stands. This page shows the pipeline, the errors it catches and what leaves the network along the way.
A wrong rule stays silent
Writing detection rules for Microsoft Sentinel is no longer the bottleneck: Microsoft’s Security Copilot generates the KQL queries, and many companies already hold the licence. The bottleneck sits behind the writing. A rule that looks plausible and is subtly wrong runs without error and never returns a row. It looks clean in review, and in production its silence reads as “no incidents”.
CardinalOps measured this on more than 13,000 rules from several hundred production SIEM environments, Sentinel included: around 13% of rules can never fire (2025 report, summary at Help Net Security). The validation layer starts at exactly this point. It was built for BLUEROPE Consult GmbH, where developing a query shrank from days to minutes.
| Tests | 192, all passing |
|---|---|
| Parser | Kusto.Language — Microsoft’s own library |
| Network access of the deterministic check | none |
| Model | cloud service or local via vLLM |
| Output | rule file, report, exit code |
One run: one rule, two repairs
The validation layer binds every rule to Microsoft’s own parser and additionally checks it against a catalogue of silent errors, meaning patterns with which a valid rule never returns a row.
The screenshot shows the interface of the validation layer with the first draft. The rule calculates with ticks as if they were seconds; the parser accepts it, the check catalogue flags line 7. The first repair swaps the arguments and fails on negative_duration, the second passes the check.
Four checks see the same query
Every generated query is bound against Microsoft’s own KQL parser, the Kusto.Language library that Microsoft publishes itself. The check resolves tables and fields against the workspace schema, not against an assumption. Findings go back to the model as a repair task, and the repaired version runs through the same four checks again.
- 01
Binding against the parser
Syntax, columns and function arguments are checked by Microsoft’s library itself. Validity is therefore decided by the parser and not by a second language model, which would only judge one guess with another.
- 02
The “unverified” mark
Whatever could not be checked against the schema appears in the report as unverified. The check reports its own gaps instead of hiding them behind a green tick.
- 03
Second opinion, advisory only
No parser can prove that the logic hunts the right threat. For that, the system queries a different model in a structured way; the decision stays with the deterministic check and with the human who holds the threat model.
Three errors the parser waves through
All three are valid KQL and run without error, and all three return practically no row in production. Two of them come from a single test run: a rule computed a rate from a duration it took for seconds and that was in ticks. When shown the error, the model corrected the unit and swapped the arguments of the difference in the process, leaving the rule dead again. Since then the check catalogue catches both patterns.
let Admins = dynamic([]); SigninLogs | where TimeGenerated > ago(1d) | where UserPrincipalName in (Admins) | where Location !in ("DE", "AT", "CH") | project TimeGenerated, UserPrincipalName, IPAddress, Location
Constantly false. An empty inclusion list never matches. For exclusion lists the same is correct — the difference is structurally decidable.
SigninLogs | where TimeGenerated > ago(1d) | where ResultType != "0" | summarize FirstSeen = min(TimeGenerated), LastSeen = max(TimeGenerated), Attempts = count() by UserPrincipalName, IPAddress | extend DurationSeconds = todouble(LastSeen - FirstSeen) | where Attempts > 10 and Attempts / DurationSeconds > 0.5
Ticks, not seconds. One second is 10,000,000 ticks; the rate comes out seven orders of magnitude too small and never exceeds the threshold. The variable is still called “…Seconds”.
SigninLogs | where TimeGenerated > ago(1d) | where ResultType != "0" | summarize FirstSeen = min(TimeGenerated), LastSeen = max(TimeGenerated), Attempts = count() by UserPrincipalName, IPAddress | extend DurationSeconds = datetime_diff('second', FirstSeen, LastSeen) | where Attempts > 10 and Attempts / DurationSeconds > 0.5
Computes the second argument minus the third. With the minimum on the left the duration is never positive and the rate never greater than zero. This error appeared while a model was repairing the ticks error: unit fixed, order reversed.
What Microsoft’s kql-guard covers and what it does not
In 2026 Microsoft published kql-guard, an offline analyser for KQL. According to its documentation it checks syntax, reports unknown tables and columns when given a schema, and scores twelve expensive query shapes, from a missing time filter to unbounded mv-expand. It thereby answers whether a query runs and what it costs.
The validation layer answers a different question: whether a running, cheap query can return a row at all. kql-guard does not document the three patterns above, and both tools can run in the same pipeline. We did not run kql-guard ourselves; this assessment rests on its README as of October 2026.
What leaves the network has been measured
The deterministic check needs only the rule and the schema. It opens no connection to the workspace and none to the outside, and it runs entirely inside the operator’s network. The advisory second opinion, by contrast, is a model call and takes the same route as generation.
Generation and the second opinion can stay in-house as well: the system runs against a locally operated model, and switching takes two environment variables. If they run through a cloud service, the requirement text and the column names leave the network. We captured exactly what that covers on a real run.
For financial entities, the DORA datasheet places this mode of operation in its regulatory context.
| HTTP calls | 2 (generation + second opinion) |
|---|---|
| Total volume | 29,032 bytes |
| Content | requirements document + column list |
| System prompt | 6,957 and 3,308 characters — house rules, no customer data |
| IP addresses in payload | none |
| Event timestamps in payload | none |
| Workspace connection | none |
No log data, no events, no production data. The measurement can be repeated at any time with the capture proxy that ships with the system — on the customer’s premises too.
What the check does not prove
The check catalogue knows the patterns we have found. In the run above the third version of the rule passes the check and still carries an error: count and duration are integers, the division truncates, and the rule fires only from one attempt per second instead of one every two. The correct form is todouble(Attempts) / DurationSeconds; the catalogue does not yet have this pattern.
Binding proves that a rule can technically fire. It does not prove that the rule describes the right threat, and it does not replace a test against real attack data. We name this limit because an overpromise would be exactly the kind of plausible-sounding false statement the system is built against.
Questions about the validation layer go to the engineer who built it →
More projects