Generating the Query Is the Easy Part
Detection engineering with LLMs works, but the work lies elsewhere than the demos suggest. Turning natural-language intent into a KQL query shows well in a meeting and is still the smaller and easier part. A capable model knows the Kusto grammar and will produce a syntactically clean query for most detection ideas with little prompting. The larger and harder part is trust: you have to know that the query does what the sentence said, on your logs, before you switch it live. A detection makes a claim about your data that is either true or false, and a false claim costs you asymmetrically and with a delay.
A Plausible Wrong Rule Is Worse Than None
A subtly wrong detection is more dangerous than none, because it fails quietly in both directions. If it is too narrow, it misses the attack, and that false negative produces no error, no alert and no other sign that anything is broken. You discover the blind spot during the incident review, if you discover it at all. If it is too broad, it fires on benign traffic and produces hundreds of false positives, until the analyst who has to triage them starts closing that rule’s alerts on sight. A noisy rule costs time and accustoms a SOC to ignoring a whole category of alert, so that a real one slips past.
In both cases the plausible query makes you less safe while looking as if it made you safer. LLMs readily write output that looks correct, and in detection “looks correct” and “is correct” can diverge for months with nobody noticing. The generator is itself an LLM application handling security-relevant intent, so it raises the same hardening questions as any LLM system exposed to untrusted input. What weighs more than the prompt surface is that a language model by default errs confidently and fluently, and with a detection you cannot afford that.
Where LLM-Written Detections Fail
The failures cluster in a few predictable places, and the validation loop targets those places.
Hallucinated tables and fields
The model references a column that does not exist, such as SourceIP where your table calls it IPAddress, or an EventID on a table that never carried one. If the query then fails to compile you are lucky, because the error is loud and easy to catch. The danger lies in a column that does exist but is almost always empty in your environment, or a table that holds a subtly different event than the model assumed. The query compiles, runs, returns nothing, and reads as “no matching activity”, so nothing distinguishes it from a healthy detection on a quiet day. A cleverer prompt does not fix that. You ground generation in the actual workspace schema and then confirm that every table and field resolves against it.
Off-by-one time windows
Detections compute with time: “N events within M minutes”, “first seen in the last 24 hours”, “a spike against last week’s baseline”. These are the places where a plausible guess goes wrong. A lookback that does not match the rule’s scheduling frequency leaves gaps between runs where a burst slips through. A window boundary that resets mid-attack counts five failed logins as three-then-two and never reaches the threshold. An ago() always counts back from the moment the rule runs; when events reach the workspace late, a tightly sized lookback has already moved past them. None of this looks wrong in the query text. The error shows up only when you run the rule against a known attack that straddles a boundary and it stays silent.
Over-broad matches
Most often a generated rule becomes useless because its scope is too broad. It uses a contains where you needed an exact match, hits on a process name that legitimate admin tooling also uses, or checks an EventID with no filter for the accounts or hosts that would make it meaningful. It fires constantly, the queue floods, and someone mutes the rule. From then on it detects as little as a rule never written and has cost more, because it consumed attention, the scarcest resource a SOC has.
Generate, Validate, Refine
A “prompt → query” architecture is therefore not enough; it takes a loop with independent checks. The multi-agent system we built for BLUEROPE Consult GmbH is built that way. It turns natural-language intent into validated Sentinel detections and takes query development from days to minutes. The loop keeps that speed safe.
One agent generates. It drafts KQL from the intent and grounds it in the real workspace schema rather than the model’s memory of what Sentinel tables usually look like. That prevents most hallucinated fields. A second agent validates. It parses the query, resolves every table and column against the actual schema, and runs it against representative sample data, including events known to be malicious and events known to be benign; this run does most of the work. A rule that stays silent on the known-bad, or fires on the known-good, fails a test that reading the query cannot replace. A third agent works adversarially. It asks what a good detection engineer asks, “how would an attacker evade this, and what benign activity would trip it?”, and turns the answers into refinement instructions. The draft loops until it survives all three. If it cannot, the system escalates to a human instead of quietly shipping something broken.
Multiple agents earn their place because they check independently. The loop checks generation against something the generator cannot argue with: a grammar, a schema, and above all the ground truth in the data. For the same reason, in other settings too, you do not let a model be the sole judge of its own work. Detection engineering fits the pattern unusually well because all three checks exist: the target is a structured language with a grammar, the output is executable, and you have labelled data to run it against. Without the loop you are left with a fast way to generate confident, unverifiable guesses about your security posture.
Where the Detection Engineer Stays in the Loop
The loop does not replace a detection engineer; it changes what they spend the day on. It absorbs the boilerplate, meaning the KQL syntax, the schema lookups and the first three broken drafts, and the engineer spends the time on the parts a model cannot do. The first is the threat model, the decision about what is worth detecting and against which adversary, given what this organisation has worth stealing. A model does not know your crown jewels or who is likely to come for them. The second is tuning. What counts as normal noise is local knowledge, and a rule well-calibrated for one company floods another. A rule also ages: a detection tuned six months ago starts throwing false positives after an infrastructure change, and that is a monitoring problem, not a generation one. The third is sign-off. Someone accountable has to decide the rule is good enough to guard production, because “the agents approved it” is not a sentence you want to say after an incident.