Generating the Query Is the Easy 20%
Detection engineering with LLMs works, but not where the demos put the emphasis. Turning natural-language intent into a KQL query — the part that shows well in a meeting — is the easy 20%. A capable model knows the Kusto grammar and will produce a syntactically clean query for most detection ideas with little prompting. The hard 80% is trust: knowing the query does what the sentence said, on your logs, before it goes live. A detection is not a paragraph you read and nod at. It is a claim about your data that is either true or false, and the cost of a false one is both asymmetric and delayed.
A Plausible Wrong Rule Is Worse Than None
A detection that is subtly wrong is more dangerous than no detection at all, because it fails quietly in both directions. If it is too narrow, it misses the attack — a false negative that produces no error, no alert, no sign that anything is broken. You discover the blind spot during the incident review, if you discover it at all. If it is too broad, it fires on benign traffic — a false positive that does produce a signal, hundreds of them, until the analyst who has to triage them starts closing that rule’s alerts on sight. A noisy rule does not merely waste time; it trains a SOC to ignore a whole category of alert, which is exactly how a real one slips past.
Either way, the plausible query has made you less safe while looking like it made you safer. That is the specific trap here: LLMs are extremely good at producing output that looks correct, and detection is a domain where “looks correct” and “is correct” can diverge for months with nobody noticing. The generator is itself an LLM application handling security-relevant intent, so it also inherits the usual hardening concerns of any LLM system exposed to untrusted input — but the deeper issue is not the prompt surface. It is that confident, fluent wrongness is the default failure mode of a language model, and a detection is precisely the artefact where you cannot afford it.
Where LLM-Written Detections Fail
The failures are not random. They cluster in a few predictable places, and a validation loop is built around knowing exactly where.
Hallucinated tables and fields
The model references a column that does not exist — SourceIP where your table calls it IPAddress, or an EventID on a table that never carried one. When the invention fails to compile you are lucky: the error is loud and easy to catch. The dangerous version is a column that does exist but is almost always empty in your environment, or a table that holds a subtly different event than the model assumed. The query compiles, runs, returns nothing, and reads as “no matching activity” — which is indistinguishable from a healthy detection on a quiet day. The fix is not a cleverer prompt. It is grounding generation in the actual workspace schema and then confirming that every table and field resolves against it.
Off-by-one time windows
Detection is temporal: “N events within M minutes”, “first seen in the last 24 hours”, “a spike against last week’s baseline”. These are exactly where a plausible guess goes wrong. A lookback that does not match the rule’s scheduling frequency leaves gaps between runs where a burst slips through. A window boundary that resets mid-attack counts five failed logins as three-then-two and never trips the threshold. An ago() measured from the wrong anchor shifts the whole window. None of it looks wrong in the query text. It shows up only when you run the rule against a known attack that straddles a boundary and watch it stay silent.
Over-broad matches
The most common way a generated rule becomes useless is scope. A contains where you needed an exact match; a hit on a process name that legitimate admin tooling also uses; an EventID with no filter for the accounts or hosts that would make it meaningful. It fires constantly, the queue floods, the rule gets muted — the same outcome as never writing it, reached more expensively. Over-broad rules are worse than they look, because their damage is to attention, the scarcest resource a SOC has.
Generate, Validate, Refine
This is why the architecture that matters is not “prompt → query”. It is a loop with independent checks — and it is the shape of the multi-agent KQL generator we built for Bluerope Consult GmbH, a system that turns natural-language intent into validated Sentinel detections and takes query development from days to minutes. The speed is the headline. The loop is the reason the speed is safe rather than reckless.
One agent generates: it drafts KQL from the intent, but grounded in the real workspace schema rather than the model’s memory of what Sentinel tables usually look like — which removes most hallucinated-field errors before they can happen. A second agent validates: it parses the query, resolves every table and column against the actual schema, and — the part that does the real work — runs it against representative sample data, including events known to be malicious and events known to be benign. A rule that stays silent on the known-bad, or fires on the known-good, has just failed a test that reading the query never could. A third agent is adversarial: it asks what a good detection engineer asks — “how would an attacker evade this, and what benign activity would trip it?” — and turns the answers into concrete refinement instructions. The draft loops until it survives all three, or until it cannot, at which point it escalates to a human instead of shipping something broken quietly.
The point of multiple agents is not novelty; it is independence. The value comes from checking generation against something the generator cannot argue with — a grammar, a schema, and above all a ground-truth oracle in the data. This is the same reason, in a different setting, that you do not let a model be the sole judge of its own work; it is the core caution in evaluating LLM systems honestly. Detection engineering happens to be an unusually good fit for the pattern because all three checks genuinely exist: the target is a structured language with a grammar, the output is executable, and you have labelled data to run it against. Skip the loop and you are left with the opposite — a fast way to generate confident, unverifiable guesses about your security posture.
Where the Detection Engineer Stays in the Loop
None of this replaces a detection engineer; it changes what they spend the day on. The loop absorbs the boilerplate — the KQL syntax, the schema lookups, the first three broken drafts — so the engineer’s time moves to the parts a model genuinely cannot do. The threat model is the first: deciding what is worth detecting, against which adversary, given what this specific organisation actually has worth stealing. A model does not know your crown jewels or who is likely to come for them. Tuning is the second: what counts as normal noise is local knowledge, and a rule well-calibrated for one company floods another. It also decays — a detection tuned six months ago starts throwing false positives after an infrastructure change, which is a monitoring problem, not a generation one. And sign-off is the third: someone accountable has to decide the rule is good enough to guard production, because “the agents approved it” is not a sentence you want to say after an incident.
Where Tippel Fits
If you run a Sentinel or SIEM environment and the bottleneck is the hours your engineers spend hand-writing and debugging detections rather than thinking about what to detect, that is precisely the boilerplate an LLM loop can take off their desk — without handing your detection quality to a model on trust. The method holds whether you are an MSSP maintaining rules across many tenants or a mid-sized security team looking after your own. What it looks like as a piece of engineering work is deliberately unglamorous: start narrow, wire the validation oracle up to your real schema and a representative sample, and prove the loop ships correct detections before you scale it.
If that is the problem you actually have, the AI Readiness Check is the cheapest way to find out whether it is worth building in your environment, with an honest go/no-go at the end rather than another convincing demo. If you would rather talk it through first, get in touch.