AI judgment
OIS — opportunity gate
OIS reads public complaint sources around the clock and hands me a short daily list of opportunities worth a closer look. Most of what it finds, it throws away on purpose — and the throwing away is the part I had to think hardest about.
My role
Sole author and operator. I designed the pipeline, wrote the gate prompt and the scoring rubric, built the workflows, and I run it on my own infrastructure. When it breaks, I am the one who fixes it — see below, because it broke.
The decision
The gate is the first stage of a two-stage pipeline: a cheap LLM call deciding whether a raw candidate is worth spending a real scoring call on. Its job is coarse sieving, not quality ranking — and the metric that matters is false-drop rate, not accuracy.
The split is the design. Sieving and ranking are different questions with different tolerances for being wrong, and one scoring pass would have to serve both with a single threshold. So the gate answers only could this be worth a closer look, and a separate, more expensive stage does the ranking on whatever survives.
That split is what makes the metric choice obvious. Accuracy is the default and it is wrong here, because the two errors are neither symmetric nor equally visible. A false pass costs a few cents and one row in a queue the second stage will kill anyway — loud, cheap, self-correcting. A false drop is gone permanently and generates no complaint. Accuracy averages the two into one number that improves as the gate gets stricter, which is exactly the failure I need to detect. So the gate is deliberately over-permissive, and ranking is a separate stage with its own budget.
Proof
Eight OIS workflows run on my n8n instance, all active and scheduled: capture jobs pulling Reddit hourly and Shopify app reviews six-hourly, the gate, the scorer, a tier-two queue and its scorer, a gated-sample job that pulls a limited pool of opportunities back out for review, and a daily digest.
The most inspectable thing I have is an incident. The gate began erroring on 2026-07-23; I found it the next day during an audit, not from an alert. The cause was a timeout chain, not the gate logic: GPU contention on my self-hosted inference host pushed calls to 133–180 seconds against 90-second timeouts downstream — provable because the router in front kept succeeding at those same durations while the gate aborted. Timeouts raised, service restarted, all four stranded items re-gated: three gated out, one passed.
Artifact placeholder — the n8n instance is private. A timestamped workflow export plus the execution log for that incident is what a reader will be able to inspect here.
What it shows I reason about which failure mode matters before optimising.
Honest limitations
There is no eval number, and I am not going to substitute accuracy for one. I have never measured this gate's false-drop rate. The argument above establishes reasoning and architecture, not a measurement — treat "the gate works" as unproven.
The measurement is specified rather than done: a harness that takes a case set, a run function, a scoring function and a metrics config, and returns a confusion matrix plus a false-drop view. Step one is a hand-labelled golden set of items the gate discarded, which does not exist yet. When the number arrives it ships with its test set and at least one failure mode it caught, not as a bare percentage.
And nothing paged me when the gate broke: it failed for roughly seventeen hours before an audit surfaced it — for a pipeline whose defining risk is silent loss, the wrong kind of irony. Alerting isn't built.
Stack
n8n on my own VPS, scheduled triggers, self-hosted LLM inference on a local GPU box behind a router workflow, Postgres/Supabase for the opportunity store.