I've spent 15 years sourcing senior talent — the last few of them embedded with Jess Mah at Mahway and now FutureProofing. I built the sourcing strategy at Shopify Logistics before this. I've sat in 200+ vetting calls watching how Jess interviews engineers — and she has one question that consistently separates real senior AI engineers from imposters in under 5 minutes.
This post is the question, why it works, and how we built our entire 5-stage funnel around the same principle.
The question
Here it is, copy-paste ready:
"Tell me about a production AI failure you owned end-to-end. What broke, what you tried first, what actually fixed it, and what you'd do differently."
That's it. One question. Five minutes. Three categories of answer.
What separates the three categories
Senior engineers answer with debugging stories. Specific, narrative, full of small details. "Our RAG pipeline started hallucinating citations three weeks after launch. I checked the embeddings first — same model, same vector DB. Turned out the chunking strategy had been tuned for the original docs and broke when the PM uploaded a new doc type with longer paragraphs. Re-chunked at the semantic boundary, hallucination rate dropped 80%. What I'd do differently: write a chunk-level eval before shipping the next doc-type expansion."
Mid-level engineers answer with architecture stories. Higher-level, framework-aware, but the specifics drop out. "We had a production issue with our LangChain pipeline. We refactored the agent design and added more observability. The issue went away." Notice: no concrete debugging path, no specific symptom, no "what I'd do differently." Just architecture nouns.
Imposters answer with hypotheticals. "In a situation like that, I would probably check the embeddings first, then look at the chunking..." — they're describing what should be done, not what they did. The tense slips into conditional. They've never actually shipped this.
The signal lands within the first 90 seconds. The remaining time is just stress-testing the story for consistency.
Why it works — the mechanics behind the filter
Most interview questions can be gamed by reading "common interview questions" articles. This one resists gaming for a structural reason: it requires a specific memory of a specific failure. You can't fabricate this without inventing details that contradict each other under follow-up.
Jess pushes on three follow-ups that expose fabrications:
- "What did you check first, and why?" — Real debuggers can describe their hypothesis tree. Fakers default to "the obvious thing" without explaining priority.
- "What was your fastest hypothesis turnaround?" — Real production debugging compresses to 5–15 minute test cycles. Fakers describe day-long investigations because that's what theory feels like.
- "What surprised you when the fix landed?" — Real failures have a "huh, that's unexpected" moment. Fabricated stories are too clean; the fix exactly matches the hypothesis.
If a candidate can't survive these three follow-ups consistently, they haven't actually shipped production AI. Doesn't matter what their resume says.
Why LeetCode is the wrong tool for AI engineers
Most companies still default to LeetCode-style algorithm puzzles for senior interviews. For AI engineers in 2026, this is testing the wrong skill.
| What LeetCode tests | What senior AI engineers actually do daily |
|---|---|
| In-place sorting algorithms | Decide whether to use LangChain or write 30 lines of custom orchestration |
| Tree traversal recursion | Debug vector DB latency spikes |
| Binary search edge cases | Write eval harnesses for hallucination rates |
| Linked list reversal | Tune chunking strategies for retrieval quality |
| O(n log n) optimizations | Pair with Claude Code on a half-broken RAG pipeline |
The skill gap between "great at algorithms" and "great at production AI judgment" is now wider than at any prior point in software hiring history. LeetCode rewards the wrong skill. The 5-minute filter rewards the right one.
How this maps to our full 5-stage funnel
The filter question is Stage 01 of our 5-stage vetting funnel (full breakdown here). We run it during a 30-minute initial screen. It kills 88% of candidates.
The remaining 12% goes through:
- Stage 02 — Technical assessment (90 min async): code review on a real production code snippet with three subtle bugs
- Stage 03 — EQ + behavioral (45 min live): how they communicate when they don't know, how they push back on a wrong PR comment
- Stage 04 — Paired AI challenge (90 min live): real work in Cursor or Claude Code on a half-broken Next.js + LangChain repo
- Stage 05 — Final founder interview (60 min live): Jess runs this herself, plus references that pass smell test
Net funnel: 12 accepted out of 2,000 contacted monthly. ~0.6%. The filter question alone cuts the funnel from 2,000 to 240 in the first 30 minutes — meaning the deeper stages only see candidates worth the senior team's time.
How to apply this in your own hiring
If you're hiring senior AI engineers and not running this question, here's the lift:
- Add it as the second question in any senior AI engineer interview. (First question: warm-up, "tell me about your current role.")
- Set a 5-minute timer mentally. If by minute 5 the candidate hasn't given you specific symptoms, specific debugging steps, and specific outcomes, you have your answer.
- Run the three follow-ups even if the answer sounds good. Fabricated stories collapse under follow-up.
- Calibrate against your seniors. Ask three senior engineers on your team to answer the same question. Their answers are your "what good looks like" baseline.
This isn't a magic question. It's just the highest-signal-per-minute question we've found for senior AI engineers — because the failure mode it surfaces (production AI debugging) is the failure mode the role actually requires.
The deeper point
Jess didn't invent this question. She inherited it from running inDinero — a fintech where production failures cost real money and could land in front of regulators. That environment taught her to interview for what production debugging actually feels like, not for what candidates think production debugging should sound like.
The 13 years between inDinero and FutureProofing didn't change the principle. The substrate did — from accounting software to RAG pipelines and multi-agent orchestrations — but the filter for "real production senior" vs "title-only senior" is the same.
If you'd like to see how we operationalize this across all 5 stages — including the actual paired AI challenge repo we use, scoring rubrics, and the per-stage rejection signals we watch for — the full vetting page is here. Or send a brief describing what you're hiring for, and we'll match you to engineers who already cleared this gate.
— Gabe
FAQ
What is the one question Jess Mah uses to filter senior AI engineers?
She asks: 'Tell me about a production AI failure you owned end-to-end — what broke, what you tried first, what actually fixed it, and what you'd do differently.' Senior engineers answer with specific debugging stories in 2 minutes. Mid-level engineers answer with architecture stories. Imposters answer with hypotheticals. The signal is audible within the first 90 seconds.
Why does Jess Mah avoid LeetCode-style questions for senior AI engineers?
LeetCode tests algorithm recall, not production AI judgment. Senior AI engineers in 2026 spend their day choosing between LangChain and a custom orchestration, debugging vector DB latency spikes, or writing eval harnesses — none of which LeetCode covers. The 5-minute filter question replaces 60 minutes of wasted algorithmic puzzles.
How does this 5-minute filter map to FutureProofing's full vetting funnel?
The filter question is what we run during the Stage 01 initial screen — it kills 88% of candidates inside 30 minutes. The remaining 12% survive into Stage 02 (technical assessment), 03 (EQ + behavioral), 04 (paired AI challenge in Cursor or Claude Code), and 05 (final founder interview). The filter isn't standalone — it's the door to a 5-stage gate that accepts 12 of every 2,000 candidates monthly.