How AI engineering interviews are structured
A typical loop has four kinds of rounds: a fundamentals screen, a system design round, a practical or coding round, and a behavioural round. The weighting has shifted in recent years. As more value moves into building on top of foundation models rather than training them, AI engineering interviews now spend more time on application architecture — retrieval, agents, evaluation, cost, and latency — and less on deriving math. You still need the fundamentals, but you need them as intuition you can explain, not as proofs you can reproduce.
This guide walks the four rounds and shows what a strong answer sounds like in each, with the recurring questions you should rehearse.
Fundamentals and LLM internals
Expect questions that test understanding, not memorisation. What is an embedding, and why is cosine similarity used? Explain that embeddings map text to vectors where distance reflects semantic similarity, and cosine similarity compares direction regardless of magnitude. What does the attention mechanism do? Give the intuition — each token weighs how much every other token matters to it — without drowning in matrix notation. What is the difference between temperature and top-p? Both control randomness; temperature scales the distribution, top-p truncates it to the smallest set of tokens above a cumulative probability. Why do models hallucinate? Because they predict plausible continuations, not verified facts; grounding and evaluation mitigate it. The pattern in every good answer is concept first, then a one-line example, then a tradeoff.
System design and the practical round
This is where AI engineering interviews are won. A canonical prompt is “design a RAG system over our internal documents.” Structure your answer: ingestion and chunking, embedding and a vector store, retrieval with reranking, prompt assembly with the retrieved context, generation, and — critically — evaluation and monitoring. Talk about caching repeated queries, controlling cost by choosing model tiers, enforcing latency budgets, and adding guardrails so the model cites sources and refuses when context is missing. The candidates who stand out always raise evaluation unprompted: “I would build an eval set so we can tell whether any change actually improves answer quality.” For an agent design question, add tool definitions, a planning loop, a step limit to prevent runaway cost, and failure handling when a tool errors.
The practical round may ask you to write a retrieval function, wire up a model call, or debug a prompt that produces bad output. Narrate your reasoning, handle errors explicitly, and mention how you would test it.
Tradeoffs and behavioural rounds
Be ready to reason about fine-tuning versus prompting versus RAG: prompting is cheapest and fastest to iterate, RAG injects fresh or proprietary knowledge, fine-tuning changes behaviour and style but is costly and slow to update — and the honest answer usually starts with “try prompting and RAG first, fine-tune only when those plateau.” Expect cost questions too: how would you cut the bill on a feature that calls a model thousands of times a day? (Caching, cheaper models for easy cases, batching, shorter prompts, and hard spend guardrails.)
For behavioural rounds, prepare concrete stories: an AI feature that misbehaved in production and how you measured and fixed it; a time you chose not to use a model; how you justified an AI feature’s cost. Lead with the measurable outcome. The whole interview rewards the same habit — pair every claim with a tradeoff and a way to measure it.