AI/ML Engineer Interview Guide: Technical & System Design

The ML engineer interview has split in two, and knowing which half you're walking into matters enormously. One track is the classic: modelling depth, evaluation rigour, and ML system design. The newer track is applied AI: retrieval-augmented generation, evaluation of LLM systems, prompt and context engineering, and shipping AI features that don't embarrass anyone in production. Senior roles increasingly test both — the fundamentals plus the new stack.

The core idea: An ML engineer is hired to make models work in production, not in a notebook. Interviews test the full lifecycle: data, training, evaluation, deployment, monitoring — and where each stage lies to you.

The question taxonomy

• ML fundamentals. Bias-variance, regularisation, metrics, leakage. Non-negotiable — the ML fundamentals guide covers it.
• ML system design. "Design a recommendation system" — data pipelines, feature stores, training/serving skew, feedback loops.
• Applied AI engineering. RAG architecture, chunking and embedding choices, evaluating LLM outputs, guardrails, latency and cost trade-offs.
• MLOps. Versioning data and models, CI/CD for models, monitoring for drift, rollback when the new model is worse.
• Coding. Real programming, usually Python — data manipulation, pipeline code, sometimes classic algorithms.

Worked example: the RAG design question

"Design a system that answers questions over our documentation." The shape of a strong answer:

# Retrieval-augmented generation, in five decisions:
# 1. Chunking: split docs by semantic boundaries, with overlap,
#    sized to fit meaningfully in the context window.
# 2. Embedding + vector store: embed each chunk; store with metadata
#    (source, section, date) for filtering and citation.
# 3. Retrieval: top-k similarity search; consider hybrid
#    (keyword + vector) for terminology-heavy docs.
# 4. Generation: stuff retrieved chunks into the prompt with
#    instructions to answer only from context and cite sources.
# 5. Evaluation: a labelled question/answer set, scored for
#    faithfulness and relevance — before and after every change.

Then volunteer the hard parts before being asked: retrieval quality is usually the bottleneck, not the model; evaluation is the part everyone skips and everyone regrets; and freshness — re-embedding when docs change — is an operational problem, not a modelling one. Those three sentences signal production experience.

How answers get scored

Rubrics reward evaluation-first thinking: candidates who ask "how will we know it's working?" before "which model should we use?" stand out immediately. In system design, handling feedback loops (the model's outputs becoming future training data) and training/serving skew separates theorists from practitioners. Coding rounds expect fluency with pandas and clean, testable pipeline code — not notebook spaghetti.

Common mistakes

• Reaching for fine-tuning when better retrieval or prompting would solve the problem cheaper and faster.
• No evaluation story for an LLM feature — "we'll eyeball it" is not a plan an interviewer can approve.
• Ignoring latency and cost: an answer that's right in eight seconds and wrong on budget fails in production.
• Treating the model as the system. The model is a component; the pipelines around it are the job.

The evaluation conversation you must lead

For LLM-powered features especially, the strongest candidates drive the evaluation discussion unprompted. The arc: build a small labelled set of representative inputs with expected behaviours before you ship anything; score both faithfulness (did the answer stick to the retrieved sources?) and task success (did it actually help the user?); automate what you can, but keep a human review loop for the failures that matter; and re-run the suite on every prompt, model, or retrieval change, because regressions in these systems are silent. Know the failure taxonomy well enough to name it live: hallucination, retrieval misses, prompt injection from retrieved content, and degradation on inputs unlike your evaluation set. When you frame evaluation as the product's safety net rather than a research nicety, you answer the question every hiring manager in this space is silently asking: will this person ship something we're embarrassed by?

FAQ

How much deep learning theory do I need? For applied AI roles, conceptual fluency (embeddings, attention at a high level, fine-tuning vs prompting) suffices. Research-leaning roles go much deeper — know which you're interviewing for.

Are classic ML topics still asked? Constantly. Fundamentals are how interviewers calibrate whether your AI knowledge has a floor. Don't skip them for the shiny parts.

Sharpen the coding half with Aissence's coding copilot, and rehearse system design narration with practice sessions.