Ace Your GenAI Engineer Interview

An LLM is a transformer‑based neural network trained on massive text corpora to predict the next token in a sequence. Key points:

• Uses self‑attention to model long‑range dependencies.
• Training objective is next‑token prediction (or masked token prediction for some models).
• Pre‑training is followed by alignment steps like instruction tuning and RLHF.

• How does the transformer architecture differ from RNNs/LSTMs?
• What is RLHF and why is it used?
• What are some trade‑offs between model size and latency?

Tokens are the units of text (sub‑words, words, or characters) that the model processes. They matter because:

• Context length is measured in tokens, not characters.
• API pricing is usually per 1K tokens.
• Tokenization can split words in non‑obvious ways, affecting prompt length.

• What happens when a prompt exceeds the context window?
• How would you estimate token usage for a given feature?
• Why do different models have different tokenizers?

Embeddings map text to dense vectors capturing semantic similarity, while generative calls produce new text by predicting tokens. Differences:

• Embeddings: representation; used for search, clustering, retrieval.
• Generation: autoregressive decoding; used for answering, summarization, etc.
• Often separate models/endpoints optimized for each task.

• How are embeddings used in RAG?
• What is cosine similarity and why is it common?
• When would you choose a smaller embedding model?

The context window is the maximum number of tokens the model can consider at once (prompt + output). It impacts:

• How much history or retrieved context you can include.
• Prompt engineering strategies (summarization, chunking, sliding windows).
• Cost and latency for long prompts.

• How would you handle very long documents?
• What are trade‑offs of using a 200K‑token model?
• How does context window relate to RAG design?

Limitations include:

• Hallucinations (confident but incorrect answers).
• Lack of up‑to‑date knowledge (frozen training data).
• Sensitivity to prompt phrasing.
• Biases from training data.
• Non‑determinism and reproducibility challenges.

• How do you mitigate hallucinations in production?
• When is fine‑tuning appropriate vs RAG?
• How do you handle safety and bias concerns?

Tokenisation converts raw text into integer IDs before feeding into an LLM:

• BPE (Byte-Pair Encoding) — most common; merges frequent character pairs into subword units. GPT models use BPE.
• Wordpiece — similar to BPE; used by BERT.
• SentencePiece — language-agnostic; handles any script without pre-tokenisation.

Practical implications:

• Cost — charged per token; verbose prompts and long outputs increase cost directly.
• Context window — measured in tokens, not words. 1 English word ≈ 1.3 tokens; code and non-English text can be 2–4× less efficient.
• Multilingual — languages underrepresented in training data tokenise less efficiently (more tokens per word), costing more and degrading quality.

• Why does English generally tokenise more efficiently than other languages?
• How can you reduce token usage without changing your prompt's intent?
• What is the relationship between token count and model latency?

Hallucinations are confident but factually incorrect outputs generated by LLMs:

• Causes: LLMs generate statistically likely tokens, not verified facts. They have no access to ground truth; training data has errors; they interpolate between seen patterns.
• Mitigation strategies:
  • RAG — ground responses in retrieved documents; most effective for factual tasks.
  • Structured output + validation — constrain model to JSON and validate against schema.
  • Self-consistency — sample multiple responses and take majority; reduces but doesn’t eliminate hallucinations.
  • Citation enforcement — require model to quote sources; helps detect but not prevent.
  • LLM-as-judge — a second model checks faithfulness of the answer against retrieved context.

• Can you completely eliminate hallucinations with RAG? Why or why not?
• What is faithfulness vs answer relevancy in RAG evaluation?
• How would you build a production hallucination monitoring pipeline?

These are decoding parameters that control the randomness of LLM outputs:

• Temperature — scales logits before softmax. Temperature < 1: more deterministic (sharper distribution). Temperature > 1: more random (flatter). Temperature = 0: greedy (always picks highest probability token).
• Top-k — restricts sampling to the k most probable tokens at each step; ignores the rest. Prevents very unlikely tokens but fixed k can be too narrow or too wide depending on context.
• Top-p (nucleus sampling) — samples from the smallest set of tokens whose cumulative probability ≥ p. Adapts dynamically to the distribution — better than top-k in most cases.

Practical defaults: temperature 0.0 for factual/deterministic tasks, 0.7–1.0 for creative tasks. Top-p 0.95 is a common default. These settings do not affect training — only inference.

• When would you set temperature to 0 vs 0.7 in a production application?
• Why is top-p generally preferred over top-k?
• What is repetition penalty and when do you need it?

RLHF (Reinforcement Learning from Human Feedback) is the primary alignment technique for modern LLMs:

• Step 1 — Supervised Fine-Tuning (SFT): Fine-tune the base model on high-quality human-written demonstrations of desired behaviour.
• Step 2 — Reward Model (RM): Collect human preference data (pairs of model outputs ranked by human raters). Train a reward model to predict which output a human would prefer.
• Step 3 — PPO optimisation: Use Proximal Policy Optimisation to fine-tune the SFT model to maximise reward model scores, with a KL-divergence penalty to prevent reward hacking (staying too far from the SFT baseline).

Alternative: DPO (Direct Preference Optimisation) — skips the reward model; optimises preferences directly. Simpler, often equally effective. Limitation: reward hacking — model learns to game the reward model rather than truly align.

• What is reward hacking and how does the KL penalty mitigate it?
• How does DPO differ from PPO-based RLHF and what are the trade-offs?
• What is Constitutional AI (CAI) and how does it extend RLHF?

A production‑ready prompt is:

• Clear about role, task, and constraints.
• Explicit about output format and style.
• Robust to minor input variations.
• Tested against edge cases and evaluated with metrics.

• How do you version prompts?
• How would you A/B test prompts?
• How do you handle localization or multi‑language prompts?

• Zero‑shot: only instructions; good for simple tasks.
• Few‑shot: add examples; good for style, format, or nuanced tasks.
• Chain‑of‑thought: encourage step‑by‑step reasoning; good for reasoning and math.

Should mention trade‑offs in context usage and latency.

• When would you avoid chain‑of‑thought?
• How do you choose examples for few‑shot prompts?
• How does this relate to evaluation?

Strategies:

• Specify exact JSON schema and field types.
• Provide one or more valid examples.
• Instruct the model to output only JSON, no extra text.
• Use validators and repair logic for malformed JSON.

• When would you use function/tool calling instead?
• How do you handle optional fields?
• How do you test robustness of structured prompts?

Steps:

• Collect failing examples and categorize failure modes.
• Simplify the prompt to isolate the cause.
• Add clarifications, constraints, or examples.
• Test across a representative evaluation set.

• How do you know when to stop tweaking prompts and change the model instead?
• How would you log prompt failures in production?
• How do you avoid overfitting prompts to a small eval set?

A typical RAG pipeline:

• Ingestion: load docs, chunk, embed, store in vector DB with metadata.
• Retrieval: embed query, similarity search, optional filters/reranking.
• Augmentation: build prompt with user query + retrieved context.
• Generation: LLM answers using augmented prompt.
• Evaluation/monitoring: track quality, latency, and failures.

• How do you choose chunk size and overlap?
• What are common failure modes of RAG?
• How would you evaluate RAG quality?

Consider:

• Semantic coherence (don't split mid‑sentence or mid‑concept).
• Model context window and cost.
• Task type (FAQ vs long‑form reasoning).
• Typical ranges: 200-800 tokens with 10-20% overlap.

• How would you empirically tune chunk size?
• What happens if chunks are too small or too large?
• How does chunking interact with reranking?

Strategies:

• Improve retrieval quality (better embeddings, chunking, filters, reranking).
• Constrain the model to answer only from provided context.
• Ask the model to cite sources or say "I don't know".
• Use secondary verification for critical domains.

• How would you detect hallucinations automatically?
• What metrics would you track in production?
• When is hallucination acceptable vs unacceptable?

Chunking splits documents into pieces for embedding and retrieval:

• Fixed-size chunking — simple; chunk by token count (e.g., 512 tokens with 50-token overlap). Risk: splits mid-sentence.
• Sentence/paragraph chunking — respects natural boundaries; better coherence. Use when documents have clear sentence structure.
• Semantic chunking — embed sentences; split when cosine distance between adjacent sentences spikes. Computationally expensive but produces semantically coherent chunks.
• Hierarchical chunking (parent-child) — embed small chunks for retrieval precision, return larger parent chunks for context. Best of both worlds.

Chunk size rule of thumb: smaller chunks = better precision (retrieval); larger chunks = better context (generation). Test empirically with your data — 256–1024 tokens is a typical starting range.

• What is the impact of chunk overlap on retrieval quality?
• How do you handle structured data (tables, code) in chunking?
• What is the parent-document retriever pattern?

Hybrid search combines dense (semantic) and sparse (keyword) retrieval:

• Dense retrieval — embedding-based; captures semantic meaning. Excels at paraphrases and concept queries. Struggles with exact terminology, codes, or rare proper nouns.
• Sparse retrieval (BM25) — keyword-based TF-IDF variant. Excels at exact terms, product codes, names. Fails at semantic similarity.
• Hybrid = dense + sparse: use Reciprocal Rank Fusion (RRF) to merge ranked results from both. Consistently outperforms either alone.
• Re-ranking: after hybrid retrieval, a cross-encoder model (e.g., Cohere Rerank, BGE-reranker) re-scores top-N results for higher precision before sending to the LLM.

Typical pipeline: Query → Dense + BM25 → RRF merge → Cross-encoder re-rank → Top-k to LLM.

• What is Reciprocal Rank Fusion and how does it combine results?
• When would dense retrieval alone fail and BM25 be critical?
• What is the computational cost trade-off of adding a cross-encoder re-ranker?

RAG evaluation covers both retrieval and generation quality:

• Retrieval metrics:
  • Context Precision — fraction of retrieved chunks that are relevant.
  • Context Recall — fraction of relevant content that was retrieved.
• Generation metrics:
  • Faithfulness — does the answer only use information from retrieved context? (Detects hallucination)
  • Answer Relevancy — does the answer actually address the question?
• RAGAs framework — open-source; evaluates all four metrics using an LLM-as-judge approach without requiring human labels.
• End-to-end — create a golden question-answer dataset; measure exact match, BLEU, or LLM-judged correctness.

• What is the difference between faithfulness and answer relevancy?
• How do you create a golden QA test set for RAG evaluation?
• What is LLM-as-judge and what are its limitations?

An agent is a system where an LLM:

• Plans steps toward a goal.
• Selects and calls tools (APIs, DBs, services).
• Iterates based on intermediate results and state.

It differs from a plain call by adding tool use, control flow, and memory around the model.

• What are risks of unconstrained agents?
• How would you debug an agent that loops?
• When would you avoid using agents?

Two dominant patterns for LLM-powered agents:

• ReAct (Reason + Act) — alternates between generating a reasoning step (Thought) and taking an action (Action/Observation) in a tight loop. Single agent; reactive; adapts in real time. Good for: short tasks, unpredictable environments, debugging (transparent chain of thought). Weakness: can loop indefinitely; no upfront plan.
• Plan-and-Execute — a planner LLM first creates a full task decomposition; executor agents carry out sub-tasks in sequence or parallel. Good for: long multi-step tasks, when a structured workflow is known upfront. Weakness: brittle if environment changes mid-execution; replanning is expensive.

Hybrid: plan first, but allow replanning if an executor step fails (LangGraph, CrewAI support this). When to use: start with ReAct for simplicity; use Plan-and-Execute when tasks have clear decomposable sub-goals.

• How does LangGraph implement the Plan-and-Execute pattern?
• What triggers replanning in a Plan-and-Execute agent?
• How do you prevent an agent from entering an infinite reasoning loop?

Agents can consume unbounded tokens/API calls without guardrails:

• Maximum iteration limits — hard stop after N reasoning steps (LangChain: max_iterations). Catch the exception and return gracefully.
• Token budget tracking — count tokens across all calls in a session; abort when budget exceeded.
• Tool call rate limiting — limit calls per tool per session; prevent accidental recursive tool use.
• Timeout per step — each tool invocation has a max execution time; agent gets an error message instead of hanging.
• Cost alerts — real-time token cost tracking; trigger alerts or circuit breakers at thresholds.
• Loop detection — hash recent (thought, action) pairs; if same state repeats 3 times, break the loop.
• Human-in-the-loop checkpoints — for high-risk actions (write access, emails), require explicit approval.

• How would you implement loop detection in a ReAct agent?
• What is a circuit breaker pattern and how does it apply to agents?
• How do you handle partial task completion when an agent hits a cost limit?

Agents use different memory types to manage context and knowledge:

• Short-term (In-context) — the active context window. Fast but limited by token window size. Cleared between sessions unless persisted.
• Long-term (External/Vector store) — user facts, learned preferences stored in a vector DB; retrieved via similarity search. Persists across sessions. Use for: personalisation, domain knowledge.
• Episodic — past conversation summaries or completed task records. Lets the agent recall “last time I helped with X, Y happened”. Use for: continuity, avoiding repeated mistakes.
• Semantic (Parametric) — knowledge baked into model weights via fine-tuning or pre-training. Doesn’t require retrieval. Use for: stable, universal knowledge.

Tools: Mem0 (managed long-term memory), LangGraph (state management), Redis (session cache).

• How does Mem0 implement long-term memory for agents?
• What is the difference between episodic and semantic memory in this context?
• How do you prevent stale facts in an agent's long-term memory?

Components:

• API gateway + auth.
• Storage for resumes and metadata.
• Queue/worker layer for LLM calls.
• LLM provider(s) with fallback and retries.
• Logging, metrics, prompt/version management.

Should mention latency vs cost trade‑offs and caching.

• How would you handle rate limits?
• What data would you log?
• How do you roll out new prompt versions safely?

A full production design covering all layers:

• Retrieval layer — hybrid search (dense + BM25) over knowledge base; re-rank with cross-encoder. Chunk size 512 tokens with parent-document retrieval.
• LLM layer — smaller model (Claude Haiku / GPT-4o Mini) for simple queries, route complex queries to larger model. System prompt enforces persona and safety.
• Guardrails — input classification (detect jailbreaks, off-topic); output validation (no PII in response, factual grounding check).
• Caching — exact-match cache for FAQs; semantic cache for near-duplicate questions (saves 30–60% of API calls).
• Latency — streaming responses; target P95 < 2s first token. Async retrieval in parallel with prompt assembly.
• Evaluation — RAGAs for faithfulness/relevancy; human review sample; A/B test new model versions on CSAT.
• Escalation — detect low-confidence responses and route to human agent.

• How would you handle multi-turn conversation history efficiently?
• How do you measure and improve CSAT for an LLM chatbot?
• What is the escalation strategy when the LLM can't answer confidently?

Production LLM systems must balance speed, cost, and reliability:

• Latency optimisations:
  • Streaming: return first token immediately rather than waiting for full response.
  • Model right-sizing: use smallest model that meets quality bar; route simple queries to fast/cheap models.
  • Prompt caching: Anthropic and OpenAI support caching prompt prefixes — reuse static system prompts to save 90% of prefix tokens.
  • Parallel calls: run retrieval, tool calls, and sub-tasks in parallel where dependencies allow.
• Reliability:
  • Multi-provider fallback: primary (Anthropic), fallback (OpenAI), emergency (self-hosted OSS).
  • Retry with exponential backoff for rate limit errors.
  • Circuit breaker: stop calling a provider if error rate > threshold for N seconds.
  • Queue-based architecture: async tasks use message queues; decouple request acceptance from LLM processing.

• What is prompt caching and which providers support it?
• How do you implement a multi-provider fallback strategy?
• What observability metrics would you track for an LLM API?

Candidate should outline:

• Load documents and split into chunks.
• Create embeddings and store in a vector store (FAISS/Chroma).
• Define a retriever from the vector store.
• Use a RetrievalQA or similar chain with an LLM.

Exact syntax isn't required, but the flow should be correct.

• How would you swap the vector store or embedding model?
• How do you log retrieved documents?
• How would you add metadata filters?