Ace Your GenAI Engineer Interview

An LLM is a transformer‑based neural network trained on massive text corpora to predict the next token in a sequence. Key points:

• Uses self‑attention to model long‑range dependencies.
• Training objective is next‑token prediction (or masked token prediction for some models).
• Pre‑training is followed by alignment steps like instruction tuning and RLHF.

• How does the transformer architecture differ from RNNs/LSTMs?
• What is RLHF and why is it used?
• What are some trade‑offs between model size and latency?

Tokens are the units of text (sub‑words, words, or characters) that the model processes. They matter because:

• Context length is measured in tokens, not characters.
• API pricing is usually per 1K tokens.
• Tokenization can split words in non‑obvious ways, affecting prompt length.

• What happens when a prompt exceeds the context window?
• How would you estimate token usage for a given feature?
• Why do different models have different tokenizers?

Embeddings map text to dense vectors capturing semantic similarity, while generative calls produce new text by predicting tokens. Differences:

• Embeddings: representation; used for search, clustering, retrieval.
• Generation: autoregressive decoding; used for answering, summarization, etc.
• Often separate models/endpoints optimized for each task.

• How are embeddings used in RAG?
• What is cosine similarity and why is it common?
• When would you choose a smaller embedding model?

The context window is the maximum number of tokens the model can consider at once (prompt + output). It impacts:

• How much history or retrieved context you can include.
• Prompt engineering strategies (summarization, chunking, sliding windows).
• Cost and latency for long prompts.

• How would you handle very long documents?
• What are trade‑offs of using a 200K‑token model?
• How does context window relate to RAG design?

Limitations include:

• Hallucinations (confident but incorrect answers).
• Lack of up‑to‑date knowledge (frozen training data).
• Sensitivity to prompt phrasing.
• Biases from training data.
• Non‑determinism and reproducibility challenges.

• How do you mitigate hallucinations in production?
• When is fine‑tuning appropriate vs RAG?
• How do you handle safety and bias concerns?

Tokenisation converts raw text into integer IDs before feeding into an LLM:

• BPE (Byte-Pair Encoding) — most common; merges frequent character pairs into subword units. GPT models use BPE.
• Wordpiece — similar to BPE; used by BERT.
• SentencePiece — language-agnostic; handles any script without pre-tokenisation.

Practical implications:

• Cost — charged per token; verbose prompts and long outputs increase cost directly.
• Context window — measured in tokens, not words. 1 English word ≈ 1.3 tokens; code and non-English text can be 2–4× less efficient.
• Multilingual — languages underrepresented in training data tokenise less efficiently (more tokens per word), costing more and degrading quality.

• Why does English generally tokenise more efficiently than other languages?
• How can you reduce token usage without changing your prompt's intent?
• What is the relationship between token count and model latency?

Hallucinations are confident but factually incorrect outputs generated by LLMs:

• Causes: LLMs generate statistically likely tokens, not verified facts. They have no access to ground truth; training data has errors; they interpolate between seen patterns.
• Mitigation strategies:
  • RAG — ground responses in retrieved documents; most effective for factual tasks.
  • Structured output + validation — constrain model to JSON and validate against schema.
  • Self-consistency — sample multiple responses and take majority; reduces but doesn’t eliminate hallucinations.
  • Citation enforcement — require model to quote sources; helps detect but not prevent.
  • LLM-as-judge — a second model checks faithfulness of the answer against retrieved context.

• Can you completely eliminate hallucinations with RAG? Why or why not?
• What is faithfulness vs answer relevancy in RAG evaluation?
• How would you build a production hallucination monitoring pipeline?

These are decoding parameters that control the randomness of LLM outputs:

• Temperature — scales logits before softmax. Temperature < 1: more deterministic (sharper distribution). Temperature > 1: more random (flatter). Temperature = 0: greedy (always picks highest probability token).
• Top-k — restricts sampling to the k most probable tokens at each step; ignores the rest. Prevents very unlikely tokens but fixed k can be too narrow or too wide depending on context.
• Top-p (nucleus sampling) — samples from the smallest set of tokens whose cumulative probability ≥ p. Adapts dynamically to the distribution — better than top-k in most cases.

Practical defaults: temperature 0.0 for factual/deterministic tasks, 0.7–1.0 for creative tasks. Top-p 0.95 is a common default. These settings do not affect training — only inference.

• When would you set temperature to 0 vs 0.7 in a production application?
• Why is top-p generally preferred over top-k?
• What is repetition penalty and when do you need it?

RLHF (Reinforcement Learning from Human Feedback) is the primary alignment technique for modern LLMs:

• Step 1 — Supervised Fine-Tuning (SFT): Fine-tune the base model on high-quality human-written demonstrations of desired behaviour.
• Step 2 — Reward Model (RM): Collect human preference data (pairs of model outputs ranked by human raters). Train a reward model to predict which output a human would prefer.
• Step 3 — PPO optimisation: Use Proximal Policy Optimisation to fine-tune the SFT model to maximise reward model scores, with a KL-divergence penalty to prevent reward hacking (staying too far from the SFT baseline).

Alternative: DPO (Direct Preference Optimisation) — skips the reward model; optimises preferences directly. Simpler, often equally effective. Limitation: reward hacking — model learns to game the reward model rather than truly align.

• What is reward hacking and how does the KL penalty mitigate it?
• How does DPO differ from PPO-based RLHF and what are the trade-offs?
• What is Constitutional AI (CAI) and how does it extend RLHF?

A production‑ready prompt is:

• Clear about role, task, and constraints.
• Explicit about output format and style.
• Robust to minor input variations.
• Tested against edge cases and evaluated with metrics.

• How do you version prompts?
• How would you A/B test prompts?
• How do you handle localization or multi‑language prompts?

• Zero‑shot: only instructions; good for simple tasks.
• Few‑shot: add examples; good for style, format, or nuanced tasks.
• Chain‑of‑thought: encourage step‑by‑step reasoning; good for reasoning and math.

Should mention trade‑offs in context usage and latency.

• When would you avoid chain‑of‑thought?
• How do you choose examples for few‑shot prompts?
• How does this relate to evaluation?

Strategies:

• Specify exact JSON schema and field types.
• Provide one or more valid examples.
• Instruct the model to output only JSON, no extra text.
• Use validators and repair logic for malformed JSON.

• When would you use function/tool calling instead?
• How do you handle optional fields?
• How do you test robustness of structured prompts?

Steps:

• Collect failing examples and categorize failure modes.
• Simplify the prompt to isolate the cause.
• Add clarifications, constraints, or examples.
• Test across a representative evaluation set.

• How do you know when to stop tweaking prompts and change the model instead?
• How would you log prompt failures in production?
• How do you avoid overfitting prompts to a small eval set?

A typical RAG pipeline:

• Ingestion: load docs, chunk, embed, store in vector DB with metadata.
• Retrieval: embed query, similarity search, optional filters/reranking.
• Augmentation: build prompt with user query + retrieved context.
• Generation: LLM answers using augmented prompt.
• Evaluation/monitoring: track quality, latency, and failures.

• How do you choose chunk size and overlap?
• What are common failure modes of RAG?
• How would you evaluate RAG quality?

Consider:

• Semantic coherence (don't split mid‑sentence or mid‑concept).
• Model context window and cost.
• Task type (FAQ vs long‑form reasoning).
• Typical ranges: 200-800 tokens with 10-20% overlap.

• How would you empirically tune chunk size?
• What happens if chunks are too small or too large?
• How does chunking interact with reranking?

Strategies:

• Improve retrieval quality (better embeddings, chunking, filters, reranking).
• Constrain the model to answer only from provided context.
• Ask the model to cite sources or say "I don't know".
• Use secondary verification for critical domains.

• How would you detect hallucinations automatically?
• What metrics would you track in production?
• When is hallucination acceptable vs unacceptable?

Chunking splits documents into pieces for embedding and retrieval:

• Fixed-size chunking — simple; chunk by token count (e.g., 512 tokens with 50-token overlap). Risk: splits mid-sentence.
• Sentence/paragraph chunking — respects natural boundaries; better coherence. Use when documents have clear sentence structure.
• Semantic chunking — embed sentences; split when cosine distance between adjacent sentences spikes. Computationally expensive but produces semantically coherent chunks.
• Hierarchical chunking (parent-child) — embed small chunks for retrieval precision, return larger parent chunks for context. Best of both worlds.

Chunk size rule of thumb: smaller chunks = better precision (retrieval); larger chunks = better context (generation). Test empirically with your data — 256–1024 tokens is a typical starting range.

• What is the impact of chunk overlap on retrieval quality?
• How do you handle structured data (tables, code) in chunking?
• What is the parent-document retriever pattern?

Hybrid search combines dense (semantic) and sparse (keyword) retrieval:

• Dense retrieval — embedding-based; captures semantic meaning. Excels at paraphrases and concept queries. Struggles with exact terminology, codes, or rare proper nouns.
• Sparse retrieval (BM25) — keyword-based TF-IDF variant. Excels at exact terms, product codes, names. Fails at semantic similarity.
• Hybrid = dense + sparse: use Reciprocal Rank Fusion (RRF) to merge ranked results from both. Consistently outperforms either alone.
• Re-ranking: after hybrid retrieval, a cross-encoder model (e.g., Cohere Rerank, BGE-reranker) re-scores top-N results for higher precision before sending to the LLM.

Typical pipeline: Query → Dense + BM25 → RRF merge → Cross-encoder re-rank → Top-k to LLM.

• What is Reciprocal Rank Fusion and how does it combine results?
• When would dense retrieval alone fail and BM25 be critical?
• What is the computational cost trade-off of adding a cross-encoder re-ranker?

RAG evaluation covers both retrieval and generation quality:

• Retrieval metrics:
  • Context Precision — fraction of retrieved chunks that are relevant.
  • Context Recall — fraction of relevant content that was retrieved.
• Generation metrics:
  • Faithfulness — does the answer only use information from retrieved context? (Detects hallucination)
  • Answer Relevancy — does the answer actually address the question?
• RAGAs framework — open-source; evaluates all four metrics using an LLM-as-judge approach without requiring human labels.
• End-to-end — create a golden question-answer dataset; measure exact match, BLEU, or LLM-judged correctness.

• What is the difference between faithfulness and answer relevancy?
• How do you create a golden QA test set for RAG evaluation?
• What is LLM-as-judge and what are its limitations?

An agent is a system where an LLM:

• Plans steps toward a goal.
• Selects and calls tools (APIs, DBs, services).
• Iterates based on intermediate results and state.

It differs from a plain call by adding tool use, control flow, and memory around the model.

• What are risks of unconstrained agents?
• How would you debug an agent that loops?
• When would you avoid using agents?

Two dominant patterns for LLM-powered agents:

• ReAct (Reason + Act) — alternates between generating a reasoning step (Thought) and taking an action (Action/Observation) in a tight loop. Single agent; reactive; adapts in real time. Good for: short tasks, unpredictable environments, debugging (transparent chain of thought). Weakness: can loop indefinitely; no upfront plan.
• Plan-and-Execute — a planner LLM first creates a full task decomposition; executor agents carry out sub-tasks in sequence or parallel. Good for: long multi-step tasks, when a structured workflow is known upfront. Weakness: brittle if environment changes mid-execution; replanning is expensive.

Hybrid: plan first, but allow replanning if an executor step fails (LangGraph, CrewAI support this). When to use: start with ReAct for simplicity; use Plan-and-Execute when tasks have clear decomposable sub-goals.

• How does LangGraph implement the Plan-and-Execute pattern?
• What triggers replanning in a Plan-and-Execute agent?
• How do you prevent an agent from entering an infinite reasoning loop?

Agents can consume unbounded tokens/API calls without guardrails:

• Maximum iteration limits — hard stop after N reasoning steps (LangChain: max_iterations). Catch the exception and return gracefully.
• Token budget tracking — count tokens across all calls in a session; abort when budget exceeded.
• Tool call rate limiting — limit calls per tool per session; prevent accidental recursive tool use.
• Timeout per step — each tool invocation has a max execution time; agent gets an error message instead of hanging.
• Cost alerts — real-time token cost tracking; trigger alerts or circuit breakers at thresholds.
• Loop detection — hash recent (thought, action) pairs; if same state repeats 3 times, break the loop.
• Human-in-the-loop checkpoints — for high-risk actions (write access, emails), require explicit approval.

• How would you implement loop detection in a ReAct agent?
• What is a circuit breaker pattern and how does it apply to agents?
• How do you handle partial task completion when an agent hits a cost limit?

Agents use different memory types to manage context and knowledge:

• Short-term (In-context) — the active context window. Fast but limited by token window size. Cleared between sessions unless persisted.
• Long-term (External/Vector store) — user facts, learned preferences stored in a vector DB; retrieved via similarity search. Persists across sessions. Use for: personalisation, domain knowledge.
• Episodic — past conversation summaries or completed task records. Lets the agent recall “last time I helped with X, Y happened”. Use for: continuity, avoiding repeated mistakes.
• Semantic (Parametric) — knowledge baked into model weights via fine-tuning or pre-training. Doesn’t require retrieval. Use for: stable, universal knowledge.

Tools: Mem0 (managed long-term memory), LangGraph (state management), Redis (session cache).

• How does Mem0 implement long-term memory for agents?
• What is the difference between episodic and semantic memory in this context?
• How do you prevent stale facts in an agent's long-term memory?

Components:

• API gateway + auth.
• Storage for resumes and metadata.
• Queue/worker layer for LLM calls.
• LLM provider(s) with fallback and retries.
• Logging, metrics, prompt/version management.

Should mention latency vs cost trade‑offs and caching.

• How would you handle rate limits?
• What data would you log?
• How do you roll out new prompt versions safely?

A full production design covering all layers:

• Retrieval layer — hybrid search (dense + BM25) over knowledge base; re-rank with cross-encoder. Chunk size 512 tokens with parent-document retrieval.
• LLM layer — smaller model (Claude Haiku / GPT-4o Mini) for simple queries, route complex queries to larger model. System prompt enforces persona and safety.
• Guardrails — input classification (detect jailbreaks, off-topic); output validation (no PII in response, factual grounding check).
• Caching — exact-match cache for FAQs; semantic cache for near-duplicate questions (saves 30–60% of API calls).
• Latency — streaming responses; target P95 < 2s first token. Async retrieval in parallel with prompt assembly.
• Evaluation — RAGAs for faithfulness/relevancy; human review sample; A/B test new model versions on CSAT.
• Escalation — detect low-confidence responses and route to human agent.

• How would you handle multi-turn conversation history efficiently?
• How do you measure and improve CSAT for an LLM chatbot?
• What is the escalation strategy when the LLM can't answer confidently?

Production LLM systems must balance speed, cost, and reliability:

• Latency optimisations:
  • Streaming: return first token immediately rather than waiting for full response.
  • Model right-sizing: use smallest model that meets quality bar; route simple queries to fast/cheap models.
  • Prompt caching: Anthropic and OpenAI support caching prompt prefixes — reuse static system prompts to save 90% of prefix tokens.
  • Parallel calls: run retrieval, tool calls, and sub-tasks in parallel where dependencies allow.
• Reliability:
  • Multi-provider fallback: primary (Anthropic), fallback (OpenAI), emergency (self-hosted OSS).
  • Retry with exponential backoff for rate limit errors.
  • Circuit breaker: stop calling a provider if error rate > threshold for N seconds.
  • Queue-based architecture: async tasks use message queues; decouple request acceptance from LLM processing.

• What is prompt caching and which providers support it?
• How do you implement a multi-provider fallback strategy?
• What observability metrics would you track for an LLM API?

Candidate should outline:

• Load documents and split into chunks.
• Create embeddings and store in a vector store (FAISS/Chroma).
• Define a retriever from the vector store.
• Use a RetrievalQA or similar chain with an LLM.

Exact syntax isn't required, but the flow should be correct.

• How would you swap the vector store or embedding model?
• How do you log retrieved documents?
• How would you add metadata filters?

Responsible AI is a framework for developing and deploying AI systems that are trustworthy, equitable, and accountable. The six core pillars (broadly consistent across EU AI Act, NIST AI RMF, and Google/Microsoft frameworks): (1) Fairness — model outputs should not discriminate against protected groups. (2) Reliability & Safety — the system behaves as intended even under adversarial inputs or distribution shift. (3) Privacy & Security — training data and user data are protected; the model cannot be tricked into leaking PII. (4) Inclusiveness — the system serves diverse users, including accessibility needs. (5) Transparency — users know they're interacting with AI; the model's limitations are documented (model card). (6) Accountability — there is a human responsible for outcomes; high-stakes decisions have human review. For LLM-based systems, add: hallucination rate, refusal policy, and content filtering as operational KPIs.

Detection: (1) Red-team with adversarial prompts targeting protected attributes (gender, race, religion, nationality). (2) Evaluate on standardised bias benchmarks: BBQ (Bias Benchmark for QA), WinoBias, StereoSet. (3) Disaggregate evaluation metrics by demographic group in your test set. (4) Use an LLM-as-judge to rate output fairness at scale. Mitigation: (1) System prompt guardrails — explicit instructions not to make assumptions about individuals based on group membership. (2) RLHF / RLAIF fine-tuning on curated datasets with balanced representation. (3) Output filtering — post-process responses through a classifier that flags biased content before it reaches the user. (4) Human review workflows for high-stakes outputs (hiring, lending, medical). Most importantly: bias audits are not a one-time step. Run them continuously as the model or prompts change, and publish results in a model card so users can make informed decisions.

An AI governance framework defines the policies, processes, and accountability structures that ensure AI systems are built and operated responsibly. Key components: (1) Risk classification — tiering AI use cases by impact (low risk: autocomplete; high risk: credit decisions, medical diagnosis). Higher-risk systems require more oversight. (2) Model cards & datasheets — standardised documentation of training data, intended use, limitations, and evaluation results. (3) Approval workflow — which stakeholders must sign off before a model ships (legal, privacy, domain expert, exec sponsor). (4) Monitoring & audit trail — logging of model inputs/outputs for audits; drift detection alerts. (5) Incident response — a playbook for when the model causes harm (who's notified, how to roll back, how to communicate). Regulatory context: EU AI Act (effective 2026) mandates conformity assessments for high-risk systems; NIST AI RMF provides a voluntary framework. In interviews, show you understand governance as an ongoing operational practice, not a one-time checklist.

Prompt injection is when a malicious user (or compromised tool output) overrides the system prompt and hijacks model behaviour. Defences: (1) Privilege separation — don't give the LLM access to tools or data it doesn't need for the current task. Apply least-privilege to tool calling. (2) Input sanitisation — strip or escape special tokens and delimiter patterns (e.g., "\nSystem:", "IGNORE PREVIOUS INSTRUCTIONS") from user input before injecting it into the prompt. (3) Output validation — parse model outputs against a strict schema; if the model returns unexpected structure, reject it before acting. (4) Instruction hierarchy — clearly delimit system instructions from user content in the prompt (use XML tags or role markers the model is trained to respect). (5) Secondary LLM as a judge — run a fast safety classifier on the final output before returning it to the user. (6) Audit logs — log all inputs and outputs so injection attempts are detectable after the fact. No single control is sufficient; defence-in-depth is required.

A model card (introduced by Google, now standard practice) is a short document that travels with a model and tells users what it is, how it was trained, what it can do, and where it fails. Sections: (1) Model details — architecture, training date, version, who built it. (2) Intended use — the tasks and contexts it was designed for; out-of-scope uses explicitly listed. (3) Training data — sources, size, preprocessing steps, known data quality issues. (4) Evaluation results — benchmark scores disaggregated by relevant subgroups (gender, language, geography). (5) Limitations — where the model underperforms, known failure modes, data recency cutoff. (6) Ethical considerations — potential for harm, bias findings, recommended safeguards. (7) How to use — example prompts, API snippet, recommended system prompt. In practice, even a one-page internal model card dramatically reduces "the model said what?!" incidents because it forces the team to document assumptions they otherwise leave implicit.

Traditional REST APIs are synchronous, deterministic, and fast (milliseconds). AI-backed APIs break all three assumptions: (1) Latency — LLM calls take 1–30 seconds, so synchronous request/response patterns cause client timeouts. Solution: use SSE (Server-Sent Events) for streaming, or async polling with a job ID. (2) Non-determinism — the same input can produce different outputs. Document this in your API contract; don't promise idempotency. (3) Cost per call — each API call has a variable token cost. Rate limiting must account for token usage, not just RPS. (4) Failure modes — LLM providers have their own rate limits, outages, and context window errors. Your API needs retry logic, fallback models, and graceful degradation. (5) Response schema — LLM outputs are freeform text; if you need structured data, use structured output / function calling and validate before returning. Treat the LLM as an unreliable upstream dependency — design your API as if it could fail or be slow at any time.

The pattern: client opens a long-lived HTTP connection; server pushes tokens as they're generated. Two protocols: (1) SSE (Server-Sent Events) — HTTP/1.1 compatible, text-based ("data: {token}\n\n"), easy to implement in any framework. Best for web clients. (2) WebSockets — bi-directional, better for conversational interfaces where the client also sends mid-stream signals (e.g., cancel). Implementation steps: set Content-Type: text/event-stream, disable response buffering (critical — most frameworks buffer by default), forward each chunk from the LLM provider stream to the client as it arrives. Handle edge cases: client disconnect (stop calling the LLM), provider stream errors (send an error event to client), and rate limit retries (tricky in streaming — better to fail fast than hang). For mobile clients with poor connections, consider an async job pattern instead: POST returns a job ID, client polls GET /job/{id}/result.

Standard RPS rate limiting is insufficient because a single prompt can use 100 tokens or 100,000 tokens — same request count, wildly different cost. Design token-aware rate limiting: (1) Count tokens before sending to the LLM (use tiktoken or the provider's token counting API). (2) Maintain a token-per-minute (TPM) budget per user/tenant in Redis using a sliding window counter. (3) Reject requests that would exceed the budget before incurring the LLM call cost. (4) Return Retry-After headers so well-behaved clients back off gracefully. Additional controls: (5) Set max_tokens limits on each request to cap worst-case cost. (6) Queue long-running requests with a job system rather than holding HTTP connections. (7) Expose a /usage endpoint so enterprise customers can monitor their own consumption. Pricing tip: track your actual cost (input + output tokens × model rate) and surface it in your internal dashboards so you can detect prompt injection attacks that inflate token usage.

Resilience patterns: (1) Circuit breaker — after N consecutive failures, stop calling the provider and return a friendly error immediately (fail fast). Reset after a cooldown window. (2) Retry with exponential backoff — for transient errors (rate limits, 429s), retry 2–3 times with jitter. Never retry on 400 (bad request) — that's a bug in your code. (3) Fallback model — primary: GPT-4o, fallback: Claude Haiku or Gemini Flash. Accept slightly lower quality rather than a 500 error. (4) Cache frequent responses — if many users ask the same question, return a cached answer during an outage. Use semantic caching (embed the query, find nearest neighbour in a vector store) to catch paraphrase matches. (5) Queue non-real-time requests — batch jobs can tolerate delays; put them in a queue and process when the provider recovers. (6) Status page subscription — subscribe to provider status pages and feed alerts into your on-call system so engineers know before users complain.

Model changes are breaking changes even if your API schema doesn't change — output tone, format, and reasoning style can shift. Strategies: (1) Pin model version in your API config, not just "gpt-4". Use "gpt-4o-2024-11-20" so upgrades are explicit. (2) Introduce API versions (v1, v2) when you change the model that backs a feature — give clients time to migrate. (3) Regression test suite — a golden set of prompt → expected output pairs that you run before any model upgrade. Use an LLM judge to evaluate whether new outputs are equivalent or better. (4) Shadow testing — route 5% of prod traffic to the new model, compare outputs, before switching over. (5) Communicate deprecations — give 90-day notice when retiring a model-backed endpoint. The practical reality: most breaking changes come not from schema changes but from prompt sensitivity to model updates. Treat your prompts as code — version them, test them, and deploy them through CI.

RAG failures fall into two buckets: retrieval failures and generation failures. Diagnose which one first. (1) Retrieval check: run the query through the retriever alone and inspect the top-K chunks. Do the retrieved chunks actually contain the answer? If no → retrieval bug. If yes → generation bug. (2) Retrieval failure causes: embedding mismatch (query and docs embedded with different models), chunking too coarse/fine, metadata filters excluding relevant docs, low similarity threshold returning junk. Fix: re-embed with consistent model, tune chunk size, adjust threshold. (3) Generation failure causes: retrieved context is correct but the LLM ignores it (context position matters — LLMs are worse at mid-context recall), prompt structure doesn't make it clear which text is context vs instruction, max_tokens too low and the answer gets cut off. Fix: move context earlier in the prompt, add explicit "Use ONLY the provided context" instructions, increase max_tokens. Log every step: query, retrieved chunks with scores, final prompt, raw LLM output. You can't debug what you can't observe.

Hallucination types: (1) Factual hallucination — model states false facts confidently. (2) Faithfulness hallucination — model ignores provided context and makes things up. (3) Format hallucination — model produces structurally invalid JSON or ignores schema. Diagnosis: (1) Use RAGAS faithfulness metric — it checks whether every claim in the answer is grounded in the retrieved context. Low faithfulness = RAG context isn't being used. (2) TruLens groundedness score — similar, useful as a continuous monitoring signal. (3) Manual audit of 50 failure cases to categorise root causes. Fixes by type: Faithfulness → strengthen the system prompt ("only answer from the provided documents"), add citation instructions, reduce temperature. Factual → add a retrieval step even for "general knowledge" questions, or use tool calling to verify claims. Format → switch to structured outputs/function calling instead of asking the model to output JSON as plain text; validate and retry on parse failure. Monitoring: track hallucination rate as a KPI. Alert when it rises above baseline — often signals a prompt change or model upgrade had an unintended side effect.

Treat it like any distributed system latency spike — isolate which component is slow. Checklist: (1) Check provider status page first. If OpenAI/Anthropic has an incident, your investigation ends there. (2) Decompose your pipeline with trace timing: embedding call → vector DB query → LLM call → post-processing. Add timing logs to each step before the next deploy. (3) Check token count distribution — did prompt length increase? A 10x token increase causes near-10x latency increase. Query for p99 input token count today vs baseline. (4) Check for retry storms — are you hitting rate limits and retrying? Each retry adds latency. Check your error rate and look for 429 responses in logs. (5) Vector DB query time — check your embedding index freshness. Index fragmentation after heavy ingestion can slow queries significantly. (6) Concurrency — check if a new code path is making sequential LLM calls where it previously ran in parallel. Fix the immediate issue first (roll back if a deploy caused it), then instrument to prevent recurrence.

Agent failures are hard to debug because the failure mode is emergent — you need to observe the full trajectory, not just input/output. Approach: (1) Log every step: thought, action, tool call args, tool result, next thought. LangSmith, Phoenix Arize, and Langfuse all provide agent trace visualisation. (2) Identify the loop trigger: is the model re-calling the same tool with the same args? Usually means it got a result it can't parse (empty string, error, unexpected format) and is retrying. Fix: handle tool errors explicitly in the system prompt ("if a tool returns an error, explain what you tried and stop"). (3) Check tool output size: if a tool returns 50,000 tokens of text, the model loses context and hallucinates the next step. Add max_results limits to every tool. (4) Add a step counter limit: never let an agent run more than N steps without a checkpoint. At N/2, inject "you have X steps remaining, summarise progress." (5) Prompt clarity: agents fail when the goal is ambiguous. Make the success condition explicit: "you are done when you can answer the original question directly without calling any more tools."

GenAI observability requires tracking dimensions traditional APM tools don't cover. Four layers: (1) Infrastructure metrics (standard) — CPU, GPU utilisation, memory, pod health, API error rates, p50/p99 latency. (2) LLM-specific metrics — token usage (input + output) per request, cost per request, model provider error rate by type (rate limit vs server error), cache hit rate. (3) Quality metrics — hallucination rate (via RAGAS or TruLens on a sample), user thumbs-up/thumbs-down rate, task completion rate for agentic flows. (4) Business metrics — feature usage, conversion rate for AI-assisted flows, support ticket deflection rate. Tools: Langfuse (open-source, traces + evals), Phoenix Arize (strong for RAG), Helicone (cost tracking), OpenTelemetry (vendor-neutral tracing). Alerting thresholds to set on day 1: hallucination rate > 5%, p99 latency > 10s, LLM cost/day > $X, provider error rate > 1%. Without observability, you're flying blind — you'll know something broke when users complain, not before.