Own Your AI Interview

An LLM is a transformer‑based neural network trained on massive text corpora to predict the next token in a sequence. Key points:

• Uses self‑attention to model long‑range dependencies.
• Training objective is next‑token prediction (or masked token prediction for some models).
• Pre‑training is followed by alignment steps like instruction tuning and RLHF.

• How does the transformer architecture differ from RNNs/LSTMs?
• What is RLHF and why is it used?
• What are some trade‑offs between model size and latency?

Tokens are the units of text (sub‑words, words, or characters) that the model processes. They matter because:

• Context length is measured in tokens, not characters.
• API pricing is usually per 1K tokens.
• Tokenization can split words in non‑obvious ways, affecting prompt length.

• What happens when a prompt exceeds the context window?
• How would you estimate token usage for a given feature?
• Why do different models have different tokenizers?

Embeddings map text to dense vectors capturing semantic similarity, while generative calls produce new text by predicting tokens. Differences:

• Embeddings: representation; used for search, clustering, retrieval.
• Generation: autoregressive decoding; used for answering, summarization, etc.
• Often separate models/endpoints optimized for each task.

• How are embeddings used in RAG?
• What is cosine similarity and why is it common?
• When would you choose a smaller embedding model?

The context window is the maximum number of tokens the model can consider at once (prompt + output). It impacts:

• How much history or retrieved context you can include.
• Prompt engineering strategies (summarization, chunking, sliding windows).
• Cost and latency for long prompts.

• How would you handle very long documents?
• What are trade‑offs of using a 200K‑token model?
• How does context window relate to RAG design?

Limitations include:

• Hallucinations (confident but incorrect answers).
• Lack of up‑to‑date knowledge (frozen training data).
• Sensitivity to prompt phrasing.
• Biases from training data.
• Non‑determinism and reproducibility challenges.

• How do you mitigate hallucinations in production?
• When is fine‑tuning appropriate vs RAG?
• How do you handle safety and bias concerns?

A production‑ready prompt is:

• Clear about role, task, and constraints.
• Explicit about output format and style.
• Robust to minor input variations.
• Tested against edge cases and evaluated with metrics.

• How do you version prompts?
• How would you A/B test prompts?
• How do you handle localization or multi‑language prompts?

• Zero‑shot: only instructions; good for simple tasks.
• Few‑shot: add examples; good for style, format, or nuanced tasks.
• Chain‑of‑thought: encourage step‑by‑step reasoning; good for reasoning and math.

Should mention trade‑offs in context usage and latency.

• When would you avoid chain‑of‑thought?
• How do you choose examples for few‑shot prompts?
• How does this relate to evaluation?

Strategies:

• Specify exact JSON schema and field types.
• Provide one or more valid examples.
• Instruct the model to output only JSON, no extra text.
• Use validators and repair logic for malformed JSON.

• When would you use function/tool calling instead?
• How do you handle optional fields?
• How do you test robustness of structured prompts?

Steps:

• Collect failing examples and categorize failure modes.
• Simplify the prompt to isolate the cause.
• Add clarifications, constraints, or examples.
• Test across a representative evaluation set.

• How do you know when to stop tweaking prompts and change the model instead?
• How would you log prompt failures in production?
• How do you avoid overfitting prompts to a small eval set?

A typical RAG pipeline:

• Ingestion: load docs, chunk, embed, store in vector DB with metadata.
• Retrieval: embed query, similarity search, optional filters/reranking.
• Augmentation: build prompt with user query + retrieved context.
• Generation: LLM answers using augmented prompt.
• Evaluation/monitoring: track quality, latency, and failures.

• How do you choose chunk size and overlap?
• What are common failure modes of RAG?
• How would you evaluate RAG quality?

Consider:

• Semantic coherence (don't split mid‑sentence or mid‑concept).
• Model context window and cost.
• Task type (FAQ vs long‑form reasoning).
• Typical ranges: 200-800 tokens with 10-20% overlap.

• How would you empirically tune chunk size?
• What happens if chunks are too small or too large?
• How does chunking interact with reranking?

Strategies:

• Improve retrieval quality (better embeddings, chunking, filters, reranking).
• Constrain the model to answer only from provided context.
• Ask the model to cite sources or say "I don't know".
• Use secondary verification for critical domains.

• How would you detect hallucinations automatically?
• What metrics would you track in production?
• When is hallucination acceptable vs unacceptable?

An agent is a system where an LLM:

• Plans steps toward a goal.
• Selects and calls tools (APIs, DBs, services).
• Iterates based on intermediate results and state.

It differs from a plain call by adding tool use, control flow, and memory around the model.

• What are risks of unconstrained agents?
• How would you debug an agent that loops?
• When would you avoid using agents?

Components:

• API gateway + auth.
• Storage for resumes and metadata.
• Queue/worker layer for LLM calls.
• LLM provider(s) with fallback and retries.
• Logging, metrics, prompt/version management.

Should mention latency vs cost trade‑offs and caching.

• How would you handle rate limits?
• What data would you log?
• How do you roll out new prompt versions safely?

Candidate should outline:

• Load documents and split into chunks.
• Create embeddings and store in a vector store (FAISS/Chroma).
• Define a retriever from the vector store.
• Use a RetrievalQA or similar chain with an LLM.

Exact syntax isn't required, but the flow should be correct.

• How would you swap the vector store or embedding model?
• How do you log retrieved documents?
• How would you add metadata filters?