Generative AI interview questions and answers: LLMs, RAG and agents

Generative AI interviews have settled into a clear pattern. Whether the role is called AI engineer, GenAI engineer, LLM engineer or "full-stack developer with AI features", interviewers test the same layers: the vocabulary (AI, machine learning, deep learning, generative models), how a large language model turns text into tokens and tokens into predictions, how retrieval-augmented generation (RAG) and agents are built around a model, and how you keep the result accurate, affordable and safe in production.

This guide collects 45 generative AI interview questions with model answers, grouped from fundamentals to system design. The answers are written to be said out loud in an interview: a direct definition first, then the trade-off or example that shows you have built something. Facts were checked against the original papers and official documentation listed at the end of the article.

Model names change every few months, so the answers below talk about model families and techniques rather than one release. Interviewers care far more that you can explain why a technique works and when to use it than that you remember the newest version number.

How to prepare for a generative AI interview

• Know the stack from the bottom up. Tokens, embeddings, attention, next-token prediction, then prompting, RAG, tool use and agents. Most follow-up questions move one layer down from where you answered.
• Build one small project end to end. A RAG assistant over your own documents, with an evaluation set, teaches more than any list of definitions. Our RAG pipeline tutorial is a good starting point.
• Prepare numbers you can defend. Context window sizes, latency, cost per request and evaluation scores from your own project are more convincing than general claims.
• Practise the trade-off answer. "It depends" is only a good answer when you say what it depends on.

Generative AI interview questions: the fundamentals

1. What is generative AI?

Generative AI is the part of AI that produces new content, such as text, code, images, audio or video, instead of only labelling or scoring existing data. A generative model learns the statistical structure of its training data and then samples new outputs that follow that structure. Large language models, image diffusion models and text-to-speech models are all generative.

2. What is the difference between AI, machine learning, deep learning and generative AI?

They are nested fields, each a subset of the one before it:

• Artificial intelligence is the broad goal of making machines perform tasks that normally need human judgement. It includes hand-written rules, search and planning, not only learning systems.
• Machine learning is AI that learns behaviour from data instead of explicit rules: regression, decision trees, gradient boosting, clustering.
• Deep learning is machine learning with multi-layer neural networks, which learn their own features from raw data such as pixels, audio or text.
• Generative AI is deep learning used to create new content. Today it is dominated by transformers and diffusion models.

3. What is the difference between a discriminative and a generative model?

A discriminative model learns the boundary between classes, written as P(y | x): given an email, how likely is it to be spam? A generative model learns how the data itself is distributed, P(x) or P(x, y), which lets it create new samples. An LLM is generative because it models the probability of the next token given the previous ones, and sampling from that distribution repeatedly produces new text.

4. What is a foundation model?

A foundation model is a large model pre-trained on broad data and then adapted to many tasks through prompting, retrieval or fine-tuning, rather than trained for one task. LLMs are the best-known example; vision-language and speech models are others. The practical point for an engineer is that you rarely train one: you choose one, then shape it with prompts, context and sometimes a small amount of fine-tuning.

Generative model interview questions

5. What are the main types of generative models?

6. How does a diffusion model generate an image?

During training, noise is added to real images over many steps until they become pure noise, and the network learns to predict and remove the noise at each step. At generation time the model starts from random noise and denoises it step by step, guided by a text prompt through conditioning, until an image appears. Most modern image systems run this process in a compressed latent space rather than on pixels, which makes it much cheaper. Samplers that take fewer steps trade a little quality for speed.

7. Why did GANs give way to diffusion models for most image generation?

GAN training is a two-player game that can oscillate or collapse onto a few output types, and it is hard to condition on long text prompts. Diffusion models train with a simple, stable objective, cover the variety in the data better and condition well on text. GANs remain useful where speed matters, such as real-time super-resolution.

LLM fundamentals interview questions

8. What is a large language model?

An LLM is a transformer network, usually with billions of parameters, trained on a very large text corpus to predict the next token. That single objective, at enough scale, produces models that can follow instructions, summarise, translate, write code and reason through multi-step problems. A good interview answer adds the limit: the model produces plausible continuations based on patterns in its training, so it can be fluent and wrong at the same time.

9. How does an LLM generate a response, step by step?

1. The prompt is split into tokens and each token is mapped to an integer ID.
2. Each ID is looked up in an embedding table to get a vector, and position information is added.
3. The vectors pass through a stack of transformer layers, where attention lets every token use information from earlier tokens.
4. The final layer produces a score for every token in the vocabulary; softmax turns the scores into probabilities.
5. A decoding strategy picks one token, which is appended to the input, and the loop repeats until a stop token or length limit.

Two production details are worth mentioning: the KV cache stores the attention keys and values of earlier tokens so each new token does not recompute them, and the first token is slower than later ones because the whole prompt must be processed first.

10. What are temperature, top-k and top-p?

They control how the next token is chosen from the probability distribution. Temperature rescales the distribution: low values make the most likely token dominate, high values flatten it and add variety. Top-k samples only from the k most likely tokens. Top-p (nucleus sampling) samples from the smallest set of tokens whose probabilities add up to p. Use low temperature for extraction, classification and code; higher values for brainstorming. Note that some newer reasoning models fix or ignore these settings, so check the provider's documentation.

11. What is the difference between pre-training, instruction tuning and preference tuning?

Pre-training teaches general language and knowledge through next-token prediction on huge unlabelled text. Instruction tuning (supervised fine-tuning) trains on prompt and response pairs so the model follows requests. Preference tuning, through methods such as RLHF or direct preference optimisation (DPO), trains the model towards responses people or reviewers rate as better, which shapes helpfulness, tone and refusals. A base model continues text; a chat model has gone through the later stages.

12. What is a context window, and what happens when you exceed it?

The context window is the maximum number of tokens the model can attend to in one request, counting the system prompt, conversation history, retrieved documents, tool results and the output. If you exceed it, the API rejects the request or older content must be truncated or summarised. A large window does not mean everything in it is used equally well: models can miss facts buried in the middle of long inputs, and every extra token adds cost and latency. That is why retrieval and context selection still matter.

13. Why do LLMs hallucinate?

The model is trained to produce likely text, not to check truth. When the answer is missing from its training data or its context, the most probable continuation can still be a confident, specific and invented answer. Causes include knowledge cut-offs, ambiguous prompts, retrieval that returned the wrong passages, and pressure to answer instead of saying "I don't know". Mitigations are covered in question 41.

Tokenization interview questions

14. What is a token, and why do models use subword tokens?

A token is the unit the model reads and writes: a whole word, part of a word, a number, punctuation or a space. Word-level vocabularies cannot cover every rare word, name or typo, and character-level input makes sequences very long. Subword tokenization sits in between: common words stay whole and rare words split into known pieces, so the vocabulary stays fixed in size and almost nothing is unknown. As a rough guide, one token is about three quarters of an English word, but other languages and code often use more tokens for the same meaning.

15. Compare BPE, WordPiece and SentencePiece.

A common trap: SentencePiece is a toolkit, and Unigram is the algorithm it often uses, so "SentencePiece vs BPE" is not a like-for-like comparison.

16. What are special tokens?

Reserved tokens that carry structure rather than text: beginning and end of sequence, padding to make a batch the same length, separators between segments, and masks used in masked-language-model training such as BERT's [MASK]. Chat models add role markers that separate system, user and assistant turns. If you build prompts by hand for an open-weight model, you must apply its chat template exactly, or quality drops.

17. Why does tokenization matter to an engineer, not just a researcher?

Pricing, rate limits and context windows are all counted in tokens. Tokenization also explains odd behaviour: models can struggle to count letters in a word because they see subword pieces, and numbers may be split inconsistently. Count tokens with the provider's tokenizer before sending large inputs, and budget for the output as well as the input.

Transformer and attention interview questions

18. What is a transformer, and why did it replace RNNs for language?

The transformer, introduced in the 2017 paper "Attention Is All You Need", processes a sequence with attention layers instead of recurrence. An RNN reads one token at a time, which is slow to train and weak at long-range dependencies. A transformer relates every position to every other position in one step, so training parallelises across the sequence on GPUs and scales to very large models and datasets.

19. Explain scaled dot-product attention.

Each token's vector is projected into three vectors: a query (what this token is looking for), a key (what it offers to others) and a value (the information it passes on). The attention score between two tokens is the dot product of one's query and the other's key. Scores are divided by the square root of the key dimension, passed through softmax to become weights, and used to take a weighted sum of the values:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · V

The scaling matters because dot products grow with the vector dimension; without it, softmax becomes very peaked and gradients become tiny, which slows learning.

20. What is multi-head attention?

Instead of one attention calculation, the model runs several in parallel, each with its own learned projections. Different heads can specialise, for example one following syntax and another following references to earlier entities. Their outputs are concatenated and projected back to the model dimension. Many recent LLMs share keys and values across groups of heads (grouped-query attention) to shrink the KV cache and speed up inference.

21. Why do transformers need positional encoding?

Attention on its own treats the input as a set: shuffling the tokens would give the same result. Positional information restores word order. The original transformer added fixed sine and cosine patterns; other models learn position embeddings; many current LLMs use rotary position embeddings (RoPE), which encode relative position inside the attention calculation and can be extended to longer contexts.

22. What is the difference between encoder-only, decoder-only and encoder-decoder models?

• Encoder-only (BERT family): bidirectional attention over the whole input, good for classification, search embeddings and extraction. It is not a text generator.
• Decoder-only (GPT-style LLMs): causal attention, each token sees only earlier tokens; built for generation and the basis of most chat models.
• Encoder-decoder (T5, the original transformer): the encoder reads the input and the decoder generates while attending to it through cross-attention; common in translation and summarisation.

23. What are padding masks and causal masks?

A padding mask stops attention from using the filler tokens added to make a batch the same length. A causal (look-ahead) mask stops each position in a decoder from seeing future tokens, so the model cannot cheat during training and generates strictly left to right.

Embeddings and vector database interview questions

24. What are embeddings?

An embedding is a dense vector of numbers that represents a piece of content so that similar meanings land close together in vector space. "Reset my password" and "I can't log in" share few words but produce nearby vectors. Embedding models are trained, often with contrastive learning, to pull related pairs together and push unrelated ones apart. Embeddings power semantic search, RAG, recommendations, clustering, deduplication and anomaly detection.

25. How do cosine similarity, dot product and Euclidean distance differ?

Cosine similarity measures the angle between two vectors and ignores their length; it ranges from −1 to 1. Dot product includes length as well as direction. Euclidean distance measures straight-line distance. If the vectors are normalised to length 1, all three produce the same ranking, which is why many embedding APIs return normalised vectors. Use the metric the embedding model was trained for.

26. What does a vector database do, and how does approximate nearest neighbour search work?

A vector database stores embeddings with their metadata and finds the vectors closest to a query vector. Exact search compares the query with every vector, which is too slow at millions of rows, so vector stores use approximate nearest neighbour (ANN) indexes. HNSW builds a layered graph that is navigated from coarse to fine; IVF clusters vectors and searches only the nearest clusters; product quantisation compresses vectors to save memory. Each trades a little recall for large gains in speed. Options range from dedicated vector databases to vector support inside PostgreSQL (pgvector), MySQL, OpenSearch and Redis; a team with one relational database often starts there.

27. What is a multimodal embedding?

A multimodal embedding model places different kinds of content, for example images and text, in one shared vector space. CLIP showed the approach: it was trained on image and caption pairs so that a photo of a dog and the text "a dog" produce nearby vectors. This enables text-to-image search, image-to-image search and zero-shot classification, and it is the basis for multimodal RAG over screenshots, diagrams and scanned documents.

RAG interview questions

28. What is retrieval-augmented generation (RAG), and why use it?

RAG retrieves relevant passages from your own data at question time and includes them in the prompt, so the model answers from current, specific sources rather than only its training. It reduces hallucination on company-specific questions, keeps answers up to date without retraining, allows citations and lets you enforce access control at retrieval. The typical pipeline:

1. Ingest: extract text from documents, clean it and split it into chunks.
2. Index: embed each chunk and store it with metadata (source, date, permissions).
3. Retrieve: embed the question and fetch the nearest chunks, often with keyword search as well.
4. Rerank and filter: reorder candidates by relevance and drop what the user may not see.
5. Generate: pass the best chunks to the LLM with instructions to answer only from them and cite sources.

For a worked example in PHP, see our RAG support assistant with Laravel and vector search.

29. What chunking strategies do you know, and how do you choose a chunk size?

• Fixed-size chunks with overlap: simple and predictable; the overlap keeps sentences that cross a boundary.
• Recursive or structure-aware splitting: split on headings, then paragraphs, then sentences, so chunks follow the document's own sections.
• Semantic chunking: start a new chunk where the topic changes, detected with embeddings.
• Document-specific splitting: by function or class for code, by row group for tables, by clause for contracts.
• Small-to-big (parent-child): search small, precise chunks but send the larger parent section to the model for context.

There is no universal best size. Small chunks retrieve precisely but lose context; large chunks keep context but dilute the embedding and cost more tokens. Choose by testing several sizes against a set of real questions and measuring retrieval quality. Adding the document title and section heading to each chunk usually helps more than tuning the size.

30. What is hybrid search, and when do you need reranking?

Hybrid search combines vector search, which captures meaning, with keyword search such as BM25, which catches exact terms like error codes, product names and IDs that embeddings can blur. The two result lists are merged, often with reciprocal rank fusion. A reranker, usually a cross-encoder that reads the question and each candidate together, then reorders the top results more accurately than vector similarity alone. A common pattern is to retrieve 50 candidates cheaply and rerank them down to the best 5.

31. What are context precision and context recall?

They measure the retrieval half of RAG, separately from the answer:

• Context precision asks whether the retrieved chunks are relevant and whether the relevant ones are ranked near the top. Evaluation libraries such as Ragas compute it as a rank-weighted average of precision at each position. Low precision means noise in the prompt, which wastes tokens and can mislead the model.
• Context recall asks whether the retrieved chunks contain everything needed to answer. Ragas splits a reference answer into claims and measures the share supported by the retrieved context. Low recall means the model cannot answer correctly however good the prompt is.

Add faithfulness (is every claim in the answer supported by the context?) and answer relevance (does it address the question?) to cover the generation half. When an answer is wrong, these metrics tell you whether to fix retrieval or generation.

32. How do you optimise token usage in a RAG system?

• Retrieve fewer, better chunks: rerank and keep only the top few instead of stuffing twenty.
• Remove duplicates and near-duplicates before building the prompt.
• Compress context: extract only relevant sentences, or summarise long sections once at ingestion time.
• Keep the system prompt short and stable, and put it first so the provider's prompt caching can reuse it across requests.
• Summarise or trim old conversation turns instead of resending the full history.
• Set a maximum output length and ask for concise formats such as JSON fields rather than prose.
• Route easy questions to a smaller, cheaper model and reserve the large model for hard ones.
• Cache answers to frequent questions, including semantic caching for paraphrased repeats.

Our guide to AI token budgets and cost limits covers enforcing these limits in code.

33. When would you choose RAG, fine-tuning or prompt engineering?

Start with prompt engineering because it is fastest to change. Add RAG when the model needs knowledge it does not have: private, frequently changing or citable information. Choose fine-tuning when you need a consistent behaviour, format or style that prompts cannot hold reliably, or want a smaller model to match a larger one on a narrow task. Fine-tuning is a poor way to add facts that change, because every update means retraining. The approaches combine well: a fine-tuned model can still use RAG. We compare them in detail in RAG vs fine-tuning vs prompt engineering.

Prompt engineering and context engineering interview questions

34. What makes a good prompt?

A good prompt states the task clearly, gives the context the model needs, sets constraints (length, tone, what not to do), shows the expected output format, and includes examples when the format or judgement is subtle. Structure helps: separate instructions from data with headings or XML-style tags, and put untrusted input inside clearly marked sections. Then test it on real inputs and iterate, because prompt quality is measured, not guessed.

35. Explain zero-shot, few-shot and chain-of-thought prompting.

Zero-shot gives only the instruction. Few-shot adds a handful of worked input and output examples, which teaches format and edge cases. Chain-of-thought asks the model to reason through intermediate steps before answering, which improved accuracy on multi-step problems in the 2022 research that named it. Many current models have built-in reasoning modes, so explicit "think step by step" instructions matter less with them; ask for the final answer in a structured field so it is easy to parse.

36. What is context engineering, and how is it different from prompt engineering?

Prompt engineering is about wording the instruction. Context engineering is about deciding everything the model sees on each call: the system prompt, which documents are retrieved, which tool definitions are offered, how much conversation history and memory is included, and in what order. In agents, the context changes on every step, so the job becomes selecting, compressing and isolating information so the model has what it needs and nothing that distracts it. Most failures in production agents are context failures rather than wording failures. Our article on short-term and long-term agent memory goes deeper.

Fine-tuning and PEFT interview questions

37. What is parameter-efficient fine-tuning, and how does LoRA work?

Full fine-tuning updates every weight, which needs a lot of GPU memory and produces a full copy of the model per task. Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a small number of new parameters. LoRA (low-rank adaptation) assumes the change a task needs is low-rank: for a weight matrix W it trains two small matrices B and A of rank r, so the effective weight is W + BA. Only A and B are trained, often well under 1% of the parameters. The adapter can be merged into the weights for inference, or kept separate so one base model serves several adapters.

38. What is QLoRA, and what are other PEFT methods?

QLoRA loads the frozen base model in 4-bit precision (the NF4 data type, with double quantisation of the scaling constants) and trains LoRA adapters on top, which made fine-tuning very large models possible on a single GPU. Other methods include adapters (small bottleneck layers inserted into each block), prefix tuning and prompt tuning (learned vectors added to the input instead of changed weights). LoRA and QLoRA are the most common in practice. Whatever the method, data quality decides the result: a few thousand clean, consistent examples beat a large noisy set, and you need a held-out set to catch overfitting.

Function calling, tools and AI agent interview questions

39. How does function calling (tool use) work?

You describe each tool to the model with a name, a description and a JSON Schema for its arguments. When a request needs a tool, the model returns a structured tool call instead of text, for example get_order_status with {"order_id": "A1024"}. Your code, not the model, validates the arguments, runs the function and returns the result; the model then uses that result to write its answer or call another tool.

{
  "name": "get_order_status",
  "description": "Look up the delivery status of one order by its ID.",
  "input_schema": {
    "type": "object",
    "properties": { "order_id": { "type": "string", "description": "Order ID such as A1024" } },
    "required": ["order_id"]
  }
}

Good practice: precise names and descriptions, enums for fixed values, strict argument validation, timeouts, concise structured results and logs of every call. The Model Context Protocol (MCP) standardises how applications expose tools and data to models, so one server can work with many clients; see our article on MCP tool approval and least privilege.

40. What is an AI agent, and how is an agentic workflow different from a chain?

An agent is an LLM running in a loop: it decides the next action, calls a tool, observes the result, updates its working context and repeats until the task is complete or a limit is reached. A chain or workflow follows steps fixed in code; an agent chooses its own steps. Workflows are more predictable, cheaper and easier to test, so use them when the steps are known, and use agents when the path genuinely depends on what the model finds. Common patterns are routing, planner and executor, reflection (the model critiques its own output) and multi-agent set-ups where specialised agents hand work to each other. Production agents need step and cost limits, idempotent tools, human approval for irreversible actions and full traces of every step; see human-in-the-loop approvals for AI agents.

Evaluation and hallucination interview questions

41. How do you reduce hallucinations in a production application?

• Ground answers in retrieved sources and instruct the model to answer only from them.
• Give the model an explicit way to say it does not know, and reward that in evaluation.
• Ask for citations and check that each cited passage exists and supports the claim.
• Use tools for facts and arithmetic (database lookups, calculators) instead of the model's memory.
• Use structured outputs with schema validation so malformed answers are caught.
• Keep a human review step where errors are costly.

Our guide to preventing AI hallucinations in business automation has examples.

42. How do you evaluate an LLM application?

Build a test set of real inputs with expected outcomes, including edge cases and adversarial inputs, and run it on every change to the prompt, model or retrieval. Score with exact checks where possible (valid JSON, correct label, required fields), task metrics such as context precision, recall and faithfulness for RAG, and an LLM-as-judge with a written rubric for open-ended quality, spot-checked by people because judges have their own biases. Track latency and cost per request alongside quality, and monitor live traffic after release, because real users find cases your test set missed. Our practical guide to LLM evals walks through a set-up.

GenAI security interview questions

43. What is prompt injection, and how do you defend against it?

Prompt injection is input that makes the model ignore or override its instructions. Direct injection comes from the user ("ignore previous instructions and..."). Indirect injection hides instructions inside content the model reads, such as a web page, email or retrieved document. No prompt wording fully prevents it, so the defence is architectural: treat all model input and output as untrusted, keep secrets out of prompts, give tools the least privilege they need, require confirmation before irreversible actions, validate model output before it reaches a database, shell or browser, and filter or label untrusted content. Assume an injection will sometimes succeed and limit what it can do.

44. What are the main risks in the OWASP Top 10 for LLM applications?

The 2025 edition lists:

1. Prompt injection
2. Sensitive information disclosure
3. Supply chain (models, datasets, plugins)
4. Data and model poisoning
5. Improper output handling
6. Excessive agency (tools or permissions beyond what the task needs)
7. System prompt leakage
8. Vector and embedding weaknesses
9. Misinformation
10. Unbounded consumption (runaway cost or resource use, sometimes called denial of wallet)

In an interview, pick two and explain the control: for excessive agency, scoped credentials and human approval; for unbounded consumption, rate limits, token caps, step limits for agents and per-user budgets. For data leakage in RAG, enforce document permissions at retrieval time, not in the prompt.

GenAI system design interview questions

45. Design a production GenAI assistant for a company's internal documents.

Walk through it in layers and state your trade-offs:

• Ingestion: connectors pull documents on a schedule, extract text (with OCR for scans), chunk by section, embed, and store chunks with source, date and access-control metadata. Re-index changed documents only.
• Retrieval: hybrid vector and keyword search, filtered by the user's permissions, then a reranker.
• Orchestration: a service builds the context (system prompt, retrieved chunks, short history), calls the model through a provider-neutral layer with timeouts, retries and a fallback model, and streams the answer with citations. See our provider-neutral AI layer design.
• Tools: read-only by default; any write action needs explicit user confirmation.
• Guardrails: input checks for injection and abuse, output checks for leaked personal data and schema validation.
• Cost and latency: prompt caching, model routing, answer caching, per-user token budgets, and long jobs moved to a queue.
• Quality: an evaluation set run in CI, user feedback buttons, and traces of each request (retrieved chunks, prompt, model, tokens, latency) for debugging.
• Governance: audit logs, data retention rules and a clear choice about which data may be sent to which provider.

Finish by naming what you would measure first: retrieval recall on real questions, answer faithfulness, cost per conversation and the share of questions the assistant correctly declines.

Where generative AI is used in real products

Interviewers often ask for examples, and the strongest answers name a problem, the technique and the measured result. Common production uses are:

• Software development: code completion, review and test generation in tools such as GitHub Copilot, Cursor and Claude; compared in our AI coding assistants comparison.
• Customer support: RAG assistants that answer from help-centre articles and hand over to people when unsure.
• Document processing: extracting fields from invoices, contracts and forms into structured data; see automating invoice processing with AI.
• Search and knowledge: semantic search over internal wikis, tickets and code.
• Content and marketing: drafts, summaries and product descriptions with human editing.
• Operations: log analysis and incident summaries; see AI for DevOps log analysis.

Browse the assistants and model platforms in our AI tools directory, including ChatGPT, Google Gemini, Mistral AI and Hugging Face for open-weight models.

Practice checklist before the interview

• Explain attention, tokens and embeddings on a whiteboard without notes.
• Build a small RAG app, write ten test questions, and measure context recall and faithfulness before and after changing chunk size.
• Write one tool definition with a JSON Schema and handle a bad argument from the model.
• Prepare a story about a failure: a hallucination, a cost spike or a prompt injection, and what you changed.
• Be ready to sketch the architecture of your project and defend each component.

Next steps

Most AI engineer roles also test general backend skills, so pair this set with our REST API design, Node.js and Docker interview questions. For anything version-specific, such as context window sizes, model names or API parameters, check the provider's current documentation before the interview, because those details change faster than the concepts in this guide.