277 AI Engineer Interview Questions & Answers

Self attention lets each token build a new representation using relevant information from allowed tokens in the same sequence. Each token creates a query, key, and value. The query is compared with keys to get scores. A mask can block some positions. Softmax turns the allowed scores into attention weights, and those weights combine the values into a contextualized representation. In causal attention, future tokens receive zero attention weight.

Self attention is a way for each piece of a sentence to use useful clues from other allowed pieces in the same sentence. For example, in "I love NLP !", the piece "love" can use the words around it to understand its role better. The system gives more importance to useful pieces and less importance to others. Some pieces can also be hidden from view. This is important when generating text because the current position must not use words that come later. The result is a richer meaning for every piece.

1. Should I explain causal masking, where a position cannot use future tokens?
2. Would you like the matrix formula as well as the intuitive explanation?

Self attention starts with the representation of each token in one sequence. Learned linear projections create three vectors for every token: a query Q, a key K, and a value V. The query represents what the current token is looking for. The key represents what a token can match against. The value carries the information that may be passed forward.

For each token, its query is compared with the keys from tokens in the same sequence. The common scaled dot product score is QKᵀ divided by the square root of dₖ, where dₖ is the key vector size. Scaling keeps the scores from becoming too large before softmax.

A mask is applied when some positions must not be used. In causal attention, future positions are blocked. The mask contributes zero to allowed positions and negative infinity to blocked positions before softmax. As a result, blocked positions receive attention weight zero.

Softmax is applied across each row of scores. It converts the allowed scores into attention weights that sum to one. A larger weight means that position contributes more information to the current token.

The model then takes a weighted sum of the value vectors. This produces one contextualized representation for every input token. The sequence length stays the same, but each output representation now contains information gathered from relevant allowed tokens.

Self attention can connect tokens that are far apart in a sequence. Its main practical limitation is cost. The attention matrix grows with the square of the sequence length, so long sequences require more computation and memory.

1. Start with the representation of every token in the same sequence.
2. Apply learned linear projections to create Q, K, and V for every token.
3. Compute scaled attention scores using QKᵀ divided by the square root of dₖ.
4. Add a mask when some positions must be blocked. For causal attention, future positions are blocked.
5. Apply softmax across each row. Allowed positions become attention weights that sum to one, while blocked positions receive weight zero.
6. Multiply the attention weights by V and sum the value vectors for each row.
7. Return one contextualized representation for every input token.

For the attention score and weighted value steps in one attention head, n tokens with vector width d require O(n²d) computation. The attention matrix itself needs O(n²) memory. The learned Q, K, and V projections have their own linear algebra cost. The quadratic attention matrix is the main reason standard self attention becomes expensive for long sequences.

A common mistake is saying that self attention always lets every token use every other token. A mask may block some positions. Another mistake is saying that Q, K, and V come from different sequences. In self attention, they are created from representations of the same sequence. It is also wrong to treat attention weights as fixed values because they depend on the current representations and learned projections. Candidates may also forget that future positions receive zero weight after causal masking or ignore the quadratic growth of the attention matrix.

Explain the flow in order: create Q, K, and V, compare queries with keys, apply the mask, use softmax to get weights, and combine the values. Use the small "I love NLP !" example and mention that causal masking gives future positions zero attention weight.

Cross attention lets one representation get information from another representation. The queries come from the representation that wants information, while the keys and values come from the representation that provides it. The model compares each query with the keys, converts those scores into weights, and uses the weights to combine the values. A common example is a decoder attending to encoder outputs during translation.

Cross attention is a way for one group of information to look at another group and choose what is useful. Imagine translating a sentence. While producing the next word, one part needs to look back at the original sentence and focus on the most helpful pieces. It can give more importance to useful pieces and less importance to unrelated ones. The same idea can connect words with image information. The important point is that one side asks for information and a different side provides the information that may be useful.

1. Do you want the explanation focused on encoder decoder models or also multimodal models?
2. Should I include the attention formula and tensor shapes?

Cross attention uses queries from one representation and keys and values from another representation. The query side is the asker. The key and value side is the provider.

For example, in an encoder decoder translation model, decoder hidden states produce the queries. Encoder outputs produce the keys and values. This lets the decoder read relevant parts of the encoded source sentence while generating the target sentence.

The model first creates Q, K, and V with learned linear projections. For each query, it compares that query with every key using a dot product. The scores are divided by the square root of the key dimension. Softmax then turns those scores into attention weights. For one query, those weights sum to one across the available key and value positions. The output for that query is the weighted sum of the values.

In matrix form, the core operation is softmax of Q times K transpose divided by the square root of d k, followed by multiplication by V. If there are n q query positions and n k key positions, the attention weight matrix has shape n q by n k. The output has one result for every query position and value dimension.

Cross attention is useful when information must move between different representations. Common examples include encoder decoder models and multimodal models where text queries attend to image features.

The main limitation is cost. Every query may compare with every allowed key, so longer query and key sequences require more computation and memory. A common mistake is confusing cross attention with self attention. In self attention, queries, keys, and values come from the same representation. In cross attention, the queries come from one representation while the keys and values come from another.

1. Start with representation A, which needs information, and representation B, which provides information.
2. Project representation A into queries Q.
3. Project representation B into keys K and values V.
4. Compute a score for every query and key pair using Q times K transpose divided by the square root of d k.
5. Apply softmax across the key positions for each query. This creates attention weights.
6. Multiply the attention weights by V. This produces one weighted value result for each query position.
7. Pass the cross attention output to the next model operation, such as the rest of a decoder block.

For n q query positions, n k key positions, key dimension d k, and value dimension d v, computing the attention scores takes work proportional to n q times n k times d k. Combining the attention weights with the values takes work proportional to n q times n k times d v. The attention weight matrix uses memory proportional to n q times n k. This is why long query and key sequences can be expensive.

A common mistake is saying that Q, K, and V always come from the same representation. That describes self attention, not cross attention. Another mistake is treating the attention scores as the final output. The scores are first converted into weights with softmax, and those weights are then used to combine V. Candidates may also forget that every query can compare with every allowed key, which affects computation and memory as the two sequence lengths grow.

Start with the source rule: queries come from one representation, while keys and values come from another. Then explain the score, softmax, and weighted value flow. Finish with one concrete example such as decoder queries attending to encoder outputs.

Causal masking makes each token position look only at itself and earlier positions. In self attention, positions to the right are blocked, which creates a lower triangular visibility pattern. This prevents future token leakage during autoregressive training and matches decoding, where the model generates each new token from the tokens already available.

Causal masking is a rule that stops one word position from looking ahead at words that come later. Imagine the sentence "The cat sat on the mat." When working at "sat," the system may use "The," "cat," and "sat," but it must not use "on," "the," or "mat." This matters because it should learn what comes next without secretly seeing future words. The same rule matches generation, because words that have not been produced yet are not available to use.

1. Should I explain both training and decoding?
2. Would you like me to show the triangular visibility pattern?

Causal masking is used inside self attention. Self attention lets one token position use information from other token positions. The causal mask limits which positions are visible.

For a query at position i, only key and value positions j where j is less than or equal to i are visible. Positions where j is greater than i are blocked. If you draw this rule as a matrix, the allowed cells form a lower triangular shape. The diagonal is allowed because a position may see itself.

The mask is applied to attention scores before softmax. Softmax is the step that turns the usable scores into attention probabilities. Scores for future positions are blocked, so those positions receive no attention probability. This prevents future token leakage.

During autoregressive training, the model can process many sequence positions together, but every position still follows the same visibility rule. The representation at position i uses only tokens from the start of the sequence through position i. Its output is then used to predict the following token. This lets training use complete sequences without allowing a position to see future answers.

During autoregressive decoding, the same rule matches generation. At each step, the model has only the prompt and tokens generated so far. It predicts one new token, adds that token to the sequence, and repeats. A KV cache may store earlier key and value states to reduce repeated computation, but it does not change the causal visibility rule.

The main limitation is that causal masking is intentionally one directional. It is correct for next token generation, but it is not the right attention pattern when every position should use both earlier and later context.

1. Take a sequence of token positions in order.
2. For query position i, allow attention only to key and value positions j where j is less than or equal to i.
3. Block every position where j is greater than i before softmax.
4. The allowed cells form a lower triangular visibility pattern.
5. During autoregressive training, the representation at each position uses only itself and earlier tokens and is used to predict the following token.
6. During autoregressive decoding, generate one token from the available prefix, append it, and repeat with the same causal rule.

A common mistake is saying that causal masking lets a token see only earlier positions and not itself. The diagonal is normally visible, so the current position is included. Another mistake is thinking the mask is only a decoding feature. It is also essential during autoregressive training because it prevents future token leakage when many positions are processed together. A third mistake is confusing causal masking with a KV cache. The mask controls visibility, while the cache stores earlier key and value states to reduce repeated computation during decoding.

Start with the rule that position i can see only positions up to and including i. Then describe the lower triangular mask. Finish by connecting the same rule to prevention of future token leakage during training and step by step generation during decoding.

Multi head attention lets a Transformer run multiple attention operations in parallel. Each attention head learns its own projection subspace and can capture different token relationships. The outputs from all heads are concatenated and passed through an output projection. We use multiple heads because they provide richer representations, but increasing the number of heads also increases computation and memory cost.

Multi head attention is a method that helps a model understand relationships between words or tokens in a sequence. Instead of looking at the sequence in only one way, the model creates several attention views and combines them. Each view can focus on different patterns in the input.

1. Are we discussing encoder self attention, decoder self attention, or cross attention?
2. Should I focus more on the architecture explanation or production tradeoffs?

Multi head attention is a core part of Transformer models. The input embeddings are projected into query, key, and value representations for each attention head. Each head performs scaled dot product attention independently. The attention scores determine how much information each token receives from other tokens.

Each head works in a learned projection subspace. This means different heads can capture different relationships or patterns in the sequence. For example, one head may capture nearby token relationships while another may capture longer range dependencies. The model does not manually assign these roles. They are learned during training.

After each head produces an output, the outputs are concatenated. The combined representation is passed through a final linear projection to return to the model dimension.

The main benefit of multiple heads is that the model can learn several views of the same input at the same time. This increases the expressive power of the Transformer. The main tradeoff is cost. More heads require more attention calculations and more memory. In production systems, engineers balance representation quality with latency, memory usage, and serving cost.

A common mistake is saying that every attention head has one fixed purpose. Heads can learn different patterns, but their exact specialization is not guaranteed. Another mistake is ignoring the cost tradeoff. More heads can improve model capacity, but they also increase computation and memory requirements.

Start with the main reason: multiple heads allow the Transformer to capture different relationships in parallel. Then explain projections, attention calculation, concatenation, and the capacity versus cost tradeoff.

Feed Forward Networks are the per token transformation layers inside a Transformer block. They take each token representation, expand it to a larger hidden dimension, apply a nonlinear activation such as GELU, and contract it back. The FFN adds learning capacity after self attention has mixed information between tokens.

A Feed Forward Network in an LLM changes the representation of each token. It helps the model learn more complex patterns after information from other tokens has been combined.

1. Are you asking about the standard FFN structure inside a Transformer block?
2. Should I focus more on the mathematical flow or production impact?

A Feed Forward Network, also called FFN, is a position wise neural network inside each Transformer block. Position wise means the same FFN weights are applied separately to every token position.

A Transformer block usually has self attention and then an FFN. Self attention allows tokens to share information with other tokens. The FFN then transforms the features of each token independently.

The FFN has three main steps. First, a linear layer expands the token representation from the model dimension to a larger intermediate dimension. Second, an activation function such as GELU adds nonlinear behavior. Third, another linear layer contracts the representation back to the original model dimension.

For example, a model can transform a token vector from 4096 dimensions to 11008 dimensions, apply GELU, and return to 4096 dimensions. Exact values depend on the model design.

The expansion gives the network more space to learn useful intermediate features. The activation allows the model to represent complex relationships that a simple linear transformation cannot capture.

In production, FFN layers contain many parameters and require significant compute and memory. Larger FFN sizes can increase model capacity but also increase inference cost.

A common mistake is saying the FFN mixes tokens. Self attention performs token mixing. The FFN mainly changes each token representation independently.

Common mistakes include confusing FFN with self attention, saying FFN stores memories, or saying FFN only increases dimensions. The FFN transforms each token representation independently while attention handles information exchange between tokens.

Start with the main conclusion that FFN is the per token nonlinear transformation inside a Transformer block. Then explain the flow: expand, activate, contract. Mention that attention mixes tokens while FFN transforms features.

Skip connections, also called residual connections, add the input of a Transformer sublayer back to the sublayer output. The model learns the change, called the residual, instead of learning a completely new representation. This creates a shorter path for information and gradients, which helps deep Transformers train more reliably and preserve useful representations.

Skip connections help a Transformer keep useful information while adding new changes. A sublayer such as self attention or a feed forward network receives an input and creates an update. The original input is added back to this update so important information can continue through the model.

1. Are we discussing the original Post Norm Transformer design or a modern Pre Norm LLM design?
2. Should I focus more on training stability, gradient flow, or Transformer block implementation details?

A Transformer block contains sublayers such as self attention and feed forward networks. Each sublayer produces a new representation from its input. A residual connection adds the original representation back to the sublayer result using element wise addition.

The basic idea is:

Output = Input + Sublayer(Input)

The sublayer learns the useful change from the current representation instead of rebuilding the entire representation. This is called learning a residual.

Residual connections matter because they create shorter paths for gradients during training. Without these paths, very deep networks can become harder to optimize because training signals must pass through many layers. The shortcut path helps gradients move through the network more effectively.

They also help preserve representations. Information that is already useful can pass through the shortcut while the sublayer adds refinements. This allows Transformers to use many layers while maintaining stable information flow.

LayerNorm placement changes the Transformer block design. In Post Norm Transformers, the sublayer output is added to the input and LayerNorm is applied after the residual addition. In Pre Norm Transformers, LayerNorm is applied before the sublayer and the residual addition happens afterward. Modern large language models commonly use Pre Norm because it generally provides more stable training for deep models.

Residual connections do not guarantee that every model will train successfully. They are one architectural mechanism that improves optimization, supports gradient flow, and enables deeper Transformer networks.

Common mistakes include confusing residual connections with attention mechanisms. Residual connections do not retrieve information or create attention scores. They only add the original input back to the sublayer output. Another mistake is saying residual connections completely remove gradient problems. They improve gradient flow but do not guarantee perfect training. Candidates also often confuse Pre Norm and Post Norm placement.

Start with the main idea that residual connections add the input back to the sublayer output. Then explain why they matter: they provide shorter gradient paths, preserve useful representations, and make deep Transformer training more stable. Mention Pre Norm and Post Norm as important design choices.

The context window is the maximum number of tokens an LLM can process in one request. It contains both input tokens and generated output tokens. It matters because the model can only use information inside this window, so AI systems must select useful context to balance answer quality, cost, and latency.

The context window is the amount of information an LLM can use during one request. A request can include instructions, conversation history, selected document information, and a user question. The input and output share the same token budget.

1. Should the explanation focus on the model limit or the application design used to manage long context?
2. Does the system need to handle long conversations or large documents?

A context window is a shared token budget that limits what an LLM can see during inference. Tokens are small pieces of text processed by the model. The input tokens and output tokens use the same limit. If the input uses more tokens, fewer tokens remain for the generated answer.

When an application prepares a request, it chooses what information should enter the context window. It usually keeps important instructions, recent conversation, and relevant retrieved information. When the content is too large, the system can remove older content, summarize information, or select the most useful parts.

This matters because the model cannot use information outside the current context window. If important evidence is removed, answers may become incomplete or incorrect. A good context selection policy helps the model receive the information needed for the task.

A larger context window can increase processing cost and latency because the model handles more tokens. Production systems balance accuracy, cost, and speed by controlling which information is included.

A common mistake is thinking an LLM automatically remembers all previous conversations. The model only uses information inside the current context window unless an external system provides memory. Another mistake is assuming a larger context window always gives better results. More tokens can increase cost and latency, and unnecessary information can reduce answer quality.

Start with the practical definition: a context window is a shared token budget for input and output. Then explain context selection, what happens when the limit is reached, and why this affects cost and reliability.

Logits are the raw unnormalized scores an LLM produces for every possible next token. The model converts these scores into probabilities using softmax, then decoding controls such as temperature, top k, and top p influence how the next token is selected. The selected token is added back to the context, and the process repeats to generate text.

Logits are numbers produced by an AI model to rank possible next tokens. They show how strongly the model prefers each possible token, but they are not probabilities yet. A larger logit means the model gives that token a higher score compared with other choices.

1. Should I focus on greedy decoding or sampling based decoding for this explanation?
2. Should I include production decoding controls such as temperature, top k, and top p?

When an LLM generates text, it starts with the current context, such as "The cat". The model produces one logit score for every token in its vocabulary. These scores are unnormalized, which means they cannot be directly interpreted as probabilities.

The logits go through softmax. Softmax converts the scores into probabilities that add up to one. The system can then use decoding methods to choose the next token. Greedy decoding selects the token with the highest probability. Sampling methods choose from the probability distribution to create more variation.

Temperature changes how focused the probability distribution is. A lower temperature makes the output more predictable because high probability tokens become more likely. A higher temperature allows more variation. Top k sampling keeps only a fixed number of highest probability tokens before selecting. Top p sampling keeps the smallest group of tokens whose combined probability reaches a chosen threshold.

After a token is selected, the system appends it to the context and runs the model again. This continues until a stopping condition is reached. Logits are important because they connect the model output to the final generated text.

In production systems, engineers choose decoding settings based on the required balance between consistency, creativity, quality, and user experience. Logits do not contain final answers and do not guarantee that the selected token is correct. They only provide the scores used during decoding.

Common mistakes include saying logits are probabilities, skipping the softmax conversion step, or saying the model directly creates the final text. Logits are only raw scores. Another mistake is assuming temperature changes the model knowledge. Temperature only changes how the probability distribution is used during decoding. Engineers should also understand that decoding settings change output behavior but do not retrain the model.

Explain the complete flow in order: input context, logits, softmax probabilities, decoding controls, next token selection, and repeat. Use the example "The cat" to make the process easy to understand.

Temperature controls how random an LLM's output is during generation. It changes token probabilities by rescaling logits before softmax. Lower temperature makes the output more focused and predictable, while higher temperature makes the output more diverse and creative. I use lower temperature for tasks that need consistency and higher temperature for tasks that need exploration.

Temperature is a setting that controls how an LLM selects the next token when generating text. It changes whether the model prefers the most likely choices or explores more possible choices.

1. What type of application is this model being used for, such as factual answers, coding, or creative generation?
2. Are other decoding settings like top p or top k used together with temperature?

An LLM first produces logits. Logits are raw scores that represent how likely each possible next token is before probabilities are created. Temperature changes these scores before softmax converts them into a probability distribution.

The rescaling step is:

z_i(T) = z_i / T

When temperature is below 1, the difference between token scores becomes larger. The probability distribution becomes sharper, so the model is more likely to select high probability tokens. This lowers entropy, which means less randomness and more predictable output.

When temperature is around 1, the original distribution is mostly maintained. This often gives a balance between output quality and diversity.

When temperature is above 1, token score differences become smaller. The distribution becomes flatter, which increases entropy and allows lower probability tokens to be selected more often. This creates more diverse responses but increases the chance of off topic or lower quality output.

Temperature does not change what the model learned. It only changes how the model samples during inference. Lower temperature is useful for factual answers, extraction, and consistent formatting. Higher temperature is useful for brainstorming and creative tasks. A common mistake is thinking temperature improves model knowledge. It only changes the selection behavior among available token choices.

Common mistakes include thinking temperature changes the model's knowledge, training data, or intelligence. Temperature only changes decoding behavior during inference. Another mistake is assuming higher temperature always creates better answers. Higher diversity can also increase off topic responses or lower quality outputs.

Explain the practical meaning first: temperature controls randomness during generation. Then explain the mechanism: logits are rescaled before softmax, changing the probability distribution. Finish with the quality versus diversity tradeoff.

Top k and Top p are decoding methods that control which tokens an LLM can choose during generation. Top k keeps a fixed number of highest probability tokens, while Top p keeps the smallest group of tokens whose combined probability reaches a chosen limit. I use them with temperature to control randomness. Top k gives a fixed candidate size, while Top p changes the candidate size based on the probability distribution.

Top k and Top p are ways to control how an LLM chooses the next word. The model first gives probabilities to possible next words. Top k keeps a fixed number of high probability words. Top p keeps enough words to reach a chosen total probability. For example, Top k with k equal to 3 always keeps three words. Top p with p equal to 0.60 keeps the smallest group of words whose total probability is at least 0.60.

1. Which output behavior is more important for this system, accuracy, creativity, or consistency?
2. Are decoding settings tuned differently for different production use cases?

An LLM generates a probability distribution for possible next tokens. The decoding method decides which tokens remain available before selecting one token.

Top k sampling creates a shortlist with a fixed size. If k is 3, only the three highest probability tokens can be selected. The remaining tokens are removed before sampling. This makes the number of choices predictable.

Top p sampling, also called nucleus sampling, creates a shortlist based on probability mass. The system sorts tokens by probability and keeps the smallest group where the combined probability reaches the chosen value. For example, with p equal to 0.60, tokens are added until their combined probability reaches 0.60. The number of tokens can increase or decrease depending on how the probabilities are distributed.

Temperature changes the probability distribution before Top k or Top p filtering. A lower temperature makes high probability tokens stronger and produces more focused output. A higher temperature makes probabilities more even and can increase variation.

Top k is useful when a fixed number of choices is preferred. Top p is useful when the system needs an adaptive candidate set. Both methods only control token selection during inference. They do not verify facts or guarantee correct answers.

Generate next token probabilities, sort candidate tokens by probability, apply Top k using a fixed token count or Top p using a cumulative probability threshold, normalize the remaining probabilities, and sample one token.

The cost depends on the number of token probabilities considered during inference. Candidate filtering requires selecting or sorting possible tokens before sampling.

A common mistake is thinking Top k or Top p selects the final answer directly. They only limit the candidate tokens before sampling. Another mistake is thinking Top p always keeps the same number of tokens. The number changes depending on the probability distribution. Temperature does not add knowledge to the model. It only changes randomness during token selection.

Start with the main difference. Explain that Top k uses a fixed number of tokens and Top p uses a probability cutoff. Then explain that temperature changes the distribution before filtering.