277 AI Engineer Interview Questions & Answers

A foundation model is a large model pretrained on broad and diverse data so it learns general patterns that can transfer to many tasks. Instead of training a new model from the beginning for every task, an AI engineer can start with the pretrained model and adapt it with prompting or fine tuning. The main benefit is reuse. Important engineering concerns include adaptation cost, data needs, model quality, and safety.

A foundation model is a large general model that first learns from a very large and varied collection of data. The goal is to learn useful patterns before the model is given one narrow job. After this broad learning stage, the same model can become a starting point for tasks such as answering questions, classifying text, judging sentiment, or working with images. Engineers can guide it with instructions or train it further on a smaller task focused data set. This reduces repeated training work, but it still requires careful testing, cost control, and safety checks.

1. Should I focus on language foundation models, or include models that also learn from images and audio?
2. Do you want me to compare prompting with fine tuning?

A useful way to think about a foundation model is as a general starting point rather than a finished task model.

First comes pretraining. The model sees broad and diverse data such as text, code, images, or audio. A common pretraining objective asks the model to predict a missing part of the input. For example, it might see a sentence with one hidden word and learn to predict the likely word. Repeating this prediction task across large amounts of data changes the model weights so they capture broad statistical patterns and useful knowledge.

The result is the foundation model. It is not limited to one task. Its pretrained weights provide a strong starting point that can transfer to downstream tasks such as question answering, text classification, sentiment analysis, or image related tasks.

Next comes adaptation. Prompting gives the model instructions and examples as input without changing its weights. Fine tuning trains the model on smaller task specific data and updates model weights. Fine tuning can produce more specialized behavior, but it requires training compute, suitable data, evaluation, and model management.

The main engineering benefit is reuse. A strong base model can make new systems faster to build, can reduce the amount of task specific training data needed, and can support many different tasks. The tradeoffs are adaptation cost and ongoing quality and safety work. Foundation models can still produce biased, incorrect, or harmful outputs. Production systems therefore need evaluation and monitoring. The model also does not automatically search, verify facts, cite sources, remember users, or call tools unless the surrounding application provides those mechanisms.

1. Pretrain one large general model on broad and diverse data such as text, code, images, or audio.
2. Use a prediction objective, such as predicting a missing part of the input, to update model weights and learn broad patterns.
3. Keep the resulting pretrained model as a general foundation model rather than a model for only one task.
4. Transfer the pretrained model to a downstream task such as question answering, text classification, sentiment analysis, or an image related task.
5. Adapt it with prompting when model weight changes are not needed, or use fine tuning on smaller task specific data when the weights should be updated.
6. Evaluate the adapted system for task quality, cost, bias, errors, misuse risk, and safety before and during production use.

A common mistake is saying a foundation model is already specialized for one task. It is a general starting point. Another mistake is treating prompting and fine tuning as the same thing. Prompting changes the input while keeping model weights unchanged. Fine tuning updates model weights using task specific training data. It is also wrong to assume the model automatically searches the web, verifies facts, cites sources, remembers users, or uses tools. Those abilities require surrounding system components. Broad pretraining also does not make the model always correct or safe.

Start with the simple idea that a foundation model is pretrained on broad data and then reused for many tasks. Explain the flow in order: pretraining objective, foundation model, knowledge transfer, prompting or fine tuning, then engineering consequences such as faster development, lower task data needs, adaptation cost, quality, and safety.

A Large Language Model is a neural network trained on large amounts of text to predict the next token from the context it receives. A Transformer processes the input tokens and produces scores for possible next tokens. Those scores become probabilities, and a decoding method selects a token. During training, the model changes its weights to reduce prediction error. During inference, the weights are normally fixed and the model generates text one token at a time. Its output is probabilistic, so it can be wrong, reflect bias, and lack current knowledge unless an external system supplies it.

A Large Language Model learns patterns from large collections of text. Its main job is simple: given some text, predict what token could come next. A token is a small piece of text. The model repeats this process to create longer responses. Training is when the model learns by changing its internal weights after comparing predictions with the real next tokens. Inference is when we use the trained model. Its weights normally stay fixed. The model can produce useful text, but its answers are not guaranteed to be correct or current.

1. Would you like the explanation to focus only on basic model behavior, or also on production concerns?
2. Should I explain the Transformer internals in detail, or keep them at a high level?

Start with a small example. Suppose the context is "The cat sat on the". The model produces a probability distribution over possible next tokens. In the diagram, "roof" has a stronger probability than alternatives such as "sofa" or "floor". A decoding method then selects a token from that distribution, for example by sampling.

The Transformer performs the main computation. Input tokens are first converted into numeric representations that include token information and position information. Masked self attention lets each position use the available earlier context while blocking future positions. The result passes through an add and normalization step, then a feed forward network, followed by another add and normalization step. These Transformer blocks can be repeated several times. The output layer then produces scores for the vocabulary, and those scores are converted into probabilities for the next token.

Training and inference are different. During training, the model sees large collections of text. At each position, it predicts the next token, compares that prediction with the actual next token, and changes its weights to reduce prediction loss. During inference, the model parameters are normally fixed. The model receives a prompt, computes probabilities for the next token, selects one token, appends it to the text, and repeats the process.

This behavior matters because an LLM is a probabilistic pattern model, not a guaranteed source of truth. It can confidently generate incorrect or made up information. It can reflect bias in its training data. Its knowledge is limited by its training data and surrounding system. It does not automatically know live events, search the web, verify claims, remember information across sessions, or use tools unless the surrounding application provides those capabilities.

1. Split the input text into tokens.
2. Convert the tokens into numeric representations with position information.
3. Pass the representations into a Transformer block.
4. Use masked self attention so each position can use the available earlier context without using future positions.
5. Combine the attention result with the residual path and apply normalization.
6. Pass the result through the feed forward network.
7. Combine that result with its residual path and apply normalization again.
8. Repeat the Transformer block for the configured number of layers.
9. Pass the final representation through the vocabulary output layer.
10. Convert the vocabulary scores into probabilities for possible next tokens.
11. Select one token using the configured decoding method.
12. Append that token to the context and repeat during generation.
13. During training, compare predictions with the actual next tokens and update model weights to reduce prediction loss.
14. During inference, normally keep the trained weights fixed and run only the prediction and decoding process.

A common mistake is saying that an LLM searches a database for the answer. The base model instead predicts tokens from its context and learned weights. Another mistake is saying that it always chooses the most likely token. A decoding method may sample from the probability distribution. Candidates also sometimes mix training with inference. Training changes the model weights, while normal inference does not. Another mistake is forgetting masking and claiming that next token prediction can use future tokens. Candidates may also assume the model automatically has current information, verifies facts, remembers past conversations, or uses tools. Those behaviors require support from the surrounding system.

Start with the one sentence definition, then walk through the small next token example. After that, explain the Transformer flow from tokens to probabilities, clearly separate training from inference, and finish with limitations. Keep the model itself separate from external retrieval, memory, search, verification, and tools.

A Transformer turns tokens into contextual representations by combining token embeddings with position information, then passing them through repeated blocks of self attention and feed forward processing. Each sublayer uses a residual path and LayerNorm, which is a normalization step. After the repeated blocks, a final LayerNorm and linear output projection produce vocabulary logits. Softmax converts those logits into next token probabilities. In the causal decoder style shown in the diagram, each token can attend to itself and earlier tokens, but not future tokens.

A Transformer is a model design that takes a sequence of text pieces and gradually builds a richer representation of what each piece means in context. It first turns every piece into numbers and adds information about its position in the sequence. It then repeats two main processing steps several times. One step lets each position gather useful information from allowed positions. The other transforms each position separately. Extra paths help keep earlier information. At the end, the model produces scores for possible next pieces and converts them into probabilities.

1. Are you asking about the causal decoder style Transformer used for next token generation?
2. Should I explain the attention calculations, or focus on the main data flow and purpose of each block?

Start with the input tokens. Each token is mapped to a learned embedding vector. Position information is added so the model can represent token order. The resulting sequence representation enters a Transformer block, and that block is repeated L times.

The first main part of each block is multi head self attention. The current representation is projected into query, key, and value matrices. Query and key values produce attention scores. Scaled dot product attention uses those scores to mix the value information. In the causal version shown in the diagram, a mask blocks future positions, so a token can attend only to itself and earlier tokens. The attention heads are concatenated, then a linear projection maps the result back to the model hidden size.

A residual path adds the original sublayer input to the attention output. LayerNorm then normalizes the hidden values. The second main part is the position wise feed forward network. It applies a linear transformation, GELU activation, and a second linear transformation independently at every sequence position. A second residual path and LayerNorm follow.

After the final repeated block, a final LayerNorm produces the final hidden states. A linear output projection maps each hidden state from the model hidden size to vocabulary logits. Logits are unnormalized scores for vocabulary items. Softmax converts them into probabilities. During generation, the system selects or samples a token from that probability distribution and repeats the process for the next position.

A practical tradeoff is attention cost. Standard self attention becomes much more expensive as sequence length grows because every position compares with many other positions. This increases compute and memory use for long contexts.

1. Split the input text into tokens.
2. Map each token to a learned embedding vector.
3. Add position information so token order is represented.
4. Send the sequence representation through L repeated Transformer blocks.
5. In each block, project the representation into query, key, and value matrices.
6. Compute scaled dot product self attention and apply the causal mask so each token can use only itself and earlier tokens.
7. Concatenate the attention heads and apply the output projection for the attention sublayer.
8. Add the residual path and apply LayerNorm.
9. Apply the position wise feed forward network using a linear transformation, GELU, and a second linear transformation.
10. Add the second residual path and apply LayerNorm.
11. After the final block, apply a final LayerNorm.
12. Apply a linear output projection from the model hidden size to the vocabulary size to produce logits.
13. Apply Softmax to convert logits into next token probabilities.

A common mistake is saying attention alone is the whole Transformer. A Transformer block also contains the feed forward network, residual paths, and normalization. Another mistake is forgetting position information. Token embeddings alone do not represent token order. Candidates also often confuse logits with probabilities. The linear output projection produces logits, while Softmax converts those logits into probabilities. In the causal architecture shown here, another mistake is allowing a token to attend to future positions during next token prediction.

Explain the flow in order: tokens, embeddings plus position information, repeated self attention and feed forward blocks, residual paths plus LayerNorm, final LayerNorm, vocabulary logits, then Softmax. State the purpose of each part in one short sentence. Also make clear that the diagram shows a causal decoder style Transformer for next token prediction.

A Transformer first turns tokens into embedding vectors and adds position information. It then passes those vectors through repeated Transformer blocks. Inside each block, self attention lets each token gather useful context, while a feed forward network transforms each token representation. Residual connections carry earlier information around both parts, and normalization keeps the representations well behaved. After the final block, the language model head projects the final hidden representation to vocabulary logits. Softmax can then turn those logits into probabilities for the next token.

A Transformer changes a sequence of text pieces into a prediction in several ordered steps. First, the text is split into tokens. Each token becomes a vector of numbers that represents learned information about that token. Position information is added so the model can tell which token came first, second, and so on. The model then passes these representations through repeated processing blocks. Each block lets tokens use information from other allowed tokens, transforms the result, and keeps useful earlier information. At the end, the model converts the final representation into scores for possible next tokens.

1. Should I explain a decoder style Transformer used for next token prediction?
2. Would you like the explanation to include the internal attention steps such as queries, keys, and values?

Start with token embeddings. An embedding maps each token to a learned vector. Position information is added because attention alone does not know token order.

The vectors then enter repeated Transformer blocks. The first major part is multi head self attention. Self attention lets each token gather context from other allowed token positions. In an autoregressive language model, a causal mask blocks attention to future tokens. Multiple heads let the model learn different attention patterns at the same time.

Attention forms queries, keys, and values from the block input. A query represents what a position is looking for. A key represents what another position can match against. A value carries the information that can be collected. Scaled dot product attention uses the query and key relationship to weight the values. The outputs from the attention heads are combined and projected.

After attention, a residual connection adds the attention output back to the sublayer input. Normalization then keeps the representation well scaled. The block next applies a position wise feed forward network. This is a small neural network applied separately to every token position. Another residual connection adds that output back to its input, followed by normalization.

The output then moves to the next Transformer block. After the final block, the language model head uses a linear projection to produce one logit for each vocabulary token. A logit is an unnormalized score. Softmax can convert those logits into probabilities used to choose or sample the next token.

1. Split the input text into tokens.
2. Map each token to a learned embedding vector.
3. Add position information so token order is represented.
4. Send the representations into a Transformer block.
5. Form queries, keys, and values for multi head self attention.
6. Apply the causal mask so each position uses only allowed token positions.
7. Combine the attention outputs, add the result back to the sublayer input through a residual connection, and normalize.
8. Apply the position wise feed forward network to each token representation.
9. Add the feed forward output back to its sublayer input through another residual connection, and normalize.
10. Repeat the Transformer block for the remaining layers.
11. Send the final hidden representations through the language model head.
12. Produce vocabulary logits and use softmax when next token probabilities are needed.

The main expensive part shown here is standard self attention. For a sequence with n tokens, attention compares token positions with other token positions, so the attention score work grows roughly with n squared. Longer context can therefore require much more computation and memory.

A common mistake is saying that attention alone is the whole Transformer. Attention is only one sublayer inside each repeated block. Another mistake is forgetting position information, which is needed because the model must represent token order. Candidates also sometimes omit the feed forward network or treat it as part of attention. Residual connections do not replace a sublayer output. They add the sublayer output back to its input. Another mistake is saying that softmax is part of every Transformer block. In this diagram, softmax comes after the language model head when vocabulary logits are converted into next token probabilities.

Explain the Transformer as one flow from tokens to probabilities. Start with embeddings and position information. Then describe one repeated block using self attention, a residual connection, normalization, feed forward processing, another residual connection, and normalization. Finish with the language model head, vocabulary logits, and softmax. This order makes each responsibility easy to follow.

Tokenization breaks text into smaller units called tokens and maps those tokens to integer IDs that a model can process. Modern tokenizers often use subword pieces, so rare or unseen words can often be represented by combining known pieces. The number of resulting tokens also matters because the model context limit is measured in tokens, not characters.

Tokenization is the step that turns written text into small pieces before a language model works with it. A short sentence may become several pieces, and a long or unusual word may become more than one piece. Each piece is also linked to a number that the model uses internally. The way text is divided depends on the set of pieces learned for that model. This matters because the model has a limited amount of text it can handle at one time, and that limit is counted using these pieces.

1. Should I explain a specific tokenizer or keep the answer provider neutral?
2. Would you like me to cover how rare or unknown text is handled?

A tokenizer has a vocabulary. The vocabulary is a set of token pieces, and each piece has an integer ID. During vocabulary construction, a tokenizer can learn useful character or subword patterns from a large text collection. One common method is byte pair encoding, which repeatedly combines frequent neighboring pieces to form useful vocabulary entries.

At inference time, the tokenizer reads the input text and produces an ordered sequence of vocabulary pieces. These pieces can be whole words, parts of words, punctuation, spaces, bytes, or other text units. In the diagram, the text "I love ice cream!" becomes several tokens. Each token then maps to an integer ID. Those IDs are used for embedding lookup before the model processes the sequence.

Subword tokenization helps because the vocabulary does not need a separate entry for every possible word. A rare or unseen word can often be represented by combining smaller known pieces. For example, a word such as "microchips" may be split into several known parts. Some tokenizers use a special unknown token when text cannot be represented. Byte based tokenizers can often avoid that fallback because they can represent text with smaller byte units.

Token count also affects context usage. A model has a maximum context length measured in tokens. More input tokens consume more of that capacity and leave less space for other context or generated output. More tokens also generally require more computation. The exact effects on latency and API cost depend on the model and serving system.

1. Build or load the tokenizer vocabulary, where each token piece has an integer ID.
2. Read the input text and split it into vocabulary pieces according to the tokenizer rules.
3. Use subword pieces when a whole word is not represented as one token.
4. If the tokenizer supports an unknown token and some text still cannot be represented, map that case to the unknown token. Byte based tokenizers can often avoid this fallback.
5. Convert every resulting token to its integer ID while keeping token order unchanged.
6. Use the token IDs for embedding lookup before the model processes the sequence.
7. Count the resulting tokens because that count consumes part of the model context capacity.

A common mistake is saying that one token always equals one word. Tokens can be whole words, parts of words, punctuation, spaces, bytes, or other units. Another mistake is assuming that every tokenizer handles unknown text in the same way. Some use an unknown token, while byte based tokenizers can often represent text using smaller units. Candidates also sometimes confuse token IDs with embeddings. A token ID is an integer lookup key, while an embedding is the numeric vector obtained through embedding lookup. Finally, character count is not the same as token count, so prompt length should be checked in tokens.

Start with the simple flow: text becomes token pieces, token pieces become integer IDs, and those IDs are used for embedding lookup before model processing. Then explain why subwords help with rare text and finish by connecting token count to context usage. Make it clear that exact token boundaries and IDs depend on the tokenizer.

Byte Pair Encoding, or BPE, is a subword tokenization method. It starts with small symbols such as characters or bytes. It repeatedly counts adjacent pairs and merges the most frequent pair into a new token. Each merge grows the vocabulary. After training, the learned vocabulary and merge rules are used to split new text into known tokens. Frequent patterns can become larger tokens, while rare words can fall back to smaller known pieces.

Byte Pair Encoding is a way to turn text into reusable pieces before an LLM processes it. Imagine the training words low, lower, and newest. We first split them into small symbols. Then we look for neighboring symbols that occur often and join them. A pair such as l followed by o can become lo. Later, lo followed by w can become low. Repeating this process creates a vocabulary containing useful pieces of different sizes. Common patterns often become larger tokens. Rare words can still be represented with smaller known pieces when the base symbol set covers their text.

1. Should I explain BPE starting from characters or from bytes?
2. Would you like me to walk through the low, lower, and newest example?

BPE learns its vocabulary through repeated pair merges. Start with the training text split into small symbols. In the diagram, the example contains low, lower, and newest. A word boundary marker may also be kept so the tokenizer can distinguish positions around word endings.

At each learning step, count every adjacent pair in the current tokenized training data. Choose the pair with the highest frequency. Replace its applicable occurrences with one new token, then add that token to the vocabulary. For example, l followed by o can become lo. A later merge can combine lo with w to create low. The process repeats. If several pairs have the same frequency, the tokenizer implementation needs a deterministic tie rule so the learned merge order is reproducible.

Training stops when the requested vocabulary size or merge limit is reached. Numbers such as 30000 or 50000 tokens are examples rather than universal values. Each additional merge can create a larger reusable token.

For new text, the tokenizer uses the learned vocabulary and merge order to produce subword tokens. Common patterns may be represented by larger pieces. Rare text can use smaller pieces instead. For example, lowest can reuse low and represent the remaining characters with smaller known tokens.

Vocabulary size creates an important tradeoff. A larger vocabulary can represent some text with fewer tokens, but the model must keep representations for more token entries. A smaller vocabulary uses fewer entries, but the same text may require a longer token sequence. With byte level BPE, every byte sequence can be represented from the base byte vocabulary. With character level BPE, coverage depends on which base characters are included.

1. Split the training text into initial symbols such as characters or bytes.
2. Count the frequency of every adjacent symbol pair.
3. Select the most frequent pair. If frequencies tie, use the tokenizer implementation's deterministic tie rule.
4. Replace applicable occurrences of that pair with one merged token.
5. Add the merged token to the vocabulary.
6. Repeat the counting and merging steps until the target vocabulary size or merge limit is reached.
7. Save the learned vocabulary and ordered merge rules.
8. Use those learned rules to segment new text into known subword tokens.
9. Represent rare text with smaller known pieces when a larger learned token is unavailable.

One common mistake is saying that BPE stores every complete word. It does not. It learns reusable subword tokens by merging repeated adjacent symbol pairs. Another mistake is saying that rare words always become an unknown token. With byte level BPE, arbitrary input bytes can fall back to the base byte tokens. With character level BPE, this depends on the base character vocabulary covering the input. Another mistake is assuming the model relearns merges during each response. The tokenizer uses merge rules learned earlier. Candidates also sometimes ignore tie handling when several pairs have the same frequency or assume that a larger vocabulary is always better.

Start with the main idea: BPE repeatedly merges frequent adjacent symbol pairs into reusable subword tokens. Then use the low, lower, and newest example to show one or two merges. Finish with the two practical points the interviewer is likely testing: rare text can fall back to smaller known pieces, and vocabulary size trades a larger token table for potentially shorter token sequences.

WordPiece and SentencePiece both break text into reusable subword pieces so many rare or unseen words can still be represented. WordPiece tokenization commonly uses greedy longest match against a learned vocabulary, with "##" marking continuation pieces. SentencePiece works directly with raw text and is a framework that can use models such as Unigram or BPE. Unigram scores possible segmentations probabilistically, while BPE follows learned merge rules. Neither approach guarantees that an unknown token can never occur.

Both methods help a language model handle words that are not stored as complete entries. They break text into smaller reusable pieces instead of needing one stored entry for every possible word. WordPiece usually chooses matching pieces from a learned list. SentencePiece can learn from the original text directly and supports more than one way to choose pieces. This difference matters because the same input can be split differently. Both methods often handle rare words well, but neither guarantees that every possible input will avoid an unknown result.

1. Do you want the comparison to focus on WordPiece tokenization versus SentencePiece Unigram, or SentencePiece as a framework?
2. Should I also explain what happens when the vocabulary cannot represent part of the input?

Start with the shared goal. A tokenizer has a fixed vocabulary, but real text contains many rare or new words. Subword tokenization reduces this problem by breaking text into smaller pieces that are already in the vocabulary.

For WordPiece, imagine the word "playing". If the vocabulary contains "play" and "##ing", tokenization can produce those two pieces. The "##" marker commonly means that the second piece continues a word. At tokenization time, WordPiece commonly uses greedy longest match. At each position, it chooses the longest vocabulary token that matches the remaining characters. If no valid sequence of pieces can cover a word, an unknown token such as "[UNK]" may be produced.

SentencePiece is different because it is a tokenizer framework rather than one scoring rule. It can train directly from raw text without requiring external pretokenization. A visible "▁" marker commonly represents a preceding space or word boundary. SentencePiece can use Unigram or BPE. Unigram assigns probabilities to candidate pieces and scores possible segmentations. BPE instead uses learned merge rules.

In production, the important point is not to assume that either method can represent every possible input. Rare text is often decomposed into known pieces, but unknown tokens can still occur unless another fallback mechanism, such as byte fallback, is configured.

1. Start with input text.
2. Use the tokenizer vocabulary learned during training.
3. With WordPiece, scan the text and choose the longest matching vocabulary piece at each position. Continuation pieces commonly use "##".
4. With SentencePiece, process raw text directly and apply the configured subword model.
5. If the model is Unigram, score candidate segmentations probabilistically. If it is BPE, apply learned merge rules.
6. Convert the selected pieces into token IDs for the language model.
7. If the input cannot be covered by available pieces, an unknown token may appear unless another fallback mechanism is configured.

A common mistake is saying that SentencePiece always uses one probabilistic algorithm. It can use different subword models, including Unigram and BPE. Another mistake is treating WordPiece vocabulary learning as the same process as runtime tokenization. Runtime WordPiece tokenization commonly uses greedy longest match. It is also wrong to claim that either approach always represents every possible input without an unknown token. Another mistake is treating "##" and "▁" as universal syntax rather than conventions used by particular tokenizer representations.

Explain the shared goal first, then give one small example. Say that WordPiece commonly uses greedy longest match and "##" continuation pieces. Then explain that SentencePiece works on raw text and is a framework that may use Unigram or BPE. Finish by mentioning that rare text is often split into known pieces, but unknown tokens are still possible depending on the tokenizer configuration.

Positional encoding gives self attention access to token order. For example, the tokens in "The cat sat" have both content and position information, so the model can distinguish that order from another ordering of the same tokens. Absolute methods represent where a token is in the sequence. Relative methods represent the distance or offset between token positions and can influence attention scores. For long sequences, behavior depends on the position method and how the model was trained.

Think about the sentence "The cat sat." The words have meaning, but their order also matters. A Transformer therefore needs information that tells it where tokens appear in the sequence. Positional encoding provides that information. It allows self attention to use token content together with token order when building contextual representations. Absolute methods describe where a token is. Relative methods describe how far token positions are from each other. The choice matters because different methods behave differently when sequences become longer than those commonly seen during training.

1. Do you want me to compare absolute and relative position methods?
2. Should I also explain behavior beyond the sequence lengths used during training?

Start with the input sequence. Tokenization creates tokens such as "The", "cat", and "sat". Each token gets an embedding that represents its content. The Transformer also receives position information before or during attention.

With an absolute method, each token receives information about where it is in the sequence. One common approach adds an absolute position representation to the token embedding. Sinusoidal encoding is one example. Learned absolute position embeddings are another example.

With a relative method, the model represents the distance or offset between token positions. This information can influence attention scores, for example through a relative bias or relative key representation. The model can then reason about relationships such as one token being two positions before another.

This information is needed because self attention without position information is permutation equivariant. If the input tokens are reordered, the corresponding outputs are reordered in the same way. Attention alone therefore does not assign sequence meaning to that ordering.

Long sequence behavior depends on the chosen method. Learned absolute embeddings may have a fixed trained position range. Some relative or functional schemes can extrapolate more naturally, but good long context quality is not guaranteed. Standard self attention also becomes more expensive as sequence length grows, and modeling very long range relationships can remain difficult.

1. Tokenize the input while keeping the sequence order.
2. Create a content embedding for each token.
3. Supply position information. Absolute methods describe where a token appears. Relative methods describe the distance or offset between token positions.
4. Let self attention use the position aware information when computing relationships between tokens.
5. Produce contextual representations that reflect both token content and sequence order.
6. For long inputs, consider the position method, the sequence lengths used during training, and the cost of standard self attention.

A common mistake is saying that every position method is simply added to token embeddings. That is true for some absolute methods, but relative methods can influence attention scores instead. Another mistake is saying that self attention already knows token order. Without position information, self attention does not assign sequence meaning to different orderings. It is also wrong to assume that a position method guarantees good performance on arbitrarily long inputs. Long context behavior depends on the position scheme, training, architecture, and attention cost.

Start with a small sentence such as "The cat sat" and explain that token meaning alone is not enough because order matters. Then explain absolute position as where a token is, relative position as the distance or offset between positions, and finish with the long sequence limitation.

Query, key, and value are three learned projections of the same token representations. The query represents what a token is looking for. Each key represents what a token can match against. Each value carries the information that can be collected. Attention compares queries with keys, scales the scores, applies softmax to get normalized weights, and uses those weights to combine the values into a new representation for each token.

Attention helps each word choose which other words matter when building its new representation. Think about the sentence "I love NLP !" and focus on the word "love". The model creates three different views of every word. One view represents what the current word wants to find. Another represents what each word can match with. The third carries the information that can be gathered. The model compares the first two views, turns the results into importance weights, and mixes information from the third view using those weights.

1. Should I explain one attention head or also discuss multiple heads?
2. Should I include the tensor shapes and the scaling term in the score calculation?

Start with input token representations X with shape (n, d_model), where n is the number of tokens and d_model is the embedding size. Attention applies three learned linear projections. These create Q, K, and V.

Q is the query matrix. A query represents what each token position is looking for. K is the key matrix. A key represents what each token position can match against. V is the value matrix. A value contains the information that can be passed into the output.

For one attention head, Q has shape (n, d_k), K has shape (n, d_k), and V has shape (n, d_v). Their learned projection matrices map the input representation into these dimensions.

The model compares every query with every key using dot products. It forms the score matrix as QK^T divided by the square root of d_k. Dividing by the square root of d_k keeps the dot product scale more controlled as the key dimension grows.

The score matrix has shape (n, n). Each row contains the scores from one query token to all key tokens. A row wise softmax converts each row into normalized attention weights. Every row sums to 1. A larger weight means the query gives more importance to that key position.

Finally, the attention weight matrix multiplies V. The result has shape (n, d_v). Each output token is therefore a weighted mixture of all value vectors.

For the "love" example, the query for "love" is compared with the keys for "I", "love", "NLP", and "!". Softmax turns those scores into weights. Those weights decide how much information to take from each value vector. The weighted sum becomes the new representation for "love".

The important limitation is that Q, K, and V do not have fixed human meanings. They are learned numerical representations. The phrases "what I am looking for", "what I can match against", and "what information I carry" are useful intuition, not literal stored questions or facts.

1. Start with token representations X of shape (n, d_model).
2. Apply three learned linear projections to create Q, K, and V.
3. Compute the score matrix as QK^T divided by the square root of d_k.
4. Apply row wise softmax to convert each score row into normalized attention weights.
5. Multiply the attention weights by V.
6. Return one weighted mixture of value vectors for each token, with shape (n, d_v).

For one attention head with n tokens, every query is compared with every key. This creates an (n, n) score matrix, so this part of the work grows roughly with n squared. The score matrix also uses memory that grows roughly with n squared. The exact computation also depends on the query, key, and value dimensions.

A common mistake is saying that query, key, and value are separate input tokens. They are learned projections of token representations. Another mistake is saying that attention directly mixes keys. Keys are used to compute relevance scores, while values provide the information that is mixed. It is also incorrect to skip softmax because the raw scores are not yet normalized attention weights. Another mistake is forgetting the division by the square root of d_k. Finally, the simple descriptions of Q, K, and V are intuition. The learned vectors do not literally contain natural language questions, labels, or facts.

Explain the flow in four steps: project X into Q, K, and V, compare Q with K, normalize the scores with softmax, then use those weights to combine V. Mention the tensor shapes if the interviewer wants more detail. A simple phrase helps: Q says what to look for, K says what can match, and V carries the information to collect.