15 Anthropic AI Engineer Interview Questions & Answers

I would keep prompts and examples private by default, control who can access them, scan content before sharing, isolate workspaces, and support immediate link revocation. The main idea is that public sharing must be an intentional action with safety checks before private examples become visible.

This question asks how to share prompts and examples without exposing private information. A safe system starts with private content and only shares content after checking access and risks.

1. Should public links allow anyone with the link to view content, or should users authenticate first?
2. Which types of private examples or sensitive data must the system protect?

I would design the sharing flow around a private first approach. When a user creates prompts and examples, the content stays inside an isolated workspace with visibility rules. The user can choose private, team, or public access, but broader access requires an explicit action.

Before creating a public link, the system checks for secrets such as tokens, passwords, and sensitive information. It also checks for private examples and personal data that should not be exposed. If risky content is found, publishing is blocked until the issue is resolved.

When a link is created, access follows the selected permission level. Public links should only expose approved content and should not provide access to unrelated workspace data. Workspace isolation prevents information from crossing between different users or teams.

Production systems should support audit logs and revocation. If a link is shared incorrectly, the owner can remove access immediately. The tradeoff is that more checks add some latency and complexity, but they reduce the chance of private prompt examples being exposed.

A common mistake is making prompts public automatically and trying to remove sensitive data later. Another mistake is trusting a hard to guess link instead of enforcing permissions. A system should also validate shared content before making it available.

Start with the main decision: keep content private first. Then explain the flow in order: workspace isolation, permissions, scanning, controlled sharing, and revocation.

I would store each repeated large prefix once in a content addressed prefix store and keep a small metadata index that maps tenant ID, prompt ID, and version to the stored content. When a new prefix or version is created, I compute its content hash, store the prefix bytes if needed, and create an immutable metadata entry. At runtime, the prompt service looks up the requested tenant and version, reads the prefix, combines it with the new user input, sends the reconstructed prompt to the model, and returns the response. Tenant access checks and immutable versions preserve isolation and reproducibility.

Many requests can use the same large block of instructions, policies, and examples. Copying that block into stored request data again and again wastes space. I would keep one stored copy and let each prompt version point to it. The important requirement is correctness. A request must still get the exact prefix that belongs to its tenant and prompt version. Old versions must not silently change. At runtime, the system rebuilds the full prompt before sending it to the model, so the model receives the same intended content.

1. Must old prompt versions remain available for audit, rollback, or reproduction?
2. Can identical prefix bytes appear in different tenants while access remains isolated by tenant?

I would separate prefix content from prefix identity.

When a new prefix or a new version is created, the application computes a content hash for the prefix bytes. That hash is a fingerprint of the stored content. The application stores the prefix content in a content addressed store and creates metadata containing tenant ID, prompt ID, version, hash, and storage location. The metadata entry is immutable for that version.

The metadata index is the lookup layer. Tenant ID keeps access inside the correct organization boundary. Prompt ID identifies the logical prompt. Version identifies the exact revision. The hash identifies the content bytes. Identical bytes can therefore reuse stored content while each tenant and version still has its own authorized metadata reference.

At runtime, the client sends the prompt identity and the changing user input. The prompt service looks up the metadata for that tenant, prompt ID, and version. It reads the matching prefix content, combines that prefix with the new user input, and sends the reconstructed full prompt to the model. The model does not need a different behavior for this optimization.

A new or changed prompt version should create a new version entry rather than overwrite an old one. This keeps previous runs reproducible and supports rollback. Hot prefixes can be cached for faster application lookup, but caching does not change the identity rules.

The main benefit is less repeated prefix storage and simpler reuse. The cost is extra metadata, lookup logic, access control, cleanup, and version management. This optimization does not guarantee lower model latency because the model still receives the reconstructed full prompt.

One mistake is using only a content hash as the prompt identity. A hash identifies content bytes, but tenant ID, prompt ID, and version are still needed for authorization and stable prompt identity. Another mistake is overwriting an existing version when the prompt changes. That breaks reproduction and rollback. Another mistake is allowing a tenant to resolve content through another tenant's metadata. Access must remain tenant scoped. A final mistake is claiming that deduplication guarantees lower model latency. It mainly reduces repeated storage and application reuse work because the model still receives the reconstructed full prompt.

Start with the practical design: store prefix content once, keep tenant and version identity in metadata, and reconstruct the full prompt at runtime. Then explain why immutable versions and tenant scoped lookups are required for correctness.

I test retrieval and generation separately. First, I judge the retrieved passages without the LLM. If they are good, I give the same question to the LLM with known-good gold context. Failure there points to the LLM; success points back to context assembly or integration.

The goal is to find which part is making the final answer weak. First, look at the information the system found for the question. Ask whether the right facts are present, complete, current, and allowed for that user. If the information itself is poor, the search side is the likely cause. If the information is good, give the model a clean, ideal set of facts and ask the same question again. If it still fails, the model is the likely cause. If it succeeds, inspect how the normal system builds and passes the information.

1. Do we have labeled questions with known relevant passages or expected facts?
2. Can we run the same production question against a controlled gold context that is known to contain the answer?
3. Should the evaluation also check freshness, authorization, and citation behavior, or only answer correctness?

I would debug the system in stages.

First, I evaluate retrieval without asking the LLM to generate anything. For each test question, I inspect the top retrieved passages and judge whether they actually support the question. I check whether the key facts are present, whether important information is missing, whether the content is current, and whether the user is authorized to receive it.

When labeled relevance data exists, I also use retrieval metrics. Recall@k asks whether the needed relevant material appears within the first k results. Precision@k asks how much of those first k results is actually relevant. Ranking metrics such as MRR@k or nDCG@k help show whether useful passages are being placed near the top. These measurements help separate a retrieval problem from a generation problem.

If retrieval is poor, I stop blaming the LLM. I investigate the retrieval path. Depending on the evidence, that can mean checking embeddings, hybrid lexical and vector search, metadata filters, reranking, chunking, query rewriting, the number of retrieved passages, or source-data quality. I change only the part supported by the failure I observed rather than changing every retrieval setting at once.

If retrieval looks good, I isolate the LLM. I ask the same question again but replace the normal retrieved context with controlled gold context. Gold context means a small, known-good set of passages that definitely contains the facts needed to answer the question. I keep the production prompt and instructions the same whenever possible, including any citation requirement, so the context is the main changed variable.

If the LLM gives a wrong or incomplete answer even with good gold context, the problem is likely on the generation side. I inspect whether the model understands the question and follows the instructions. I may simplify the prompt, add clearer examples, reduce unnecessary complexity, make the expected output format clearer, or evaluate whether a different model is needed.

If the LLM answers correctly with gold context but the production answer is still poor, I inspect the RAG pipeline between retrieval and generation. The retrieved passages may be relevant before assembly but then be truncated, ordered badly, mixed with distracting text, or omitted when the final model context is built. Prompt construction can also place instructions or evidence in an ineffective form.

The key decision is simple: poor retrieved evidence points to retrieval; good retrieval plus failure with gold context points to LLM comprehension or instructions; good retrieval plus success with gold context points to context assembly or integration. I repeat this across several representative questions instead of trusting one example, because a single query can give a misleading result.

1. Choose representative test questions and, when available, known relevant passages or expected facts.
2. Run retrieval only. Do not generate an answer yet.
3. Inspect the top-k passages for relevance, missing facts, freshness, and authorization.
4. When labels exist, compute retrieval measures such as Recall@k, Precision@k, MRR@k, or nDCG@k.
5. If retrieval is poor, investigate retrieval components such as embeddings, hybrid search, metadata filters, reranking, chunking, query rewriting, k, and source-data quality.
6. If retrieval is good, ask the same question using controlled gold context known to contain the answer while keeping the production prompt and instructions the same when possible.
7. Grade the gold-context answer for correctness and completeness.
8. If the answer is still poor, investigate the LLM, prompt instructions, examples, output constraints, or model choice.
9. If the gold-context answer is correct but production remains poor, inspect context assembly for missing, truncated, badly ordered, or distracting passages.
10. Repeat the test across multiple questions before deciding on the root cause.

The main cost is repeated evaluation. Retrieval checks need test questions and, for strong offline metrics, human or trusted relevance labels. Looking at more retrieved passages increases review work and can increase retrieval and reranking cost. Gold-context tests add extra LLM calls. Larger context also uses more model input tokens. Operationally, the evaluation is easier to maintain when retrieval scores, retrieved document identifiers, assembled context, and final-answer grades are recorded separately, because each failure can then be traced to the correct stage.

A common mistake is to judge retrieval only from the final generated answer. A strong LLM can sometimes answer despite weak retrieval, while a weak answer can occur even when retrieval is correct. Another mistake is testing the LLM with the same bad retrieved context instead of replacing it with known-good gold context. Teams also sometimes change embeddings, chunking, reranking, prompts, and models at the same time, which makes the cause impossible to isolate. Other mistakes include checking only one query, ignoring whether key facts are missing from the retrieved passages, treating high similarity as proof of usefulness, and forgetting that good retrieved passages can still be lost or truncated during context assembly.

Describe the diagnosis as a simple decision tree. First test retrieved evidence without the LLM. Then, only if retrieval is good, test the same question with gold context. State the three outcomes clearly: bad retrieval means retrieval problem; failure with gold context means generation problem; success with gold context means inspect context assembly or integration.

At a high level, I would make email retries safe without sending the same email twice. The hard part is that a timeout can happen after the provider accepted the request. I would separate workflow replay from the real side effect. The Tool Gateway uses an idempotency key, meaning the same email intent reuses one key. It returns stored SUCCESS results and resolves UNKNOWN attempts with that same key. The downside is that safe recovery needs the provider to ignore repeated keys or let us check the outcome.

The goal is simple. An agent may send an email, then see a timeout or server error. The difficult part is that the email may already have been accepted even though the agent did not receive a success response. Sending the request again without checking could send two emails. The diagram solves this by giving each email intent a stable key, saving the attempt state, and separating normal orchestration retries from the real external side effect. Unknown outcomes are checked again with the same key instead of blindly sending another email.

1. Does the external email service support deduplication using an idempotency key?
2. Can we query the provider later to learn whether an uncertain request succeeded?
3. How long should we keep the saved key, state, and result?

I would begin by saying that the same email intent must always use the same idempotency key. Idempotency means repeating the same request does not create another side effect.

The Orchestrator decides to send an email. Before the external side effect, it saves its run state. It then calls the Tool Gateway with the tool name, normalized arguments, and the idempotency key.

The Tool Gateway first checks the durable Idempotency Store. Durable means the state survives a normal process restart. The store keeps the key, attempt state, and any known result.

If the state is SUCCESS, the gateway returns the stored result. It does not call the Email Service again. If the state is NEW, this is the first execution. During that attempt, the store can track the key as IN_PROGRESS so a retry does not look like new work.

For a NEW attempt, the Tool Gateway sends the email request to the Email Service with the same idempotency key. The diagram shows the external service ignoring repeated requests that carry the same key.

After a known success, the Tool Gateway saves the SUCCESS result. A later retry can then return that stored response instead of sending another email.

The most important failure case is a timeout or 5xx after the request was sent. The caller may not know whether the Email Service accepted it.

In that case, the attempt is IN_PROGRESS or UNKNOWN. The Tool Gateway does not treat it as NEW. It resolves the uncertain attempt by querying the downstream service, retrying with the same key, or checking the recorded outcome. This avoids a blind second send.

The Orchestrator may retry or resume its workflow. It reuses the same tool-call key. If the Tool Gateway finds SUCCESS, it returns the stored result. If the result is still UNKNOWN, it follows the safe recovery path with the same key.

The main trade-off is important. If the external provider cannot ignore repeated keys and cannot be queried or checked later, exactly-once email delivery cannot be guaranteed after an ambiguous timeout. In that case, blindly retrying is unsafe.

The benefit is that orchestration retries become much safer. The agent can resume its run without automatically sending the email again. A stable key lets the Tool Gateway recognize the same email intent and return a saved SUCCESS result. UNKNOWN attempts are handled with the same key instead of being treated as new work. The downside is extra state and recovery logic. The Idempotency Store must keep the key, status, and known result. Safe recovery also depends on the email provider. If it cannot ignore repeated keys or let us check an uncertain result, we cannot promise exactly-once delivery after a timeout.

At a high level, I would build a bounded agent that can understand a new goal, plan steps, choose tools, and learn from results. The main challenge is allowing useful adaptation without giving the agent unlimited freedom. I would explain three flows: task understanding and planning, safe tool execution, and memory-based adaptation. Governance controls permissions, budgets, approvals, retries, and stopping. The trade-off is that stronger controls make the system safer, but they can reduce flexibility.

The system must accept a task it has not seen before and decide how to complete it safely. For example, a user might ask for a market research report about solar panels in India. The difficult part is not only making a plan. The agent must also choose suitable tools, remember useful results, learn from success or failure, and stop at the right time. The diagram organizes this into an Agent Core, Memory, Tools & Services, and Governance & Controls that limit what the agent can do.

1. Which tools is the agent allowed to use for a new task?
2. Which actions need human approval before execution?
3. How long should learned task history and user preferences be kept?
4. Which limits matter most: time, cost, API calls, rate, or memory?

I would start by turning the user's goal into a structured representation. Task Understanding & Representation records the objective, success criteria, constraints, available tools, preferences, and context. This matters because the Planner needs a clear description of what success means. The example also includes limits such as time, budget, and scope. Those limits become part of the task before execution starts.

Next, the Planner breaks the task into ordered steps. It produces a plan, the next action, the reason for that action, and a stopping condition. The Planner is adaptive, which means the plan can change after new results arrive. The system does not need one perfect plan at the start. It can take a step, inspect the result, and decide what should happen next. The loop continues until a stopping condition is met.

The Tool Selector chooses the best tool for the next action. It considers tool descriptions, past success, cost, latency, and permissions. The Executor then runs the chosen tool or sub-agent and gets the result. Tools & Services include Information, Utilities, Data & Files, Sub-Agents, and Integrations. The Executor validates inputs and uses timeouts, retries, and idempotency. Idempotency means a safe retry should not repeat the same side effect.

Memory feeds state back into task understanding, planning, tool selection, and execution. Working Memory keeps the current context. Long-term Memory keeps past tasks and lessons. Tool Experience stores success rate, latency, and cost. User Preferences & Constraints keep user-specific choices and limits. Results can update this state, so later plans and tool choices can use what the agent has learned.

Governance & Controls limits adaptation and execution. Authorization & IAM uses least privilege, so each tool gets only the access it needs. Guardrails & Policies restrict allowed actions and content. Risky actions can require human approval. Budgets & Limits cap time, cost, API calls, rate, and memory. Monitoring & Audit records plans, actions, results, and state changes. Recovery & Reliability uses retries, fallbacks, checkpoints, and error handling. Stop Conditions & Completion ends the loop when success criteria are met, a budget or time limit is reached, or the user stops the task.

The benefit is flexibility. The same agent can handle a new goal by building a plan and choosing from available tools. Memory also helps later decisions use past results. The downside is that adaptation can become unsafe or expensive without limits. That is why permissions, approvals, budgets, guardrails, and stop conditions bound execution. Retries and recovery improve reliability, but they add more control logic. Long-term memory can improve future choices, but keeping more history increases storage, cost, and privacy concerns. We accept some limits on flexibility because controlled execution is safer and easier to audit.

At a high level, I would use deterministic orchestration to control a team of isolated coding agents. The main challenge is giving agents enough freedom to change code without allowing unsafe access or endless retries. I would explain three flows: task planning, isolated agent work, and quality gates before review. Shared workflow state records progress, guardrails control permissions and budgets, and failed gates return work to the right agent. The trade-off is stronger safety and recovery with more orchestration complexity.

The goal is to let several coding agents work on one software task without losing control of the repository. One agent can clone the code, another can write tests, another can debug failures, and another can prepare the pull request. The difficult part is keeping their work isolated while still sharing enough workflow information to coordinate them. The diagram solves this with deterministic orchestration, separate Ephemeral Workspaces, a shared Workflow State Store, automated Quality Gates, bounded retries, and Human Review before Merge.

1. Should every change require Human Review before Merge?
2. Which tools and repository actions may each agent use?
3. What retry and runtime limits should apply to each task?
4. Which Quality Gates must pass before the pull request can move forward?

I would start with the User or Maintainer submitting a repository and objective. The task may arrive from the Web UI, CLI, GitHub Webhook, or Scheduler. The API / Gateway handles authentication, authorization, rate limiting, and request validation.

The Orchestrator / Planner breaks the objective into smaller tasks. It selects the next agent and applies retry and timeout rules. It also records state transitions. This keeps workflow control in deterministic application logic instead of letting an agent decide when the whole process should continue or stop.

The Workflow State Store is the source of truth for workflow progress. It keeps workflow status, the task queue, artifact references, logs, and events. This lets the system know what finished, what failed, and what should run next.

The Event Bus carries events such as task scheduled, task completed, gate result, and errors. These events help the orchestration layer react to progress without making one coding agent responsible for coordinating everything.

Each Agent Worker has a focused job. The Repo Agent clones the repository and checks out the branch. The Test Agent writes or updates unit tests and runs them. The Debug Agent analyzes failures, proposes fixes, changes code or tests, and re-runs tests. The PR Agent prepares changes, creates the pull request, adds its description, and requests review.

Each agent runs in its own Ephemeral Workspace. That is a temporary isolated container or virtual machine holding its own repository copy. Agents can read and write only inside their workspace. External network access is disabled by default, and resources are cleaned up after use.

Agent results pass through automated Quality Gates. The system checks the build, tests, coverage threshold, lint and static analysis, and security scan. Each gate returns pass or fail.

If a gate fails, the system returns failure details to the appropriate agent. It increments the retry count and respects retry and time limits. This creates a bounded retry loop instead of an endless debugging cycle.

When all gates pass, the pull request moves to Human Review. A reviewer can approve it or request changes. Approved work reaches Merge, which merges to main, closes the workflow, and cleans up resources.

Guardrails & Policies control allowed tools, maximum retries, maximum runtime, concurrency limits, secrets management, and approval rules. Least privilege means each agent receives only the access needed for its task.

The workflow ends after successful review and Merge, or when retry or runtime limits prevent more work. Audit & Logs record actions, tool calls, diffs, results, and decisions. The main trade-off is extra control-plane complexity in exchange for safer, repeatable agent execution.

At a high level, I want to give developers a secure way to access Anthropic AI models through APIs. I separate the design into the developer request layer, Edge & API Gateway, Core API Services, Model-Serving Backends, and Data & State. The request first uses HTTPS, then authentication, quotas, and validation. Core services route requests, apply guardrails, manage streaming, and call model backends. The response returns through the API path. The main trade-off is adding security and reliability controls while keeping latency low.

This question asks how to build an API that lets developers safely use Anthropic AI models. The goal is to make model access simple while protecting the platform, controlling usage, and returning reliable responses. The main challenge is designing a secure path from a developer request to model-serving systems while handling authentication, validation, streaming, errors, and versioning. I will explain the design by following the diagram flow from the developer app to the model backend and back.

• How many developers and organizations will use this API?
• Is real-time streaming required for responses?
• What usage limits and security controls are required?
• How should model versions be managed?

The API provides developers with controlled access to Anthropic AI models. The developer application sends requests with model, messages, tools, and streaming options. The platform controls security, validation, routing, and model access.

The developer does not directly call model-serving backends. This boundary allows the platform to protect models and manage traffic.

The request first reaches the Edge & API Gateway through HTTPS. TLS protects the connection. Authentication and Authorization checks the x-api-key and resolves organization or workspace permissions. Authentication verifies the caller. Authorization decides what the caller can access.

The gateway also manages rate limits, quotas, request validation, and idempotency keys. Invalid or unauthorized requests are rejected before reaching model services.

Validated requests move to Core API Services. The Routing & Model Gateway selects the correct model path based on policy and capacity. Context & Prompt Guardrails apply safety checks and policy rules.

Request Orchestration prepares model requests, tool calls, and system instructions. Prompt Caching reuses eligible prompt prefixes to reduce repeated processing. It does not guarantee identical model outputs.

The Streaming Orchestrator manages token streaming, connection handling, and backpressure.

Core services send model requests to Model-Serving Backends. These systems run inference and generate responses. Tool Execution provides controlled tool access when needed.

The model backend focuses on generation. API services handle user-facing security and API behavior.

Responses return from model services through Core API Services to the developer application. When streaming is enabled, Server-Sent Events send partial updates such as message start, content blocks, and message completion.

Logging, traces, and metrics support operations but are not part of the user response path.

Data & State stores API keys, usage information, request logs, traces, metrics, and developer feedback. Security controls include HTTPS, scoped API keys, input validation, tenant isolation, audit logs, and abuse detection.

The API returns clear errors for invalid requests, authentication failures, permission failures, rate limits, internal errors, and temporary overload. Versioning uses the API version header and explicit beta controls for enabled features.

The main trade-off is that more validation and security controls add complexity and some latency, but they improve safety and reliability.

This design separates API access from model execution. The benefit is clearer security ownership, easier scaling, and safer model access. The gateway handles authentication, authorization, quotas, and validation. Core services handle routing, guardrails, caching, and streaming. Model backends focus on generating responses.

The downside is more components to operate and monitor. Security checks add some latency. Rate limits protect the platform but may restrict heavy usage. Prompt caching improves performance for supported repeated content but has limits. Versioning reduces breaking changes but requires maintenance. We accept these trade-offs because secure and predictable AI access is more important than removing every extra step.

At a high level, I would build a classifier service that converts LLM scores into binary predictions. The client sends inputs, and the system batches requests before calling score_batch(prompt, inputs). The LLM returns scores, and the application converts them into class probabilities. A threshold then creates the final prediction. The system also exposes calibration, threshold selection, and failure analysis to improve trust. The main trade-off is balancing accuracy, latency, and cost while keeping predictions reliable.

We are solving a decision problem. We want an LLM system that can classify inputs into two classes, such as positive or negative. The goal is to turn an LLM score into a useful probability and final decision. The main challenge is that model scores are not automatically perfect probabilities. We need batching, calibration, and evaluation to make results reliable. This answer follows the diagram from request input, scoring, probability conversion, decision, output, and improvement.

• What traffic level and latency target should the classifier support?
• Do we have labeled data for calibration and evaluation?
• Which mistakes are more costly, false positives or false negatives?

I start with the client application. The client sends classification inputs with the prompt and data that should be scored. The request enters the system and moves to the ingestion and batching layer.

Batching groups multiple requests together before scoring. This improves efficiency because the system can process several inputs in one score_batch(prompt, inputs) call. The trade-off is that waiting for larger batches can increase latency.

The dynamic batcher receives requests and creates batches using size or time limits. This component owns request grouping only. It does not decide the class result.

The batch is sent to the LLM scoring boundary. The goal is to improve throughput and reduce repeated model calls.

The scoring component calls score_batch(prompt, inputs). The LLM returns one boundary score for each input.

The model output is probabilistic. The application does not directly use the raw score as the final decision. Instead, deterministic application logic converts the score into a probability and applies business rules.

The system converts the score into a class probability. The probability represents the likelihood that the input belongs to class one.

Calibration makes the probability more trustworthy. For example, predictions with a probability of 0.8 should generally be correct close to that level when measured on validation data.

The system then compares the probability with a configurable threshold. If the probability is greater than or equal to the threshold, it predicts class one. Otherwise, it predicts class zero.

The response returns per-item results. It includes the prediction, probability, threshold used, and metadata shown in the design.

Returning the probability helps users understand confidence instead of only seeing a binary answer.

Calibration uses labeled validation data. The system checks reliability diagrams, Brier score, expected calibration error, and calibration methods such as temperature scaling or isotonic regression.

Threshold selection chooses the decision point based on goals such as precision, recall, F1 score, ROC or PR trade-offs, and cost-sensitive decisions.

Failure analysis reviews errors by segment, topic, and input type. The team checks confusion patterns, false positives, false negatives, and score distribution changes.

Monitoring tracks latency, throughput, error rate, queue behavior, score distribution, calibration drift, logs, and traces. Feedback from human labels and outcomes helps improve prompts, calibration, and threshold settings.

The trade-off is that evaluation and monitoring add operational effort, but they make LLM classification safer and easier to improve.

This design separates the LLM score from the final business decision. The benefit is better control because the application can calibrate probabilities and choose the right threshold. Batching improves cost and throughput, but waiting for a batch can increase latency. Calibration needs labeled data, but it makes probabilities easier to trust. Threshold selection allows different choices for false positives and false negatives. Monitoring and failure analysis require extra engineering work, but they help find problems early. The system accepts more operational complexity to make predictions measurable, explainable, and reliable.

At a high level, I would build a prompt playground that lets developers create, test, and run prompts safely. I would separate the design into the web app, control plane, data plane, execution plane, and supporting systems. Users create prompts and runs through the Web App. The Run Service sends work to the execution plane, where workers call model providers and stream tokens back. Tenant isolation protects customer data, and cost tracking records usage. The trade-off is that stronger reliability and isolation require more services and operational complexity.

We are building a prompt playground for developers and prompt engineers. The system helps users create prompts, save versions, run experiments, and understand results. The main challenge is supporting 100,000 monthly users and 1 million runs per day while keeping data isolated, tracking cost, and handling model provider failures. I will explain the design using the Web App, control plane, data plane, execution plane, and cross-cutting systems.

• Should users only test prompts, or also manage team collaboration?
• Do we need multiple model providers from the beginning?
• How much run history and cost detail should users keep?

The goal is to provide a safe workspace for prompt development. Developers and prompt engineers use the Web App Playground UI to author prompts, test outputs, compare results, evaluate runs, and monitor cost.

The Web App sends requests into the platform. The system separates control operations from execution work. This keeps user actions responsive while allowing many prompt runs.

The API Gateway receives requests and performs request validation, tenant resolution, and access checks.

The Auth & Org Service manages users, organizations, projects, API keys, and roles. This provides tenant isolation between customers.

The Prompt Service manages prompts and immutable versions. Immutable versions mean saved prompt versions cannot be changed, which helps reproduce earlier runs.

The Run Service creates runs, tracks status, and connects execution results with usage information. Settings & Config stores model settings and environment configuration.

The data plane stores system information. PostgreSQL stores metadata such as users, organizations, projects, and run information.

The Prompt Store keeps versioned prompt content. The Run Store keeps status, timing, tokens, and cost data. Event and Stream Log stores token events, logs, and errors. Object Storage stores larger artifacts and files.

When a user starts a run, the Run Service creates the execution request. The request moves to the durable queue. Workers read jobs from the queue and execute them.

Workers are stateless, so more workers can be added as traffic grows. The Provider Gateway routes requests to model providers based on health, cost, and capacity.

The Response Processor formats results, records usage, and sends output to the Streaming Gateway. The Streaming Gateway sends tokens back through WebSocket or SSE.

Model providers can have timeouts, network failures, or temporary errors. The Provider Gateway can use another provider when supported by the routing policy.

Cancellation stops work cooperatively. The system records partial results when possible. Run state tracking helps prevent duplicate processing.

Observability collects logs, metrics, and traces. Cost Tracking records token usage and cost per run, user, and organization. Quotas and Rate Limiting protect shared resources.

Audit & Security records important actions. Cache improves performance for reusable data. Feedback & Evaluations help improve prompts.

The main trade-off is complexity versus control. More services provide better isolation, scaling, and reliability, but they require more operations work.

The design separates responsibilities so each service has one clear job. The benefit is easier scaling, better security, and simpler troubleshooting. The downside is that more services increase deployment and monitoring work.

Tenant isolation protects customer data. Versioned prompts improve repeatability. The queue helps handle large workloads without blocking users. Streaming improves user experience but requires connection management.

Provider routing improves availability, but it adds decision logic. Cost tracking improves visibility, but it requires storing more usage data. We accept these trade-offs because the system must support many users and many daily runs reliably.