8 NVIDIA Cloud Engineer Interview Questions & Answers

At a high level, I would collect GPU telemetry once and let both real-time and offline processing use the same recorded events. The main challenge is getting fast operational results while keeping later recomputation consistent. I would divide the design into secure ingestion, a real-time path, and an offline path. Both paths use the same event identity, event time, schema version, and stored history. Real-time results arrive quickly, while offline jobs can recompute complete results and publish corrected derived data.

The goal is to collect GPU telemetry once and reuse the same information for two needs. Operations teams need fresh metrics, alerts, and scaling signals quickly. Analytical jobs need a longer history for reports, planning, and model features. The hard part is making both paths understand an event in the same way, even when events arrive twice, arrive late, or must be processed again. The diagram solves this with one append-only telemetry stream, shared storage tiers, separate real-time and offline processing, and a replay path that makes recomputed results converge with earlier results.

1. How quickly must dashboards, alerts, and scaling signals react?
2. How long should recent raw telemetry stay in Hot Log Storage?
3. How long should Cold Object Storage keep historical telemetry?
4. Which real-time results may be approximate before recomputation?
5. What schema compatibility rules are required during producer changes?

I would start by sending GPU node or agent telemetry through one controlled entry path. The diagram applies mTLS authentication, authorization, schema validation, rate limits, and quotas before an event enters the Telemetry Stream.

Each event carries an event_id, source_id, event_time, ingest_time, schema_version, payload, partition_key, and checksum. The event_id identifies the same event during retries or replay. Event time says when the GPU produced the event. Ingest time says when the platform received it.

The Telemetry Stream is the single shared record for both processing paths. It is append-only, so existing events are not changed in place. It is durable, partitioned, replicated, and ordered within a partition.

The diagram shows at-least-once ingestion. That means the same event can appear again. Consumers therefore use event_id to make repeated processing safe. The source_id can be used as the partition key so telemetry from one source keeps a useful processing order.

Stream Processing reads new events and performs windowed aggregation, anomaly detection, threshold alerts, and fast derived metrics. Real-Time State keeps the latest values per GPU, sliding windows, and short-lived state.

These results feed dashboards, on-call alerting, auto-scaling controllers, and runbooks or webhooks. This path is optimized for low delay. Results inside an active time window may still change as more telemetry arrives.

Processing progress can be represented by partition offsets. Those offsets act as restart points, so processing can continue from a known position after a restart.

The same telemetry is also placed into storage tiers. Hot Log Storage keeps recent raw telemetry for fast replay. Cold Object Storage keeps a compressed, long-term archive partitioned by date and source. The Catalog / Index tracks schemas, versions, partitions, offsets, lineage, and retention.

Batch / Lake Processing reads this stored history for aggregations, ML feature jobs, capacity planning, and cost or usage reports. Results go into the Analytical Store for BI, data science, planning, finance, and ad-hoc queries.

If processing logic changes or data must be corrected, the system can replay an offset or time range. It can also backfill corrected data and run the same deterministic processing again.

The replay path compares real-time and recomputed results using defined differences and thresholds. Reconciled results are published into shared derived data such as canonical metrics, capacity signals, normalized events and alerts, and ML features. Repeated writes remain safe. Schema rules, lineage, data quality, retention, access controls, and observability support both processing paths.

The benefit is that both processing paths start from the same recorded telemetry. This reduces disagreements caused by separate input systems. The real-time path gives dashboards and alerts low delay, but values inside an active window may still change. Offline processing takes longer, but it can scan stored history and recompute results. Replay makes recovery and corrections possible. The downside is extra storage and compute because telemetry is kept in several tiers and may be processed again. We accept that cost because the design gives fast operations, long-term analysis, and a clear way to make recomputed results replace earlier results when needed.

I would build the image once, add useful human readable tags, authenticate to the registry with short lived credentials, push every required tag, then capture the published registry manifest digest and use that digest as the approved artifact identity. Tags are convenient pointers and may move unless registry policy blocks reassignment. The digest identifies the exact published content. I would also record source revision, build metadata, provenance, scan evidence, and any signature against that same digest.

The pipeline should publish one built image in a way that is easy for people to recognize and exact for machines to verify. It gives the image useful names, sends it to the chosen registry, records where the build came from, checks the final stored result, and keeps evidence for later deployment. It also needs safe login behavior and a clear response when only part of the publishing work succeeds. The key idea is simple: friendly names can change, while the final stored fingerprint identifies the exact approved image.

1. Can existing tags move to a new image, or does the registry enforce tag immutability?
2. Which human readable tags are required for a release?
3. Is signing or provenance attestation required?

I would build the container image once, then add the required human readable tags to that same local image. Useful tags can include a version, a branch plus commit value, a build ID, and optionally latest for discovery. I would not treat any tag as the immutable identity.

The registry path should be explicit, such as registry.example.com/team/service. The pipeline should authenticate with short lived workload credentials or another least privilege identity, and it should avoid printing secrets in logs.

Next, I would push every required tag. After the push, I would inspect the remote image and capture the published registry manifest digest. The approved reference should use the repository plus digest form, such as repository@sha256:<digest>. Production deployment should pin that digest so the exact approved content is selected even if a named tag later moves.

I would also record the source revision, build ID, creation time, version, image metadata, SBOM, vulnerability scan result, and provenance information when those controls are used. If signing or attestation is enabled, it should bind to the same published digest.

If a named tag already exists, pushing it can move the tag to a new digest unless registry policy prevents reassignment. The old digest remains a distinct content identity. On a transient push failure, the pipeline should retry with bounded exponential backoff. The client or registry may reuse uploaded blobs, but that behavior is implementation specific. If only some required tags are published, the stage should fail and promotion should wait until every required tag resolves to the intended digest.

Common mistakes include treating a tag as an immutable identity, trying to derive the final registry digest before publication, rebuilding separately for different tags, deploying production by latest, storing long lived registry passwords in CI, printing credentials in logs, assuming every client and registry resumes interrupted uploads in the same way, ignoring partial tag publication, and recording a digest without verifying it against the remote registry. Another mistake is approving one digest but deploying through a mutable tag that can later point somewhere else.

Start with the main rule: tags are convenient names, while the registry manifest digest is the immutable published identity. Then walk through build, tagging, secure login, push, remote digest capture, metadata, failure handling, and deployment by digest. Mention that partial publication must block promotion. That shows both CI knowledge and production judgment.

I would give every physical GPU a unique resource identity and let each matrix cell acquire exactly one compatible GPU before the test starts. Jenkins can expand the test combinations in parallel, while executor and concurrency limits control how many cells may run. The scheduler checks GPU type and driver affinity before allocation. Redis stores the scheduler allocation lease with an owner and a short lifetime. On timeout or cancellation, the Pipeline stops the test work, runs cleanup, and releases the allocation. If an agent disappears, I quarantine that GPU until cleanup or host recovery is confirmed instead of assigning it again.

The goal is to run many GPU tests at the same time without letting two tests use the same physical accelerator. Each GPU gets its own identity. A test waits until a compatible GPU is available, uses it, and releases it after cleanup. The queue also limits how much capacity one job can take, so other jobs still get a chance to run. If a machine disappears, its GPUs are not reused immediately. They stay unavailable until the system confirms that the old test can no longer be using them.

1. Do tests require an exact GPU model, or can they run on any compatible model?
2. Are driver and CUDA versions exact requirements or minimum compatible versions?
3. How much concurrency should one job or branch be allowed to consume?
4. What recovery signal proves a GPU is safe to reuse after an agent disappears?

I would model every physical GPU as a unique schedulable resource, such as agent gpu 01 GPU 0. Jenkins labels describe which agents are compatible with each matrix cell, while the unique GPU identity prevents two cells from claiming the same accelerator.

The Pipeline expands combinations such as GPU type, driver requirement, operating system, and test suite. These cells can enter the Jenkins queue in parallel. The scheduler checks executor capacity, GPU availability, and driver affinity before dispatch. I would also use global, per label, and per job concurrency limits. A weighted fairness policy gives competing jobs access to the fleet and reduces starvation risk.

When a cell is selected, it acquires one GPU allocation. The diagram uses Redis for scheduler allocation state. Each entry records the GPU identity, current owner, and a short lease lifetime. The running agent renews that lease with heartbeats. The test receives the chosen GPU index, then publishes logs, test reports, and artifacts.

Timeout and cancellation must interrupt the test work and run cleanup. The scheduler removes the allocation only after cleanup is confirmed. Lease expiry by itself does not prove that the physical GPU is idle.

That rule is critical when an agent disappears. The lease metadata may expire, but I would quarantine the GPU. The scheduler must not reuse it until the agent reconnects, the old process is cleaned up, or the host is reset.

In production I would watch GPU utilization, queue depth, wait time, allocation latency, agent health, and allocation failures. The main tradeoff is safety versus utilization. Conservative quarantine may leave capacity idle longer, but it prevents duplicate GPU ownership.

Common mistakes are using only an agent label and assuming that it protects individual GPUs, setting Jenkins executors equal to GPU count without a separate GPU allocation rule, letting one large matrix consume the whole fleet, ignoring driver compatibility, and assuming that lease expiry proves the accelerator is free. Another serious mistake is immediately retrying work on the same GPU after an agent disappears. The old process may still be running. Cancellation also needs explicit cleanup so test processes and allocation state do not remain behind.

Start with the safety rule: one active allocation for each physical GPU. Then explain compatibility, queue limits, cleanup, and lost agent recovery in that order. Call out that lease expiry is scheduler state, not proof that the physical accelerator is idle. That distinction shows strong production judgment.

I would give every build output an immutable identity based on the application version and reviewed Git commit, then store each CPU, GPU, operating system, and hardware variant under a predictable repository path. CI would create a SHA 256 checksum, sign the artifact with GPG or Cosign, generate an SBOM, and publish build metadata. I would promote the same verified bytes from snapshot to staging and then release instead of rebuilding them. Access would follow least privilege. Remote runners would use replication or repository caching. Every test or deployment would record the exact repository path, checksum, signature result, build metadata, and source commit.

The goal is to make every build output easy to find, safe to keep, and possible to prove later. One source change can create several files for different processors, graphics hardware, operating systems, and machines. I would give each output a permanent identity and keep information that explains where it came from. Only approved outputs move toward release. People and automation receive only the access they need. Remote runners receive nearby copies or cached downloads. Temporary builds expire by policy, while release records keep the evidence needed to prove exactly what was tested and deployed.

1. Which artifact types and CPU, GPU, operating system, and hardware combinations must the repository support?
2. Do remote runners need full repository replication, pull through caching, or both?
3. What retention period and recovery objectives are required for snapshot, staging, and release repositories?
4. Which signing system and approval process are already used?

I would separate repositories by lifecycle. A practical layout is libs snapshot local for development outputs, libs staging local for verified candidates, libs release local for production releases, and libs release remote for disaster recovery or another region. Consumers read approved release content rather than writing into release storage.

The artifact identity must not depend on a mutable label such as latest. I would use the application version plus the reviewed Git commit, for example version 1.4.2 plus commit abc123. Architecture details belong in the path and metadata. Each matrix entry records values such as x86_64 or arm64, operating system, GPU presence, hardware target, artifact path, build time, and builder identity.

For every build, CI calculates a SHA 256 checksum, creates a GPG or Cosign signature, generates an SBOM such as CycloneDX, runs the required tests and scans, and publishes build metadata. Promotion is one way. The same verified artifact bytes move from snapshot to staging and then release after policy checks, tests, scans, and approval. Promotion must not rebuild the release artifact.

Permissions follow least privilege. CI service accounts can write only to the repositories needed by their pipeline. Release automation receives the permission needed to promote approved content. Developers and deployment systems mainly read release content. Audit users receive access to metadata and audit records. TLS, repository access controls, and signed content checks protect the path.

Remote runners can use an Artifactory remote repository, a Nexus proxy repository, or repository replication when a durable regional copy is required. A cache reduces repeated network transfers, while replication improves regional availability but adds storage cost and replication lag.

Retention depends on repository purpose. Snapshot builds can expire after a configured age or build count. Staging content can expire after its release window. Released artifacts should follow the release retention policy and must not be deleted while still referenced. Cleanup should also remove safe unreferenced blobs, enforce storage quotas, and check references before deletion.

For availability, I would use redundant repository services, durable metadata and blob storage, backups, health checks, and a tested recovery path. Depending on the product and deployment model, replication can support another region or disaster recovery site.

Finally, every test and deployment records the exact repository path, SHA 256 checksum, signature verification result, source commit, build metadata, SBOM, and promotion record. That creates the evidence chain from reviewed code to CI build to stored artifact to promoted release to the exact artifact consumed.

Common mistakes are using a mutable name such as latest as the release identity, rebuilding an artifact during promotion, mixing architecture variants without enough metadata, trusting a filename without verifying its SHA 256 checksum and signature, giving broad write access to release repositories, deleting release artifacts only because they are old, treating a cache as the only durable copy, and recording only a version label in deployment logs instead of the exact repository path, checksum, signature result, build metadata, and source commit.

Explain the flow in lifecycle order: reviewed source, matrix build, verification, repository storage, promotion, distribution, consumption, and evidence. Emphasize that promotion reuses the same verified artifact bytes and that the final audit trail links the reviewed Git commit to the checksum of the artifact that was actually tested or deployed.

At a high level, I would choose the fabric from the workload, not from a brand preference. The main challenge is balancing predictable accelerator communication against interoperability, operations, and cost. I would compare transport behavior, latency, scaling, RDMA, and cluster fit. InfiniBand fits tightly coupled, communication-heavy training that needs predictable low latency. RoCEv2 Ethernet fits mixed workloads and IP-based services better. Ethernet needs more congestion tuning, while InfiniBand needs a more specialized fabric and operating model.

The question asks which network fabric better fits a large AI cluster. The hard part is that different workloads care about different things. Large synchronized training jobs may spend a lot of time moving data between accelerators. Inference, storage, and normal services may care more about easy IP connectivity and familiar operations. The diagram compares Ethernet using RoCEv2 with InfiniBand across transport behavior, latency, scaling, RDMA, congestion handling, topology, tooling, interoperability, cost, and workload fit. I would use those areas to make the recommendation.

1. Is the cluster mainly running large synchronized training jobs?
2. How important is predictable low latency between accelerators?
3. Will the fabric also carry inference, storage, or normal IP traffic?
4. Does the team already operate a large Ethernet environment?
5. Is easy integration more important than maximum collective communication performance?

I would start by asking what dominates the cluster. The diagram shows training jobs with heavy all-reduce traffic, inference services, and data or storage services. Large synchronized training can become communication-bound. That means accelerator communication limits job speed. In that case, predictable collective performance matters greatly.

For a mixed cluster, Ethernet becomes more attractive. It connects naturally to IP-based services and fits normal data-center operations.

RoCEv2 provides RDMA over UDP/IP. RDMA moves data between hosts with low software overhead. The Ethernet design uses PFC and ECN to help create a low-loss fabric. PFC can pause selected traffic classes. ECN marks congestion so endpoints can react before queues become too full. The downside is extra tuning and congestion-management work.

InfiniBand provides native RDMA. Its fabric has built-in credit-based flow control to help prevent buffer overflow and packet loss. It is designed for tightly coupled HPC and accelerator communication.

The diagram shows Ethernet with generally higher latency and jitter than a well-designed InfiniBand fabric. Modern high-bandwidth Ethernet can still support very large leaf-spine networks. Results depend on topology, oversubscription, congestion control, and tuning.

InfiniBand uses HPC-oriented designs such as fat-tree and rail-optimized topologies. These designs target high bisection bandwidth. In simple words, many accelerators can exchange data across the fabric at the same time. This helps collective-heavy training jobs.

Ethernet has a broad operating ecosystem. The diagram shows familiar tools such as SNMP, NetFlow, and sFlow. It also connects naturally to IP-based cloud, storage, and service networks.

InfiniBand uses specialized tools such as OpenSM, ibdiagnet, performance counters, and fabric telemetry. The diagram also shows cluster services including job scheduling, monitoring, storage, and identity systems. If InfiniBand must reach Ethernet services, cross-fabric access may use the shown IP or service gateway, or suitable host connectivity.

I would choose InfiniBand when large synchronized training is communication-bound and predictable low latency and collective performance dominate. I would choose RoCEv2 Ethernet when mixed traffic, IP interoperability, existing Ethernet skills, and cost flexibility matter more.

The trade-off is operational. Ethernet can reuse familiar infrastructure but needs careful PFC, ECN, topology, and congestion tuning. InfiniBand can provide more predictable training-fabric behavior, but it brings specialized hardware, tools, and operations.

The benefit of RoCEv2 Ethernet is flexibility. It works naturally with IP-based systems and can reuse existing Ethernet skills and tools. The downside is more tuning. Large AI fabrics need careful congestion control, oversubscription planning, PFC, and ECN settings. InfiniBand gives more predictable low-latency behavior for tightly coupled training. The downside is a more specialized fabric, toolset, and operating model. Cost is not automatically lower for either choice. Ethernet can reduce integration cost when existing infrastructure is reused. InfiniBand cost depends on fabric size, topology, hardware, and purchasing choices.

At a high level, I would let all three traffic types share one fast fabric without treating them as one traffic class. The main challenge is protecting latency-sensitive service traffic while AI training and storage move large flows. I would divide the design into logical isolation, QoS protection, and operational control. Separate VRFs isolate each workload. Trusted ingress classifies traffic, while egress queues enforce bandwidth protection. Weighted shares, bounded low-latency priority, congestion control, and rate caps prevent starvation. The trade-off is more policy and tuning work.

The network must carry three very different kinds of traffic over the same physical fabric. AI training sends large collective flows between GPU nodes. Storage replication needs steady bandwidth. Service-mesh traffic is usually smaller, but delay can directly hurt application response time. The difficult part is sharing links without allowing one workload to consume all available capacity. The diagram handles this with separate routing boundaries, traffic classification, protected queues, security controls, monitoring, failure isolation, and controlled rollout of network changes.

1. Does the AI training traffic use RoCE and require PFC?
2. Which service-mesh latency targets must we protect?
3. How should available bandwidth be divided during congestion?
4. Which failures must stay isolated to one tenant or traffic class?

I would first separate reachability before tuning bandwidth. AI TRAINING, STORAGE-REPLICATION, and SERVICE-MESH each use a dedicated VRF or tenant. A VRF gives each workload its own routing table on the shared fabric.

Each domain also has its own subnet, VNI, and routing domain. IPAM / DHCP manages addressing. The SDN Controller manages VXLAN / EVPN state. This limits how far routing and policy mistakes can spread.

The High-Speed Shared Fabric uses a leaf-spine design with several available paths. GPU Nodes exchange collective traffic through RDMA / NCCL. Storage Nodes send Replication Traffic. Service Pods exchange east-west traffic through the service mesh.

The key idea is that logical separation does not require three physical networks. All three classes can share the same links when congestion is controlled correctly.

At trusted ingress, the Policy Engine validates or remarks the traffic class. When an egress link becomes congested, each class enters its configured queue.

AI Training uses a weighted high-throughput queue with a large guaranteed share. If RoCE needs lossless behavior, PFC applies only to the RoCE priority. ECN and RoCE congestion control tell senders to slow down before queues stay overloaded.

Storage Replication uses a weighted throughput queue. It gets a guaranteed share plus a bandwidth cap. Service Mesh uses a bounded low-latency priority queue. Its reserved share protects latency, while its rate cap prevents it from starving training or storage.

Policy & Security uses VRF segmentation, Firewall / ACL rules, RBAC / IAM, and per-tenant rate limiting. Service-mesh traffic uses mTLS and Zero Trust controls.

VRFs and ACLs contain reachability and policy faults. The PFC domain stays limited to the RoCE training priority. A shared-link failure can still affect several classes, so ECMP uses surviving paths where available.

The Telemetry Pipeline watches per-tenant flows, latency, drops, PFC pauses, queue depth, ECN marks, and bandwidth. Dashboards and alerts expose SLO or bandwidth problems.

Change Orchestration uses GitOps and automation. Changes canary by tenant, rack, or zone. Synthetic traffic and SLO checks validate them. Automated rollback stops a bad change from spreading. The main trade-off is extra configuration and tuning complexity.

The benefit is that one physical fabric can serve all three workloads efficiently. Separate VRFs give each workload its own routing boundary. Different queues then protect bandwidth and latency when links become busy. Training keeps a large guaranteed share without becoming an unlimited priority class. Service traffic gets low-latency treatment, but a rate cap stops it from starving other traffic. Storage gets steady bandwidth with its own cap. The downside is more operational work. Queue settings, congestion control, PFC scope, classifications, and policies must stay consistent. A physical link failure can also affect several classes even though their routing and security boundaries remain separate.

At a high level, the NVIDIA device plugin connects Kubernetes with GPUs that already exist on a node. The main challenge is keeping Kubernetes scheduling separate from low-level GPU access. I would explain it in three steps: discover and advertise GPUs, schedule a Pod that requests them, then expose the allocated devices inside its container. The plugin helps with discovery and allocation, but it does not install drivers or manage application-level GPU sharing.

The goal is to let Kubernetes place a Pod on a node that has an available NVIDIA GPU. The difficult part is that Kubernetes does not directly control the GPU hardware. The node already needs an NVIDIA Driver that can talk to the GPUs. The NVIDIA Device Plugin connects that node-level GPU information to Kubernetes. The diagram separates this into GPU discovery, Kubernetes scheduling, GPU allocation, and device access inside the running container. It also shows an important boundary: the plugin does not install drivers or control how an application shares a GPU.

1. Are Pods requesting whole GPUs through nvidia.com/gpu?
2. Should I focus on the normal scheduling and allocation path shown here?
3. Should application-level GPU sharing remain outside this basic device-plugin flow?

I would start with the physical GPU because Kubernetes cannot use it directly. The NVIDIA GPU is on the Kubernetes Node. The NVIDIA Driver runs on the host operating system and provides low-level access to that hardware.

The driver must already be installed and working. The NVIDIA Device Plugin does not install or manage it. This gives the plugin a working GPU environment that it can discover.

Next, the NVIDIA Device Plugin runs as a DaemonSet Pod on the node. It discovers GPUs through the NVIDIA Driver. It then registers with kubelet through the Kubernetes Device Plugin Interface, which uses gRPC.

The plugin reports GPU devices and their health to kubelet. Kubelet then exposes the GPU capacity to Kubernetes as the extended resource nvidia.com/gpu. The Kubernetes Scheduler can see that node capacity when it decides where a GPU Pod can run.

A user creates a Pod with a GPU resource request. In the diagram, the Pod uses limits with nvidia.com/gpu: 1.

The Kubernetes Scheduler checks the nvidia.com/gpu capacity available on nodes. It chooses a node with enough free GPU resources. Kubernetes then binds the Pod to that selected node. The scheduler chooses the node, but it does not choose the exact physical GPU device.

When the Pod starts, kubelet works through its Device Plugin Interface. The diagram shows operations such as ListAndWatch and Allocate.

Kubelet asks the NVIDIA Device Plugin to allocate the required GPU. The plugin returns the device information needed to prepare the container. This keeps Kubernetes scheduling separate from NVIDIA-specific device details on the node.

The Container Runtime, such as containerd or CRI-O, receives the device information through the kubelet-controlled startup path. It prepares the container with the assigned GPU devices and required environment.

The running Pod Container can then see NVIDIA device files such as /dev/nvidia* and use NVIDIA libraries. The application can now use its assigned GPU.

The important boundary is that the device plugin connects GPU resources to Kubernetes scheduling and allocation. It does not install NVIDIA drivers, set application-level CUDA options, manage application-level GPU sharing, enforce application GPU memory limits, or monitor the application's GPU usage.

The benefit is that Kubernetes can treat NVIDIA GPUs as schedulable resources without understanding every hardware detail. The device plugin gives kubelet GPU inventory and health information and helps allocate the correct devices to containers. This keeps the scheduler focused on choosing the right node. The downside is that the plugin does not solve every GPU problem. The NVIDIA Driver must already exist on the node. Application-level sharing, CUDA settings, memory control, fairness, and GPU monitoring need separate mechanisms. This separation keeps responsibilities clear, but operators still need to manage those other parts.