9 Google Cloud Engineer Interview Questions & Answers

At a high level, I would build a cloud platform that safely lets many tenants create and run resources. The main challenge is separating global management from regional customer workloads while keeping failures isolated. I would explain three parts: how requests enter the platform, how the global control plane manages desired state, and how regional data planes run workloads. Strong control-plane state and multi-zone regional placement improve reliability. The trade-off is more safety and isolation at the cost of greater complexity and infrastructure.

The goal is to build a cloud platform where many customers can safely create compute, networking, storage, database, container, and serverless resources. The difficult part is keeping management reliable without putting customer workloads inside the management system. The design solves this by separating a global control plane from regional data planes. The control plane decides what resources should exist. Regional data planes run those resources. Clear tenant, project, region, zone, and security boundaries also stop one failure from spreading across the whole platform.

1. Which resource types must the first version support?
2. Should customers choose the region for every regional resource?
3. Which data-plane resources need optional cross-region copies?
4. How much networking and security control should each tenant receive?

I would start by defining who owns each resource. The hierarchy is Organization, then Folders, then Projects, then Resources. Projects are important isolation boundaries. IAM, quotas, and policies can be applied above a resource and inherited downward. Billing and Metering also track usage at the project level. This gives every later service a clear tenant and ownership context.

Management requests first reach Global Anycast DNS. They then pass through the Global Edge at layer 7. The edge handles DDoS protection, WAF checks, TLS termination, rate limiting, and bot protection. The request then reaches the Control Plane API Gateway. Access and Security services check identity, authorization, policies, keys, and audit requirements before the requested operation continues.

The control plane manages desired state. Compute Service, Container Service, Serverless Service, Networking Service, Storage Service, Database Service, and Messaging Service expose management operations. Policy and Quota applies limits and guardrails.

The Provisioning Orchestrator turns requests into workflows, schedules, and repeated checks. Resource Inventory tracks the catalog and topology. Configuration Service stores desired state and templates. Multi-region State and Metadata Stores keep strongly consistent control information. Caching and Leader Election support fast access and coordination. The Event Bus carries control-plane events between services.

Cross-Cutting Services provide observability, notifications, the service catalog, Billing and Metering, and SLA and reliability information. Global Platform Services provide global DNS, global load balancing, certificate management, the artifact registry, CI/CD automation, chaos testing, and support functions.

The control plane sends work to the selected region. Each region contains availability zones. Compute Node Pools run virtual machines. Container Node Pools run container workloads. Serverless Runtimes run functions and event-driven work.

Regional Data Plane Services provide virtual networking, load balancing, block storage, object storage, file storage, and DNS or DHCP. Regional Data Stores provide SQL or NoSQL storage, cache, messaging, backup and snapshots, and Local KMS. Customer workload execution stays in this regional data plane rather than inside the global control plane.

Under every layer are the hardware fleet, regional network fabric, time and coordination, software-defined infrastructure, image and patch management, capacity management, and failure domains. These foundations let the higher services operate consistently.

The control plane keeps multi-region, strongly consistent metadata. Data-plane resources use multiple availability zones by default, while cross-region replication is optional. Failures are isolated by tenant, project, zone, and cell. Automatic failover, retries, idempotency, and backoff help the platform recover safely.

The main trade-off is consistency versus availability and latency. Strong control-plane consistency makes management safer but requires more coordination. More regions and replication improve resilience but increase cost and operational complexity.

The benefit is strong separation between management and customer workloads. A problem in one region or zone does not need to stop the entire platform. Multi-zone placement also protects regional workloads from smaller failures. The downside is more infrastructure and more operational work. Strong consistency for control-plane state makes resource management safer, but coordination across regions can add delay. Optional cross-region replication can improve recovery, but it costs more. Global services make the platform easier to use, but they can increase latency and make data-location rules harder. More tenant isolation also improves safety, but it reduces how much infrastructure can be shared.

At a high level, the goal is to let users discover and stream video smoothly from many devices. The main challenge is keeping playback fast while authorization, metadata, media processing, and analytics scale safely. I would explain three flows: application requests, media ingest and playback delivery, and background telemetry. Stateless services scale behind load balancing, while CDN edge caches handle heavy media reads. The trade-off is that critical user and billing data stays strongly consistent, while catalog and recommendation data can accept small delays.

The system must let people browse titles, manage their accounts, start authorized playback, and watch video smoothly. It must also accept new media, create several streaming versions, protect those files, and deliver them close to viewers. The difficult part is that video creates very heavy read traffic, while account and billing information must remain correct. The diagram separates normal application requests, media ingest and processing, playback delivery, and background telemetry. Security, regional recovery, caching, and operations support all of these paths without choosing one cloud provider.

1. Which user, subscription, and billing data must stay strongly consistent across regions?
2. How quickly should another region serve traffic after a regional failure?
3. What playback authorization and DRM rules must every supported device follow?
4. How long should original media, processed renditions, and analytics data be kept?

I would begin with the path used by normal viewers. Smart TVs, mobile apps, web browsers, and game consoles enter through DNS using geo or Anycast routing. Static assets can be served from the CDN Edge. Traffic then passes WAF and DDoS Protection, the Global Load Balancer, and the API Gateway. The gateway applies rate limits and authentication checks.

The Application Services are separate microservices. User Service handles profiles and settings. Auth Service handles login, OAuth, and tokens. Catalog Service handles titles and metadata. Watch Service stores continue-watching history. Playlist Service manages My List. Recommendations, Search, and Notification Service handle their own work. The Service Mesh uses mTLS, which means internal service calls are encrypted and authenticated.

Different data uses different stores. The Relational DB keeps user and billing data. The NoSQL DB keeps catalog metadata. The Time Series DB keeps metrics. A Distributed Cache keeps frequently used profiles and catalog data close to the services.

Object Storage keeps metadata, thumbnails, images, and documents. Background work goes through Event Bus, Stream Bus, Task Queue, and Scheduler components. The diagram shows Pub/Sub, Kafka, workers, and cron jobs as examples. This work runs separately so slower background jobs do not block normal requests.

For new content, media enters through Live Ingest or File Upload. Transcoding creates several bitrate and resolution versions, from lower quality through 4K. Packaging creates HLS or DASH output. Media Storage keeps the original files and processed renditions in Object Storage.

DRM and Encryption protect the content. DRM licensing supports PlayReady, Widevine, and FairPlay. Key Management handles key rotation. The Global CDN for Media then places media closer to viewers with multi-region edge caching.

For playback, the app sends a Manifest Request. Playback Auth Service validates the token and entitlements. Manifest Service creates the Dynamic Manifest. CDN or Edge Cache serves manifests and HLS or DASH media segments. Rendition Selection chooses an appropriate bitrate based on available bandwidth. Playback reaches the user through secure HTTPS, with QUIC shown as an option.

Playback Events such as startup, quality-of-experience data, and errors enter the Telemetry and Analytics pipeline. Stream Processing handles real-time analytics. The Data Warehouse supports longer-term analysis. Dashboards and Alerts show business and playback health.

Stateless microservices scale horizontally behind load balancers. CDN capacity absorbs massive read traffic. Strong consistency is used for user and billing data, while catalog and recommendation data can accept small delays. Regional failure uses active-active multi-region service, global DNS failover, and replicated data. IAM and RBAC, Secrets Manager, private networking, audit logs, encryption, compliance controls, backups, and snapshots protect the system. Operations use centralized logs, metrics, traces, alerting, SLO monitoring, cost monitoring, CI/CD, and infrastructure as code. Retries use backoff, circuit breakers, idempotent jobs, and dead-letter queues for resilience.

The benefit is that each part can scale for its own workload. CDN caching keeps most video reads away from the application services and media storage. The downside is that many services, queues, caches, and data stores create more operational work. Strong consistency keeps user and billing data correct, but cross-region coordination can add delay and cost. Catalog and recommendation data can accept small delays, which makes scaling easier. Active-active regions improve availability, but duplicated capacity and replicated data cost more. Tiered hot, warm, and cold storage lowers storage cost. Retries improve resilience, but they need backoff and circuit breakers to avoid making failures worse.

At a high level, TCP uses a three-step handshake to open a reliable, two-way connection. The main challenge is confirming both sides can communicate and synchronizing their starting sequence numbers. I would explain it as SYN, SYN-ACK, and ACK. The client sends SYN with sequence x. The server replies with sequence y and acknowledges x + 1. The client then acknowledges y + 1. Both sides reach ESTABLISHED, and application data can flow. The trade-off is extra setup time before normal communication begins.

The goal is to let a client and server prepare a connection before normal application data starts flowing. Both sides need to confirm communication and establish starting numbers for tracking data correctly. The diagram organizes this into three messages. First, the client asks to connect. Second, the server confirms that request and sends its own starting number. Third, the client confirms the server's response. The diagram also shows the TCP states, how lost handshake segments are retried, how a connection can be rejected, and when normal data transfer can begin.

1. Should I explain both the normal handshake and common failure cases?
2. Should I include the TCP connection states on both sides?
3. Should I explain how the initial sequence numbers are used?

I would start with the client in the CLOSED state. The server is waiting for connection requests in the LISTEN state.

The client sends a SYN segment to the server. SYN means synchronize. Its purpose is to request a connection and send the client's initial sequence number. In the diagram, that number is x. After sending SYN, the client enters SYN-SENT and waits for the server's response.

When the server receives SYN, it sends SYN-ACK back to the client. This segment performs two jobs.

The SYN part carries the server's initial sequence number, shown as y. The ACK part confirms the client's SYN. A SYN consumes one sequence number, so the server sends Ack = x + 1. The server then enters SYN-RECEIVED because it received SYN and sent SYN-ACK.

The client receives SYN-ACK and confirms the server's SYN. It sends an ACK segment back to the server.

The client's next sequence number is x + 1. Its acknowledgment number is y + 1 because the server's SYN also consumes one sequence number. After sending this ACK, the client is ESTABLISHED. When the server receives it, the server also enters ESTABLISHED.

Once the handshake completes, the connection is established. Application data can now flow in both directions.

TCP is full-duplex, which means both sides can send and receive independently. Sequence numbers identify positions in each byte stream. An acknowledgment number tells the sender which byte the receiver expects next. These numbers help TCP provide ordered and reliable data transfer after connection setup.

I would also explain what happens when a handshake segment is lost. If SYN or SYN-ACK is lost, that segment is retransmitted after a retransmission timeout, or RTO. If the final ACK is lost, the server retransmits SYN-ACK, and the client sends ACK again.

If the server is not listening, it may respond with RST, which means reset. The connection is refused, so the client returns to CLOSED. If retries are exhausted without success, the client also gives up and returns to CLOSED. The handshake synchronizes sequence numbers and establishes the TCP connection, but it does not provide encryption or authentication.

The benefit is that both sides confirm communication before normal data transfer begins. They also establish starting sequence numbers for tracking bytes in each direction. The downside is that connection setup needs three segments before both sides are fully established. Lost handshake segments can add delay because TCP may need to retransmit them after a timeout. A rejected connection stops the handshake completely. TCP gives applications reliable and ordered delivery after setup, but the handshake does not encrypt traffic or prove the identity of the other side. Those security features must come from another protocol or layer.

At a high level, the OSI model gives us seven logical layers for understanding network communication. The main challenge is knowing what each layer does and where a failure belongs. I would explain it in three parts: the seven layer responsibilities, encapsulation and decapsulation, and top-down troubleshooting. Data moves from Application down to Physical on the sender, then back up on the receiver. The trade-off is that real TCP/IP networks do not map perfectly to every OSI layer.

The OSI model is a simple way to organize what happens when one computer communicates with another. It divides communication into seven logical layers. Each layer solves a different part of the problem. These layers are not seven separate physical routes. They describe different responsibilities in one client-to-server conversation. I would first explain what each layer does and what kind of addressing or framing it uses. Then I would show how data moves down and back up the stack. Finally, I would use the same layers to troubleshoot a failed connection.

1. Would you like all seven layers with representative protocol examples?
2. Should I also explain how I would troubleshoot a client-to-server failure?

I would say that the OSI model separates networking into seven logical responsibilities. Layers 7 through 5 are closest to applications. Layers 4 through 1 handle transport, routing, local-link delivery, and the physical medium. Each layer serves the layer above it. The model gives us a common way to describe one end-to-end conversation.

Layer 7 is the Application layer. It provides network services used by applications and users. The diagram includes HTTP/HTTPS, DNS, DHCP, SMTP, IMAP, FTP, SSH, and SNMP.

Layer 6 is Presentation. It translates, encrypts, compresses, and formats data. Examples shown include TLS/SSL, JPEG, MPEG, ASCII/UTF-8, JSON, and XML.

Layer 5 is Session. It establishes, manages, and ends communication sessions between applications. The diagram shows RPC, NetBIOS, SIP, PPTP, and SMB Session as representative behaviors.

Layer 4 is Transport. It provides end-to-end delivery, multiplexing, flow control, and reliability when the transport protocol supports those features. TCP and UDP are the examples. Port numbers identify source and destination application endpoints. Its data unit is a TCP segment or UDP datagram.

Layer 3 is Network. It provides logical addressing and routes packets across networks. Source and destination IP addresses are the important addressing boundary. Its data unit is a packet. The diagram includes IPv4, IPv6, ICMP, IPsec, OSPF, and BGP.

Layer 2 is Data Link. It handles framing, MAC addressing, error detection, and media access on a local link. Its data unit is a frame. The diagram includes Ethernet, Wi-Fi, VLAN, PPP, and ARP. Source and destination MAC addresses describe the current link-level delivery and can change when traffic crosses routed links.

Layer 1 is Physical. It transmits raw bits as electrical, optical, or radio signals. Examples include copper, fiber, and radio. It has no network addressing responsibility.

On the client, application data is encapsulated as it moves downward. Relevant layers add control information such as headers. The Physical layer transmits the resulting bits across the medium. The server receives those bits and decapsulates the data upward until the application receives it.

For troubleshooting, I can start at Layer 7 and move downward. I check application behavior, TLS and formatting, session state, ports and TCP state, IP reachability and routing, MAC or VLAN problems, and finally cables, signals, and link status. This gives me a structured diagnosis without pretending each OSI layer is a separate physical route.

The benefit is that the OSI model gives engineers a shared language for network problems. A failed request can be checked one layer at a time instead of treating networking as one large mystery. Encapsulation also explains how application data becomes transport data, packets, frames, and finally bits. The downside is that OSI is a conceptual model. Real TCP/IP networks do not always separate these jobs into seven exact implementation layers. Some protocols also span ideas from more than one OSI layer. We still use the model because it makes responsibilities and troubleshooting easier to explain.

I first check the user's UID and groups, then inspect the target and every parent directory for normal permissions and ACLs. Finally, I verify access as that user with a read-only test -x. On files, x means executable; on directories, it means searchable or traversable.

The goal is to find out whether one person is allowed to use a particular item in the required way. I first confirm exactly which person is being checked and which groups apply to that person. Then I inspect the item and every folder leading to it, because access can be blocked before the final item is reached. I also check for extra access rules that may grant or restrict permission. Finally, I safely try the permission check as that person without changing anything, so I can confirm what the system will actually allow.

1. Is the target a regular file, a directory, or should I explain both cases?
2. Should I include POSIX ACLs in addition to normal owner, group, and other permission bits?
3. May I assume I have authorized administrative access to perform a read-only check as the target user?

I would separate identity from authorization. Authentication establishes which user is being evaluated. Authorization determines whether that user is allowed to execute a file or traverse a directory.

First, I confirm the target user's UID, primary group, and supplementary groups with id USER. Group membership matters because Linux may authorize access through the owning group, a supplementary group, or a POSIX ACL entry.

Next, I inspect the complete path rather than checking only the final object. namei -l PATH is useful because it shows the owner, group, and mode bits for every component of the path. To reach a file or lower directory, the user needs execute permission on every parent directory. On a directory, execute means search or traversal: the user can pass through it and access a known entry name when other permissions allow it. Directory read permission is different; it normally controls whether directory entry names can be listed.

For the final object, I use stat PATH or ls -ld PATH to examine its type, owner, group, and normal rwx mode bits. Linux evaluates the permissions that apply to the target identity rather than combining owner, group, and other permissions together.

I also inspect POSIX ACLs with getfacl. An ACL, or Access Control List, can define permissions for named users and groups in addition to the normal mode bits. The ACL mask can limit the effective permissions of named-user entries, named-group entries, and the owning-group entry, so I check the effective ACL result rather than assuming the displayed rwx characters alone tell the whole story. Because traversal permission is required on every directory in the path, I would inspect ACLs on parent directories too when normal mode bits do not fully explain access.

After the metadata checks, I verify the effective result without modifying permissions. If I am authorized to run commands as the target identity, I can use sudo -u USER -- test -x PATH and inspect its exit status. This provides a read-only check using the target user's effective credentials instead of testing as root.

For a regular file, test -x checks whether the process has execute permission for that path. For a directory, test -x checks search or traversal permission. It does not mean the user can list the directory; listing generally requires read permission as well.

If the permission metadata appears correct but direct program execution still fails, I would distinguish permission authorization from other execution restrictions. For example, a filesystem mounted with noexec can prevent execution even when the user's file permission check succeeds. A script can also fail because its interpreter is missing or inaccessible. Those are runtime or filesystem restrictions, not evidence that the basic user permission calculation was wrong.

Throughout the investigation, I keep the checks read-only. I do not use chmod, chown, setfacl, or group changes merely to discover whether access exists. That preserves least privilege, avoids changing production state, and gives an auditable result.

1. Identify the exact target user and target path.
2. Run id USER to confirm the user's UID, primary group, and supplementary groups.
3. Run namei -l PATH to inspect ownership and mode permissions on every path component.
4. Inspect the final object with stat PATH or ls -ld PATH.
5. Run getfacl PATH and inspect parent-directory ACLs when needed.
6. Determine whether the applicable owner, named-user, group, named-group, mask, or other permissions provide execute access.
7. Verify execute permission on the file or search/traversal permission on every required directory.
8. When authorized, run a read-only sudo -u USER -- test -x PATH check and inspect its exit status.
9. If program execution still fails despite a successful permission check, investigate separate restrictions such as a noexec filesystem mount or an unavailable script interpreter.

These checks are inexpensive. Identity and file metadata commands use very little memory. The amount of work mainly grows with the number of directories in the path and the number of ACL entries that must be examined. A path with more components requires more permission checks. Operational risk is low because the recommended commands are read-only. Maintenance cost is also low because they use standard Linux identity, filesystem, and ACL tools. The main security consideration is that impersonating another user for verification requires explicitly authorized administrative privilege.

A common mistake is checking only ls -l on the final file. That can miss supplementary groups, POSIX ACLs, ACL masks, and blocked traversal on a parent directory. Another mistake is assuming directory execute permission means the same thing as file execute permission; for directories it means search or traversal. Testing as root is also misleading because root does not represent the target user's normal authorization. Candidates should not use chmod, chown, setfacl, or group changes simply to test access. Finally, a successful test -x does not guarantee that a program will run successfully, because separate restrictions such as a noexec mount or a missing script interpreter can still prevent execution.

Present the answer in five layers: identity and groups, every parent directory, final-object mode bits, ACLs, and a read-only verification as the target user. Explicitly explain that execute permission means execution for a regular file but search or traversal for a directory.

SIGKILL, signal 9, forcibly terminates a Linux process and cannot be caught, blocked, or ignored. The process cannot perform its own cleanup. Prefer SIGTERM for graceful shutdown, and use SIGKILL mainly when graceful termination fails or an immediate forced stop is necessary.

This question asks what happens when a computer program is forced to stop immediately. The important point is that the program gets no warning and no chance to finish its work neatly. It cannot save unfinished work, close things in its normal way, or prepare for shutdown. The computer takes back resources that belong to the stopped program, but work outside the program may still be unfinished or inconsistent. You should also explain why a normal, polite stop request is usually tried first and when an immediate forced stop is justified.

1. Should I explain the difference between SIGKILL and SIGTERM?
2. Should I include how a parent process observes SIGKILL termination?
3. Should I discuss the uninterruptible-sleep, or D-state, edge case?

SIGKILL is Linux signal 9. Its significance is that it requests forced termination and the target process cannot catch, block, or ignore it. No signal handler runs, so the application has no opportunity to perform user-space cleanup before termination.

The kernel reclaims kernel-managed resources such as the process's memory and file descriptors. Closing file descriptors normally releases kernel-managed file locks. However, application or external state can still be inconsistent because application cleanup did not execute. For example, an application workflow may be only partly complete, user-space buffered data may not have been explicitly flushed by the application, and an external or distributed lock may remain until its own lease or recovery mechanism clears it.

Sockets owned by the process are closed by the kernel, but the application cannot perform its own graceful protocol shutdown or send a final message. For database work, an uncommitted transaction may be rolled back when its connection closes, depending on the database, but a larger application-level business workflow can still remain incomplete.

The kernel records that the process was terminated by signal 9. For a child process, the parent can use wait or waitpid to obtain the termination status and determine that SIGKILL caused the termination. Many shells and container tools commonly represent signal 9 termination as status 137 because 128 + 9 = 137, but 137 is a tool convention rather than a universal kernel exit status.

SIGTERM, signal 15, is normally preferred first. SIGTERM can be caught and handled, so an application can stop accepting work, flush application buffers, close connections gracefully, release application-managed resources, and complete its shutdown logic. SIGKILL is appropriate as a last resort when a process is stuck or unresponsive, ignores SIGTERM or cannot complete graceful termination, or must be stopped immediately to protect the system.

One important edge case is a task in uninterruptible sleep, commonly shown as D state. SIGKILL can be pending for that task, but the task may not actually exit until the kernel wait causing that state finishes.

After using SIGKILL, verify that the process is gone and check application or external state that normally depends on cleanup before restarting the process or retrying work.

1. Prefer a graceful termination such as SIGTERM when the situation allows it.
2. Give the process enough time to run its shutdown handling and exit.
3. If graceful termination fails, decide whether forced termination is necessary to restore or protect the system.
4. Use SIGKILL only when that forced stop is justified.
5. Confirm that the process has actually exited.
6. Check application and external state that may have depended on cleanup before restarting or retrying work.
7. If the task is in uninterruptible sleep, understand that SIGKILL may remain pending until the kernel wait finishes.

SIGKILL itself is operationally simple because the target application does not get a chance to handle it. The main cost is recovery risk rather than CPU or memory complexity. Forced termination can leave unfinished application work, external locks, partial workflows, or state that must be checked before restart. SIGTERM can take longer because it allows graceful cleanup, but that extra time usually reduces recovery work and the chance of inconsistent application state.

Common mistakes are saying SIGKILL can be caught or handled, assuming application cleanup runs, claiming kernel-managed resources remain allocated, or saying every SIGKILL always produces exit code 137. The kernel reclaims kernel-managed resources, while application or external state may remain incomplete. Another mistake is using SIGKILL as the normal first choice instead of trying SIGTERM. Also, SIGKILL does not guarantee immediate disappearance while a task remains blocked in uninterruptible sleep.

Start with the key contrast: SIGTERM asks a process to shut down gracefully, while SIGKILL forces termination with no process-level cleanup. Then explain kernel resource reclamation, possible application-state risks, termination reporting, the D-state edge case, and why SIGKILL should normally be the last resort.

Run sar -d -p 1 300 and inspect tps, rkB/s, wkB/s, areq-sz, aqu-sz, await, and %util. Look for sustained changes from baseline and corroborate them with other host and application evidence before blaming storage.

The goal is to find out whether the part of the machine that reads and saves data is struggling while the slowdown is happening. I would watch it for several minutes instead of judging one moment. I would separate how many jobs are arriving, how much data is moving, whether work is piling up, how long each job takes, and how busy the device stays. Then I would compare those observations with what is normal for that machine and check other parts of the system before deciding what actually caused the problem.

1. Are we investigating a live incident, or should I describe the normal troubleshooting procedure?
2. Do we have a known healthy baseline for this host or device?
3. Is the workload using local disks, virtual block storage, or another storage layer that also needs to be checked?

I would start with a defined sampling window rather than a one-time snapshot. On standard sysstat, I would run sar -d -p 1 300. The -d option reports activity for each block device. The -p option makes device names easier to read; it does not take an ALL argument. By default, -d reports all devices unless the command is restricted with a device filter. The final 1 300 means one sample every second for 300 samples, so the observation lasts about five minutes.

For each device, I would read several fields together. tps is the number of I/O transfers per second, so it represents request volume. A high value means many I/O operations, but it does not tell me how much data is moving.

rkB/s and wkB/s show read and write throughput in kibibytes per second. These tell me how much data is being transferred. High throughput may be healthy if it is normal for the workload and within the device's capability.

areq-sz is the average request size in kibibytes. It gives context to tps and throughput. For example, many small requests can produce high tps without equally high throughput. Request size alone does not prove a random or sequential access pattern.

aqu-sz is the average queue length of requests issued to the device. A sustained rise above the normal baseline can indicate that more I/O is outstanding or queueing. I would not use a universal threshold because healthy values depend on the device, concurrency, workload, and storage stack.

await is the average time, in milliseconds, for I/O requests issued to the device to be served. It includes time spent waiting and being serviced. If await rises above baseline while aqu-sz also rises, requests are taking longer to complete and queueing is stronger evidence of storage pressure.

%util is the percentage of elapsed time during which I/O requests were issued to the device. For devices that serve requests serially, a value near 100% can indicate that the device is busy almost continuously. For modern SSDs, RAID, and other devices that handle requests in parallel, %util does not by itself show the device's performance limit. I would treat it as supporting evidence, not proof of saturation.

I would therefore read the signals as one story: tps shows request volume; rkB/s and wkB/s show throughput; areq-sz describes average request size; aqu-sz shows outstanding or queued work; await shows I/O completion delay; and %util shows device busy time.

The final decision must use correlated evidence. If aqu-sz and await rise above baseline, %util is also high where meaningful, and application latency or user impact worsens during the same interval, storage becomes a stronger hypothesis. I would still check CPU, memory, load, disk-space pressure, application metrics, logs, latency or timeout signals, recent changes, and the underlying cloud-storage service. Storage pressure is a possible cause, not a conclusion from one sar field.

1. Run sar -d -p 1 300 during the suspected problem window.
2. Identify the device or devices serving the affected workload.
3. Read tps for I/O request volume.
4. Read rkB/s and wkB/s for read and write throughput.
5. Use areq-sz to understand average request size.
6. Compare aqu-sz with the normal baseline to identify sustained queueing or outstanding work.
7. Check await to see whether I/O completion time is elevated.
8. Use %util as supporting device-busy evidence, with extra caution for parallel devices such as SSDs and RAID.
9. Correlate the same time window with CPU, memory, load, disk-space pressure, application metrics, logs, latency, timeouts, user impact, recent changes, and the underlying storage service.
10. Call storage a likely bottleneck only when multiple observations support that conclusion and competing explanations have been checked.

The command is lightweight because it reads operating-system counters instead of scanning application data. Sampling every second for five minutes produces 300 observations per device, so the amount of output grows with the number of devices and samples. The main cost is analysis: you must compare several fields over time and against a healthy baseline. A window that is too short can miss sustained pressure, while a very long or very frequent collection produces more data to review and retain. Exact field availability and interpretation can also vary with the installed sysstat version and the storage technology.

Common mistakes are taking one sample instead of observing a useful interval; treating high tps as a problem without considering throughput and request size; using an absolute aqu-sz threshold as proof of trouble; assuming high await identifies its own cause; and treating %util near 100% as automatic proof that storage is saturated. Another mistake is using legacy sector-based field names while describing their values as kilobytes. Also, on standard sysstat, -p is a standalone pretty-print flag, so sar -d -p ALL 1 300 should not be copied literally; use sar -d -p 1 300 to show all block devices unless you intentionally restrict them with a device filter.

Explain the fields as a diagnostic story: request volume, throughput, request size, queueing, wait time, then device busy time. Emphasize trends and correlation instead of one high number, and finish by saying you would corroborate the evidence before naming storage as the root cause.

I would capture on the interface carrying the suspected traffic, filter narrowly by host, protocol, and port, bound the capture by time and packet count, and save a pcap. Then I would analyze direction, timing, TCP flags, retransmissions, and responses to support or reject the network-path hypothesis.

The goal is to watch only the small amount of information related to the connection problem, save it so it can be checked later, and avoid collecting more than needed. First, find which connection on the machine carries the traffic between the two computers. Then narrow the collection to the other machine and the service being tested. Keep the collection short and limited. Afterward, compare what leaves the machine with what comes back, how long replies take, and whether the expected exchange finishes. Use that evidence to decide what part of the path to check next.

1. Which source, destination, protocol, and port are involved in the suspected network problem?
2. Which Linux host and interface are on the path I should observe?
3. Is the hypothesis about outbound connectivity, the return path, a firewall, routing, or the backend service?
4. Are there restrictions on capturing payload data or retaining packet files?

I would start by defining the hypothesis and choosing the correct capture boundary. In the diagram, the Linux host has ens4 toward the client side and ens5 toward the backend. Because the hypothesis concerns the backend at 10.20.40.20 on TCP port 5432, I would capture on ens5. I would verify interface names and addresses first with ip -brief addr instead of assuming an interface name.

Next I would use a Berkeley Packet Filter, or BPF, to collect only relevant traffic. The focused filter for this case is host 10.20.40.20 and tcp port 5432. Other valid examples are dst host 10.20.40.20 and tcp port 5432, src net 10.20.30.0/24 and (tcp port 80 or tcp port 443), icmp and host 203.0.113.10, or tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0 when I only need TCP control packets. Narrow filters reduce noise, disk use, processing, and unnecessary exposure of unrelated traffic.

For a bounded production capture, I would use a command such as sudo timeout 300 tcpdump -i ens5 -p -nn -s 128 -c 10000 -w /var/tmp/backend-5432.pcap 'host 10.20.40.20 and tcp port 5432'. -i ens5 selects the correct interface. -p avoids promiscuous mode when it is unnecessary. -nn disables hostname and service-name lookups. -s 128 limits bytes captured from each packet to reduce payload exposure, although that can truncate data that might later be needed. -c 10000 caps the packet count. -w preserves the packets in a pcap file, and timeout 300 limits the capture to five minutes.

I would treat the pcap as sensitive evidence. I would use the least privilege needed to run tcpdump, keep the capture window short, store the pcap in a protected location, and capture full packet payloads only when the investigation requires them. Before drawing conclusions, I would confirm that packets are actually visible on ens5, that the peer and TCP port 5432 are correct, and that the capture contains the packet directions needed for the hypothesis.

Then I would analyze four kinds of evidence. Direction tells me whether the request leaves and whether a response returns. Timing shows round-trip delay and unusually large gaps. TCP flags such as SYN, SYN/ACK, ACK, and RST show connection-state behavior. Responses such as ICMP errors and retransmissions can support hypotheses about loss, blocking, reachability, or latency.

The interpretation must remain limited to what the capture proves. If I see a SYN leave but no SYN/ACK return, that supports a downstream-loss or return-path hypothesis, but it does not prove which firewall or router is responsible. If I see SYN, SYN/ACK, and ACK, the TCP path was established, but that does not prove the application is healthy. If I see an RST response, that supports the conclusion that the peer or host networking stack rejected the connection, but the packet capture alone may not explain why. Repeated retransmissions or large timing gaps support a loss or latency hypothesis, but they do not identify the exact failing device.

For offline analysis, I can read the saved capture with tcpdump -r backend-5432.pcap -nn or review timing and direction with tcpdump -r backend-5432.pcap -nn -tttt. If deeper decoding is required, I can open the same pcap in Wireshark. I would correlate packet evidence with routing, firewall, service, and host evidence. If the evidence is inconclusive, I would move the capture point to another boundary and repeat the same hypothesis-driven test.

1. Define the network-path hypothesis and identify the source, destination, protocol, and port.
2. Inspect the Linux host interfaces and select the interface that actually carries the suspected traffic.
3. Build the narrowest useful BPF filter for the relevant peer and service.
4. Bound the capture by duration, packet count, and snap length, and save it to a protected pcap file.
5. Confirm that the expected peer, port, and packet directions appear in the capture.
6. Analyze request and response direction, timing gaps, TCP flags, retransmissions, and ICMP responses.
7. Compare the observed packet pattern with the predicted failure pattern.
8. Support or reject the hypothesis without claiming more than the capture proves.
9. Correlate the result with routing, firewall, service, and host evidence.
10. Move the capture to another boundary if the evidence remains inconclusive.

The main cost is proportional to how much traffic is captured. A broad or long capture can consume disk space, add processing overhead, and collect sensitive data. Narrow filters, a limited snap length, a packet-count cap, and a short time window reduce those risks. Using a smaller snap length can hide payload or header details that later analysis needs, so it is a privacy-versus-diagnostic-detail tradeoff. Offline analysis adds storage and operator time but lets engineers study evidence without repeatedly capturing production traffic.

Common mistakes are capturing on the wrong interface, using an overly broad filter, forgetting to save the capture for offline analysis, running an unbounded capture, capturing full payloads unnecessarily, leaving sensitive pcap files unprotected, and assuming one packet pattern proves an exact root cause. A SYN with no visible SYN/ACK does not prove a particular firewall failed. A completed TCP handshake does not prove the application is healthy. An RST does not by itself explain why the connection was rejected. Other mistakes include ignoring packet direction and timing, using invalid BPF syntax, or accidentally using options from another packet-analysis tool as if they were tcpdump options.

Present the answer as a hypothesis-driven sequence: choose the capture boundary, filter narrowly, bound and preserve the capture, confirm that you captured the expected traffic, inspect direction, timing, flags, and responses, and then state both what the evidence supports and what it does not prove. Emphasizing safe production capture and evidence limits shows stronger troubleshooting judgment than simply listing tcpdump flags.