EnglishPublished Jun 27, 2026Updated Jun 27, 202621 min read

50 Backend & Distributed Systems Scenario Questions (With Answers)

Tips

Fifty production-style debugging and design questions, each answered with a focus on what's happening internally and how to fix it. These are the kinds of scenarios that come up in senior backend interviews and, more importantly, in real on-call shifts. Grouped into three parts: ten core questions, then two batches of follow-ups.

Part 1 — Core Questions

1. A HashMap with millions of records suddenly degrades. What's happening internally?

Hash collisions. Bad/weak hashCode() clusters entries into few buckets. Each bucket is a chain — lookups degrade from O(1) toward O(n). Java 8+ converts a bucket to a red-black tree once it holds 8+ entries (O(log n)), which helps but is still far slower than O(1).
Resizing / rehashing. When size > capacity × loadFactor (default 0.75), the map doubles capacity and rehashes every entry. With millions of records this is expensive and creates GC pressure.
GC pressure. Millions of Node/Entry objects bloat the heap → longer, more frequent GC pauses that look like "the map got slow."
Concurrency misuse. HashMap is not thread-safe. Concurrent writes corrupt the structure; pre-Java 8 they could spin into an infinite loop during resize.

Fixes: size the map up front (new HashMap<>(expected / 0.75 + 1)), use a strong hashCode(), use immutable keys, switch to ConcurrentHashMap for concurrent access, and consider primitive-specialized maps (fastutil, Koloboke) to cut object overhead.

2. Two threads update the same account balance. How do you prevent data inconsistency?

The core problem is a read-modify-write race: both threads read the old balance, compute, and overwrite each other (lost update).

Don't read-then-write in the app. Do a conditional atomic update in the DB: UPDATE account SET balance = balance - :amt WHERE id = :id AND balance >= :amt — the database serializes it.
Optimistic locking — version column (@Version in JPA). The update checks the version; on conflict you get OptimisticLockException and retry. Best for low contention.
Pessimistic locking — SELECT ... FOR UPDATE holds a row lock for the transaction. Best for high contention, costs throughput.
In a single JVM — synchronized / ReentrantLock / AtomicLong, but this breaks the moment you scale to multiple instances.
Across instances — a distributed lock (Redis, ZooKeeper), but prefer letting the DB enforce it.

3. Why might a Spring `@Transactional` method fail to roll back?

Checked exceptions don't trigger rollback by default — only RuntimeException and Error do. Use @Transactional(rollbackFor = Exception.class).
Self-invocation. Calling a @Transactional method from another method in the same class bypasses the Spring proxy — no transaction is started. Call it through an injected bean.
Non-public method. Proxy-based AOP only advises public methods.
Exception swallowed. A try/catch inside the method eats the exception, so Spring never sees it.
Wrong propagation (e.g. NOT_SUPPORTED, or a nested call that doesn't open a new tx).
Storage engine / datasource doesn't support transactions (e.g. MySQL MyISAM).
Multiple datasources with the wrong PlatformTransactionManager.
Bean not Spring-managed (created with new).

4. API works locally but returns 503 in production. What do you investigate first?

503 = Service Unavailable — usually the load balancer has no healthy upstream, or the app is overloaded.

Investigate, roughly in order:

Is the instance registered and healthy in the LB / ingress? A failing readiness probe pulls it out of rotation.
Is the app actually up? Check for crash-on-startup, OOMKilled containers, restart loops.
Resource limits — thread pool / connection pool exhausted, container CPU/memory limits, OOM.
Downstream timeouts — a slow dependency causes requests to pile up and the LB to mark the app unhealthy.
Config drift — prod env vars, secrets, DB URLs, active Spring profile differ from local.

The key insight: local has one instance, no LB, no real traffic, no resource caps. 503 is almost always infrastructure / capacity, not your business logic.

5. A Kafka consumer processes the same message twice. How do you handle it?

Kafka is at-least-once by default — duplicates are expected (consumer rebalance, redelivery before offset commit, retries). Don't try to eliminate them; make processing idempotent.

Dedup by a stable ID (message key / business ID) in a processed-messages table or via an upsert.
Commit offsets after processing, not before (manual ack).
Idempotency table keyed on message ID, written in the same transaction as the side effect.
For end-to-end exactly-once semantics within Kafka: idempotent producer + transactional reads/writes (read_committed, EOS). But idempotent consumers are simpler and robust.

6. DB connections are getting exhausted but traffic hasn't increased. Why?

Almost always a leak or things holding connections too long — not load.

Connection leak — connections not returned to the pool (missing close(), exception path skips cleanup). Most common cause.
Long-running / hung queries holding connections open.
Transactions left open — no commit/rollback, so the connection never frees.
A new code path opening raw connections outside the pool.
DB-side — locks/contention making queries hang, or a lowered max_connections.

Diagnose: pool metrics (HikariCP active/idle/pending), enable leakDetectionThreshold, and SHOW PROCESSLIST / pg_stat_activity to find stuck queries.

7. One microservice gets slow and starts affecting others. How do you isolate it?

This is a cascading failure through synchronous call chains — slow callee fills the caller's threads, which backs up its callers.

Distributed tracing (Jaeger/Zipkin, propagated trace IDs) to find the slow hop.
Timeouts on every remote call — never wait indefinitely.
Circuit breaker (Resilience4j) — stop calling a failing dependency, fail fast, return a fallback.
Bulkheads — isolate each dependency in its own thread pool so one slow one can't starve the rest.
Backpressure / rate limiting, and decouple with a queue where the call doesn't need to be synchronous.

8. A Spring Boot app gradually consumes more memory. How do you debug it?

Suspect a memory leak (heap grows and doesn't recover after GC).

Confirm it's a leak — GC logs / heap graph. If old-gen keeps climbing after full GC, it's a leak, not just churn.
Heap dump — -XX:+HeapDumpOnOutOfMemoryError, or jmap. Analyze in Eclipse MAT / VisualVM → dominator tree → what's retaining memory.
Common culprits: unbounded caches (no eviction/TTL), growing static collections, ThreadLocals never cleared, unclosed resources, listeners/subscriptions never removed, classloader leaks on hot redeploy.
Profilers: async-profiler, Java Flight Recorder.
Don't forget non-heap — Metaspace, direct/off-heap buffers (Netty), thread stacks.

9. Retry logic prevents failures, but users report duplicate transactions. Why?

The original request actually succeeded, but the response was lost (timeout, dropped connection). The client retries a non-idempotent write → the operation runs twice.

Fix: idempotency.

Client generates an idempotency key (UUID) per logical operation and sends it on every retry. The server records it and returns the same result for a repeat key instead of re-executing.
Make the write itself idempotent (upsert / conditional update).
Only auto-retry operations that are idempotent or protected by a key.

10. Service passes health checks but customers still hit failures. How do you troubleshoot?

The health check is too shallow — liveness only proves the process is up, not that real functionality works.

A /health returning 200 says nothing about a specific broken endpoint, a degraded downstream, or one bad instance behind the LB.
Look at real signals: per-endpoint error rates, latency percentiles, logs and traces — not just /health.
Synthetic / black-box monitoring of actual user flows (login, checkout), not a ping.
Partial failures: one downstream down, one region, one bad pod, certain user segments.
Add deeper readiness checks for critical dependencies — but carefully, so a single flaky dependency doesn't take the whole fleet out of rotation (avoid cascading via health checks).

Part 2 — More Scenarios

11. API latency spikes every few minutes, then recovers. What's happening?

GC stop-the-world pauses (large heap, allocation spikes) — check GC logs.
Cache stampede — many keys expire at once → thundering herd to the DB.
Scheduled/batch jobs or cron tasks competing for CPU/IO.
Connection pool / thread pool briefly saturating under bursts.

Fix: tune GC (or reduce allocation), add TTL jitter, move batch work off the request path.

12. Reads return stale data right after a write. Why?

Read-replica lag — write hits primary, read hits a replica that hasn't caught up (eventual consistency).
Stale cache — cache not invalidated/updated on write.

Fix: read-your-writes (route the user's reads to primary briefly, or pin to a session), write-through / invalidate cache on write, or accept and design for eventual consistency.

13. You added a cache, but DB load still spikes periodically. Why?

Cache stampede / thundering herd. Many entries share a TTL and expire simultaneously, or a hot key expires and thousands of concurrent requests all miss and hit the DB at once.

Fix: TTL jitter (randomize expiry), single-flight / mutex so only one request rebuilds a key while others wait, and refresh-ahead (proactively refresh before expiry).

14. Two services occasionally deadlock at the database. Why?

Lock-ordering deadlock — transaction A locks row 1 then row 2; transaction B locks row 2 then row 1; they wait on each other. The DB's deadlock detector kills one.

Fix: acquire locks in a consistent global order, keep transactions short, lower isolation where safe, and retry the victim transaction.

15. An ORM query gets dramatically slower as data grows. Why?

N+1 query problem — lazy associations loaded inside a loop fire one query per parent row (1 + N).

Fix: JOIN FETCH / entity graphs / batch fetching, project only needed columns (DTOs), and verify with SQL logging. Also check for missing indexes on filter/join columns.

16. You need a transaction spanning two microservices. How?

Distributed 2-phase commit is fragile and rarely worth it. Use the Saga pattern — a sequence of local transactions, each with a compensating action to undo on failure.

Pair with the Transactional Outbox pattern: write the state change and an outbox event in one local tx, then publish the event reliably (CDC / poller) — avoids the dual-write problem.

17. Kafka messages are processed out of order. Why?

Kafka only guarantees ordering within a partition. Messages for the same entity landed in different partitions (or multiple consumers / concurrent processing reordered them).

Fix: partition by a key (e.g. accountId) so all events for one entity share a partition, and process a partition single-threaded for that key.

18. Request threads are exhausted under modest load. Why?

Blocking calls inside a fixed thread pool — every thread is parked waiting on slow I/O (DB, remote call), so new requests queue.

Fix: right-size the pool, add timeouts, move to async/reactive (WebClient, virtual threads), and use bulkheads to isolate slow dependencies.

19. How would you design a rate limiter, and a distributed one?

Algorithms: fixed window (simple, bursty at edges), sliding window (smoother), token bucket (allows bursts up to bucket size), leaky bucket (smooth output rate).
Distributed: keep counters/tokens in Redis (atomic INCR/Lua script for token bucket) keyed by client + window, so all app instances share state. Watch for clock skew and the cost of a Redis round-trip per request (mitigate with local pre-checks).

20. App is slow right after deploy, then speeds up. Why?

JIT warmup — the JVM interprets before compiling hot paths.
Cold caches — application and DB buffer caches empty.
Cold connection pools — connections established lazily under first load.

Fix: warmup requests before adding to the LB, pre-populate critical caches, and pre-size pools (min idle).

21. A distributed system misbehaves due to clock skew. Why, and what do you do?

Wall clocks on different machines drift. Relying on System.currentTimeMillis() for ordering, token expiry, or "last write wins" produces wrong results when clocks disagree.

Fix: sync with NTP, but don't trust wall clocks for ordering — use logical clocks (Lamport / vector clocks) or monotonic sequence numbers, and allow leeway on token expiry validation.

22. Deploys drop in-flight requests. How do you fix it?

The process is killed (SIGTERM/SIGKILL) before active requests finish.

Fix: graceful shutdown — on SIGTERM, deregister from the load balancer, stop accepting new requests, drain in-flight ones within a timeout, then exit. Spring Boot: server.shutdown=graceful + a termination grace period in the orchestrator (e.g. k8s terminationGracePeriodSeconds and a preStop hook).

23. One shard/partition is overloaded while others are idle. Why?

Hot key / hot partition — uneven key distribution (e.g. a celebrity user, a monotonically increasing timestamp key, or a low-cardinality partition key).

Fix: choose a higher-cardinality / better-distributed partition key, add a salt/suffix to spread a hot key, or split the hot partition. For writes on sequential IDs, hash the key.

24. How do you design an idempotency key for a payments API?

Client sends a unique Idempotency-Key header per logical request.
Server stores (key → request hash, status, response); first request executes and persists the result atomically with the side effect.
A repeat key returns the stored response instead of re-executing.
Include a request-fingerprint check so the same key with a different body is rejected, and set a retention TTL.

25. When do you choose availability over consistency (CAP)?

Under a network partition you can't have both.

Choose CP (consistency) when correctness is non-negotiable — payments, balances, inventory you can't oversell.
Choose AP (availability) when staleness is acceptable and uptime matters more — feeds, recommendations, social counts, presence.

Most real systems are a mix: strong consistency on the money path, eventual consistency everywhere else.

Part 3 — Even More Scenarios

26. An index exists, but the query still does a full table scan. Why?

The planner decided the index isn't usable or isn't worth it:

Function/expression on the column — WHERE LOWER(email) = ? can't use a plain index on email (need a functional index).
Leading-column rule — a composite index (a, b) can't serve a query filtering only on b.
Implicit type cast — comparing a varchar column to a number disables the index.
Low selectivity — if the predicate matches a large fraction of rows, a scan is genuinely cheaper.
Stale statistics — the optimizer mis-estimates cardinality. Run ANALYZE.

Check with EXPLAIN ANALYZE.

27. Writes got slower after you added several indexes. Why?

Every index is a secondary structure the DB must keep in sync. Each INSERT/UPDATE/DELETE now updates the table and every affected index (B-tree maintenance, page splits, more WAL/redo). More indexes = faster reads, slower writes, more disk.

Fix: keep only indexes that earn their keep, drop unused/duplicate ones, prefer composite indexes over many single-column ones, and don't index low-selectivity columns.

28. `OFFSET`-based pagination gets slower on deep pages. Why?

LIMIT 20 OFFSET 100000 makes the DB scan and discard 100,000 rows before returning 20. Cost grows linearly with page depth.

Fix: keyset / cursor pagination — WHERE id > :last_seen_id ORDER BY id LIMIT 20. Constant time, uses the index, and is stable under concurrent inserts.

29. After a replica failover, some recently-committed writes vanished. Why?

Asynchronous replication — the primary acked the write before the replica received it. On failover, the replica is promoted without those last writes (non-zero RPO).

Fix: use semi-synchronous / synchronous replication for critical data (primary waits for ≥1 replica to ack), accept the latency cost, and understand your RPO/RTO tradeoff.

30. A Redis-based distributed lock occasionally lets two clients into the critical section. Why?

The lock expired (TTL) while client A was still working (GC pause, slow I/O), so client B acquired it → two holders.
A client deletes a lock it no longer owns (deletes B's lock thinking it's its own).

Fix: store a unique owner token and release with a compare-and-delete Lua script; use a fencing token (monotonic number) that the protected resource checks, so a stale holder's writes are rejected. For correctness-critical locks, a single Redis node isn't safe.

31. Kafka consumer lag keeps growing. How do you diagnose and fix it?

Consumers can't keep up with the produce rate.

Diagnose: per-partition lag, processing time per message, rebalance frequency, downstream (DB/API) latency inside the consumer.
Fix: add consumers up to the partition count (parallelism is capped by partitions — add partitions if needed), batch processing, move slow work async, tune max.poll.records, and make sure one slow message isn't blocking the poll loop.

32. A single bad message blocks the whole consumer. How do you handle it?

A poison message that always fails keeps getting retried and stalls the partition.

Fix: bounded retries, then route it to a Dead Letter Queue (DLQ) and move on. Alert on DLQ depth, store enough context to replay after a fix, and never retry a non-transient failure forever.

33. What's the difference between at-most-once, at-least-once, and exactly-once delivery?

At-most-once — fire and forget; may lose messages, never duplicates. (Commit offset before processing.)
At-least-once — never lose, may duplicate; the common default. (Process, then commit.) Requires idempotent consumers.
Exactly-once — no loss, no duplicates. Expensive; achieved end-to-end via idempotency + transactions (Kafka EOS), not magic. In practice: at-least-once + idempotency ≈ exactly-once effect.

34. With cache-aside, why do you sometimes serve stale data after an update?

Classic cache-aside race: reader misses cache, reads old DB value; meanwhile a writer updates the DB and deletes the cache key; the reader then writes its stale value back into the cache. Now the cache is wrong until TTL.

Fix: delete-on-write (not write-on-write), short TTL as a safety net, versioned keys, or write-through. For strict cases, single-flight the rebuild.

35. What cache-write strategy should you pick: write-through, write-back, or cache-aside?

Cache-aside (lazy) — app reads cache, on miss loads DB and populates. Simple, most common; risk of staleness.
Write-through — write goes to cache and DB synchronously. Cache always fresh; slower writes.
Write-back (write-behind) — write cache, flush to DB async. Fast writes; risk of data loss if the cache dies before flush.

Choose by tolerance for staleness vs. write latency vs. durability.

36. After an outage, services recover then immediately fall over again. Why?

Retry storm / thundering herd — every client retries at once the instant the service comes back, spiking load far above normal and knocking it down again.

Fix: exponential backoff with jitter, a circuit breaker that reopens gradually, request deduplication, and load shedding / admission control so the recovering service isn't flooded.

37. Requests time out at the gateway even though the backend eventually responds. Why?

Timeout budget misconfiguration — the gateway timeout is shorter than the downstream chain, so the gateway gives up (and often retries) while work is still in flight, multiplying load.

Fix: set a timeout budget that decreases down the call chain (each hop's timeout < its caller's remaining budget), propagate deadlines, and don't retry non-idempotent calls.

38. Explain circuit breaker states and when each transitions.

Closed — calls pass through; failures are counted.
Open — failure threshold exceeded; calls fail fast (no downstream hit) for a cooldown.
Half-open — after cooldown, a few trial calls are allowed; success → Closed, failure → back to Open.

Purpose: stop hammering a failing dependency and give it room to recover, while failing fast for callers.

39. How do you size a database connection pool?

Bigger is not better — too many connections cause context-switching and lock contention at the DB. A common starting point: connections ≈ ((core_count × 2) + effective_spindle_count), then measure.

Key idea: the pool should be just large enough to keep the DB busy, not to mirror your thread count. The pool sits in front of a finite DB resource; the sum of pools across all app instances must stay under the DB's max_connections.

40. How do you run a schema migration with zero downtime?

Use the expand / contract (parallel change) pattern:

Expand — add the new column/table, backward-compatible (nullable, no rename, no drop).
Migrate — backfill data; deploy code that writes both old and new, reads old.
Switch — deploy code that reads new.
Contract — once nothing uses the old schema, drop it.

Never rename/drop in one step while old code is still running. Avoid long table locks (use online DDL / pt-online-schema-change / gh-ost for big tables).

41. Your webhook deliveries are sometimes missed by receivers. How do you make delivery reliable?

Treat it like a queue, not a fire-and-forget HTTP call:

Persist the event, deliver via a worker with retries + exponential backoff.
Expect the receiver to be down → keep retrying within a window, then DLQ.
Send an idempotency key / event ID and a signature so receivers can dedup and verify.
Provide at-least-once semantics and tell consumers to be idempotent; offer a replay/re-send endpoint.

42. Sessions break when you scale to multiple instances. Why, and how do you fix it?

In-memory sessions live on one instance; a later request hitting a different instance has no session.

Options: sticky sessions (LB pins a client to one instance — simple but breaks on instance loss and hurts balancing), or go stateless — store session in a shared store (Redis) or use a signed token (JWT). Stateless scales cleanly and survives instance failure.

43. A single counter (likes, inventory) becomes a write hotspot. How do you scale it?

Everyone updating one row serializes on that row's lock → contention.

Fix: sharded counters — split into N sub-counters (counter_0..counter_N), increment a random/hashed shard, sum on read. Or buffer increments and flush periodically, or use an approximate/probabilistic counter where exactness isn't required.

44. Two clients update the same record concurrently in an eventually-consistent store. How do you resolve the conflict?

Last-write-wins (LWW) — simple, but silently drops one update; needs reliable ordering (risky with clock skew).
Vector clocks — detect concurrent updates and surface a conflict for the app/user to merge.
CRDTs — data types that merge deterministically without coordination (counters, sets, registers).

Choose by whether losing an update is acceptable (LWW) or every update must survive (CRDT/merge).

45. A query returns correct results in one region but wrong/missing in another. What would you investigate?

Replication lag between regions (data not propagated yet).
Routing — reads hitting a regional replica that's behind or partitioned.
Data residency / partial replication — some data deliberately not replicated cross-region.
Cache divergence — per-region caches holding different states.
Time zone / UTC handling in date filters differing by region config.

Start with: is the row actually present in that region's DB, and how stale is that replica?

46. You write to the DB and publish an event, but the two sometimes disagree. Why?

The dual-write problem — two separate systems (DB + message broker) updated in one logical operation with no shared transaction. A crash between them leaves the DB updated but no event published (or vice versa).

Fix: the Transactional Outbox — write the state change and an outbox row in one local DB transaction, then a separate relay (CDC like Debezium, or a poller) publishes outbox rows to the broker at-least-once. Single source of truth, no lost events.

47. Traffic for keys that don't exist still hammers the database. Why?

Cache penetration — requests for non-existent keys always miss the cache and fall through to the DB every time (often malicious or buggy clients hitting random IDs).

Fix: cache the negative result (store a short-TTL null/sentinel for missing keys), and/or front the cache with a Bloom filter to reject keys that definitely don't exist before touching the DB.

48. A request fails somewhere across ten services. How do you find where?

Logs per service aren't enough — you can't stitch one request's path together.

Fix: distributed tracing. Propagate a correlation/trace ID (W3C traceparent) through every hop and log it everywhere; use OpenTelemetry → Jaeger/Zipkin to see the full span tree, per-hop latency, and exactly which service errored. Bake the trace ID into error responses so support can jump straight to the trace.

49. A fast producer overwhelms a slow consumer in a streaming pipeline. What do you do?

Without flow control the consumer's buffers grow unbounded → memory blowup or dropped data.

Fix: backpressure — the consumer signals demand and the producer slows to match (reactive streams / Flow, bounded queues that block or reject). Options when you can't slow the source: buffer (bounded), drop/sample (shed load), or spill to durable storage (a queue) and process at the consumer's pace.

50. Under extreme overload, the whole system collapses instead of degrading. How do you prevent that?

No admission control → every request is accepted, queues grow, latency explodes, everything times out at once (congestion collapse).

Fix: load shedding + graceful degradation. Reject or queue-limit excess requests early (return 429/503 fast), prioritize critical traffic, serve cached/stale or reduced responses, disable non-essential features under pressure, and apply backpressure upstream. Better to serve 80% well than 100% badly.