# How do you trace root causes in interleaved production logs? > Learn how to separate the first loud ERROR symptom from the earlier root-cause event when reading interleaved async production logs. - Canonical HTML: [https://unrust.dev/guides/reading-production-incident-logs](https://unrust.dev/guides/reading-production-incident-logs) - Markdown representation: [https://unrust.dev/guides/reading-production-incident-logs/index.md](https://unrust.dev/guides/reading-production-incident-logs/index.md) Published: 2026-10-22 Published by Unrust. Scoring and editorial method: https://unrust.dev/help#methodology ## Direct answer To trace root causes in interleaved production logs, group entries by request or correlation ID first, then scan backward from the first visible ERROR to find the earliest state transition, timeout, or fallback event that broke an invariant—never assume the loudest stack trace is where the fault started. ## Why the first ERROR line is usually a downstream symptom In asynchronous services, a quiet warning or state transition—such as a lock lease expiring early, a retry starting before a rollback completes, or a stale cache write—often happens hundreds of milliseconds before a downstream consumer throws a loud exception. If you stop at the first red ERROR line in a log viewer, you end up patching the victim function while leaving the earlier race condition or state corruption untouched. - [Try the interactive async log timeline explorer](https://unrust.dev/tools/async-log-timeline-explorer/index.md) - [Read the worked walkthrough on symptom vs root cause in logs](https://unrust.dev/blog/trace-first-symptom-vs-root-cause-in-logs/index.md) ## Group by correlation ID, then check shared resource overlap Raw log streams interleave unrelated requests, background workers, and health checks. Start by filtering or grouping lines by requestId, jobId, or traceId so you can read a single execution timeline in timestamp order. Next, look for where two timelines touch the same shared resource—such as the same orderId, account row, or lease key—and compare the timestamps of acquire, expire, retry, and commit events. ## Verify causal order with timestamps and state preconditions For every error message, ask what precondition had to become false for that error to be reachable, then scan earlier log lines for the event that invalidated that precondition. In Unrust's log-timeline workspace drills and public browser explorer, you practice reading structured log streams alongside handler code to distinguish the first symptom from the true root-cause event. ## An incident log root-cause decision table Use this table when inspecting interleaved application and worker log lines during an incident review. ### Multiple concurrent requests and background jobs appear interleaved in the log stream - **Review decision:** Partition log lines by correlation ID (reqId / jobId) and shared entity key (orderId / userId) - **Evidence to verify:** You can separate unrelated background noise from the specific request or entity timeline that failed ### A worker or handler logs a fatal ERROR on commit, write, or status update - **Review decision:** Scan earlier WARN or INFO events on the same entity for lease expiry, timeout, or duplicate retry dispatch - **Evidence to verify:** You identify the earlier timestamp where two actors overlapped or a precondition was lost ### A retry attempt logs a duplicate-key or invalid-state failure - **Review decision:** Compare the timeout of the caller against the completion timestamp of the first attempt - **Evidence to verify:** Verify whether the first attempt continued running and mutated state after the caller timed out and retried ## Try the log trace: symptom line vs root-cause line You inspect four log lines for order_42: 10:00:00.100 [job=A] INFO acquired lock order_42 (ttl=500ms) 10:00:00.650 [job=A] WARN lock lease expired order_42 while awaiting payment gateway 10:00:00.700 [job=B] INFO acquired lock order_42 and committed status=PAID 10:00:00.820 [job=A] ERROR optimistic-lock conflict writing status=PAID for order_42 Which line marks the root-cause transition that made the conflict possible? - 10:00:00.820 [job=A] ERROR optimistic-lock conflict — because it is the only ERROR line. - 10:00:00.650 [job=A] WARN lock lease expired while job=A continued running past its 500ms TTL without aborting. - 10:00:00.100 [job=A] INFO acquired lock — because acquiring locks causes conflicts. - The database optimistic-lock check is broken and should be removed. **Reasoning:** The ERROR line shows where the safety check fired, while the earlier WARN line at 10:00:00.650 shows the invariant breach (continuing work after lease expiry without fencing) that allowed concurrent execution. ## Practice loop 1. Filter the log stream to the correlation IDs and entity keys involved in the failure before reading stack traces. 2. Find the first ERROR line, state what invariant was violated, and scan upward for the earliest timestamp where that invariant drifted. 3. Practice isolating root-cause events from downstream symptoms in the free Async Log Timeline Explorer at /tools/async-log-timeline-explorer. ## Common questions ### Can I practice log-timeline root-cause tracing on Unrust without signing in? Yes. You can use the interactive Async Log Timeline Explorer at /tools/async-log-timeline-explorer and read the worked walkthrough at /blog/trace-first-symptom-vs-root-cause-in-logs right in your browser. ### How do log drills fit into Unrust's practice loop? Alongside single-snippet drills, Unrust's practice workspace includes log-timeline drills that pair structured production-style logs with service code so you can practice isolating the exact event and code path where state diverged. ## Limitations Log-timeline drills teach structured causal reasoning on self-contained event streams; real production incidents may also involve missing logs, clock skew across hosts, or infrastructure metrics outside application logs. ## Provenance Original editorial guide by the Unrust team synthesizing distributed-systems log analysis and incident postmortem practices. ## Sources - [Site Reliability Engineering: Effective Troubleshooting](https://sre.google/sre-book/effective-troubleshooting/) — Google SRE Book / O'Reilly. Explains how to form falsifiable hypotheses from telemetry and distinguish cascading symptoms from triggering faults. - [OpenTelemetry Logs Data Model and Context Correlation](https://opentelemetry.io/docs/specs/otel/logs/data-model/) — OpenTelemetry Specification. Defines how trace, span, and correlation identifiers connect interleaved log records across asynchronous execution paths. ## Related guides - [How do you debug code without AI?](https://unrust.dev/guides/debugging-without-ai/index.md): A practical debugging loop for developers who want to reason through a failure before asking an AI assistant or search engine for a fix. - [How do you trace a bug across multiple files without an IDE?](https://unrust.dev/guides/debugging-across-multiple-files/index.md): Learn a structured method for tracing contracts, data shapes, and state mutations across module boundaries when a bug spans multiple files. - [How do you review retry logic safely?](https://unrust.dev/guides/reviewing-retry-logic/index.md): Review retry logic by checking retryable failures, idempotency, attempt budgets, backoff, jitter, and the outcome a caller sees when retries stop. Next step: [Take the free diagnostic](https://unrust.dev/diagnostic/index.md)