Every entry in this journal started the same way: a terminal window, a problem that didn’t have a clean answer yet, and a habit of writing down what actually worked instead of trusting memory to hold onto it. What follows is a running log of the tools, mistakes, and small discoveries that came out of that habit.
Where the log started
The first entries were rough — half command history, half diary. Over time the format settled into something more useful: a short reproduction of the problem, the commands that solved it, and a note on why the obvious first attempt didn’t work.
Keep a notes.md next to any long-running project. Future-you will not remember why a flag was necessary six months from now.
Most entries follow the same shape: something breaks, strace or a stray log line points at the real cause, and a two-line fix replaces twenty minutes of guessing.
A journal isn’t useful because it’s well written. It’s useful because it’s there when the same bug shows up again.
Setting up the workflow
Every story in this log gets written the same way: from inside a terminal, using a small set of tools that stayed consistent across two years of entries.
Debugging the Real World
A few habits that made the log worth keeping:
- Timestamp every entry, even rough ones — order matters more than polish
- Paste the exact command that worked, not a paraphrased version
- Link related entries instead of repeating the same explanation
- Tag entries by tool (
docker,nginx,git) so they’re searchable later
The format evolved through a few clear stages:
- Plain text file, appended to whenever something broke
- Markdown files split by month, still mostly unsearchable
- Static site generated from the notes, one entry per problem
- Tags and cross-links between related debugging sessions
None of this needed special tooling to start. A single text file was enough for the first year.
A debugging session, start to finish
One of the more useful entries came from a deploy script that silently failed on one server but not another. The fix was small once the log was captured properly:
#!/usr/bin/env bash
set -euo pipefail
echo "Checking service health..."
if ! curl -fsS "http://localhost:8080/health" > /dev/null; then
echo "Health check failed, dumping recent logs:" >&2
journalctl -u deploy.service -n 50 --no-pager
exit 1
fi
echo "Deploy verified."
set -euo pipefail turned a silent failure into a loud one. Without it, the script kept going after the health check failed and reported success anyway.
The tools used across these sessions differ a lot in how much they assume about the reader:
| Tool | Best for | Notes |
|---|---|---|
| journalctl | Service logs on systemd hosts | Filters by unit, time, and priority |
| strace | Watching syscalls in real time | Verbose, but finds permission issues fast |
| tcpdump | Diagnosing network-level failures | Steep learning curve, invaluable remotely |
| tmux | Keeping sessions alive over SSH | Turns a dropped connection into a non-event |
What the screenshots don’t show
Every entry in this log has a tidy code block, but the actual terminal at the time usually had six panes open, a tail -f running somewhere, and at least one command re-run three times before it worked.
Gallery
Screenshots in a debugging log age fast. Redact hostnames and tokens before saving one — it’s easy to forget what’s visible in a terminal prompt.
Why the log kept going
Most personal documentation projects die within a month. This one lasted because each entry paid for itself the first time it got reused — a fix found in thirty seconds instead of thirty minutes1, a command remembered instead of relearned.
A debugging log is only useful if it’s honest about what didn’t work, not just the final fix. Skipping the dead ends makes the next entry harder to trust.
If there’s one thing worth taking from this project, it’s that the format doesn’t matter nearly as much as the habit of writing the entry down the same day the problem got solved2.
