Debugging With Nothing But a Terminal

The first move in any remote incident — and the config trick that stops you retyping it.

SSH mastery isn't about the connection itself, it's ~/.ssh/config doing the memorizing for you — named hosts, jump-host tunneling via ProxyJump, and local port forwarding that makes a remote database answer on localhost. One config file replaces a dozen half-remembered command lines.

The system is slow and users are timing out — what do you check first, in what order?

A fixed first-60-seconds checklist — load/CPU, free -h, df -h, iostat, ss -tulpn, dmesg — turns "something's wrong" into a specific bottleneck before you've been on the box a full minute. The one gotcha worth knowing cold: low "free" memory with high "available" is Linux caching doing its job, not a leak.

Every serious one-liner you're about to use is just small tools wired together this way.

A ten-minute investigation compressed into cat access.log | grep 500 | awk '{print $1}' | sort | uniq -c | sort -nr | head -10 isn't one complex tool — it's five simple ones connected by pipes, each doing one job. Understanding stdin/stdout/stderr as three separate channels is what makes that chain composable instead of a fragile trick.

Every production incident eventually comes down to the same question — what does the log say?

grep reads input line by line and prints what matches — the power is entirely in pattern thinking, not flag memorization, and the regex you write here (grep -E) is the same regex Python, JavaScript, and Go use. It's composable by design: pipe into it to filter, pipe out of it to chain further, which is why it sits inside almost every real investigation pipeline rather than running alone.

The "survival syntax" that solves 80% of log-searching problems with six characters.

A working SRE doesn't need the whole regex spec — ., *, ^, $, [ ], and \ cover finding an IP in a log file or cleaning a host:port string down to just the host with sed. It's the same six characters doing the work whether you're driving grep, sed, awk, or a script.

Why a regex once took down Cloudflare's entire network for 27 minutes — and why that wasn't a bug.

Regex engines compile your pattern into a state machine, and which kind — a DFA (guaranteed linear time) or an NFA with backtracking (fast normally, exponential on the wrong input) — determines whether a pattern is just slow or a live outage waiting to happen; that's exactly what caused Cloudflare's 2019 WAF incident. It also proves a hard limit: no regex can correctly match balanced parentheses or arbitrary nesting, the actual reason "you can't parse HTML with regex" is true and not snobbery.

The response body just told you it failed. curl -v tells you why.

curl -v labels every line of the conversation — * for curl's own notes, > for what it sent, < for what came back — exposing the DNS lookup, TLS handshake, and headers that a plain response body hides entirely. -i and -w '%{http_code}' give lighter cuts of the same visibility when you don't need the whole transcript, perfect for scripts and health checks.

That JSON you just curled — here's what actually turns it into something jq can query.

Parsing is a two-phase pipeline — lexing turns raw text into tokens, then parsing builds a tree from them — the exact process running every time json.loads() or yaml.safe_load() succeeds, or a SyntaxError names the precise character where it gave up. The same grammar-driven structure explains why JSON, YAML, and Kubernetes manifests all reject malformed input the same way: you've violated the grammar, not just "the format."

500 lines of JSON, one error message buried somewhere inside. This is the tool built for exactly that.

jq is sed/awk/grep purpose-built for JSON — it walks the same tree structure the last step just explained, letting you filter and reshape an API response or log line without regexing against raw text. It ships as a single binary (apt, brew, choco, scoop all carry it), so there's no reason not to have it on every box you SSH into.

K8s manifests, Helm values, Ansible playbooks — all the same tree jq just showed you, in a different skin.

yaml.safe_load() turns a YAML file into the same Python dicts and lists that json.load() produces — one data model underneath two syntaxes. The one rule that isn't optional: always safe_load, never yaml.load(), since the unsafe version can deserialize arbitrary Python objects and execute code.

sed and awk don't understand indentation. yq does — because it's jq for YAML.

yq's syntax deliberately mirrors jq — yq '.metadata.name' pod.yaml reads a field, yq -i '.spec.replicas = 3' updates one in place — because it treats YAML as the structured tree from the last two steps, not text to pattern-match. That's what makes auditing a live Deployment for missing resource limits, or merging a base config with an environment overlay, a one-liner instead of a sed script waiting to break on the next re-indent.

You typed vi config.yaml and the arrow keys feel wrong. Four commands get you out alive.

Vim's one real trick is that it's modal — Normal mode for moving and deleting, Insert mode for typing, Command mode for saving and quitting — and confusing the two is the entire reason it feels broken to a first-timer. Survival mode is deliberately small: i to type, Esc to stop, :wq to save and quit, :q! to bail without saving — enough to fix one config and get out.

Your SSH connection just dropped mid-migration. Without a multiplexer, that process's state just went unknown.

tmux runs a server that keeps sessions alive independent of your terminal connection — tmux new -s work, detach with the Ctrl+b d prefix, and tmux attach picks the exact session back up, panes and all, even after your laptop's Wi-Fi drops. Sessions, windows, and panes nest (one incident session, one window per concern, one pane per running command) — the same structure a later step's multi-agent trick runs on top of.

Ctrl+r history search is the beginner move. fzf is a universal filter you can wire into anything.

The whole tool is one primitive — pipe any list in, filter it interactively, get a selection out — which is why it turns into a file opener, a process killer, or a git-branch switcher with nothing more than a one-line alias piping into fzf. It removes the need to memorize exact names (pod names, branch names, PIDs) since you're filtering a live list instead of typing one from memory.

One model's answer is an opinion, not a verdict. tmux is what lets you get a second one without breaking your flow.

Split a tmux session across two or three AI CLI panes running different models, work the incident in one, then paste its proposed fix or diagnosis into another and ask "what do you think of this?" — treating the panes as independent reviewers instead of one oracle. The technique is entirely tmux plumbing you already have from the last step: no new tool, just panes, copy-mode, and the discipline of asking twice before you ship a fix.

backup.sh, backup_v2.sh, backup_FINAL_USE_THIS_ONE.sh — the mess Git exists to end.

Git tracks every change as a Directed Acyclic Graph of commits, each pointing to its parent — a verifiable history that never loses data, unlike a folder full of hand-numbered file copies. That structure is also what makes safe experimentation possible: branch, try the change, roll back cleanly if it breaks something, without touching the version everyone else depends on.

Someone hands you a repo link. Clone? Where does the code even go?

A remote repository is the shared source of truth your local clone syncs against — git clone gets you a copy, git push sends your commits up, git pull brings the team's back down. This is also where code review and CI/CD actually plug in: a push is the trigger that starts both.

Two people touch the same 500-line YAML file on different branches. The workflow decides whether that's a routine merge or an afternoon.

The feature-branch workflow — branch, commit, rebase onto main before you push, open a PR, merge only after review — turns concurrent infrastructure changes into a routine review instead of a conflict fire drill. YAML conflicts need one extra step past a normal merge: re-validate with yq eval '.' config.yaml after resolving, since a broken indent won't necessarily show up as a Git conflict marker.

Opening a PR shouldn't mean leaving the terminal you've been living in for the last hour.

gh pr create, gh run watch, and gh pr merge --auto --squash move the entire open-review-merge loop into the shell — including watching CI finish in real time instead of refreshing a browser tab. gh api goes further, returning raw JSON you can pipe straight into the jq from a few steps back for anything the built-in commands don't cover.