Incident Response and On-Call Basics for a Small Team

Two of our earlier guides sit close to this topic without actually covering it. Our guide on production-grade software and technical debt covers the engineering practices that prevent incidents from happening in the first place — testing, code review, deployment discipline. Our guide on API and integration strategy covers the risk of depending on a third-party vendor whose own infrastructure might fail. Neither one covers what this guide covers: the actual operational process for what happens once something breaks in your own production system, regardless of whose fault it is or how well-tested your code was going in. Prevention and vendor risk are both real and worth taking seriously, but they are not incident response — incident response is what a team actually does in the minutes, hours, and days after an alert fires, and most small teams building their first product have never designed that process deliberately at all.

Beyond Prevention: What This Guide Covers

It is worth being specific about why this deserves its own guide rather than a section inside an existing one. A small team's first instinct, when something breaks, is usually improvisation: whoever notices first starts investigating, other people join in an ad hoc way, someone eventually fixes it, and then everyone moves on without writing anything down. This works, sometimes, for a small enough problem. It reliably fails for a large one — multiple people investigating the same thing without coordinating, nobody communicating status to anyone outside the room, no record afterward of what actually happened or why, and the same root cause quietly recurring months later because nobody ever formally closed the loop. The frameworks in this guide — largely developed at Google and PagerDuty, at a scale far larger than most startups will ever reach — scale down cleanly to a team of two or three engineers, and adopting even a lightweight version of them from the start avoids relearning these lessons the expensive way, during an actual incident, under real pressure.

The Incident Command System

Google's Site Reliability Engineering book is a real, freely available source worth reading directly rather than taking on faith — the full text, including Chapter 14 on managing incidents, is published at sre.google/sre-book/managing-incidents. The chapter explicitly borrows its structure from the Incident Command System used in emergency services, and defines four core roles for a software incident: the Incident Commander, who “holds the high-level state about the incident” and structures the response by assigning responsibilities; the Operations Lead, who works with the Incident Commander to apply operational tools to the actual problem; the Communications Lead, described as “the public face of the incident response task force,” responsible for periodic updates to stakeholders outside the immediate response; and the Planning Lead, who supports operations by handling longer-term concerns during a drawn-out incident, such as filing follow-up bugs and arranging handoffs as the incident stretches past a single shift.

The book is direct about why this separation of roles matters, independent of team size: “a clear separation of responsibilities allows individuals more autonomy than they might otherwise have,” and the Incident Commander's single most important responsibility, in the book's own words, is “to keep a living incident document” — a single, continuously updated record of what is known, what has been tried, and what is still uncertain, so that anyone joining partway through, or picking the incident up after a handoff, can get oriented without re-asking questions someone already answered. For a team of two or three people, this doesn't mean staffing four separate roles with four separate people — it means explicitly assigning who is playing which role for a given incident, even if one person plays two of them, so that decisions get made by someone specific rather than by whoever happens to speak up in the moment.

PagerDuty's real-world version of the same roles

PagerDuty runs its own internal incident response process according to a similar structure, and has published the actual process publicly under an open license at response.pagerduty.com, which is worth reading directly since it is a real company's actual internal process rather than a theoretical model. PagerDuty's Incident Commander role is described specifically as acting “as the single source of truth of what is currently happening and what is going to happen during a major incident,” and, notably, the IC “delegates all repair actions and is NOT a resolver” — a deliberate design choice that keeps the person coordinating the response from getting pulled into the technical weeds of actually fixing the problem, which is precisely the role confusion that causes coordination to break down during a real incident. PagerDuty also names a Deputy, who serves as a “hot standby” ready to take over the IC role if needed; a Scribe, who documents the incident timeline and captures important decisions as they're made; a Subject Matter Expert, who does the actual diagnosis and repair work and reports status using a structured “CAN” report (Condition, Actions, Needs); and separate Customer and Internal Liaison roles for handling external and internal communications respectively.

The two frameworks aren't identical, but the underlying agreement between them is the more important point for a small team: both separate who decides from who fixes from who communicates from who records, because collapsing all four into one overwhelmed person during a real incident is exactly how things get missed. A team of three engineers doesn't need seven distinct named roles — it needs an explicit, agreed-upon answer, decided before an incident happens rather than during one, to a much simpler question: when something breaks at 2 a.m., who is actually in charge of the response, and who is writing down what's happening as it happens?

Severity Levels: SEV1 Through SEV4

GitLab is a useful, real, publicly documented example to work from directly, since GitLab publishes its actual incident severity definitions in its public documentation rather than keeping them internal (GitLab Docs, Incident Management). GitLab defines five labels: S1 (Critical), S2 (High), S3 (Medium), S4 (Low), and Unknown for incidents whose severity hasn't yet been assessed. A concrete example GitLab has used publicly to illustrate an S1 event, drawn from its own public issue tracker, is a deployment that breaks an important workload — such as CI pipelines failing for every user on the platform — which is a useful, concrete anchor point rather than an abstract definition: an S1 isn't “something feels seriously wrong,” it's “a core piece of functionality has stopped working for everyone.”

Adapting severity levels for a team that doesn't need five tiers

A five-tier scale built for a company running thousands of services is more granularity than a small team running one or two products typically needs, and the more useful exercise is adapting the underlying logic rather than copying the exact label count. The core distinction worth preserving from GitLab's framework is the gap between “this affects everyone, right now, and needs an immediate response regardless of the hour” and everything else. A workable three-tier version for a small team: a Critical tier for anything that takes the product down entirely or affects billing/data integrity for all users, which justifies waking someone up outside working hours; a High tier for a significant but partial problem — a single feature broken, a subset of users affected — that needs same-day attention but not necessarily an immediate 2 a.m. response; and a Low tier for everything else, handled during normal working hours through the regular bug-fixing process rather than through the incident process at all. The specific number of tiers matters less than having an explicit, agreed-upon line between “wake someone up for this” and “this can wait until morning” decided in advance, rather than argued about in the moment by whoever is on call.

Alerting Philosophy: Actionable vs. Noisy

Rob Ewaschuk, a former Google site reliability engineer, wrote “My Philosophy on Alerting” as an internal Google document that was later made publicly available and is still directly readable today (Rob Ewaschuk, “My Philosophy on Alerting”). The document's central claims are worth quoting directly rather than paraphrasing, since the phrasing itself is part of why the document became so widely referenced: “every time the pager goes off, I should be able to react with a sense of urgency,” “every page should be actionable,” and “every page response should require intelligence — if a page merely merits a robotic response, it shouldn't be a page.” Ewaschuk's framing directly names the failure mode this guides against: pages should be about a novel problem, not a recurring, already-understood condition that a human has already learned to ignore — which is precisely the mechanism behind alert fatigue, covered in the burnout data below.

Google's SRE book formalizes a closely related distinction in its own chapter on monitoring distributed systems, published at sre.google/sre-book/monitoring-distributed-systems: symptom-based alerting versus cause-based alerting. The book asks, of any candidate alerting rule, “does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?” and argues that “it's better to spend much more effort on catching symptoms than causes; when it comes to causes, only worry about very definite, very imminent causes.” In practice, this means alerting on “users are experiencing errors right now” rather than on every internal condition that might theoretically contribute to an error eventually — a distinction that matters enormously for a small team, since a team of two or three engineers cannot sustainably absorb a high volume of low-value, cause-based alerts without burning out the person on call.

What this means in practice for a first alerting setup

For a small team setting up alerting for the first time, the practical translation of this philosophy is to start with a genuinely small number of alerts tied directly to user-visible symptoms — the product is down, requests are failing at an elevated rate, a critical background job hasn't completed — and resist the temptation to add an alert for every metric that's technically possible to monitor. Every additional alert added to an on-call rotation is a real, ongoing cost paid by whoever is holding the pager, not a free safety margin, and a team that starts with ten carefully chosen, genuinely actionable alerts is in a materially better position than a team that starts with fifty and spends the next year discovering, incident by incident, which ones nobody actually needed to see at 3 a.m.

On-Call Burnout: What the Data Shows

This data comes from a real, named, dated source: PagerDuty's “State of Digital Operations 2022” report, published directly by PagerDuty at pagerduty.com/state-of-digital-ops-2022. It is worth being precise about the survey's own methodology rather than treating the figures as a universal industry statistic: PagerDuty surveyed its own customer base specifically, drawing 293 responses from roughly 5,849 invited “Power Users,” a response rate the report itself frames as falling within a normal range for this kind of survey. This means the population is PagerDuty's own existing customers — teams already using incident-management tooling, which may skew toward organizations with more mature (and therefore more heavily used) incident processes rather than a fully representative cross-section of the entire technology industry. That caveat doesn't undermine the finding so much as sharpen what it actually shows: even among teams that have already invested in incident-management tooling, more than half report being interrupted outside working hours, and the trend between 2020 and 2021 was toward more hours worked, not fewer.

It is also worth naming, directly, a set of related burnout statistics that circulate widely in secondary coverage of this topic but that this guide could not independently verify against a primary source with confidence: specific claims about companies risking the loss of “1 in 4” IT professionals to poor work-life balance, or specific figures from PagerDuty's earlier 2020 “Digital Pressures” report, appear across secondary sources without a directly confirmed exact figure and methodology in this research pass. Rather than repeat an unverified number, this guide relies on the 2022 report's figures above, which were confirmed directly against PagerDuty's own published report.

Structuring On-Call for a Small Team

A useful, real source for thinking about on-call specifically at a smaller scale, rather than at Google or PagerDuty's own scale, is Increment magazine's dedicated “On-Call” issue, published by Stripe and archived at increment.com/on-call, which surveyed more than thirty industry practitioners from companies including Amazon, Dropbox, Facebook, Google, and Netflix specifically about incident response and on-call practices. The issue is worth reading directly for the range of perspectives it collects, since the practical answer to “how should on-call actually work” varies meaningfully by team size and product type even among practitioners who all agree on the same underlying principles.

Bringing the frameworks above together into concrete guidance for a team of two to five engineers: with a team this size, a weekly rotation is usually more sustainable than a daily one, since daily rotations multiply the number of handoffs (and handoff gaps, where knowledge doesn't transfer cleanly) without meaningfully reducing any individual engineer's total burden across a month. A single person should rarely be on call more than one week in three or four at this team size, both because sustained on-call duty is a documented burnout risk per the data above, and because a rotation with too few people in it means the same person is perpetually context-switching between building the product and being ready to drop everything for it. Every handoff between rotations should include a short, explicit conversation — not just a calendar event firing silently — covering any open issues, anything unusual currently happening in production, and confirmation that the incoming engineer actually has working access to whatever tools and runbooks they might need at 2 a.m., not discovering a missing permission in the middle of an actual incident.

The specific mistake of treating on-call as unpaid, invisible labor

One structural mistake worth naming directly, because it is common enough at small companies specifically: treating on-call responsibility as an unstated, uncompensated expectation rather than an explicit part of a role, with explicit boundaries. This research could not find a verifiable, dated, named source for a specific industry-average on-call compensation figure, so this guide does not cite one — but the underlying principle doesn't require a specific number to be actionable: whatever a team decides is fair, it should be decided explicitly and communicated clearly, rather than left as an ambiguous, unacknowledged expectation that quietly erodes goodwill on a small team where every person's ongoing buy-in matters disproportionately.

Runbooks: reducing reliance on any one person's memory

A closely related practice worth adopting early, alongside the rotation structure itself, is the runbook — a written, specific set of steps for handling a known, recurring type of incident, kept somewhere the entire on-call rotation can find and follow it without needing to interrupt whoever originally built the system. This connects directly to the bus-factor concept our technical due diligence guide covers in a different context: a system that only one engineer can operate under pressure, because the steps for recovering it live only in that engineer's head, is a bus-factor-of-one risk wearing an on-call disguise. A small team doesn't need a runbook for every conceivable failure on day one, but it is worth writing one the first time any incident takes meaningfully longer to resolve than it should have because the person on call that week wasn't the person who originally built the system in question. Treating each real incident as a prompt to write the runbook that would have made the next one faster is a cheap, compounding habit — each one takes minutes to write immediately after the incident, while the details are fresh, and saves far more than that the next time the same failure mode recurs, whoever happens to be holding the pager when it does.

A runbook doesn't need to be exhaustive to be useful; a short, specific list of the exact commands or dashboard checks that resolved a past incident, plus a note on what the underlying cause actually was, is enough to turn a repeat incident from a 45-minute investigation into a 5-minute fix. What matters most is that it actually gets used and kept current — a runbook nobody has looked at in a year, describing a system that has since been rebuilt, is arguably worse than no runbook at all, since it gives the person on call false confidence in a procedure that no longer matches reality. Reviewing and updating runbooks should be a standing part of whatever cadence a team already uses to check in on its own engineering practices, not a one-time artifact created once and never revisited.

Blameless Postmortems

Allspaw's underlying argument, as consistently described across independent secondary sources discussing the post, is that engineers involved in an incident acted with good intentions given the information genuinely available to them at the time — which means a useful postmortem asks what about the system allowed a reasonable, well-intentioned action to still produce a bad outcome, rather than asking who to blame for it. This reframing has a direct, practical consequence for how honestly a team can actually investigate its own failures: an engineer who fears personal blame for an incident has a real incentive to omit details, minimize their own role, or avoid flagging a near-miss before it becomes a real incident, while an engineer operating inside a genuinely blameless culture has every incentive to describe exactly what happened, including their own mistakes, because doing so is understood as helping fix the system rather than risking punishment.

Google's SRE book devotes its own chapter to postmortem culture, and is generally understood to have helped formalize and popularize the practice industry-wide roughly four years after Allspaw's original post — which is itself a useful data point for a small team: this isn't a fringe practice a single company happened to adopt, it is a documented pattern that spread from one real, named practitioner's specific experience to become standard guidance at one of the largest technology companies in the world, precisely because teams that adopted it found real, ongoing value in it.

1. 1

   Write the postmortem soon after the incident, while details are fresh

   Waiting even a few days lets details blur — timelines, exact error messages, the order decisions were actually made in — that matter for an accurate account.

2. 2

   Describe the timeline before drawing any conclusions

   Establish what actually happened, in order, before jumping to root cause — a rushed jump to "why" tends to anchor on the first plausible explanation rather than the real one.

3. 3

   Focus contributing factors on systems and process, not individuals

   Ask what about the alerting, the deployment process, or the documentation allowed a reasonable action to produce a bad outcome — not who made the mistake.

4. 4

   Assign concrete follow-up actions with owners and dates

   A postmortem that ends with "we should be more careful" produces no actual change. Specific, owned, dated action items are what turn a review into a real fix.

A Real Incident: GitLab, January 2017

This incident is unusually well-documented for a real production failure, because GitLab chose to handle it with an unusual degree of public transparency: publishing a detailed postmortem on its own engineering blog (“Postmortem of database outage of January 31,” published February 10, 2017) and live-streaming part of the actual recovery effort. The broad shape of the incident is independently corroborated by contemporaneous press coverage — TechCrunch reported on it directly the day after it happened (“GitLab suffers major backup failure after data deletion incident,” February 1, 2017) — and the key facts are consistent across GitLab's own account and independent reporting: roughly 300GB of production data was affected, the data-loss window was roughly six hours because the most recent good snapshot was that old, and the incident affected approximately 5,000 projects, 5,000 comments, and 700 new user accounts. Recovery took roughly 18 hours in total.

The detail worth sitting with longest is the backup failure, not the initial mistake. A single engineer running the wrong command against the wrong server is precisely the kind of human error no process can fully eliminate — people make mistakes under pressure, and a resilient incident response system has to assume that will keep happening rather than trying to design it away entirely. What turned a single mistaken command into an 18-hour, 300GB-scale incident was that the safety net meant to catch exactly this kind of error — five separate backup and replication mechanisms — had, by GitLab's own account, quietly stopped working well before the incident, and nobody had discovered that until it was needed. This is the single clearest, most concrete argument in this entire guide for testing your recovery process before you need it, not after: a backup you have never actually restored from is not a verified backup, it is an assumption.

“Five of the six backup and replication mechanisms GitLab believed it had in place had failed or were not actually running at the time of the incident.”

— Summarized from GitLab's own account of the January 31, 2017 database outage, corroborated by contemporaneous press coverage

GitLab's decision to publish this level of detail publicly, and to livestream its own recovery, is itself a real, direct application of the blameless-postmortem principle covered above — the company chose transparency over minimizing its own exposure, and the incident is now widely cited across the industry, years later, as a useful teaching example precisely because of that openness rather than in spite of it. A small team without GitLab's public profile has no obligation to publish its own postmortems externally, but the same internal principle applies at any scale: an incident that gets documented honestly and specifically becomes a genuine asset the team can learn from; one that gets smoothed over or under-examined becomes a recurring risk waiting for the next opportunity to happen again.

Building Your First Process

Bringing everything above together into an actual process a small team can put in place before its first real incident, rather than during one:

One further, practical point worth naming directly: none of this needs to be built before it's needed in its most complete form. A team shipping its first product doesn't need a formal five-tier severity scale, a dedicated Scribe role, and a polished internal postmortem template on day one — it needs an explicit, if simple, answer to the single most important question this guide has covered: when something breaks, who is in charge, and what happens afterward so it doesn't just happen again. Everything else — more tiers, more formal roles, a more polished process — can be added later, as the team and the product actually grow into needing it, in the same way GitLab's own five-tier system and dedicated incident tooling grew out of years of real incidents rather than existing from the company's first day.

Frequently Asked Questions

None of the frameworks in this guide were designed for a two-person startup — they were built at Google, at PagerDuty, and refined publicly by GitLab, all operating at a scale most companies reading this guide will never reach. That scale gap is precisely why they're worth borrowing rather than reinventing: these are lessons a small team would otherwise have to relearn the expensive way, incident by incident, over years. The version that matters for a small team isn't the five-tier severity scale or the seven-role incident command structure — it's the underlying discipline those things encode: decide who's in charge before you need to know, alert on what actually matters, and write down what really happened afterward, honestly enough that it actually helps the next time.