9.3 — Regression triage

Module 9 · Lesson 3 · 🟡 Intermediate · ~30 min

An on‑call runbook. When "LCP jumped 400 ms overnight" lands in your channel, work through this in order.

---

Step 0 — Confirm it's real (5 minutes)

Before investigating, rule out measurement artifacts. Roughly a third of reported regressions aren't regressions.

-- Did sample volume change? A traffic-mix shift moves p75 without anything getting slower.
SELECT DATE(ts) AS day,
       COUNT(*) AS samples,
       APPROX_QUANTILES(value, 100)[OFFSET(75)] AS p75
FROM vitals
WHERE name = 'LCP' AND page_type = 'pdp' AND device_class = 'mobile'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY 1 ORDER BY 1;

The four false alarms:

Pattern	Signal	Verdict
Sample count changed sharply	40% more samples	Traffic mix shift, not a regression
Only one segment moved	Desktop flat, mobile up	Real, but scoped — check device mix too
Device/connection mix shifted	More mobile-low this week	Population change (campaign, new market)
The reporter changed	A deploy touched the RUM code	Measurement bug

-- Population check: did WHO we're measuring change?
SELECT DATE(ts) AS day, device_class, connection,
       COUNT(*) / SUM(COUNT(*)) OVER (PARTITION BY DATE(ts)) AS share
FROM vitals WHERE name = 'LCP' AND page_type = 'pdp'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY 1, 2, 3 ORDER BY 1 DESC;

If the mix shifted (a marketing campaign brought in lower‑end devices), the metric is correct and the "regression" is a population change. Say so — it's still worth knowing, but it isn't a bug.

---

Step 1 — Scope it (10 minutes)

Narrow down what regressed before asking why.

-- Which page types?
SELECT page_type,
       APPROX_QUANTILES(IF(ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY), value, NULL), 100)[OFFSET(75)] AS p75_now,
       APPROX_QUANTILES(IF(ts <  TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY), value, NULL), 100)[OFFSET(75)] AS p75_before
FROM vitals WHERE name = 'LCP' AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY 1;

-- Which release? THIS is usually the answer.
SELECT release_sha, MIN(ts) AS first_seen, COUNT(*) AS samples,
       APPROX_QUANTILES(value, 100)[OFFSET(75)] AS p75
FROM vitals WHERE name = 'LCP' AND page_type = 'pdp' AND device_class = 'mobile'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 DAY)
GROUP BY 1 HAVING samples > 1000 ORDER BY first_seen;

release_sha   first_seen           samples   p75
a3f9c21       2026-03-10 09:14     412,003   2,140
b7e2d84       2026-03-11 14:32     388,221   2,180
c9a1f56       2026-03-12 11:08     401,887   2,610   ← here

If a specific release is the boundary, you're done scoping. Read that diff.

Then narrow further:

Question	Query dimension	What it tells you
Which page type?	page_type	Feature scope
Which device class?	device_class	CPU vs network
Which connection?	connection	Bandwidth vs CPU
Which country?	country	Infrastructure/regional
Which experiment arm?	experiments	An A/B test is causing it
First visit or returning?	is_returning	Cache‑related

-- Is it an experiment? This catches a surprising number of "mystery" regressions.
SELECT experiments, COUNT(*) AS n,
       APPROX_QUANTILES(value, 100)[OFFSET(75)] AS p75
FROM vitals WHERE name = 'LCP' AND page_type = 'pdp'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 2 DAY)
GROUP BY 1 HAVING n > 500 ORDER BY p75 DESC;

---

Step 2 — Use attribution (10 minutes)

The sub‑part breakdown tells you the mechanism without any code reading.

SELECT
  DATE(ts) AS day,
  AVG(CAST(JSON_VALUE(attribution, '$.ttfb') AS FLOAT64))        AS ttfb,
  AVG(CAST(JSON_VALUE(attribution, '$.loadDelay') AS FLOAT64))   AS load_delay,
  AVG(CAST(JSON_VALUE(attribution, '$.loadTime') AS FLOAT64))    AS load_time,
  AVG(CAST(JSON_VALUE(attribution, '$.renderDelay') AS FLOAT64)) AS render_delay
FROM vitals WHERE name = 'LCP' AND page_type = 'pdp' AND device_class = 'mobile'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
GROUP BY 1 ORDER BY 1;

day         ttfb   load_delay   load_time   render_delay
2026-03-08   180          90       1,240            120
2026-03-09   178          88       1,255            118
2026-03-12   182          94       1,720            121    ← load_time +38%

Load time jumped and everything else held. That means the LCP resource got bigger or slower — an image change, a sizes change, or a CDN problem. You now know which of four sections of the LCP playbook to read, and you haven't opened a code editor.

Also check whether the LCP element itself changed:

SELECT DATE(ts) AS day, JSON_VALUE(attribution, '$.element') AS element, COUNT(*) AS n
FROM vitals WHERE name = 'LCP' AND page_type = 'pdp'
  AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY 1, 2 ORDER BY 1 DESC, n DESC;

A changed LCP element is a strong signal — someone restructured the page, and the browser is now picking a different (later) element.

---

Step 3 — Find the change (15 minutes)

# What shipped between the last-good and first-bad releases?
git log --oneline b7e2d84..c9a1f56

# Filter to what plausibly matters
git log b7e2d84..c9a1f56 --stat -- \
  'app/**' 'components/**' 'next.config.ts' 'package.json' 'middleware.ts'

# Dependency changes are a common cause and easy to miss
git diff b7e2d84..c9a1f56 -- package-lock.json | grep -E '^\+.*"version"' | head -30

The usual suspects, in order of frequency:

Cause	How to spot it
A new dependency	package.json diff; bundle size jump
A new third‑party tag	Added outside the codebase — check the tag manager's change log
An image change	New hero, changed sizes, quality setting
A route became dynamic	Build output diff; TTFB sub‑part jumped
A 'use client' moved up the tree	Bundle size jump on many routes
A CDN/infra config change	Not in git — check your infra change log
An A/B test launched	experiments dimension
A vendor changed their script	Nothing in your repo changed at all

Not everything that changes is in your repository. Tag manager publishes, CDN config, DNS, vendor script updates, and CMS content changes are all invisible to git log. Keep a shared change log across all of them, or you'll spend a day reading a diff that contains nothing.

---

Step 4 — Reproduce (15 minutes)

# Build both versions and compare
git checkout b7e2d84 && npm ci && npm run build && npm start &
./scripts/perf-compare.sh http://localhost:3000/p/test-sku > /tmp/before.json

git checkout c9a1f56 && npm ci && npm run build && npm start &
./scripts/perf-compare.sh http://localhost:3000/p/test-sku > /tmp/after.json

diff /tmp/before.json /tmp/after.json

# Bundle comparison — often the answer on its own
git checkout b7e2d84 && npm run build && node scripts/bundle-budget.mjs > /tmp/bundle-before.txt
git checkout c9a1f56 && npm run build && node scripts/bundle-budget.mjs > /tmp/bundle-after.txt
diff /tmp/bundle-before.txt /tmp/bundle-after.txt

If the range is large, bisect:

git bisect start c9a1f56 b7e2d84
git bisect run bash -c '
  npm ci --prefer-offline --silent &&
  npm run build --silent &&
  node scripts/bundle-budget.mjs
'

git bisect run with a deterministic check (bundle size, route mode) is fast and reliable. With a noisy check (Lighthouse LCP) it will give you a wrong answer — use it only with deterministic gates.

---

Step 5 — Decide

Situation	Action
Clear cause, small fix	Fix forward, ship today
Clear cause, large fix	Revert the change, schedule the fix
Third‑party caused it	Disable the tag if you can, escalate to the vendor and the tag owner
Infra caused it	Roll back the infra change
No clear cause, impact large	Revert the whole release, investigate calmly
No clear cause, impact small	Open a ticket with everything you've learned; monitor

Bias toward reverting. A revert takes ten minutes; a root cause investigation under time pressure takes hours and produces worse fixes. Revert, then investigate.

---

The triage template

Fill this in as you go — it's the artifact that makes the post‑mortem quick.

## Perf regression: PDP mobile LCP +470ms

**Detected:** 2026-03-13 08:15 (alert: LCPRegression)
**Impact:** PDP mobile p75 LCP 2,140ms → 2,610ms (+22%). 31% of sessions.
            Estimated conversion impact: -0.3% to -0.9% relative (~$3-9M/yr annualized).

### Confirmation
- [x] Sample volume stable (± 4%)
- [x] Device/connection mix stable
- [x] Not an experiment (all arms affected equally)
- [x] Desktop also affected, smaller magnitude (+180ms)

### Scope
- Page types: PDP only. PLP and home unaffected.
- Devices: all, mobile worse
- Boundary release: c9a1f56 (deployed 2026-03-12 11:08)

### Attribution
LCP sub-parts: load_time 1,240ms → 1,720ms (+38%). Others flat.
→ The LCP resource got bigger or slower.

### Root cause
PR #4821 changed the PDP gallery to render at 2x resolution for "retina
sharpness", removing the responsive `sizes` attribute. Mobile now downloads
a 1,600px image for a 390px slot.

### Fix
Restore `sizes`, use `quality={72}`. PR #4839.

### Verification
- Lab: LCP 2,580ms → 2,090ms (median of 7)
- Field: monitor p75 for 48h post-deploy

### Prevention
- The `uses-responsive-images` Lighthouse assertion was set to `warn`.
  Promoting to `error`. (PR #4840)
- Adding a Playwright assertion that PDP images are ≤ 900px intrinsic on
  a 390px viewport.

---

Common regression causes, ranked

From experience across large commerce codebases:

#	Cause	Detection
1	New third‑party tag	Third‑party origin gate; TBT jump
2	New dependency / barrel import	Bundle budget gate
3	Route became dynamic	Route mode gate; TTFB jump
4	Image sizes/format regression	LCP load_time sub‑part; Lighthouse image audits
5	'use client' moved up the tree	Bundle gate; boundary ratchet
6	Cache config change (headers, keys)	CDN hit ratio drop; TTFB jump
7	A/B test with a heavy variant	experiments dimension
8	CMS content change (huge hero image)	LCP load_time; not in git
9	Backend service degradation	Server-Timing; TTFB
10	Vendor script update	Nothing changed in your repo

Note that 1, 6, 8, and 10 don't appear in your git history at all. That's why the change log across systems matters more than the code review.

---

Building the change log

# A shared timeline everyone writes to. It can be a channel, a spreadsheet,
# or annotations on your dashboards — but it must be one place.
2026-03-12 11:08  DEPLOY   c9a1f56  web  "PDP gallery retina update"     @pdp-squad
2026-03-12 14:20  TAG      GTM      "Added TikTok pixel"                 @growth
2026-03-12 16:45  CDN      fastly   "Enabled brotli on JSON"             @platform
2026-03-13 02:00  VENDOR   —        "Chat vendor auto-updated to 4.2"    (detected)
2026-03-13 09:00  CMS      —        "New homepage hero published"        @merch

Annotate your performance dashboards with these events. Half of triage is "what changed at 11:08?" and this answers it in five seconds instead of an hour of asking around.

---

Common mistakes

Mistake	Cost
Investigating before confirming the regression is real	Hours spent on a traffic‑mix shift
Not checking the population mix	Wrong conclusion entirely
Skipping attribution	Reading code instead of reading data
Only looking at your own git history	Misses third‑party, CDN, and CMS causes
Bisecting with a noisy metric	Wrong commit identified
Fixing forward under pressure	Slower and riskier than reverting
No post‑mortem prevention item	The same regression next quarter
No cross‑system change log	Triage takes 10× longer

---

Lab 9.3 — Practice on a real one

1. Find a past regression in your dashboard history — any visible step change.
2. Work the runbook retrospectively. How long would each step have taken with your current tooling?
3. Note where you got stuck. Missing releaseSha? No attribution? No change log? Those gaps are your action items.
4. Build the change log if you don't have one. Start with deploys and tag manager publishes.
5. Add dashboard annotations for deploys.
6. Write the triage template into your runbook repository.
7. Run a game day: deliberately ship a regression to staging (a removed sizes attribute) and have someone else triage it. Time it.

---

Checklist

• releaseSha in RUM, with dashboard annotations for deploys
• Attribution data available for all three vitals
• Cross‑system change log covering deploys, tags, CDN, CMS, and vendors
• Triage template in the runbook
• Bisect only with deterministic checks
• Revert is the default under time pressure
• Every regression produces a prevention item, usually a new gate
• Game day run at least once so the runbook is tested

Next: 9.4 Scaling the practice