Incident Response Process & Postmortem Template

Overview

This document outlines the incident escalation process and blameless postmortem template for production incidents on Learnille.

🚨 1. Incident Severity Levels

Severity	Definition	Response SLA	Target Resolution
SEV-1 (Critical)	Core service outage (e.g. payment gateway failing, database down, total API failure).	`< 15 mins`	`< 2 hours`
SEV-2 (Major)	Major feature degraded (e.g. video streaming buffering, notification delays).	`< 1 hour`	`< 6 hours`
SEV-3 (Minor)	Non-critical bug (e.g. cosmetic UI glitch, analytics logging drop).	Next business day	Next sprint release

📝 2. Blameless Postmortem Template

# Incident Postmortem: [INC-YYYY-MM-DD-ShortName]

## Executive Summary
- **Date & Time**: YYYY-MM-DD HH:MM UTC
- **Duration**: XX minutes
- **Severity**: SEV-1 / SEV-2
- **Impact**: XX% of users affected. XX failed course purchases / booking attempts.
- **Root Cause**: Short description of failure mechanism.

## Timeline (All times in UTC)
- **10:00** - Automated Sentry alert triggered for high error rate on `/payments/webhook`.
- **10:05** - On-call engineer acknowledged incident and began log investigation.
- **10:20** - Mitigating hotfix deployed to staging and verified.
- **10:35** - Hotfix deployed to production; error rate returned to normal.

## Root Cause Analysis (5 Whys)
1. **Why did the payment webhook fail?** Signature verification failed due to missing environment variable.
2. **Why was the env variable missing?** Deployment script overwrote environment configuration during deployment.

## Corrective Actions & Preventative Tasks
- [ ] Add CI environment variable validation check before deployment script execution.
- [ ] Add automated E2E test for webhook signature verification.