Skip to content
Learnille Documentation

Rollback Instructions

Rollback Instructions

Overview

This document provides procedures for rolling back deployments in the Learnille platform. Rollbacks may be necessary due to application failures, performance issues, or other critical problems.

Rollback Types

1. Automated Rollback

  • Trigger: Health check failures, error thresholds exceeded
  • Execution: Immediate, no manual intervention required
  • Duration: < 5 minutes
  • Scope: Application container rollback

2. Manual Rollback

  • Trigger: Detected issues requiring human judgment
  • Execution: Manual command execution
  • Duration: 10-30 minutes
  • Scope: Application, database, or infrastructure

3. Emergency Rollback

  • Trigger: Critical system failure, security breach
  • Execution: Immediate action by on-call engineer
  • Duration: < 15 minutes
  • Scope: Complete environment restoration

Pre-Rollback Checklist

Assessment

  • Identify the issue and impact
  • Determine rollback scope (application, database, infrastructure)
  • Check current system status and monitoring
  • Notify stakeholders if needed
  • Gather evidence (logs, metrics, user reports)

Preparation

  • Verify backup availability and integrity
  • Check previous version stability
  • Prepare rollback commands/scripts
  • Ensure access to deployment tools
  • Coordinate with team members if needed

Automated Rollback Procedures

Application Rollback (Kubernetes)

Immediate Rollback

Terminal window
# Check current deployment status
kubectl get deployments -n learnille-prod


# Rollback to previous revision
kubectl rollout undo deployment/learnille-api -n learnille-prod


# Monitor rollback progress
kubectl rollout status deployment/learnille-api -n learnille-prod

Blue-Green Rollback

Terminal window
# Check which environment is active
kubectl get service learnille-api -n learnille-prod -o yaml


# Switch traffic back to blue environment
kubectl patch service learnille-api -n learnille-prod -p '{"spec":{"selector":{"version":"blue"}}}'


# Verify traffic switch
kubectl get endpoints learnille-api -n learnille-prod

Database Rollback

Migration Rollback

Terminal window
# Check migration status
npm run migration:status -- --env production


# Rollback last migration
npm run migration:rollback -- --env production


# Verify database integrity
npm run migration:status -- --env production

Data Restore (if needed)

Terminal window
# Restore from backup (if data corruption)
pg_restore -h production-db -U learnille -d learnille_prod backup_file.sql


# Verify data integrity
psql -h production-db -U learnille -d learnille_prod -c "SELECT COUNT(*) FROM users;"

Manual Rollback Procedures

1. Application Rollback

Self-Hosted PM2 / Git Rollback

Terminal window
# SSH into self-hosted production server
ssh admin@server.learnille.dev


# Navigate to application root
cd /opt/learnille


# Checkout previous stable Git commit / tag
git checkout v1.2.0


# Rebuild and reload PM2 process
cd server && npm install && npm run build
pm2 reload learnille-api --update-env

—service learnille-api
—task-definition learnille-api:1.2.2
—force-new-deployment

Monitor deployment

aws ecs describe-services
—cluster learnille-prod
—services learnille-api
—query ‘services[0].deployments’

#### Using Docker Compose
```bash
# Stop current containers
docker-compose -f production.yml down


# Update image version in compose file
sed -i 's/learnille\/api:1.2.3/learnille\/api:1.2.2/' production.yml


# Start previous version
docker-compose -f production.yml up -d


# Verify containers
docker-compose -f production.yml ps

2. Infrastructure Rollback

Terraform Rollback

Terminal window
# Check current state
terraform show


# Rollback to previous state
terraform plan -state=terraform.tfstate.backup
terraform apply -state=terraform.tfstate.backup


# Verify infrastructure
terraform show

CloudFormation Rollback

Terminal window
# Check stack status
aws cloudformation describe-stacks --stack-name learnille-prod


# Rollback to previous version
aws cloudformation update-stack \
  --stack-name learnille-prod \
  --template-body file://template-v1.2.2.yaml \
  --parameters file://parameters.json


# Monitor stack update
aws cloudformation describe-stack-events --stack-name learnille-prod

3. Configuration Rollback

Environment Variables

Terminal window
# Backup current environment
cp .env.production .env.production.backup


# Restore previous environment
cp .env.production.v1.2.2 .env.production


# Restart application
kubectl rollout restart deployment/learnille-api -n learnille-prod

Configuration Files

Terminal window
# Backup current config
cp config/production.yaml config/production.yaml.backup


# Restore previous config
cp config/production-v1.2.2.yaml config/production.yaml


# Apply configuration
kubectl apply -f config/production.yaml

Emergency Rollback Procedures

Critical System Failure

  1. Immediate Assessment

    Terminal window
    # Check system status
    curl -f https://api.learnille.com/health || echo "API down"
    

    # Check database connectivity
    psql -h production-db -U learnille -d learnille_prod -c "SELECT 1;"
    

    # Check infrastructure status
    aws ecs describe-services --cluster learnille-prod --services learnille-api

  2. Execute Emergency Rollback

    Terminal window
    # Force rollback to last known good state
    kubectl set image deployment/learnille-api app=learnille/api:1.2.2 -n learnille-prod
    kubectl rollout status deployment/learnille-api -n learnille-prod
    

    # Switch to backup database if needed
    aws rds failover-db-cluster --db-cluster-identifier learnille-prod

  3. Traffic Management

    Terminal window
    # Enable maintenance mode
    kubectl patch configmap learnille-config -p '{"data":{"maintenance":"true"}}'
    

    # Switch to backup load balancer
    aws elbv2 set-ip-address-type \
      --load-balancer-arn $BACKUP_LB_ARN \
      --ip-address-type ipv4

Security Incident Response

  1. Isolate Affected Systems

    Terminal window
    # Block suspicious traffic
    aws waf update-ip-set \
      --name suspicious-ips \
      --scope REGIONAL \
      --id $IP_SET_ID \
      --addresses $SUSPICIOUS_IP
    

    # Disable compromised service
    kubectl scale deployment learnille-api --replicas=0 -n learnille-prod

  2. Data Recovery

    Terminal window
    # Restore from clean backup
    aws s3 cp s3://learnille-backups/clean-backup.sql /tmp/
    psql -h production-db -U learnille -d learnille_prod < /tmp/clean-backup.sql

  3. Security Validation

    Terminal window
    # Run security scan
    docker run --rm -v $(pwd):/src securecodebox/scanner
    

    # Update security groups
    aws ec2 revoke-security-group-ingress --group-id $SG_ID --protocol tcp --port 80 --cidr 0.0.0.0/0

Post-Rollback Procedures

Validation Steps

  • Application health checks passing
  • Database connectivity verified
  • User functionality tested
  • Performance metrics normal
  • Monitoring alerts cleared

Documentation

Terminal window
# Document rollback in incident log
echo "$(date): Rollback executed - $(whoami)" >> rollback-log.txt
echo "Reason: $ROLLBACK_REASON" >> rollback-log.txt
echo "Impact: $ROLLBACK_IMPACT" >> rollback-log.txt


# Update status page
curl -X POST https://status.learnille.com/api/incidents \
  -H "Authorization: Bearer $STATUS_API_KEY" \
  -d '{"status": "resolved", "message": "Issue resolved via rollback"}'

Communication

  • Notify internal team via Slack
  • Update external status page
  • Send customer communication if needed
  • Schedule post-mortem meeting

Investigation

  • Analyze root cause of deployment failure
  • Review monitoring data during incident
  • Update deployment processes if needed
  • Document lessons learned

Monitoring During Rollback

Key Metrics to Monitor

  • Application response time
  • Error rates by endpoint
  • Database connection pool usage
  • Infrastructure resource utilization
  • User session success rate

Alerting During Rollback

# Temporary rollback monitoring
groups:
  - name: rollback-monitoring
    rules:
      - alert: RollbackHealthCheck
        expr: up{job="learnille-api"} == 0
        for: 2m
        labels:
          severity: critical
          rollback: "true"
      - alert: RollbackErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
          rollback: "true"

Rollback Testing

Regular Testing

  • Monthly rollback drills
  • Automated rollback testing in staging
  • Documentation review and updates
  • Team training sessions

Test Scenarios

  • Application deployment failure
  • Database migration failure
  • Infrastructure configuration error
  • Security vulnerability discovery
  • Performance degradation

Tools and Resources

Rollback Scripts

emergency-rollback.sh
#!/bin/bash
set -e


echo "Starting emergency rollback..."


# Get last stable version
STABLE_VERSION=$(curl -s https://api.github.com/repos/learnille/learnille/releases/latest | jq -r .tag_name)


# Execute rollback
kubectl set image deployment/learnille-api app=learnille/api:$STABLE_VERSION
kubectl rollout status deployment/learnille-api


echo "Rollback completed successfully"

Monitoring Dashboard

  • Rollback status dashboard
  • Real-time metrics during rollback
  • Historical rollback data
  • Success/failure analytics

Contact Information

Emergency Contacts

  • Primary On-call: +1-555-0123 (DevOps Engineer)
  • Secondary On-call: +1-555-0124 (Senior Engineer)
  • Management: +1-555-0125 (Engineering Manager)

Communication Channels

Continuous Improvement

Metrics Tracking

  • Rollback frequency and success rate
  • Mean time to rollback
  • Customer impact assessment
  • Process efficiency improvements

Process Updates

  • Regular review of rollback procedures
  • Incorporation of lessons learned
  • Tool and automation improvements
  • Team training and preparedness