incident-commander

CategoryData
AuthorAlireza Rezvani
LicenseMIT
Rating4.70/5
Uses11.7K

Incident Commander Skill

Category: Engineering Team
Tier: POWERFUL
Author: Claude Skills Team
Version: 1.0.0
Last Updated: February 2026

Overview

Incident response framework for availability/reliability incidents (outages, degradations, failed deploys): severity classification, timeline reconstruction, and post-incident review.

This is NOT security incident triage. For security events (ransomware, intrusion, data exfiltration, IOC analysis, NIST SP 800-61 forensics), route to incident-response. Both skills use SEV1-SEV4 labels; this one scores operational impact (users, revenue, SLA), while incident-response classifies attack types and forensic handling.

Key Features

  • Automated Severity Classification - Intelligent incident triage based on impact and urgency metrics
  • Timeline Reconstruction - Transform scattered logs and events into coherent incident narratives
  • Post-Incident Review Generation - Structured PIRs with multiple RCA frameworks
  • Communication Templates - Pre-built templates for stakeholder updates and escalations
  • Runbook Integration - Generate actionable runbooks from incident patterns

Skills Included

Core Tools

1. Incident Classifier (incident_classifier.py)
- Analyzes incident descriptions and outputs severity levels
- Recommends response teams and initial actions
- Generates communication templates based on severity

2. Timeline Reconstructor (timeline_reconstructor.py)
- Processes timestamped events from multiple sources
- Reconstructs chronological incident timeline
- Identifies gaps and provides duration analysis

3. PIR Generator (pir_generator.py)
- Creates comprehensive Post-Incident Review documents
- Applies multiple RCA frameworks (5 Whys, Fishbone, Timeline)
- Generates actionable follow-up items

Incident Response Framework

Severity Classification System

#### SEV1 - Critical Outage
Definition: Complete service failure affecting all users or critical business functions

Characteristics:

  • Customer-facing services completely unavailable

  • Data loss or corruption affecting users

  • Security breaches with customer data exposure

  • Revenue-generating systems down

  • SLA violations with financial penalties

Response Requirements:

  • Immediate escalation to on-call engineer

  • Incident Commander assigned within 5 minutes

  • Executive notification within 15 minutes

  • Public status page update within 15 minutes

  • War room established

  • All hands on deck if needed

Communication Frequency: Every 15 minutes until resolution

#### SEV2 - Major Impact
Definition: Significant degradation affecting subset of users or non-critical functions

Characteristics:

  • Partial service degradation (>25% of users affected)

  • Performance issues causing user frustration

  • Non-critical features unavailable

  • Internal tools impacting productivity

  • Data inconsistencies not affecting user experience

Response Requirements:

  • On-call engineer response within 15 minutes

  • Incident Commander assigned within 30 minutes

  • Status page update within 30 minutes

  • Stakeholder notification within 1 hour

  • Regular team updates

Communication Frequency: Every 30 minutes during active response

#### SEV3 - Minor Impact
Definition: Limited impact with workarounds available

Characteristics:

  • Single feature or component affected

  • <25% of users impacted

  • Workarounds available

  • Performance degradation not significantly impacting UX

  • Non-urgent monitoring alerts

Response Requirements:

  • Response within 2 hours during business hours

  • Next business day response acceptable outside hours

  • Internal team notification

  • Optional status page update

Communication Frequency: At key milestones only

#### SEV4 - Low Impact
Definition: Minimal impact, cosmetic issues, or planned maintenance

Characteristics:

  • Cosmetic bugs

  • Documentation issues

  • Logging or monitoring gaps

  • Performance issues with no user impact

  • Development/test environment issues

Response Requirements:

  • Response within 1-2 business days

  • Standard ticket/issue tracking

  • No special escalation required

Communication Frequency: Standard development cycle updates

Incident Commander Role

#### Primary Responsibilities

1. Command and Control
- Own the incident response process
- Make critical decisions about resource allocation
- Coordinate between technical teams and stakeholders
- Maintain situational awareness across all response streams

2. Communication Hub
- Provide regular updates to stakeholders
- Manage external communications (status pages, customer notifications)
- Facilitate effective communication between response teams
- Shield responders from external distractions

3. Process Management
- Ensure proper incident tracking and documentation
- Drive toward resolution while maintaining quality
- Coordinate handoffs between team members
- Plan and execute rollback strategies if needed

4. Post-Incident Leadership
- Ensure thorough post-incident reviews are conducted
- Drive implementation of preventive measures
- Share learnings with broader organization

#### Decision-Making Framework

Emergency Decisions (SEV1/2):

  • Incident Commander has full authority

  • Bias toward action over analysis

  • Document decisions for later review

  • Consult subject matter experts but don't get blocked

Resource Allocation:

  • Can pull in any necessary team members

  • Authority to escalate to senior leadership

  • Can approve emergency spend for external resources

  • Make call on communication channels and timing

Technical Decisions:

  • Lean on technical leads for implementation details

  • Make final calls on trade-offs between speed and risk

  • Approve rollback vs. fix-forward strategies

  • Coordinate testing and validation approaches

Communication Templates

#### Initial Incident Notification (SEV1/2)

code
Subject: [SEV{severity}] {Service Name} - {Brief Description}

Incident Details:
  • Start Time: {timestamp}

  • Severity: SEV{level}

  • Impact: {user impact description}

  • Current Status: {investigating/mitigating/resolved}


Technical Details:
  • Affected Services: {service list}

  • Symptoms: {what users are experiencing}

  • Initial Assessment: {suspected root cause if known}


Response Team:
  • Incident Commander: {name}

  • Technical Lead: {name}

  • SMEs Engaged: {list}


Next Update: {timestamp}
Status Page: {link}
War Room: {bridge/chat link}


---
{Incident Commander Name}
{Contact Information}

#### Executive Summary (SEV1)

code
Subject: URGENT - Customer-Impacting Outage - {Service Name}

Executive Summary:
{2-3 sentence description of customer impact and business implications}


Key Metrics:
  • Time to Detection: {X minutes}

  • Time to Engagement: {X minutes} 

  • Estimated Customer Impact: {number/percentage}

  • Current Status: {status}

  • ETA to Resolution: {time or "investigating"}


Leadership Actions Required:
  • [ ] Customer communication approval

  • [ ] PR/Communications coordination  

  • [ ] Resource allocation decisions

  • [ ] External vendor engagement


Incident Commander: {name} ({contact})
Next Update: {time}


---
This is an automated alert from our incident response system.

#### Customer Communication Template

code
We are currently experiencing {brief description of issue} affecting {scope of impact}.

Our engineering team was alerted at {time} and is actively working to resolve the issue. We will provide updates every {frequency} until resolved.


What we know:
  • {factual statement of impact}

  • {factual statement of scope}

  • {brief status of response}


What we're doing:
  • {primary response action}

  • {secondary response action}


Workaround (if available):
{workaround steps or "No workaround currently available"}


We apologize for the inconvenience and will share more information as it becomes available.


Next update: {time}
Status page: {link}

Stakeholder Management

#### Stakeholder Classification

Internal Stakeholders:

  • Engineering Leadership - Technical decisions and resource allocation

  • Product Management - Customer impact assessment and feature implications

  • Customer Support - User communication and support ticket management

  • Sales/Account Management - Customer relationship management for enterprise clients

  • Executive Team - Business impact decisions and external communication approval

  • Legal/Compliance - Regulatory reporting and liability assessment

External Stakeholders:

  • Customers - Service availability and impact communication

  • Partners - API availability and integration impacts

  • Vendors - Third-party service dependencies and support escalation

  • Regulators - Compliance reporting for regulated industries

  • Public/Media - Transparency for public-facing outages

#### Communication Cadence by Stakeholder

| Stakeholder | SEV1 | SEV2 | SEV3 | SEV4 |
|-------------|------|------|------|------|
| Engineering Leadership | Real-time | 30min | 4hrs | Daily |
| Executive Team | 15min | 1hr | EOD | Weekly |
| Customer Support | Real-time | 30min | 2hrs | As needed |
| Customers | 15min | 1hr | Optional | None |
| Partners | 30min | 2hrs | Optional | None |

Runbook Generation Framework

#### Dynamic Runbook Components

1. Detection Playbooks
- Monitoring alert definitions
- Triage decision trees
- Escalation trigger points
- Initial response actions

2. Response Playbooks
- Step-by-step mitigation procedures
- Rollback instructions
- Validation checkpoints
- Communication checkpoints

3. Recovery Playbooks
- Service restoration procedures
- Data consistency checks
- Performance validation
- User notification processes

#### Runbook Template Structure

markdown
# {Service/Component} Incident Response Runbook

Quick Reference


  • Severity Indicators: {list of conditions for each severity level}


  • Key Contacts: {on-call rotations and escalation paths}


  • Critical Commands: {list of emergency commands with descriptions}


Detection


Monitoring Alerts


  • {Alert name}: {description and thresholds}


  • {Alert name}: {description and thresholds}


Manual Detection Signs


  • {Symptom}: {what to look for and where}


  • {Symptom}: {what to look for and where}


Initial Response (0-15 minutes)

1. Assess Severity
   - [ ] Check {primary metric}
   - [ ] Verify {secondary indicator}
   - [ ] Classify as SEV{level} based on {criteria}

2. Establish Command
   - [ ] Page Incident Commander if SEV1/2
   - [ ] Create incident tracking ticket
   - [ ] Join war room: {link/bridge info}


3. Initial Investigation
   - [ ] Check recent deployments: {deployment log location}
   - [ ] Review error logs: {log location and queries}
   - [ ] Verify dependencies: {dependency check commands}


Mitigation Strategies


Strategy 1: {Name}

Use when: {conditions}
Steps:
1. {detailed step with commands}
2. {detailed step with expected outcomes}
3. {validation step}

Rollback Plan:
1. {rollback step}
2. {verification step}


Strategy 2: {Name}

{similar structure}

Recovery and Validation

1. Service Restoration
   - [ ] {restoration step}
   - [ ] Wait for {metric} to return to normal
   - [ ] Validate end-to-end functionality

2. Communication
   - [ ] Update status page
   - [ ] Notify stakeholders
   - [ ] Schedule PIR


Common Pitfalls


  • {Pitfall}: {description and how to avoid}


  • {Pitfall}: {description and how to avoid}


Reference Information

→ See references/reference-information.md for details

Usage Examples


Example 1: Database Connection Pool Exhaustion
bash

Classify the incident

echo '{"description": "Users reporting 500 errors, database connections timing out", "affected_users": "80%", "business_impact": "high"}' | python scripts/incident_classifier.py

Reconstruct timeline from logs

python scripts/timeline_reconstructor.py --input assets/sample_timeline_events.json --output timeline.md

Generate PIR after resolution

python scripts/pir_generator.py --incident assets/sample_incident_data.json --timeline timeline.md --output pir.md
code
### Example 2: API Rate Limiting Incident
bash

Quick classification from stdin

echo "API rate limits causing customer API calls to fail" | python scripts/incident_classifier.py --format text

Build timeline from multiple sources

python scripts/timeline_reconstructor.py --input assets/simple_timeline_events.json --detect-phases --gap-analysis

Generate comprehensive PIR

python scripts/pir_generator.py --incident assets/sample_incident_pir_data.json --rca-method fishbone --action-items ```

Best Practices

During Incident Response

1. Maintain Calm Leadership
- Stay composed under pressure
- Make decisive calls with incomplete information
- Communicate confidence while acknowledging uncertainty

2. Document Everything
- All actions taken and their outcomes
- Decision rationale, especially for controversial calls
- Timeline of events as they happen

3. Effective Communication
- Use clear, jargon-free language
- Provide regular updates even when there's no new information
- Manage stakeholder expectations proactively

4. Technical Excellence
- Prefer rollbacks to risky fixes under pressure
- Validate fixes before declaring resolution
- Plan for secondary failures and cascading effects

Post-Incident

1. Blameless Culture
- Focus on system failures, not individual mistakes
- Encourage honest reporting of what went wrong
- Celebrate learning and improvement opportunities

2. Action Item Discipline
- Assign specific owners and due dates
- Track progress publicly
- Prioritize based on risk and effort

3. Knowledge Sharing
- Share PIRs broadly within the organization
- Update runbooks based on lessons learned
- Conduct training sessions for common failure modes

4. Continuous Improvement
- Look for patterns across multiple incidents
- Invest in tooling and automation
- Regularly review and update processes

Integration with Existing Tools

Monitoring and Alerting

  • PagerDuty/Opsgenie integration for escalation
  • Datadog/Grafana for metrics and dashboards
  • ELK/Splunk for log analysis and correlation

Communication Platforms

  • Slack/Teams for war room coordination
  • Zoom/Meet for video bridges
  • Status page providers (Statuspage.io, etc.)

Documentation Systems

  • Confluence/Notion for PIR storage
  • GitHub/GitLab for runbook version control
  • JIRA/Linear for action item tracking

Change Management

  • CI/CD pipeline integration
  • Deployment tracking systems
  • Feature flag platforms for quick rollbacks
Join our Telegram