Incident Response Stories

MLOps engineers are expected to handle production incidents. Behavioral questions test your crisis management, debugging skills, and communication under pressure.

The STAR-L Framework for Incidents

Component	What to Include
Situation	Production context, business impact
Task	Your specific role and responsibility
Action	Technical steps you took
Result	Resolution, metrics, timeline
Learning	What you changed to prevent recurrence

Interview Question: Critical Incident

Question: "Tell me about a time when a production ML system failed and you had to debug it under pressure."

Strong Answer Structure:

incident_story:
  situation:
    context: "Our fraud detection model suddenly started flagging 40% of legitimate transactions as fraudulent"
    impact: "Customer complaints spiked, support queue grew to 500+ tickets in 2 hours"
    severity: "P1 - direct revenue impact, estimated $50K/hour in false declines"

  task:
    role: "On-call MLOps engineer"
    responsibility: "Identify root cause, coordinate rollback if needed, restore normal operation"
    stakeholders: "Product, customer support, executive team on the incident bridge"

  action:
    timeline:
      - "0-15 min: Checked monitoring dashboards, saw prediction distribution shift"
      - "15-30 min: Ruled out model drift - model hadn't been updated in 2 weeks"
      - "30-45 min: Investigated feature pipeline, found upstream data source returning nulls"
      - "45-60 min: Traced to partner API rate limiting us after contract change"

    technical_steps:
      - "Queried feature store for recent feature values vs baseline"
      - "Compared input feature distributions across 24-hour windows"
      - "Found 'merchant_risk_score' feature was null for 60% of requests"
      - "Null values were being imputed as high-risk (default behavior)"

  result:
    resolution: "Implemented feature fallback to cached values, restored normal operation"
    timeline: "Total incident duration: 75 minutes"
    metrics:
      - "False positive rate returned to baseline 2%"
      - "Processed backlog of 500 tickets within 4 hours"
      - "No customer churn attributed to incident"

  learning:
    immediate: "Added null rate monitoring with alerts on feature pipeline"
    long_term: "Implemented circuit breaker for external data dependencies"
    process: "Updated runbook with feature-level debugging steps"

Common Incident Scenarios to Prepare

mlops_incident_scenarios = {
    "model_degradation": {
        "symptoms": "Accuracy dropping gradually over days/weeks",
        "common_causes": ["Data drift", "Feature drift", "Upstream schema change"],
        "investigation_steps": [
            "Compare current vs training data distributions",
            "Check feature store for schema changes",
            "Review upstream data source changes"
        ]
    },

    "sudden_performance_drop": {
        "symptoms": "Metrics cliff overnight or within hours",
        "common_causes": ["Bad deployment", "Infrastructure change", "Data pipeline failure"],
        "investigation_steps": [
            "Check recent deployments (git log, ArgoCD)",
            "Verify model artifact integrity",
            "Check feature pipeline health"
        ]
    },

    "latency_spike": {
        "symptoms": "P99 latency exceeds SLA",
        "common_causes": ["Resource exhaustion", "Model size increase", "Batching issues"],
        "investigation_steps": [
            "Check CPU/memory/GPU utilization",
            "Review recent model size changes",
            "Analyze request queue depth"
        ]
    },

    "silent_failure": {
        "symptoms": "Model returns predictions but they're wrong",
        "common_causes": ["Feature serving mismatch", "Model/preprocessing version mismatch"],
        "investigation_steps": [
            "Compare feature values at training vs serving",
            "Verify model version matches preprocessing version",
            "Check for training-serving skew"
        ]
    }
}

Communication During Incidents

incident_communication:
  executive_update:
    format: "Impact → Current status → ETA → Next update time"
    example: "40% of transactions flagged incorrectly. Root cause identified - upstream data issue. Fix deploying now, ETA 15 minutes. Next update at 3:30 PM."

  technical_update:
    format: "What we know → What we're investigating → What we need"
    example: "merchant_risk_score feature returning nulls since 2 PM. Investigating partner API. Need access to their status page."

  post_mortem:
    sections:
      - "Timeline of events"
      - "Root cause analysis"
      - "Impact assessment"
      - "Action items with owners"
      - "Lessons learned"

Red Flags to Avoid

Red Flag	Better Approach
"I fixed it alone"	Show collaboration and communication
Blaming others	Take ownership, focus on solutions
No metrics	Quantify impact and resolution time
No learning	Always include process improvements

Interview Signal: The best incident stories show both technical depth (you understand the system) and leadership (you coordinated the response and improved processes).

Next, we'll cover cross-team collaboration stories. :::