Skip to content

Incident: Exception spike on intel #46

Description

@yannvr

Incident Summary

An exception spike has been detected on the intel application, which is outside of the automated operations scope (no operable runbook is configured, and it is not an operable service under our deployment tool).

Event Details

  • App: intel
  • Source: signal_watcher
  • Kind: exception_spike
  • Last Hour Metrics:
    • Exceptions: 8 (vs 0 in the previous hour)
    • Events: 35 (vs 14 in the previous hour)
    • Pageviews: 3 (vs 4 in the previous hour)
    • Visitors: 2 (vs 2 in the previous hour)
  • Last 24 Hours Metrics:
    • Exceptions: 8 (vs 0 in the previous 24 hours)
    • Events: 75 (vs 6 in the previous 24 hours)

Fleet Probes / App Details

  • Final URL: https://intel.hyperdrift.io
  • HTTP Status: 200 (The landing page itself is responding with HTTP 200, but background exceptions are spiking)
  • Server: nginx
  • Page Size: 30176 bytes
  • Snippet: <!DOCTYPE html><html lang="en" data-palette="midnight"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/_next/static/css/b496025c9cd768cc.css" data-precedence="next"/><link rel="stylesheet" href="/_next/static/css/531dc02

Actions Taken & Results

  1. Heal Service Attempt:
    • heal_service(app="intel") -> {"healed": false, "reason": "no automated runbook for 'intel' — escalate to a human"}
  2. Service Configuration Check:
    • get_service_config(app="intel") -> {"error": "'intel' is not an operable service"}

Escalation

Since intel has no automated runbook and is not an operable service, human intervention is required to investigate the logs and address the underlying cause of the exceptions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions