> ## Documentation Index
> Fetch the complete documentation index at: https://docs.niro.cx/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts & Incidents

> Automatic incident detection with notifications via Slack, Discord, PagerDuty, and webhooks.

## Overview

Niro automatically detects operational incidents by analyzing status data from your fleet. When a problem is detected, an **incident** is opened. When the condition clears, the incident is resolved.

<CardGroup cols={2}>
  <Card title="Automatic detection" icon="magnifying-glass">
    Niro continuously evaluates cluster data for crash loops, offline clusters, node failures, and more.
  </Card>

  <Card title="Smart notifications" icon="paper-plane">
    Route alerts to Slack, Discord, PagerDuty, or a custom webhook with per-channel severity filters.
  </Card>

  <Card title="AI investigation" icon="brain">
    One-click root cause analysis gathers logs and cluster state to explain what went wrong.
  </Card>

  <Card title="Incident lifecycle" icon="circle-exclamation">
    Incidents open when conditions fire and resolve automatically when they clear.
  </Card>
</CardGroup>

## Detected incident types

| Type                  | Condition                                                                    |
| --------------------- | ---------------------------------------------------------------------------- |
| **Cluster offline**   | No status update received within the expected window                         |
| **Pod crash-looping** | A pod has restarted more than a threshold number of times in a recent window |
| **Node not ready**    | A node has stopped reporting as healthy                                      |
| **Deployment failed** | A deployment apply resulted in a Failed or Partial status                    |
| **Drift detected**    | Live cluster state diverges from the repo's desired state                    |
| **Image pull error**  | A pod can't pull its container image                                         |

Each incident has a stable fingerprint based on cluster, type, and scope. Niro won't fire duplicate incidents for the same ongoing condition.

## Incident lifecycle

<Steps>
  <Step title="Condition detected">
    Niro detects the condition from incoming cluster data.
  </Step>

  <Step title="Incident opened">
    An incident is created in the Alerts page. Notifications are sent to all matching channels.
  </Step>

  <Step title="Investigation (optional)">
    If auto-investigation is enabled, Niro runs an AI root-cause analysis in the background.
  </Step>

  <Step title="Incident resolved">
    When the condition clears, Niro resolves the incident and sends resolution notifications.
  </Step>
</Steps>

<Note>
  You can't manually delete a **firing** incident — if the condition is still active, Niro would reopen it immediately. Resolved incidents can be deleted from the Alerts page.
</Note>

## Notification channels

<AccordionGroup>
  <Accordion title="Discord" icon="bell">
    <Badge>Free</Badge>

    Niro posts a formatted message to a Discord channel via webhook.

    **Setup:**

    1. In Discord: Server Settings → Integrations → Webhooks → New Webhook
    2. Copy the webhook URL
    3. In Niro: Settings → Alerts → Add Discord channel
  </Accordion>

  <Accordion title="Slack" icon="paper-plane">
    <Badge>Pro</Badge>

    Niro posts a formatted message to a Slack channel via an Incoming Webhook.

    **Setup:**

    1. Create an Incoming Webhook in your Slack workspace
    2. In Niro: Settings → Alerts → Add Slack channel with the webhook URL
  </Accordion>

  <Accordion title="PagerDuty" icon="bell">
    <Badge>Pro</Badge>

    Niro creates PagerDuty incidents via the Events API v2, sending `trigger` on open and `resolve` on close.

    **Setup:**

    1. In PagerDuty: create a service and copy its Integration Key
    2. In Niro: Settings → Alerts → Add PagerDuty channel with the Integration Key
  </Accordion>

  <Accordion title="Webhook" icon="link">
    <Badge>Pro</Badge>

    Niro POSTs a JSON payload to your URL on every incident state change.

    ```json theme={"theme":{"light":"github-light","dark":"vesper"}}
    {
      "event": "incident.opened",
      "incident": {
        "id": "inc_abc123",
        "type": "pod_crash_looping",
        "severity": "high",
        "cluster": "my-cluster",
        "scope": "default/my-app-5d9b8c-xyz",
        "fired_at": "2024-01-15T10:30:00Z"
      },
      "org": "my-org"
    }
    ```

    Events: `incident.opened` · `incident.resolved`
  </Accordion>

  <Accordion title="Incident.io" icon="circle-exclamation">
    <Badge>Enterprise</Badge>

    Niro creates Incident.io incidents directly via their API.
  </Accordion>
</AccordionGroup>

## Acknowledging incidents

You can acknowledge a firing incident to signal that your team is aware and investigating. Acknowledging an incident doesn't resolve it — it continues to track the condition and resolves automatically when the condition clears. Acknowledgements appear in the incident timeline.

## Minimum severity filter

Each channel has a minimum severity setting — incidents below the threshold don't trigger that channel.

| Severity     | Typical use                                             |
| ------------ | ------------------------------------------------------- |
| **Info**     | Logging webhook — capture everything                    |
| **Warning**  | Slack — catch problems early                            |
| **Error**    | On-call Slack — significant failures only               |
| **Critical** | PagerDuty — severe outages requiring immediate response |

## Auto-investigation

Niro can automatically run an [AI investigation](/features/ai-investigation) when incidents change state. Configure in **Settings** → **Auto-investigation**:

| Mode       | When investigation runs   |
| ---------- | ------------------------- |
| `off`      | Never — manual only       |
| `firing`   | When an incident opens    |
| `resolved` | When an incident resolves |
| `both`     | On both open and close    |

<Tip>
  `firing` mode is most useful — you get root-cause analysis the moment something breaks, before you've opened the dashboard.
</Tip>

Available on <Badge>Pro</Badge> and above.

## Related

<CardGroup cols={2}>
  <Card title="AI Investigation" icon="brain" href="/features/ai-investigation">
    What the investigator analyzes and what it returns.
  </Card>

  <Card title="Monitor Your Fleet" icon="gauge" href="/guides/fleet-monitoring">
    Pod inventory and cluster health views.
  </Card>

  <Card title="Plans" icon="credit-card" href="/reference/plans">
    Notification channel availability by plan.
  </Card>

  <Card title="Multi-tenancy" icon="building" href="/features/multi-tenancy">
    Org-level alert channel configuration.
  </Card>
</CardGroup>