Correlated events

GET /api/events/mtr

A correlated event says that packet loss starting at one hop reached the destination on several paths at about the same time. Several targets, several agents, or both were affected through that hop. Because the evidence comes from independent paths, an event usually points more directly at the faulty router or link than any single traceroute does.

Events are built from the traceroutes of your ICMP targets, so they cover the traceroute retention period of your plan.

How events are detected

Observiant examines every traceroute that reached its destination with packet loss:

1. Find where the loss starts. Walking back from the destination, the fault hop is the earliest answering hop in an unbroken run of hops that all show loss. Hops that do not answer are skipped. A hop that answers without loss ends the run. Loss that appears at one router but not at the hops after it is therefore ignored: that pattern is usually ICMP rate limiting, not forwarding loss.
2. Group by fault hop. Observations with the same fault hop address belong together when each one is no more than the correlation window after the previous one.
3. Require independent evidence. A group becomes an event only when it includes at least minimum evidence distinct target-and-agent paths, and those paths involve at least two targets or at least two agents. Repeated reports on the same path do not add evidence.

Every response includes the values in effect, as correlation_window_ms and min_evidence. By default, the window is 15 minutes and the minimum evidence is 3 paths.

An event is active while its most recent observation is no older than one correlation window. After that it is resolved.

Parameters

Parameter	Description
start	Optional. Range start, Unix milliseconds. Default: 24 hours before end.
end	Optional. Range end, Unix milliseconds. Default: now.
page	Optional. Page number, starting at 1. Default 1. A page past the last one returns the last page.
page_size	Optional. Events per page, 1–500. Default 100. Larger values are reduced to 500.

The range can be at most 30 days.

Request

curl

curl --header "Authorization: Bearer $OBSERVIANT_API_KEY" \
  "$OBSERVIANT_URL/api/events/mtr?page_size=50"

Python

response = session.get(f"{base_url}/api/events/mtr", params={"page_size": 50}, timeout=60)
response.raise_for_status()
events = response.json()["events"]

Response

{
  "events": [
    {
      "event_id": "mtr-hop-3f9c2a7b1d4e8f60-5a1c9e2b7d3f4a81",
      "alert_key": "mtr-hop-3f9c2a7b1d4e8f60",
      "kind": "mtr_propagating_loss",
      "status": "active",
      "started_at": 1791701890114,
      "last_seen": 1791702747302,
      "fault_hop": {
        "index": 4,
        "host": "203.0.113.130",
        "hostname": "xe-0-1-0.fra-core3.example.org",
        "country": "DE",
        "asn": "AS64502",
        "loss_pct": 30
      },
      "affected_targets": ["01M4MZ5T3A76E4RBFKGBD5MP8V", "01M4MZ5TCN74PFRTXDKXYE4XWX"],
      "affected_agents": ["dc1-probe", "sys-amsterdam"],
      "evidence_count": 3,
      "observation_count": 7,
      "max_destination_loss_pct": 30,
      "evidence": [
        {
          "result_id": "01M4N1Q5D8C2X7V4B9N3M6K1JA",
          "target_id": "01M4MZ5TCN74PFRTXDKXYE4XWX",
          "agent_label": "sys-amsterdam",
          "display_name": "Amsterdam",
          "timestamp": 1791702747302,
          "destination": "8.8.8.8",
          "destination_loss_pct": 20,
          "fault_hop": {"index": 4, "host": "203.0.113.130", "hostname": "xe-0-1-0.fra-core3.example.org",
                        "country": "DE", "asn": "AS64502", "loss_pct": 30},
          "previous_hop": {"index": 3, "host": "198.51.100.66", "hostname": "be20.ams-edge2.example.net",
                           "country": "NL", "asn": "AS64500", "loss_pct": 0}
        }
      ]
    }
  ],
  "page": 1,
  "page_size": 50,
  "total_events": 1,
  "total_pages": 1,
  "has_next": false,
  "correlation_window_ms": 900000,
  "min_evidence": 3,
  "truncated": false
}

The example shows one of the event's three evidence entries.

Event fields

Events are sorted by last_seen, most recent first.

Field	Description
event_id	Identifies this occurrence: the fault hop together with the time the occurrence started.
alert_key	Identifies the fault hop. It is the same for every occurrence at the same hop, so use it to group repeated incidents.
kind	Always mtr_propagating_loss.
status	active or resolved, as described in How events are detected.
started_at, last_seen	First and most recent observation in the event, Unix milliseconds.
fault_hop	The hop where the loss starts, as seen in the most recent observation. See Hop fields.
affected_targets	IDs of the affected targets, sorted. Look up labels with GET /api/targets.
affected_agents	Labels of the agents that saw the loss, sorted.
evidence_count	Number of distinct target-and-agent paths in the event.
observation_count	Number of traceroutes in the event, including repeats on the same path.
max_destination_loss_pct	Highest packet loss at a destination among the observations, percent.
evidence	One entry per path: the most recent traceroute on that path, most recent first.

Evidence fields

Field	Description
result_id	The traceroute. Fetch it with GET /api/mtr/{result_id}.
target_id, agent_label, display_name, timestamp	The path and time of the traceroute.
destination	Destination address of the traceroute.
destination_loss_pct	Packet loss at the destination, percent.
fault_hop	Where the loss starts on this path.
previous_hop	The last answering hop before the fault hop, which had no loss. Its index is 0 when the fault hop is the first answering hop.

Hop fields

Field	Description
index	Hop number, counting from 1, as in MTR output.
host	Hop address.
hostname	Reverse DNS name, or an empty string.
country	ISO 3166-1 alpha-2 country code, or an empty string.
asn	Autonomous system, such as AS64502, or an empty string.
loss_pct	Packet loss at this hop, percent.

Response fields

Field	Description
page, page_size	The page returned and its size.
total_events, total_pages, has_next	Totals for the requested range.
correlation_window_ms, min_evidence	Detection settings in effect.
truncated	true when the range held more than 50,000 traceroutes with propagating loss and only the most recent 50,000 were evaluated. Request a shorter range.

Events are computed for the range you request

Observiant computes events from the traceroutes in the requested range every time you call the endpoint. As a result:

• An incident that began before start is reported with started_at at its first observation inside the range. Its event_id then differs from the one you would get with an earlier start.
• The same incident keeps the same event_id across calls only as long as its first observation stays inside the range.

For alerting, query a fixed look-back window that is longer than your incidents usually last, such as the last 24 hours, and remember which event_id values you have already handled. Use alert_key to recognize a hop that has failed before.

Example: notify on new incidents

This script is meant to run every five minutes from cron or a scheduler. It reports each active event once and reports again when that event resolves. It keeps the IDs it has handled in a small state file.

watch_events.py

"""Report new and resolved Observiant correlated events.

Run periodically, for example every 5 minutes. Requires OBSERVIANT_URL and
OBSERVIANT_API_KEY in the environment.
"""
import json
import os
import pathlib

import requests

STATE_FILE = pathlib.Path("observiant-events.json")

base_url = os.environ["OBSERVIANT_URL"]
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['OBSERVIANT_API_KEY']}"


def fetch_events():
    """Return every event of the last 24 hours, across all pages."""
    events, page = [], 1
    while True:
        response = session.get(
            f"{base_url}/api/events/mtr",
            params={"page": page, "page_size": 500},
            timeout=60,
        )
        response.raise_for_status()
        body = response.json()
        events.extend(body["events"])
        if not body["has_next"]:
            return events
        page += 1


def notify(message):
    # Replace with your chat, paging or ticketing integration.
    print(message)


state = json.loads(STATE_FILE.read_text()) if STATE_FILE.exists() else {"active": {}}
seen_active = state["active"]

events = fetch_events()
current = {event["event_id"]: event for event in events}

for event_id, event in current.items():
    hop = event["fault_hop"]
    details = [hop["host"], hop["hostname"], hop["asn"], hop["country"]]
    where = f"hop {hop['index']} " + " ".join(value for value in details if value)
    if event["status"] == "active" and event_id not in seen_active:
        notify(
            f"Packet loss starting at {where}: {event['evidence_count']} paths, "
            f"{len(event['affected_targets'])} targets, agents {', '.join(event['affected_agents'])}, "
            f"up to {event['max_destination_loss_pct']:g}% loss at destinations"
        )
        seen_active[event_id] = where
    elif event["status"] == "resolved" and event_id in seen_active:
        notify(f"Resolved: packet loss starting at {where}")
        del seen_active[event_id]

# Forget events that have left the 24-hour window.
for event_id in list(seen_active):
    if event_id not in current:
        del seen_active[event_id]

STATE_FILE.write_text(json.dumps(state))

$ python watch_events.py
Packet loss starting at hop 4 203.0.113.130 xe-0-1-0.fra-core3.example.org AS64502 DE: 3 paths, 2 targets, agents dc1-probe, sys-amsterdam, up to 30% loss at destinations