Alert to root cause: Zabbix, PagerDuty and Grafana MCP with rConfig

Alert to root cause: Zabbix, PagerDuty and Grafana MCP with rConfig

The page goes off at 02:14: an uplink on a distribution switch is down. The first question on any call is “did anyone change something?” With your monitoring and rConfig both connected to one agent, that question takes one prompt instead of three browser tabs and a manual diff.

Best for: on-call triage, the first ten minutes of an incident, post-incident timelines. Tools: rConfig, plus one alert source (Zabbix, PagerDuty or Grafana). Optional: Grafana for metrics around the alert. rConfig tools used: get_devices, get_config_history, get_config_diff.

How it flows

sequenceDiagram
    participant E as Engineer
    participant A as Agent
    participant M as Monitoring MCP<br/>(Zabbix / PagerDuty / Grafana)
    participant R as rConfig MCP
    E->>A: "Did a config change cause this alert?"
    A->>M: Get the alert: device, trigger, start time
    A->>R: get_devices (name) → device ID
    A->>R: get_config_history (device, "show run")
    R-->>A: Versions with timestamps
    A->>R: get_config_diff (last version before alert, first after)
    R-->>A: Changed lines only
    A-->>E: What changed, when, and whether it fits the symptom

The agent works on one device, so every call is small: a page of history and one diff.

Example session

The prompt names the alert and asks for timestamps, which keeps the answer honest about what rConfig can and can’t see:

Zabbix has a problem “Interface Gi1/0/48 down” on dub-dist-02 that started at 02:14 today. Check rConfig for configuration changes on that device in the last 24 hours. Show me the diff, say whether the change could explain the alert, and quote the backup times you’re comparing.

What a well-briefed agent does:

1. Reads the alert from the Zabbix MCP server (host, trigger, start time). With PagerDuty it reads the incident and its alerts; with Grafana, the alert rule and firing time.
2. Finds the device in rConfig with get_devices filtered by name. If rConfig’s inventory comes from Zabbix through Device Sync, the hostnames match exactly.
3. Lists config versions with get_config_history, filtered to the running-config command so show version captures don’t crowd the list.
4. Diffs the two versions either side of 02:14 with get_config_diff.
5. Answers, citing the evidence.

An illustrative answer:

rConfig has two running-config backups either side of the alert:
  01:00 (version 412) and 03:00 (version 413).

The diff shows one change on GigabitEthernet1/0/48:
  - switchport trunk allowed vlan 10,20,30
  + switchport trunk allowed vlan 10,20

VLAN 30 was removed from the uplink's allowed list. If the alert is
based on a VLAN 30 SVI or a monitored host in VLAN 30, this change
fits the symptom. A physical link-down would not be explained by it.
Check the interface's physical status before acting on this.

The answer distinguishes “fits the symptom” from “caused it”. That distinction is worth putting in your standing agent instructions (see Write the routing down once).

What each tool contributes

Limits to know

• rConfig sees changes at backup time. If the device is backed up nightly, a change made at 01:55 shows up after the next backup, not at 02:14. For devices where this matters, schedule backups more often, or run a backup from rConfig during the incident. The MCP server can’t trigger one: it is read-only.
• No change is also an answer. If the versions either side of the alert are identical, get_config_diff reports identical: true, and the agent should say so plainly. Config is ruled out for that window; move on to hardware, carriers and neighbours.
• Correlation isn’t causation. The agent compares timestamps and reads a diff. An engineer confirms the cause before rolling anything back.
• History pages at 50. For a device backed up very frequently, the agent may need a second page of get_config_history to reach versions from yesterday.

Guardrails

• Keep every server read-only: rConfig always is; run Grafana with --disable-write; use a PagerDuty user in a restricted role; enable your Zabbix server’s read-only mode.
• The agent proposes the rollback as text. Applying it goes through your change process.
• Use a token for a user whose role covers the devices the on-call team supports. See Tokens and what the agent can see.

Variations

• PagerDuty first. Start from the incident: “Summarise incident Q1ABC2D and check rConfig for changes on the affected devices.” Useful when one incident groups alerts from several devices.
• Many devices, one site. For a site-wide outage, have the agent run the same history check across the site’s devices (filter get_devices by group or tag) and list only the devices whose config changed in the window.
• Post-incident timeline. After the fact, ask for a timeline that merges alert times, config change times and backup times. It drops straight into the incident review.

Related reference

• Connect an AI agent over MCP
• Config diffs
• Drift Detection and Narration
• Zabbix Device Sync