[Grafana]: Base: Block ingestor lagging behind

Incident Report for The Graph

Resolved

This incident has been resolved.

Posted Apr 23, 2025 - 22:02 UTC

Update

[Comment from Opsgenie]Vlad Sirenko noted: "https://graphprotocol.slack.com/archives/CL43LRZC6/p1745438908907459?thread_ts=1745433420.253159&cid=CL43LRZC6

syncing, should be self resolved soon " on alert: "[Grafana]: Base: Block ingestor lagging behind"

Posted Apr 23, 2025 - 20:10 UTC

Update

[Comment from Opsgenie]Vlad Sirenko noted: "The problem is being resolved on the Pinax end. they have just rolled out the fix. " on alert: "[Grafana]: Base: Block ingestor lagging behind"

Posted Apr 23, 2025 - 20:05 UTC

Update

[Comment from Opsgenie]Vlad Sirenko acknowledged alert: "[Grafana]: Base: Block ingestor lagging behind"

Posted Apr 23, 2025 - 20:04 UTC

Investigating

[FIRING:1] Base: Block ingestor lagging behind Production (partial_outage indexers P2-High true Infrastructure)
https://thegraph.grafana.net/alerting/list

**Firing**

Value: B0=0
Labels:
- alertname = Base: Block ingestor lagging behind
- cmp = partial_outage
- component = indexers
- grafana_folder = Production
- priority = P2-High
- statuspage = true
- team = Infrastructure
Annotations:
- description = Triggered when the block ingestor is lagging
- statuspage = Base: Block ingestor lagging behind
Source: https://thegraph.grafana.net/alerting/grafana/ddjeivil1kc8we/view?orgId=1
Silence: https://thegraph.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dddjeivil1kc8we&matcher=cmp%3Dpartial_outage&matcher=component%3Dindexers&matcher=priority%3DP2-High&matcher=statuspage%3Dtrue&matcher=team%3DInfrastructure&orgId=1
Dashboard: https://thegraph.grafana.net/d/7rcuDImZk?from=1745434600000&orgId=1&to=1745438235315
Panel: https://thegraph.grafana.net/d/7rcuDImZk?from=1745434600000&orgId=1&to=1745438235315&viewPanel=118

Posted Apr 23, 2025 - 19:57 UTC