Resolved -
There are no reports of issues, and things look stable. We will close out the alert in 24 hours if things continue to look good.
Jul 24, 16:27 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Jul 24, 08:35 UTC
Update -
We are actively investigating and debugging the 'Internal Server Errors' while updating the IPFS Kubo nodes. Our team is closely monitoring the situation and will provide a further update in the coming hours.
Jul 24, 06:10 UTC
Update -
Some subgraph deployments and IPFS requests are going through at this time, although we're still seeing a large volume of "500 internal server errors". The team is working around the clock to continue debugging this. We will provide another update in the coming hours as we continue to make progress.
Jul 24, 01:17 UTC
Update -
We first identified a major issue with our IPFS instance on July 18th and began troubleshooting immediately. Our devops team worked over the weekend to determine whether the root cause was related to our IPFS configurations or our internal deployment router.
The team increased timeouts, restarted services, and blocked IPs, but the issue persisted. We have now pulled in IPFS experts to help resolve the issue, and are in the process of reconfiguring our IPFS kubo nodes, improving node peering, and updating our load balancing services. We are monitoring the situation and will provide another update in the coming hours.
Jul 23, 20:17 UTC
Identified -
The issue has been identified and a fix is being implemented.
Jul 22, 14:31 UTC
Monitoring -
The issue has been identified and a solution is being implemented.
Jul 22, 10:38 UTC
Identified -
The issue has been identified and a fix is being implemented.
Jul 18, 10:16 UTC