There are no reports of issues, and things look stable. We will close out the alert in 24 hours if things continue to look good.
Posted Jul 24, 2024 - 16:27 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 24, 2024 - 08:35 UTC
Update
We are actively investigating and debugging the 'Internal Server Errors' while updating the IPFS Kubo nodes. Our team is closely monitoring the situation and will provide a further update in the coming hours.
Posted Jul 24, 2024 - 06:10 UTC
Update
Some subgraph deployments and IPFS requests are going through at this time, although we're still seeing a large volume of "500 internal server errors". The team is working around the clock to continue debugging this. We will provide another update in the coming hours as we continue to make progress.
Posted Jul 24, 2024 - 01:17 UTC
Update
We first identified a major issue with our IPFS instance on July 18th and began troubleshooting immediately. Our devops team worked over the weekend to determine whether the root cause was related to our IPFS configurations or our internal deployment router.
The team increased timeouts, restarted services, and blocked IPs, but the issue persisted. We have now pulled in IPFS experts to help resolve the issue, and are in the process of reconfiguring our IPFS kubo nodes, improving node peering, and updating our load balancing services. We are monitoring the situation and will provide another update in the coming hours.
Posted Jul 23, 2024 - 20:17 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 22, 2024 - 14:31 UTC
Monitoring
The issue has been identified and a solution is being implemented.
Posted Jul 22, 2024 - 10:38 UTC
Identified
The issue has been identified and a fix is being implemented.