On August 4th (08:36am, EST), our US and EU customers experienced a 189-minute incident, including approximately 70 minutes of downtime. During this incident, customers in the US and EU servers experienced degraded performance in our core services with high latency. Between 9:41am and 9:48am EST, we initiated a complete shutdown of our services to help our systems recover.
While the disruption lasted for an extended period, no data loss occurred during the incident, and we were able to ensure the integrity of your information.
A third party infrastructure provider deployed a security update which resulted in unexpected behavior and temporary instability within the platform. While the issue was outside of our direct control, it impacted how our service operated.
Our team scaled the system to support the load that was created as a result of the third party security update. By scaling the system and working with the third party provider to revert the security update, we restored normal service by 11:45am EST.
We’re working to sharpen our processes for blocking impacts of live updates performed by third party providers, and improve metrics that will help to identify third party provider issues proactively.
We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability.
Your team at monday.com