Summary

On July 18, 2023, Hookdeck's ingestion service experienced an unexpected disruption due to an outage at Cloudflare's Workers KV store. This incident led to an increase in 500 HTTP status codes at our ingestion endpoints. The issue began at 05:49 UTC and was resolved by 07:08 UTC. The issue was contained to a handful of sources, and localized mainly in the UK datacenter, with some incidents in the Sydney data centre. During that interval, 1.23% of our total traffic was affected.

Timeline

• 05:49 UTC: Onset of the outage. Hookdeck began observing increased 500 HTTP status codes at ingestion, indicating a problem with the Cloudflare's Workers KV store.
• 07:08 UTC: The incident was resolved by Cloudflare, and the 500 error rate returned to normal.
• 15:15 UTC: Hookdeck implemented a workaround to bypass Cloudflare's KV at ingestion in case of error.

Root Cause

The increase in 500 errors at ingestion was traced back to an issue with Cloudflare's Workers KV store, which started experiencing increased 500 HTTP errors. Hookdeck relies on Cloudflare KV store to retrieve ingestion endpoint preferences, integrations and quotas.

Lessons Learned

What went wrong

• Our monitoring system did not detect the outage, as the threshold of errors was too low to be detected

Where we got lucky

• The outage was somewhat localized, as it mainly affected only a handful of sources

Corrective actions

• Improve monitoring sensitivity to detect such errors
• Modify ingestion code to account for KV possibly being unavailable, which assumes highest limits, and defers verification to our main system after ingestion

Conclusion

We understand the crucial role we play in our customers' operations and sincerely apologize for the inconvenience this outage might have caused. We take this incident very seriously and are taking proactive measures to prevent any similar occurrences in the future. We appreciate your understanding and trust as we continue to work towards providing seamless services.