On May 24th, 2026, at approximately 17:05 UTC, Hookdeck experienced a complete disruption of request, attempt, and event processing lasting approximately 3 hours and 10 minutes. Processing dropped to near-zero affecting requests, and downstream events and attempts, all stopped processing as our data syncing service was unable to write to Google Cloud Bigtable.
Hookdeck's architecture uses a two-tier storage model for request and event data: an in-memory store provides short-term access to recent data (approximately 5 minutes), while Bigtable serves as the durable long-term persistence layer. Our processing pipeline relies on Bigtable for longer-window idempotency checks (between 5 minutes and 2 hours) avoiding reprocessing of entities on message retries if they were already processed. When Bigtable writes began failing, these lookups failed, causing messages to be retried repeatedly. This stormed the processing pods with retries, leading to OOM crashes that brought ingestion to a crawl.
The root cause was a Google Cloud Bigtable infrastructure issue in the us-east1-b zone. Google confirmed that an issue with their resource allocation and accounting system for storage resources caused write operations to fail across the zone. Read operations also experienced elevated latency, though Google’s initial report stated they were unaffected.
No data was lost during this incident. Google Pub/Sub and Kafka buffered all incoming writes during the outage window. After Google resolved the underlying Bigtable issue and we restarted our syncing service at 20:15 UTC, the full backlog was processed successfully, with traffic peaking at approximately 6x normal volume before returning to steady state around 21:30 UTC.
As a side effect of the outage, some events that were successfully delivered to their destination appear in a PENDING state in the dashboard. We are actively remediating these misrepresented states. In the meantime, clicking on an individual event in the dashboard will show its correct state.
us-east1-b affecting our Bigtable instance. Internal investigation is already underway.us-east1-b).