Between May 13 and May 29, 2025, the Salesforce write-back integration failed for users in US-based instances. This resulted in delayed updates from our systems to Salesforce, primarily impacting Salesforce-based reporting for a subset of customers. EU instances were unaffected.
During the outage, approximately 6m messages accumulated in the processing queue. No immediate user-facing functionality was impacted beyond reporting accuracy.
Root Cause: The integration failure stemmed from a deployment issue that blocked the write-back process, compounded by a lack of logging and monitoring resulting in 6m backed up queue messages that were subsequently all released. Additionally, events were processed as individual API calls rather than in bulk, which led to some customers hitting Salesforce API rate limits.
Resolution & Improvements:
The queue eventually cleared after manual intervention. We are implementing the following to prevent recurrence:
- Monitoring for the queue size and write-back errors
- Logging improvements
- Bulk processing and flow control to reduce API load
- Internal alert tuning to reduce noise
We apologize for the oversight and thank affected customers for their patience!