Transaction Processing Issues
Incident Report for Braintree
Postmortem

Impact Description

During the incident window (21:50 UTC, 9 Aug 2019 - 03:28 UTC, 10 Aug 2019, some transactions and authorizations issued during the incident window are not available via the API or through the Control Panel. Additionally, merchants will experience HTTP 404 Not Found errors when attempting to search, void, or Submit for Settlement on the affected transactions.

Root Cause

A wrongly configured connection from the Braintree gateway deployed to a small group of servers resulted in requests and transactions sent to a newly provisioned database that was not ready to receive production traffic. When these transactions were written to the newly provisioned database, they were issued public ids that did not match the primary transaction sequence, and therefore the public transaction ids returned to merchants were not correct. As a result, merchants are unable to reference or take actions on these affected transactions.

Affected transactions will be individually recovered in coordination with individual affected merchants, however, these affected transactions will not be available through the Control Panel or Braintree API.

Corrective Actions & Preventative Measures

  • Engineers mitigated the impact and prevented ongoing transaction impact at 03:28 UTC.
  • Database inconsistency detection and mitigation processes are scheduled for improvement to ensure that engineers are able to detect and recover impact more quickly.
  • New transaction inconsistency recovery processes have been implemented to decrease time to incident resolution.
Posted 8 days ago. Aug 13, 2019 - 14:42 UTC

Resolved
All impacted transactions are being tracked, and we will reach out to coordinate recovery. A full root cause analysis will be provided.
Posted 11 days ago. Aug 10, 2019 - 15:50 UTC
Monitoring
Due to an issue relating to a deploy, a small number of transactions created between 21:50 and 03:28 UTC are not available. Transactions with inline settlement may have already settled. No data was lost, but due to the nature of the issue, these transactions are not currently visible in the Control Panel or API, or incorrect transaction details may appear.

Merchants may also experience HTTP 404 "Not Found" errors when attempting to search or void an affected transaction, or submit an affected transaction for settlement. Merchants *should not* re-submit affected transactions, or attempt to capture or void affected transactions. Engineers are working to assess full impact and will provide more information when it becomes available.
Posted 12 days ago. Aug 10, 2019 - 04:57 UTC