PayPal and American Express Processing Issues
Incident Report for Braintree
Postmortem

Root Cause Analysis

Date: Monday, 27 August 2018 19:40 - 19:50

All Times UTC

Executive Summary

During a routine database operation at 19:40, the database backing American Express and Pay with PayPal processing became unresponsive, preventing any operations on these payment methods, and was unable to recover on its own. Engineers manually intervened, restoring normal service at 19:50. Transaction creation requests failed with a Processor Network Unavailable (3000) response and submit for settlement requests failed with an HTTP 500 response.

Merchants can retry failed American Express and PayPal transactions. Customers may temporarily see duplicate authorizations for retried Pay with PayPal transactions if the transactions failed during an attempted submit for settlement call (and not on transaction creation) during the incident window of 19:40 - 19:50.

Technical Summary

A planned deploy of the internal service providing American Express and Pay with PayPal processing included a database migration to add a foreign key index before the new service code was deployed. During the rollout of this service at 19:40, engineers discovered that the creation of this index was unintentionally written to be blocking, preventing writes to that table. Engineers manually terminated the index creation at 19:50, and the service was able to recover.

The database tables involved in the migration log Pay with PayPal and American Express transactions, which prevented any operations on these payment methods during the incident.

Follow-Up

  • Braintree is deploying existing tooling to this service to prevent engineers from adding indexes in an unsound manner.
  • Database changes for this service will be centrally reviewed by our database team going forward.
Posted 26 days ago. Aug 29, 2018 - 21:12 UTC

Resolved
PayPal service has returned to normal. The majority of failed Amex and PayPal transactions occurred between 19:40 - 19:50 UTC and contain a Processor Network Unavailable (3000) response. Some merchants may have noticed additional PayPal errors or connectivity issues until 20:10 UTC.
Posted 28 days ago. Aug 27, 2018 - 20:17 UTC
Monitoring
Amex transactions are now succeeding. A small number of PayPal requests may still fail. Engineers are working on it.
Posted 28 days ago. Aug 27, 2018 - 20:00 UTC
Investigating
An elevated number of PayPal and Amex transactions are currently failing. Engineers are investigating.
Posted 28 days ago. Aug 27, 2018 - 19:49 UTC
This incident affected: United States Processing, Canadian Processing, European Processing, APAC Processing, and PayPal Processing.