All Times UTC
During a routine database operation at 19:40, the database backing American Express and Pay with PayPal processing became unresponsive, preventing any operations on these payment methods, and was unable to recover on its own. Engineers manually intervened, restoring normal service at 19:50. Transaction creation requests failed with a Processor Network Unavailable (3000) response and submit for settlement requests failed with an HTTP 500 response.
Merchants can retry failed American Express and PayPal transactions. Customers may temporarily see duplicate authorizations for retried Pay with PayPal transactions if the transactions failed during an attempted submit for settlement call (and not on transaction creation) during the incident window of 19:40 - 19:50.
A planned deploy of the internal service providing American Express and Pay with PayPal processing included a database migration to add a foreign key index before the new service code was deployed. During the rollout of this service at 19:40, engineers discovered that the creation of this index was unintentionally written to be blocking, preventing writes to that table. Engineers manually terminated the index creation at 19:50, and the service was able to recover.
The database tables involved in the migration log Pay with PayPal and American Express transactions, which prevented any operations on these payment methods during the incident.