Transaction Processing Issues
Incident Report for Braintree

Impact Description

During the incident window, we experienced an intermittent outage across different processing networks which resulted in unsuccessful transactions. Merchants processing in Europe, Australia, Canada, and New Zealand received an elevated rate of Processor Network Unavailable (3000) responses for the failed transactions.

Root Cause

An infrastructure operation was being carried out to restore the SSH File Transfer Protocol (SFTP) connection for a specific region. During this operation, several processor-connecting hosts were incorrectly taken out of service. As a result, processor networks on these hosts became unavailable, resulting in failed transactions. Engineers were immediately notified, quickly restored these processor-connection hosts, and traffic recovered.

Corrective Actions & Preventative Measures

  • Engineers immediately restored the servers and prevented ongoing transaction impact at 15:12:52 UTC.
  • Safeguards have been implemented for future processor operations to reduce the likelihood of accidental outages.
Posted Aug 16, 2019 - 21:13 UTC

Between 15:10:31 - 15:12:52 UTC on Aug 13, 2019, we experienced an elevated rate of transaction failures across different processing networks due to an infrastructure operation. Traffic is currently back to normal and transactions are processing successfully.

An elevated rate of Processor Network Unavailable (3000) responses for transactions.

Engineers are investigating internally.
Posted Aug 13, 2019 - 15:32 UTC
This incident affected: Transaction Processing (United States Processing, Canadian Processing, European Processing, APAC Processing).