Transaction Processing Issues
Incident Report for Braintree
Postmortem

Impact Description

During the incident window, we experienced an intermittent outage across different processing networks which resulted in unsuccessful transactions. Merchants processing in Europe, Australia, Canada, and New Zealand received an elevated rate of Processor Network Unavailable (3000) responses for the failed transactions.

Root Cause

An infrastructure operation was being carried out to restore the SSH File Transfer Protocol (SFTP) connection for a specific region. During this operation, several processor-connecting hosts were incorrectly taken out of service. As a result, processor networks on these hosts became unavailable, resulting in failed transactions. Engineers were immediately notified, quickly restored these processor-connection hosts, and traffic recovered.

Corrective Actions & Preventative Measures

  • Engineers immediately restored the servers and prevented ongoing transaction impact at 15:12:52 UTC.
  • Safeguards have been implemented for future processor operations to reduce the likelihood of accidental outages.
Posted 5 days ago. Aug 16, 2019 - 21:13 UTC

Resolved
Between 15:10:31 - 15:12:52 UTC on Aug 13, 2019, we experienced an elevated rate of transaction failures across different processing networks due to an infrastructure operation. Traffic is currently back to normal and transactions are processing successfully.

Symptoms
An elevated rate of Processor Network Unavailable (3000) responses for transactions.

Cause
Engineers are investigating internally.
Posted 8 days ago. Aug 13, 2019 - 15:32 UTC
This incident affected: Transaction Processing (United States Processing, Canadian Processing, European Processing, APAC Processing).