First Data Processing Issues
Incident Report for Braintree
Postmortem

RCA: Elevated Visa/MC Declines in US & APAC

Date: Saturday, 10 March 2018 09:22 - 11:32

All Times in UTC

Executive Summary
Beginning at 09:22 on Saturday, 10th March, we observed an increase in the decline rate across all Visa and MasterCard transactions for some US and APAC merchants. Braintree engineers were engaged and were able to determine that the increased decline rate was not a result of any issues within the Braintree gateway. It was later discovered that one of our processing partners was experiencing issues resulting in an increased decline rate for Visa and MasterCard transactions.

The impacted processing partner deployed a fix at 11:30 and Braintree engineers observed decline rates return to normal at 11:32.

Follow-Up

  • We will be reviewing our monitoring and alerting processes around transaction decline rates.
  • For US merchants processing in USD, we will be deploying an authorization redundancy solution that will allow us to failover to a secondary authorization platform when our primary authorization partners have issues that impact Braintree’s ability to authorize transactions normally.
Posted 3 months ago. Mar 12, 2018 - 20:58 UTC

Resolved
The transaction decline rate has returned to normal levels. The full impact window was 4:22am to 6:32am CST.
Posted 3 months ago. Mar 10, 2018 - 12:40 UTC
Investigating
An issue upstream of our gateway is causing a higher than normal number of Visa and MasterCard transactions on the First Data network to decline. We are aware of the issue and are working urgently to resolve it as quickly as possible.

Scope: This issue affects some merchants in the United States and Asia. The Braintree gateway is currently up and accepting requests at this time.
Posted 3 months ago. Mar 10, 2018 - 12:25 UTC
This incident affected: United States Processing and APAC Processing.