Impact Description
Between 8:44 UTC and 9:46 UTC on 13 June 2019, impacted merchants encountered timeouts when attempting to reach Braintree’s API endpoint during the incident window. Some Control Panel users may have also had difficulty using features such as downloads.
Root Cause
Braintree routes traffic to the Gateway API via multiple internet service providers (ISPs) across multiple datacenters. One ISP servicing one of our datacenters had an upstream issue preventing inbound traffic from reaching our servers. While engineers were quickly alerted to the connectivity errors, it took some time to properly determine which ISP was having issues. Once engineers were able to confirm the impacted ISP, they removed both the ISP and the datacenter from serving public traffic and traffic levels and error rates recovered.
Corrective Actions & Preventative Measures
We are auditing and refining our automation around removing ISPs and datacenters from serving traffic (de-peering)
We are introducing higher-precision ISP-level monitoring into primary system dashboards
We are improving application resiliency during prolonged outbound connectivity issues
Symptoms
Timeouts or increased latency when making API calls.
Cause
An upstream networking issue with a third-party Internet Service Provider (ISP).