During the incident window between 17:56 and 18:18 UTC on 12 December 2019, an elevated rate of merchant traffic received connection timeout errors when attempting to call the Gateway API. This includes approximately 100% of Pay with Venmo traffic. Merchants hosted within US AWS regions may have been unable to reach the Gateway API.
At 17:56 UTC, we encountered a hardware failure for one of our load balancers servicing one of our data centers. This load balancer became unable to service traffic, which is typically quickly resolved by automatic failover mechanisms. In this case, the automatic failover mechanisms did not trigger and engineers had to manually intervene to mitigate the issue at 18:18 UTC.