During the incident window, impacted merchants experienced an elevated rate of failed Processor Network Unavailable (3000) responses for Pay with Venmo transactions. Consumers may have been unable to complete their transactions using Pay with Venmo. Other payment methods were not impacted.
A configuration change to use a new internet proxy was initiated on a gateway connection. At 19:15 UTC, the change was applied in a subsequent production deploy and no widespread issues were initially detected.
At 20:11 UTC, a wrong proxy usage was identified. While investigating, the earlier configuration change was determined to be incorrect, and as a result, the proxy was not properly used. Our monitoring also alerted us to an increased rate of Pay with Venmo failures and engineers were engaged. Engineers quickly initiated a rollback which completed at 21:35 UTC. Errors immediately subsided, restoring the Pay with Venmo service.