Pay with Venmo Processing Issues
Incident Report for Braintree
Postmortem

Impact Description

During the incident window, impacted merchants experienced an elevated rate of failed Processor Network Unavailable (3000) responses for Pay with Venmo transactions. Consumers may have been unable to complete their transactions using Pay with Venmo. Other payment methods were not impacted.

Root Cause

A configuration change to use a new internet proxy was initiated on a gateway connection. At 19:15 UTC, the change was applied in a subsequent production deploy and no widespread issues were initially detected.

At 20:11 UTC, a wrong proxy usage was identified. While investigating, the earlier configuration change was determined to be incorrect, and as a result, the proxy was not properly used. Our monitoring also alerted us to an increased rate of Pay with Venmo failures and engineers were engaged. Engineers quickly initiated a rollback which completed at 21:35 UTC. Errors immediately subsided, restoring the Pay with Venmo service.

Corrective Actions & Preventative Measures

  • We are working to establish better testing and alerts around internet proxy changes.
  • More granular monitoring for different payment methods in line with our incident thresholds will be implemented and surfaced for subsequent gateway deployments.
  • Alert timing and context will also be improved to inform rapid decision making in response to potential outages.
  • We are also working to improve knowledge sharing around dependencies to reduce the recurrence of a similar issue.
Posted Mar 27, 2020 - 17:07 UTC

Resolved
A fix was successfully implemented and no further impact observed since 21:35 UTC.
Posted Mar 24, 2020 - 21:43 UTC
Update
The issue has been identified and a fix is currently being implemented.
Posted Mar 24, 2020 - 21:08 UTC
Identified
We're currently investigating an elevated rate of Pay with Venmo transaction failures. Other payment methods are not impacted. An update will be provided as soon as possible.

Symptoms
An elevated rate of Processor Network Unavailable (3000) responses for Pay with Venmo transactions.

Cause
Engineers are investigating internally.
Posted Mar 24, 2020 - 20:48 UTC
This incident affected: Transaction Processing (United States Processing).