[SEV-1] Togetherpay Production Outage | Transaction Failures
Incident Report for Togetherpay
Resolved
At 11:51 a.m. eastern, CPU usage spiked to 100%. At 2:18 p.m. eastern the instance was rebooted to free up resources. Intermittent transaction processing issues could have occurred between 11:51 a.m. - 2:30 p.m. eastern with all transactions failing during a 10 window during the reboot (2:18 p.m. - 2:28 p.m. eastern). CPU usage is hovering around 90% at the moment. IT Ops plans to size up the database this evening to handle the load. A separate maintenance event will be posted with additional details about that event.
Posted Oct 18, 2022 - 16:23 EDT
Update
Teams continue to investigate this issue and monitor production performance.
Posted Oct 18, 2022 - 15:16 EDT
Update
Systems are recovering and teams are monitoring.
Posted Oct 18, 2022 - 14:29 EDT
Update
IT Ops is in the process of rebooting the system.
Posted Oct 18, 2022 - 14:22 EDT
Investigating
We are experiencing an issue that may impact your ability to process transactions. We are working as quickly as possible to identify and resolve the issue. Additional information will be provided when known, or when the situation is resolved.
Posted Oct 18, 2022 - 14:15 EDT
This incident affected: Togetherpay API - Production (Transaction Processing, Payment Tokenization, Merchant Endpoints).