An effort to clean up merchant level users in Togetherpay led to the archiving of inactive users - those that had not logged in for over one year and those that have never logged in. Identified user records were archived. This inadvertently broke authentication for Togetherpay merchant API calls, because some of the archived user records were actually merchant API keys.
Work to restore the archived users led to additional issues including database connectivity errors and replica host failures.
The incident was resolved by reverting the affected user records and fixing database replica configurations.
Teams evaluated the sequence of events and implemented steps to prevent such events from happening in the future.