500 HTTP Errors being returned for services IDMatch5.0.fullreturn and IDMatchComply5.1.SSN
Incident Report for Veratad
Postmortem

Issue

The 500 HTTP errors were the result of a deployment that occurred at the following production servers on the approximate timestamps. 

s0000: 2021-09-24 12:16:22

s0001: 2021-09-24 13:30:33 

s0002: 2021-09-24 13:43:02

s0005: 2021-09-24 13:51:24

Impact

We can assert that there were 327 errors from 2021-09-24 12:20:59 to 2021-09-24 20:06:30. This lines up approximately with the number of IDMatch5.0.fullreturn, IDMatchComply5.1.SSN transactions that were run after each server was updated (345). The difference is mostly likely due to not knowing exactly when each server patch was applied.

Resolution

We will add functional tests for IDMatch5.0.fullreturn and IDMatchComply5.1.SSN.

Posted Sep 29, 2021 - 14:29 UTC

Resolved
Users were receiving 500 HTTP errors from two Veratad services:

- IDMatch5.0.fullreturn
- IDMatchComply5.1.SSN
Posted Sep 24, 2021 - 18:00 UTC