A Hockey Stick found due to ChatGPT Outage => Double Traffic AIPRM => AIPRM Outage

As often OpenAI ChatGPT has some outages,

that looks like this at the moment.

This seems to lead to an average of 1 reload of all active users.

This doubled the traffic to AIPRM immediately.

While there are many optimizations in place, with a young system like AIPRM some lesser needed functions were untouched since January.

These logging functions suddenly caused a hockey-stick or at least exponential time delay behavior, slowing down everything else.

For the technically interested, this non-linearity in response times can happens in all system, at some point very surprised - as was the case here too. Nonlinear problems like that are described here also.

where I took this example illustration of a hockey stick curve on response times from

AIPRM service is restored, but at the time of writing this ChatGPT issue still exists:

This gives the impression that AIPRM has a problem - not anymore.

We continue looking into things, but service for users should be fully restored.

2 Likes

Just like ChatGPT being down partially still, we see another surprising large wave of traffic coming in,
causing temporary outage. We are working on it.

And another day of crazy traffic spikes just now, that we mitigate