*** PagerDutyBot: SEV-0: GLOBAL CONTROL PLANE UNREACHABLE. API ERROR RATE > 95%.
*** sev_manager has joined
<sev_manager>
Status?
<net_ops_jen>
It’s bad. Traffic to the auth service just verticalized. We’re seeing 50M RPS.
<sev_manager>
DDoS? Scrubbing center active?
<net_ops_jen>
That’s the weird part. It’s bypassing the WAF. It looks like legit traffic. TLS fingerprints are valid.
<sev_manager>
Did someone let Tyler push a new WAF regex? Please tell me we didn’t backpedal into 2019.
<backend_bob>
I’m looking at the sample logs. These are all authenticated requests.
<backend_bob>
They are all hitting
POST /api/v4/user/token/refresh.
<sev_manager>
Why is everyone refreshing their token at the exact same second?
<net_ops_jen>
It’s not one refresh. I’m seeing the SAME user IDs hitting it 500 times per second.
<frontend_felix>
Uh, guys?
<sev_manager>
Go ahead Felix.
<frontend_felix>
We pushed the “Seamless Session” update to the dashboard 15 minutes ago.
<frontend_felix>
The goal was to silently refresh the token in the background so users don’t get logged out.
<backend_bob>
Felix… look at the code.
<frontend_felix>
I am.
<frontend_felix>
…
<backend_bob>
Felix?
<frontend_felix>
Oh no.
<sev_manager>
REPORT.
<frontend_felix>
Okay, so, in React…
useEffect runs when a dependency changes.
<frontend_felix>
We have:
useEffect(() => { refreshToken() }, [token])
<backend_bob>
And
refreshToken()… updates the token?
<frontend_felix>
Yes.
<backend_bob>
Which triggers the
useEffect again?
<frontend_felix>
Yes.
<backend_bob>
So you turned every single customer’s browser tab into an infinite loop cannon pointed at our auth servers?
<frontend_felix>
In my defense, the tokens are incredibly fresh.
<sev_manager>
Rollback the frontend.
<sev_manager>
I’ll start writing another ridiculously well edited postmortem blog post.
<backend_bob>
Hackernews and primeagen is going to going to love this one.