How to break production on Black Friday

submited by

Style Pass

2024-11-25 10:00:03

Users won’t do this! No way! Why would they enter such values? It doesn’t make sense! Does it sound familiar? How many times have you said it to yourself?

Black Friday and Cyber Monday are approaching. Preparations for this time of the year are underway. Business and tech departments are working together to boost your company’s sales. System admins, site readability engineers (SREs), and developers are on high alert.

So far, so good. Traffic to services is growing, sales are growing. Infrastructure and business services are performing as predicted. Your Slack channels are full of “Congrats to the team” and “excellent job, guys” messages. Happy days!

Meanwhile, your boss, encouraged by the higher sales, decided to launch a few more web services to boost sales further. In the end, the engineers tested and verified the new software. They didn’t find any issues. You are confident that the team can go ahead.

You run production and test infrastructure in many Kubernetes clusters. You know that the new services you are about to deploy require API keys and secrets for authentication.