Patching requests for fun and (concurrent) profit

Let's consider the following script. It runs a bunch of concurrent requests against a URL using the requests library, both with certificate verification enabled and disabled, and outputs the time it takes to do it in both cases.

What's the time difference between the two? It turns out it is highly dependent on your local configuration. On my local machine, with a relatively modern config (Python 3.12 + OpenSSL 3.0.2), the times are ~1.2s for verify=True and ~0.5s for verify=False.

It's a >100% difference, but we initially blamed it on cert verification not being trivial and taking some time. However, we observed even larger differences (>500%) in some of our environments, and decided to find out what was going on.

Our main use case for requests is running lots of requests concurrently, and we spent some time bisecting this oddity to see if there was room for a performance optimization.

The issue is a bit more clear after profiling the concurrent executions. When verifying certs, these are the top 3 function calls by total time spent in them among all threads:

Leave a Comment