Python Concurrency: The Tricky Bits
An exploration of threads, processes, and coroutines in Python, with interesting examples that illuminate the differences between each.
As a data scientist who is spending more time on software engineering, I was recently forced to confront an ugly gap in my knowledge of Python: concurrency. To be honest, I never completely understood how the terms async, threads, pools and coroutines were different and how these mechanisms could work together. Every time I tried to learn about the subject, the examples were a bit too abstract for me, and I hard time internalizing how everything worked.
Because of restrictions with this YouTube video, I couldn’t embed the video in this article, so you will have to open it in a different window.
This talk is incredibly intimidating at first. Not only is it coded live from scratch, but it also jumps immediately into socket programming, something that I had never encountered as a data scientist. However, if you go through it slowly and understand all the components (as we do in this blog post) it turns out to be the best educational material on Python concurrency I have ever encountered. This blog post documents what I learned along the way so others can benefit, too.
To demonstrate concurrency, it is useful to create a task that can saturate your CPU (such as mathematical operations) for a noticeable period of time. David uses a function that computes a Fibonacci number.
