Go Profiling in Production

At Oodle, we use Golang for our backend services. In an earlier blog post (Go faster!), we discussed optimization techniques for performance and scale, with a specific focus on reducing memory allocations. In this post, we'll explore the profiling tools we use. Golang provides a rich set of profiling tools to help uncover code bottlenecks.

We use the standard net/http/pprof package to register handlers for profiling endpoints. As part of our service startup routine, we run a profiler server on a dedicated port.

CPU profiling helps understand where CPU time is being spent in the code. To collect a CPU profile, we can hit the /debug/pprof/profile endpoint with a seconds parameter.

This command starts a local HTTP server and opens the profile in the browser. We frequently use [flame graphs] (https://www.brendangregg.com/flamegraphs.html) for profile analysis.

As a multi-tenant SaaS platform, Oodle serves many tenants from the same deployment. Ingestion and query loads vary significantly among tenants. To understand CPU profiles for individual tenants, we use [profiler labels] (https://rakyll.org/profiler-labels/) to add tenant identifiers to the profiles.

Leave a Comment