The Mysterious Gotcha of gRPC Stream Performance
Here at Ably, we deal with hard engineering problems all the time and pushing at the edges often results in us running up against all sorts of interesting gotchas. We recently made some AWS NLBs fall over, and had a hazardous encounter with Cassandra counter columns.
In our day-to-day, we use gRPC for fast and efficient data exchange with mutual client/server state synchronization. It’s generally the right tool for the job, but we have come across some performance issues that left us scratching our heads.
Recently, we spent longer than planned trying to work out why the performance of a gRPC streaming server was worse than expected. Given how commonly used gRPC is, we expected lots of blog posts and Stack Overflow questions, but most of the performance analysis out there was focused on unary requests and not streams. So we rolled up our sleeves and went sleuthing in-house.
gRPC was developed by Google as an open source Remote Procedure Call (RPC) framework designed to connect services in and across data centers. It supports load balancing, tracing, health checking, and authentication. Due to its high-performance design and common language support, it has widespread use for connecting up microservices which is the thing we use it for. It is language agnostic and lightweight so is just as suited to connecting end devices to cloud servers. It uses HTTP2 as transport protocol and Protocol Buffers as the interface designation.
