Data Gateway — A Platform for Growing and Protecting the Data Tier

The Netflix Online Datastore team has built a platform we call the Data Gateway to enable our datastore engineers to deliver powerful data abstractions which protect Netflix application developers from complex distributed databases and incompatible API changes. In this opening post, we cover the platform as the first part of a series which shows how we use this platform to raise the level of abstraction that application developers use every day to create, access, and maintain their online data.

At Netflix, we have adopted and contributed to a large number of open-source (OSS) technologies and databases in the data tier, including Apache Cassandra, EVCache (memcached), OpenSearch, and more. Historically, the Online Data Platform operated these datastores and provided their OSS API via client libraries. For example, we operated Apache Cassandra clusters and provided developers client libraries based on Thrift or Cassandra Query Language (CQL) protocols. This strategy gave Data Platform leverage because OSS meant fewer engineers could operate more types of databases for a larger number of users. However, while this enabled rapid expansion, coupling applications to a multitude of APIs Netflix did not control had significant maintenance costs in the long-run.

Most databases have a large API surface area and few guardrails, which result in a number of usage anti-patterns that require advanced knowledge to avoid and may take years to be revealed. For example, developers must avoid writing too much data to one row or field, and the limits change for every datastore. As Netflix’s engineering organization grew and the number of use cases proliferated, engineers experienced more burden from mitigating database misuse and re-designing applications. This also increased the risk of product outages as most critical applications rely on database services and database migrations are intrinsically hazardous.

Leave a Comment