“Take the first step in faith. You don’t have to see the whole staircase. Just take the first step.” — Dr. Martin Luther King Jr. PayPal has e

PayPal’s Data Warehouse Migration to Google BigQuery

submited by

Style Pass

2021-05-28 20:30:07

“Take the first step in faith. You don’t have to see the whole staircase. Just take the first step.” — Dr. Martin Luther King Jr.

PayPal has experienced record growth since the beginning of the global pandemic. To keep up with the demand from growth, we decided to migrate PayPal Analytics platforms to the public cloud. The first big migration of a warehouse workload to BigQuery in Google Cloud took less than a year. Along the way, the PayPal team built a platform which would support many other use cases as well.

This writeup captures a milestone migration experience. We migrated half of our data and processing from Teradata systems to Google Cloud Platform’s BigQuery.

As organizations and consumers ventured into new ways of doing business during the pandemic, PayPal experienced record-high transaction volumes. This put a lot of pressure on the offline analytics systems used for compliance, risk processing, product and financial analytics, marketing, customer success, and fraud protection. These analytics systems were all in on-premises data centers. The systems were powered by Teradata and Hadoop at the core, with additional software and workflows in place to manage resources across these systems.

Demands for processing data were far outstripping the existing capacity on-premises. Adding capacity quickly during the pandemic had its own share of challenges. Data platform teams managed the crisis with manual intervention to prioritize various workloads that demanded additional processing time. Given the business outlook of continuing growth, PayPal realized that the analytics ecosystem required a change.