Pinterest Open-Sources a Production-Ready PubSub Java Client for Kafka, Flink, and MemQ
InfoQ Homepage News Pinterest Open-Sources a Production-Ready PubSub Java Client for Kafka, Flink, and MemQ
Pinterest open-sourced its generic PubSub client library, PSC, which has been heavily used in production for a year and a half. The library helped the engineering teams by increasing developer velocity, and the scalability and stability of services using it. Over 90% of Java applications have migrated to PSC with minimal changes.
Pinterest uses messaging infrastructure throughout its platform, including Apache Kafka, Apache Flink, and MemQ. Jeff Xiang, a software engineer at Pinterest, provides a summary of some of the challenges resulting from using different messaging backends:
Over the years, operational experience has taught us that our customers and business would greatly benefit from a unified PubSub interface that the platform team owns and maintains, so that application developers can focus on application logic instead of spending precious hours debugging client-server connectivity issues.
