A Practical Method for Testing Differential Privacy
At Sarus, we build privacy-safe analytics tools, including an open-source library called Qrlew, designed to rewrite SQL queries into Differential Privacy (DP). Qrlew’s methodology has been peer-revewed and its code is open for anyone to check. To further reinforce its reliability and prevent future code-changes to break DP guarantees, Sarus decided to build an empirical testing framework.
Our approach bridges the gap between theory and application, introducing methods that enable developers to validate DP mechanisms effectively. Leveraging the adversary’s perspective, hypothesis testing, and empirical approximations, we show how to test privacy mechanisms with SQL queries and real-world datasets. By simplifying dataset creation, partitioning result into buckets and computing the empricial privacy loss, our methods provide a robust framework for DP validation.
We also introduce the dp_tester library, which implements this testing technique, making it easy to verify your DP mechanisms. Using interactive experiments in a ready-to-run Jupyter Notebook in colab, developers and researchers can empirically assess privacy guarantees with minimal effort.
