Bug which took months to debug…
So you want to hear about debugging hell? Buckle up, because I’m about to tell you the story of a bug that took few months. The solution was embarrassingly simple once we found it.
Let me paint you a picture of our delightfully over-engineered system. We had this web service running on Tomcat that would take UI filters and convert them into Spark-compatible SQL. Think of it as a translation layer between human-readable queries and Spark’s particular brand of SQL masochism.
Instead of running the SQL directly in the same Java program (which would have been easier), we had to send it to a separate Jetty service running on a Unix system where the actual Spark session was running. This was done to keep things separate and because the Spark session and the web server used different runtime libraries (JAR files). Also, running Spark in different modes like cluster or client added complexity. To keep it simple, the Jetty service was set up to handle requests on top of the Spark session. The reasons for this setup included avoiding JAR conflicts, handling caching properly, and maybe someone trying to use microservices before they became popular.
For other databases (DB2, Greenplum, SQL Server), we kept it simple and ran queries directly. But Spark? Oh no, Spark had to be special.
