“Most information is irrelevant and most effort is wasted, but only the expert knows what to ignore.” — James Clear, Atomic Habits You have a fa

The Impedance Mismatch Test: Is Your Data Layer a Complex Mess?

submited by

Style Pass

2021-07-22 20:30:09

“Most information is irrelevant and most effort is wasted, but only the expert knows what to ignore.” — James Clear, Atomic Habits

You have a fancy data pipeline with lots of different systems. It looks very sophisticated on the surface, but it’s actually a complex mess under the hood. It might need a lot of plumbing work to connect different pieces; it might need constant monitoring; it might require a large team with unique expertise to run, debug and manage it. The more systems you use, the more places you are duplicating your data and the more chances of it going out of sync or stale. Furthermore, since each of these subsystems is developed independently by different companies, their upgrades or bug fixes might break your pipeline and your data layer.

If you aren’t careful, you might end up with the following situation as depicted in the three-minute video below. I highly recommend you watch it before you proceed.