The trouble with openness | anderegg.ca

Back in the early 2000s, there was this nebulous idea called the semantic web. It never really went anywhere, but I found it exciting at the time.

One piece that particularly spoke to me was the notion of including data in websites so that web scrapers could easily get at it. This was supposed to make the web more open and interoperable. The problem was that the data needed to be well structured and rigorously defined. Only a few nerds cared to put in the effort for that, and even those nerds had endless arguments about how it should work.

Fast forward a bit and a few things happened. First, the hippie optimism of the early web wore off. Few people wanted their data to be “open”, and most large companies took pains to keep their data to themselves. Second, APIs took off in a huge way. If you were going to share data with people, it was likely to be using an API rather than to encode it directly in the website content. This provided more control over who could access the data. Third, even most of the nerds from earlier gave up on building websites in a rigorous way. It became easier to use query selectors or natural language tools to scrape data from the few sites that would still let you.

The semantic web was an idealistic dream, but it was a nice one for a time. Now things are much more closed, for better and worse.

Leave a Comment