Wikidata : SPARQL query service/WDQS graph split

The Search team is currently running an experiment to split the Wikidata Query Service graph and use federation for the queries that need access to all subgraphs. This is a breaking change, which will require a number of queries to be rewritten, either to access a new SPARQL endpoint, or to use federation. We want to have a good understanding of the trade-offs before we commit to any long-term solution.

The underlying issue that we are trying to address is the medium term scalability and stability of Wikidata Query Service, which can hinder access to and the possibility to query the data in Wikidata. The Wikidata Query Service runs on top of Blazegraph, and comprises 15 billion triples. The graph is currently growing at the rate of 1 billion triples per year.

The ability to reload the graph is a critical function in order to ensure data consistency and be able to recover from potential critical data issues. It is an indication of the stability and scalability of the system. Furthermore, the instability of the data reload process is directly linked to the size of the graph, in a similar way that the runtime stability of WDQS is linked to the size of the graph.

Leave a Comment