dennis-tra / nebula-crawler

A libp2p DHT crawler that gathers information about running nodes in the network. Starting with a set of bootstrap nodes it recursively asks all peers for their DHT neighbors and follows them until all peers were dialed.

The crawler is in a working state as it's successfully visiting and following all nodes in the network. However, the project is very young and thus has its sharp edges here and there in the codebase and documentation. Most importantly, the gathered numbers are in line with existing data like the wiberlin/ipfs-crawler. This crawler also powers a Grafana dashboard which can be found here.

Nebula is a command line tool and provides the three sub-commands crawl, monitor and daemon. See the command line help page below for configuration options:

The crawl sub-command starts by connecting to a set of bootstrap nodes and constructing the routing tables (kademlia k-buckets) of the remote peers based on their PeerIds. Then nebula builds random PeerIds with a common prefix length (CPL) and asks each remote peer if they know any peers that are closer to the ones nebula just constructed. This will effectively yield a list of all PeerIds that a peer has in its routing table. The process repeats for all found peers until nebula does not find any new PeerIds.

Leave a Comment