Search code, repositories, users, issues, pull requests...
This very useful database is somewhat challenging to use locally because of its sheer size. These utiliities make it easy and fast to deal with the large data volume while being very effiicent on disk and memory resouces.
Here is hibp-download running on a 400Mbit/s connection, averaging ~48MB/s which is very close to the theorectical maximum. At this network speed, a download of the entire HIBP database, including prefixing and joining the over 1 million files, converting to our binary format and writing to disk, takes under 12 minutes.
Download "Have I been pawned" database. 38GB download, uses 21GB of disk space and takes ~6/12 minutes on 1Gbit/400Mbit connection. Detailed progress is shown.
The output will be the number of times that pasword has appeared in leaks. Integrate this into your signup and login processes to show warnings to the user that they using a compromised password.
These utilities are designed to have a very modest resource footprint. By default, they use a binary format to store and search the data. The orginal text records are about 45 bytes per password record, our binary format is 24 bytes, so storage requirements are almost halved (21GB currently).
