Katana: web crawler for offensive security and web exploration – Trickster Dev
Katana is CLI tool and Go library for automatically traversing (crawling) across all pages of given website(s) to map them out. It can work in two main modes - requests-based and through browser automation (headful or headless). To allow for discovery of API endpoints it can optionally do JavaScript parsing even when running in requests-based mode. Furthermore, Katana can do passive crawling by leveraging pre-crawled data from Internet Archive Wayback Machine, CommonCrawl and Alien Vault OTX. Since mapping out site pages and APIs is useful for security research activities (e.g bug bounty hunting) Katana is designed to fit into larger automation workflows, esp. when used together with other tooling from Project Discovery.
On macOS Katana is available through Homebrew and there is also official Docker image on Docker Hub that includes a Chromium browser.
But now we don’t see status codes and we’re not even sure if all of these pages load properly. We can use -mdc argument with Project Discovery DSL snippet to filter out only responses with status code 200:
