# crawley

crawley is a Security Scanning product. It runs on-premises. Pricing is free.

A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.

Checked October 2026.

## What is crawley?

Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection.

The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources.

Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface.

The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.

Tags: Network Discovery, Reconnaissance, Security Tools

## Where it fits in your security program (NIST CSF 2.0)

- [Identify](https://cybersectools.com/guides/building-identify-program-small-team): helps you know your assets and your risks.

## Quick facts

- Category: [Exposure & Vulnerability Management](https://cybersectools.com/categories/exposure-vulnerability-management) · [Security Scanning](https://cybersectools.com/categories/security-scanning)
- Deployment: On-Premises
- Company size fit: Startup, SMB, Mid-Market, Enterprise
- Pricing model: Free
- GitHub stars: 308
- Website: github.com

## crawley FAQ

### What is crawley?

crawley is a product in the Security Scanning category. It covers Network Discovery, Reconnaissance and Security Tools.

### What is the pricing for crawley?

crawley is free. Download and setup instructions are at https://github.com/s0rg/crawley/.

### What are alternatives to crawley?

Alternatives to crawley listed on CybersecTools:

1. [DNSAudit.io](https://cybersectools.com/tools/dnsauditio) - Free DNS security scanner that checks domains for misconfigs and exposure. (Free)
2. [RoboShadow Mobile Network Scanner](https://cybersectools.com/tools/roboshadow-mobile-network-scanner) - Android app for scanning networks to identify security vulnerabilities (Free)
3. [Aikido EOL Scanner](https://cybersectools.com/tools/aikido-eol-scanner) - Detects end-of-life and outdated software in code and containers (Commercial)
4. [Jsmon 2.0](https://cybersectools.com/tools/jsmon-20) - JavaScript security scanner for detecting vulnerabilities in third-party scripts (Commercial)
5. [Sucuri Website Malware Scanner](https://cybersectools.com/tools/sucuri-website-malware-scanner) - Website malware scanner with remote & server-side scanning capabilities (Commercial)

Compare all crawley alternatives at [https://cybersectools.com/alternatives/crawley](https://cybersectools.com/alternatives/crawley)

### What deployment and organization size does crawley fit?

crawley is available as on-premises software. It is listed for startups, small businesses, mid-market companies and enterprises. It is free.

## Compare crawley with similar solutions

- [crawley vs DNSAudit.io](https://cybersectools.com/compare/crawley-vs-dnsauditio)
- [crawley vs RoboShadow Mobile Network Scanner](https://cybersectools.com/compare/crawley-vs-roboshadow-mobile-network-scanner)
- [crawley vs Aikido EOL Scanner](https://cybersectools.com/compare/aikido-eol-scanner-vs-crawley)

All alternatives to crawley: [/alternatives/crawley](https://cybersectools.com/alternatives/crawley)

---

Source: https://cybersectools.com/tools/crawley

For AI agents: every CybersecTools shortlist is listed at https://cybersectools.com/shortlists.md, and any compare page has a Markdown copy too (add .md to its URL).

More on crawley: funding, momentum, hiring and news, plus the full product and company record, are available through the Mandos MCP Server (paid plans), which connects to AI assistants: https://mandos.io/mcp?utm_source=cybersectools.com&utm_medium=product-md