A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.
Checked October 2026.
Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection.
The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources.
Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface.
The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.
No core features are listed for this product yet, so check the vendor's documentation for the full feature list.
No integrations are listed for this product yet. Check the vendor's documentation, or ask the vendor which integrations and APIs it supports.
Common questions about crawley: features, pricing and alternatives.
crawley is a product in the Security Scanning category. It covers Network Discovery, Reconnaissance and Security Tools.
crawley is free. Download and setup instructions are at https://github.com/s0rg/crawley/.
Alternatives to crawley listed on CybersecTools:
Compare all crawley alternatives at https://cybersectools.com/alternatives/crawley
crawley is available as on-premises software. It is listed for startups, small businesses, mid-market companies and enterprises. It is free.
Side by side: features, pricing model and the company behind each one.
CorsMe is a specialized scanner that identifies Cross-Origin Resource Sharing (CORS) misconfigurations in web applications and provides remediation recommendations.
A security scanner that identifies Cross-Origin Resource Sharing (CORS) misconfigurations in web applications to detect potential vulnerabilities.
A Python-based command-line tool that scans websites for CORS misconfigurations by analyzing HTTP response headers to identify potential security vulnerabilities.
Command line tool for testing CRLF injection on a list of domains.
CryptoLyzer is a cryptographic protocol analyzer that examines TLS, SSL, SSH, and DNSSEC server implementations with fingerprinting capabilities and multiple output formats.
Automated web vulnerability scanner with 60+ security checks
Get a summary of this page from your assistant.
Mandos MCP: ask about all 9,033 products.
Connect your AI
