crawley
A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.
Free308
Free308
crawley
A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.
Data verified May 2026
ADYour product here. Reach security decision-makers.Launch a campaigncrawley Description
Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection. The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources. Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface. The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.
crawley FAQ
Common questions about crawley including features, pricing, alternatives, and user reviews.
crawley is A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection. It is a Attack Surface solution designed to help security teams with Network Discovery, Reconnaissance, Security Tools.
crawley is a free Attack Surface tool. This makes it accessible for organizations of all sizes, from startups to enterprises. Visit https://github.com/s0rg/crawley/ for download and installation instructions.
Popular alternatives to crawley include:
1.Censys Internet Map - Internet intelligence platform for asset discovery and attack surface mapping (Commercial)
2.Vulneri ASM - ASM platform for continuous discovery and risk validation of internet-exposed assets. (Commercial)
3.Cylana EASM - AI-powered EASM platform for digital asset discovery and monitoring. (Commercial)
4.Attaxion - Agentless EASM platform for asset discovery, exposure mgmt & risk reduction. (Commercial)
5.AttackSurfaceMapper - Automate your reconnaissance process with AttackSurfaceMapper, a tool for mapping and analyzing network attack surfaces. (Free)
Compare all crawley alternatives at https://cybersectools.com/alternatives/crawley
crawley is for security teams and organizations that need Network Discovery, Reconnaissance, Security Tools. It's particularly suitable for small to medium-sized teams looking for cost-effective solutions. Other Attack Surface tools can be found at https://cybersectools.com/categories/attack-surface
Have more questions? Browse our categories or search for specific tools.
