Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection. The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources. Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface. The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.