crawley

A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.

308

Attack Surface

0

crawley

A Go-based web crawler that supports multiple protocols and authentication methods for systematic web resource discovery and collection.

Attack Surface•External Attack Surface Management

Network Discovery Recon Asset Discovery

308stars

GitHub Stars

15 Aug

Last commit

0

Updated 12 Oct 25

crawley Description

Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection. The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources. Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface. The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.

crawley Description

Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection. The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources. Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface. The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.

FEATURED

Password manager with end-to-end encryption and identity protection features

VPN service providing encrypted internet connections and privacy protection

Mandos Fractional CISO Services

Fractional CISO services for B2B companies to accelerate sales and compliance

Stay Updated with Mandos Brief

Get the latest cybersecurity updates in your inbox

TRENDING CATEGORIES

Digital Forensics and Incident Response

Digital Forensics and Incident Response (DFIR) tools for digital forensic analysis, evidence collection, malware analysis, and cyber incident investigation.

Offensive Security

Offensive security tools for penetration testing, red team exercises, exploit development, and ethical hacking activities.

Honeypots & Deception

Honeypots and cyber deception solution that simulate vulnerable systems to detect, divert, and analyze attacker activities in real time.

Threat Intelligence Platforms

TIP for collecting, analyzing, and sharing cyber threat data, indicators of compromise (IOCs), and threat feeds.

Comprehensive cybersecurity guides, best practices documentation, and implementation tutorials for security professionals.

View All Categories →

POPULAR

Automated vulnerability assessment and remediation platform

10

Security platform that provides protection, monitoring and governance for enterprise generative AI applications and LLMs against various threats including prompt injection and data poisoning.

5

A threat intelligence aggregation service that consolidates and summarizes security updates from multiple sources to provide comprehensive cybersecurity situational awareness.

5

Fabric Platform by BlackStork

Fabric Platform is a cybersecurity reporting solution that automates and standardizes report generation, offering a private-cloud platform, open-source tools, and community-supported templates.

5

Mandos Brief Newsletter

A weekly newsletter providing cybersecurity leadership insights, industry updates, and strategic guidance for security professionals advancing to management positions.

5

View Popular Tools →