crawley Logo

crawley

0
Free
308
15 Aug 2025
10 September 2025
Visit Website

Crawley is a web crawler written in Go that follows the unix-way philosophy for web resource discovery and collection. The tool supports multiple network protocols including HTTP, HTTPS, FTP, and SFTP for comprehensive web crawling capabilities. It provides various authentication methods such as basic authentication, digest authentication, and Kerberos to access protected resources. Key features include the ability to crawl nested resources recursively, filter crawled content based on specified criteria, and save discovered resources directly to disk for offline analysis. The crawler can be programmatically integrated into Go applications through its API interface. The tool is designed for security professionals who need to systematically discover and collect web resources for reconnaissance, asset discovery, or security assessment purposes.

FEATURES

SIMILAR TOOLS

Catalyst is a SOAR system that automates alert handling and incident response processes, adapting to your workflows and being open source.

IRIS-SOAR is a Python-based modular SOAR platform that automates security incident response workflows and integrates with DFIR-IRIS for enhanced digital forensics operations.

Fast Intercept is a security automation platform that empowers users to maximize their existing security products and automate routine tasks.

Automated Digital Forensics and Incident Response (DFIR) software for rapid incident response and intrusion investigations.

An open-source, drag-and-drop security workflow builder with integrated case management for automating security workflows and tackling alert fatigue.

Fabric Platform is a cybersecurity reporting solution that automates and standardizes report generation, offering a private-cloud platform, open-source tools, and community-supported templates.

JIMI is a flow-based orchestration automation platform that combines low-code and no-code capabilities for multi-team collaboration across IT, security, and development operations.

A community repository of workflow templates for the Ayehu NG platform that enables automated IT and business process execution.

Shuffle Automation provides an open-source platform for security orchestration, automation, and response.

PINNED

Proton Pass Logo

Proton Pass is a cross-platform password manager that provides encrypted storage, password generation, and security monitoring features with integrated 2FA and dark web monitoring capabilities.

Data Protection
NordVPN Logo

NordVPN is a commercial VPN service that encrypts internet connections and hides IP addresses through a global network of servers, featuring integrated threat protection and multi-device support.

Network Security
Mandos Logo

Fractional CISO service that helps B2B companies implement security leadership to win enterprise deals, achieve compliance, and develop strategic security programs.

Consulting
Checkmarx SCA Logo

Checkmarx SCA

A software composition analysis tool that identifies vulnerabilities, malicious code, and license risks in open source dependencies throughout the software development lifecycle.

Application Security
Orca Security Logo

Orca Security

A cloud-native application protection platform that provides agentless security monitoring, vulnerability management, and compliance capabilities across multi-cloud environments.

Cloud Security
DryRun Logo

DryRun

A GitHub application that performs automated security code reviews by analyzing contextual security aspects of code changes during pull requests.

Application Security
CybersecTools logoCybersecTools

Explore the largest curated directory of cybersecurity tools and resources to enhance your security practices. Find the right solution for your domain.

Operated by:

Mandos Cyber • KVK: 97994448

Netherlands • contact@mandos.io

VAT: NL005301434B12

Copyright © 2025 - All rights reserved