crawley
The unix-way web crawler Crawley is a simple web crawler written in Go. Features: * Supports multiple protocols (http, https, ftp, sftp) * Supports multiple authentication methods (basic auth, digest auth, kerberos) * Supports crawling of nested resources * Supports filtering of crawled resources * Supports saving of crawled resources to disk Example usage: ```go import "github.com/s0rg/crawley" func main() { crawler := crawley.NewCrawler() crawler.AddURL("https://example.com") crawler.Start() } ```
FEATURES
ALTERNATIVES
Akamai Enterprise Application Access is a ZTNA solution that provides secure, identity-based access to private applications without exposing the network.
A specialized packet sniffer for displaying and logging HTTP traffic, designed to capture, parse, and log traffic for later analysis.
A multi-threading tool for sniffing HTTP header records with support for offline and live sniffing, TCP flow statistics, and JSON output.
A collection of PCAPs for ICS/SCADA utilities and protocols with the option for users to contribute.
Unfurl is a URL analysis tool that extracts and visualizes data from URLs, breaking them down into components and presenting the information visually.
A daemon for blocking USB keystroke injection devices on Linux systems
Open source software for leveraging insights from flow and packet analysis to identify potential security threats or attacks.
PINNED
Mandos Brief Newsletter
A weekly newsletter providing cybersecurity leadership insights, industry updates, and strategic guidance for security professionals advancing to management positions.
PTJunior
An AI-powered penetration testing platform that autonomously discovers, exploits, and documents vulnerabilities while generating NIST-compliant reports.
OSINTLeak
OSINTLeak is a tool for discovering and analyzing leaked sensitive information across various online sources to identify potential security risks.
ImmuniWeb® Discovery
ImmuniWeb Discovery is an attack surface management platform that continuously monitors an organization's external digital assets for security vulnerabilities, misconfigurations, and threats across domains, applications, cloud resources, and the dark web.
Checkmarx SCA
A software composition analysis tool that identifies vulnerabilities, malicious code, and license risks in open source dependencies throughout the software development lifecycle.
Orca Security
A cloud-native application protection platform that provides agentless security monitoring, vulnerability management, and compliance capabilities across multi-cloud environments.
DryRun
A GitHub application that performs automated security code reviews by analyzing contextual security aspects of code changes during pull requests.