crawley
The unix-way web crawler Crawley is a simple web crawler written in Go. Features: * Supports multiple protocols (http, https, ftp, sftp) * Supports multiple authentication methods (basic auth, digest auth, kerberos) * Supports crawling of nested resources * Supports filtering of crawled resources * Supports saving of crawled resources to disk Example usage: ```go import "github.com/s0rg/crawley" func main() { crawler := crawley.NewCrawler() crawler.AddURL("https://example.com") crawler.Start() } ```
FEATURES
EXPLORE BY TAGS
SIMILAR TOOLS
A suite for man in the middle attacks, featuring sniffing of live connections, content filtering, and protocol dissection.
WiGLE.net is a platform that collects and provides data on WiFi networks and cell towers, with over 1.3 billion networks collected.
Unfurl is a URL analysis tool that extracts and visualizes data from URLs, breaking them down into components and presenting the information visually.
A free, open-source network protocol analyzer for capturing and displaying packet-level data.
A Bluetooth 5 and 4.x sniffer using TI CC1352/CC26x2 hardware with advanced features and Python-based host-side software.
Suricata offers real-time intrusion detection, intrusion prevention, and network monitoring.
A website scanner that provides a sandbox for the web, allowing users to scan URLs and websites for potential threats and vulnerabilities.
A Linux command-line tool that allows you to kill in-progress TCP connections based on a filter expression, useful for libnids-based applications that require a full TCP 3-way handshake for TCB creation.
PINNED
Proton Pass is a cross-platform password manager that provides encrypted storage, password generation, and security monitoring features with integrated 2FA and dark web monitoring capabilities.
NordVPN is a commercial VPN service that encrypts internet connections and hides IP addresses through a global network of servers, featuring integrated threat protection and multi-device support.
Fractional CISO service that helps B2B companies implement security leadership to win enterprise deals, achieve compliance, and develop strategic security programs.
Checkmarx SCA
A software composition analysis tool that identifies vulnerabilities, malicious code, and license risks in open source dependencies throughout the software development lifecycle.
Orca Security
A cloud-native application protection platform that provides agentless security monitoring, vulnerability management, and compliance capabilities across multi-cloud environments.
DryRun
A GitHub application that performs automated security code reviews by analyzing contextual security aspects of code changes during pull requests.