crawley
The unix-way web crawler Crawley is a simple web crawler written in Go. Features: * Supports multiple protocols (http, https, ftp, sftp) * Supports multiple authentication methods (basic auth, digest auth, kerberos) * Supports crawling of nested resources * Supports filtering of crawled resources * Supports saving of crawled resources to disk Example usage: ```go import "github.com/s0rg/crawley" func main() { crawler := crawley.NewCrawler() crawler.AddURL("https://example.com") crawler.Start() } ```
FEATURES
EXPLORE BY TAGS
SIMILAR TOOLS
WiGLE.net is a platform that collects and provides data on WiFi networks and cell towers, with over 1.3 billion networks collected.
Netis Cloud Probe is an open source project for capturing and analyzing network packets across different machines.
Netcap efficiently converts network packets into structured audit records for machine learning algorithms, using Protocol Buffers for encoding.
A technique to associate applications with TLS parameters for identifying malware and vulnerable applications.
Open source framework for network traffic analysis with advanced features.
SentryPeer is a fraud detection tool that monitors and detects fraudulent activities on SIP servers, capturing IP addresses and phone numbers of suspicious activities and providing a notification system to service providers.
A multiplatform C++ library for capturing, parsing, and crafting network packets with support for various network protocols.
PINNED
Mandos
Fractional CISO service that helps B2B companies implement security leadership to win enterprise deals, achieve compliance, and develop strategic security programs.
Checkmarx SCA
A software composition analysis tool that identifies vulnerabilities, malicious code, and license risks in open source dependencies throughout the software development lifecycle.
Orca Security
A cloud-native application protection platform that provides agentless security monitoring, vulnerability management, and compliance capabilities across multi-cloud environments.
DryRun
A GitHub application that performs automated security code reviews by analyzing contextual security aspects of code changes during pull requests.