crawley
The unix-way web crawler Crawley is a simple web crawler written in Go. Features: * Supports multiple protocols (http, https, ftp, sftp) * Supports multiple authentication methods (basic auth, digest auth, kerberos) * Supports crawling of nested resources * Supports filtering of crawled resources * Supports saving of crawled resources to disk Example usage: ```go import "github.com/s0rg/crawley" func main() { crawler := crawley.NewCrawler() crawler.AddURL("https://example.com") crawler.Start() } ```
FEATURES
ALTERNATIVES
JARM is a TLS server fingerprinting tool used for identifying server configurations and malicious infrastructure.
Akamai Enterprise Application Access is a ZTNA solution that provides secure, identity-based access to private applications without exposing the network.
A tool for discovering open S3 Buckets starting from a domain using various techniques such as crawling and DNS crawling.
A set of Go-based emulators for testing network security and analyzing network traffic.
High-performance remote packet capture and collection tool used for forensic analysis in cloud workloads.
Express middleware for detecting and redirecting Tor or Surface users.
PINNED
InfoSecHired
An AI-powered career platform that automates the creation of cybersecurity job application materials and provides company-specific insights for job seekers.
Mandos Brief Newsletter
A weekly newsletter providing cybersecurity leadership insights, industry updates, and strategic guidance for security professionals advancing to management positions.
Kriptos
An AI-driven data classification and governance platform that automatically discovers, analyzes, and labels sensitive information while providing risk management and compliance capabilities.
System Two Security
An AI-powered platform that automates threat hunting and analysis by processing cyber threat intelligence and generating customized hunt packages for SOC teams.
Aikido Security
Aikido is an all-in-one security platform that combines multiple security scanning and management functions for cloud-native applications and infrastructure.
Permiso
Permiso is an Identity Threat Detection and Response platform that provides comprehensive visibility and protection for identities across multiple cloud environments.
Wiz
Wiz Cloud Security Platform is a cloud-native security platform that enables security, dev, and devops to work together in a self-service model, detecting and preventing cloud security threats in real-time.
Adversa AI
Adversa AI is a cybersecurity company that provides solutions for securing and hardening machine learning, artificial intelligence, and large language models against adversarial attacks, privacy issues, and safety incidents across various industries.