Automate the discovery of organizational footprints across the internet using a single domain or IP
Imagine entering just a domain or IP address and having the tool automatically determine where that target is exposed on the internet. SpiderFoot is a tool designed to automate the open-source intelligence (OSINT) process for threat intelligence analysis and attack surface mapping. It collects and analyzes data using over 200 modules via a web interface or command line.
A structure where modules exchange information
SpiderFoot uses over 200 modules to collect data. Each module is connected via a publisher/subscriber model, allowing one module to pass information it finds to another as input to extract more data. This enables tasks such as subdomain enumeration, email address extraction, Bitcoin address verification, DNS zone transfer attempts, threat intelligence lookups, social media account discovery, cloud bucket enumeration, IP geolocation, web scraping, dark web searches, port scanning, and data breach searches. Most modules do not require an API key, and when they do, a free tier is often available.
Scan targets and result utilization
You can scan various entities including IP addresses, domains, hostnames, network subnets, ASNs, emails, phone numbers, usernames, person names, and Bitcoin addresses. Collected data can be exported in CSV, JSON, and GEXF formats, or queried directly through a SQLite backend. Since version 4.0, a correlation engine configurable via YAML has been introduced, interpreting data based on 37 predefined rules. Additionally, it can extend functionality by calling external tools like DNSTwist, Whatweb, Nmap, and CMSeeK.
Installation and execution methods
Python 3.7 or higher is required. For stability, it is recommended to use packaged release versions, as the master branch may contain less tested, newer features. During installation, install the necessary libraries using pip, then run the python3 ./sf.py -l 127.0.0.1:5001 command to start the web server. A Dockerfile is also provided for deployment in container environments. Dark web searches are supported through TOR integration.
Differences between the open-source version and HX
SpiderFoot HX is a cloud-based managed service that offers different features from the open-source version. HX sends notifications via email, REST, and Slack for changes in attack surface monitoring, scans multiple targets at once, and supports multi-user collaboration, 2FA authentication, investigation features, customer support, pre-installed third-party tools, a complete RESTful API, screenshot functionality, importing custom Python modules, and data transmission to Splunk/ElasticSearch/REST endpoints. The open-source version lacks these cloud-managed features and must be operated locally.
Pre-use considerations
This tool can be used aggressively for red team exercises or penetration testing, or defensively to understand an organization's internet exposure. It is distributed under the MIT license and has been actively developed since 2012. A Python 3.7 or higher environment is required, and since some modules may require external API keys, you should verify the free tier or cost structure of those APIs before use. If using the master branch, features may be unstable, so packaged release versions are recommended for production environments.
