Skip to content

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

Updated README.md.

smicallef8d0a71b77e06
diff --git a/README.md b/README.md
index e9c2d05d..ce8cd616 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,243 @@
-ABOUT
-------
+<h1 align="center">
+ <br>
+ <a href="https://www.spiderfoot.net"><img src="https://www.spiderfoot.net/wp-content/themes/spiderfoot/img/spiderfoot-wide.png"></a>
+</h1>
-SpiderFoot is an open source intelligence (OSINT) automation tool. Its goal is to automate the process of gathering intelligence about a given target, which may be an IP address, domain name, hostname, network subnet, ASN, e-mail address or person's name.
+### ABOUT
-SpiderFoot can be used offensively, i.e. as part of a black-box penetration test to gather information about the target, or defensively to identify what information you or your organisation are freely providing for attackers to use against you.
+SpiderFoot is an open source intelligence (OSINT) automation tool. It integrates with just about every data source available and utilises a range of methods for data analysis, making that data easy to navigate.
-Read more at the project website: https://www.spiderfoot.net.
+SpiderFoot has an embedded web-server for providing a clean and intuitive web-based interface but can also be used completely via the command-line. It's written in **Python 3** and **GPL-licensed**.
+
+<img src="https://www.spiderfoot.net/wp-content/themes/spiderfoot/img/spiderfoot-browse.png">
+
+### FEATURES
+
+- Web based UI or CLI
+- Over 170 modules (see below)
+- Python 3
+- CSV/JSON/GEXF export
+- API key export/import
+- SQLite back-end for custom querying
+- Highly configurable
+- Fully documented
+- Visualisations
+- TOR integration for dark web searching
+- Dockerfile for Docker-based deployments
+- Can call other tools like DNSTwist, Whatweb and CMSeeK
+- Actively developed since 2012!
+
+### USES
+
+SpiderFoot's 170+ modules feed each other in a pub/sub model to ensure maximum data extraction to do things like:
+
+- Host/sub-domain/TLD enumeration/extraction
+- E-mail address enumeration/extraction
+- Phone number extraction
+- Bitcoin and Ethereum address extraction
+- DNS zone transfers
+- Threat intelligence and Blacklist queries
+- API integraiton with SHODAN, HaveIBeenPwned, Censys, AlienVault, SecurityTrails, etc.
+- Social media account enumeration
+- S3/Azure/Digitalocean bucket enumeration/scraping
+- IP geo-location
+- Web scraping, web content analysis
+- Image and binary file meta data analysis
+- Office document meta data analysis
+- Dark web searches
+- So much more...
+
+### PURPOSE
+
+SpiderFoot can be used offensively (e.g. in a red team exercise or penetration test) for reconnaissance of your target or defensively to gather information about what you or your organisation might have exposed over the Internet.
+
+You can target the following entities in a SpiderFoot scan:
+
+ - IP address
+ - Domain/sub-domain name
+ - Hostname
+ - Network subnet (CIDR)
+ - ASN
+ - E-mail address
+ - Phone number
+ - Username
+ - Person's name
+
+### MODULES
+
+| Module | Name | Description |
+| :------------- |:-------------| :------------|
+sfp_abusech.py|abuse.ch|Check if a host/domain, IP or netblock is malicious according to abuse.ch.|
+sfp_abuseipdb.py|AbuseIPDB|Check if a netblock or IP is malicious according to AbuseIPDB.com.|
+sfp_accounts.py|Accounts|Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.|
+sfp_adblock.py|AdBlock Check|Check if linked pages would be blocked by AdBlock Plus.|
+sfp_ahmia.py|Ahmia|Search Tor 'Ahmia' search engine for mentions of the target domain.|
+sfp_alienvaultiprep.py|AlienVault IP Reputation|Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.|
+sfp_alienvault.py|AlienVault OTX|Obtain information from AlienVault Open Threat Exchange (OTX)|
+sfp_apility.py|Apility|Search Apility API for IP address and domain reputation.|
+sfp_archiveorg.py|Archive.org|Identifies historic versions of interesting files/pages from the Wayback Machine.|
+sfp_arin.py|ARIN|Queries ARIN registry for contact information.|
+sfp_azureblobstorage.py|Azure Blob Finder|Search for potential Azure blobs associated with the target and attempt to list their contents.|
+sfp_badipscom.py|badips.com|Check if a domain or IP is malicious according to badips.com.|
+sfp_bambenek.py|Bambenek C&C List|Check if a host/domain or IP appears on Bambenek Consulting's C&C tracker lists.|
+sfp_base64.py|Base64|Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.|
+sfp_bgpview.py|BGPView|Obtain network information from BGPView API.|
+sfp_binaryedge.py|BinaryEdge|Obtain information from BinaryEdge.io's Internet scanning systems about breaches, vulerabilities, torrents and passive DNS.|
+sfp_bingsearch.py|Bing|Obtain information from bing to identify sub-domains and links.|
+sfp_bingsharedip.py|Bing (Shared IPs)|Search Bing for hosts sharing the same IP.|
+sfp_binstring.py|Binary String Extractor|Attempt to identify strings in binary content.|
+sfp_bitcoin.py|Bitcoin Finder|Identify bitcoin addresses in scraped webpages.|
+sfp_blockchain.py|Blockchain|Queries blockchain.info to find the balance of identified bitcoin wallet addresses.|
+sfp_blocklistde.py|blocklist.de|Check if a netblock or IP is malicious according to blocklist.de.|
+sfp_botscout.py|BotScout|Searches botscout.com's database of spam-bot IPs and e-mail addresses.|
+sfp_builtwith.py|BuiltWith|Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.|
+sfp_callername.py|CallerName|Lookup US phone number location and reputation information.|
+sfp_censys.py|Censys|Obtain information from Censys.io|
+sfp_cinsscore.py|CINS Army List|Check if a netblock or IP is malicious according to cinsscore.com's Army List.|
+sfp_circllu.py|CIRCL.LU|Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.|
+sfp_citadel.py|Citadel Engine|Searches Leak-Lookup.com's database of breaches.|
+sfp_cleanbrowsing.py|Cleanbrowsing.org|Check if a host would be blocked by Cleanbrowsing.org DNS|
+sfp_cleantalk.py|CleanTalk Spam List|Check if an IP is on CleanTalk.org's spam IP list.|
+sfp_clearbit.py|Clearbit|Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.|
+sfp_coinblocker.py|CoinBlocker Lists|Check if a host/domain or IP appears on CoinBlocker lists.|
+sfp_commoncrawl.py|CommonCrawl|Searches for URLs found through CommonCrawl.org.|
+sfp_comodo.py|Comodo|Check if a host would be blocked by Comodo DNS|
+sfp_company.py|Company Names|Identify company names in any obtained data.|
+sfp_cookie.py|Cookies|Extract Cookies from HTTP headers.|
+sfp_crossref.py|Cross-Reference|Identify whether other domains are associated ('Affiliates') of the target.|
+sfp_crt.py|Certificate Transparency|Gather hostnames from historical certificates in crt.sh.|
+sfp_customfeed.py|Custom Threat Feed|Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.|
+sfp_cybercrimetracker.py|cybercrime-tracker.net|Check if a host/domain or IP is malicious according to cybercrime-tracker.net.|
+sfp_darksearch.py|Darksearch|Search the Darksearch.io Tor search engine for mentions of the target domain.|
+sfp_digitaloceanspace.py|Digital Ocean Space Finder|Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.|
+sfp_dnsbrute.py|DNS Brute-force|Attempts to identify hostnames through brute-forcing common names and iterations.|
+sfp_dnscommonsrv.py|DNS Common SRV|Attempts to identify hostnames through common SRV.|
+sfp_dnsneighbor.py|DNS Look-aside|Attempt to reverse-resolve the IP addresses next to your target to see if they are related.|
+sfp_dnsraw.py|DNS Raw Records|Retrieves raw DNS records such as MX, TXT and others.|
+sfp_dnsresolve.py|DNS Resolver|Resolves Hosts and IP Addresses identified, also extracted from raw content.|
+sfp_dnszonexfer.py|DNS Zone Transfer|Attempts to perform a full DNS zone transfer.|
+sfp_dronebl.py|DroneBL|Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.|
+sfp_duckduckgo.py|DuckDuckGo|Query DuckDuckGo's API for descriptive information about your target.|
+sfp_emailformat.py|EmailFormat|Look up e-mail addresses on email-format.com.|
+sfp_email.py|E-Mail|Identify e-mail addresses in any obtained data.|
+sfp_emailrep.py|EmailRep|Search EmailRep.io for email address reputation.|
+sfp_errors.py|Errors|Identify common error messages in content like SQL errors, etc.|
+sfp_ethereum.py|Ethereum Finder|Identify ethereum addresses in scraped webpages.|
+sfp_filemeta.py|File Metadata|Extracts meta data from documents and images.|
+sfp_flickr.py|Flickr|Look up e-mail addresses on Flickr.|
+sfp_fortinet.py|Fortiguard.com|Check if an IP is malicious according to Fortiguard.com.|
+sfp_fraudguard.py|Fraudguard|Obtain threat information from Fraudguard.io|
+sfp_fringeproject.py|Fringe Project|Obtain network information from Fringe Project API.|
+sfp_fsecure_riddler.py|F-Secure Riddler.io|Obtain network information from F-Secure Riddler.io API.|
+sfp_fullcontact.py|FullContact|Gather domain and e-mail information from fullcontact.com.|
+sfp_github.py|Github|Identify associated public code repositories on Github.|
+sfp_googlemaps.py|Google Maps|Identifies potential physical addresses and latitude/longitude coordinates.|
+sfp_googlesearch.py|Google|Obtain information from the Google Custom Search API to identify sub-domains and links.|
+sfp_gravatar.py|Gravatar|Retrieve user information from Gravatar API.|
+sfp_greynoise.py|Greynoise|Obtain information from Greynoise.io's Enterprise API.|
+sfp_h1nobbdde.py|HackerOne (Unofficial)|Check external vulnerability scanning/reporting service h1.nobbd.de to see if the target is listed.|
+sfp_hackertarget.py|HackerTarget.com|Search HackerTarget.com for hosts sharing the same IP.|
+sfp_haveibeenpwned.py|HaveIBeenPwned|Check HaveIBeenPwned.com for hacked e-mail addresses identified in breaches.|
+sfp_honeypot.py|Honeypot Checker|Query the projecthoneypot.org database for entries.|
+sfp_hosting.py|Hosting Providers|Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.|
+sfp_hostsfilenet.py|hosts-file.net Malicious Hosts|Check if a host/domain is malicious according to hosts-file.net Malicious Hosts.|
+sfp_hunter.py|Hunter.io|Check for e-mail addresses and names on hunter.io.|
+sfp_iknowwhatyoudownload.py|Iknowwhatyoudownload.com|Check iknowwhatyoudownload.com for IP addresses that have been using BitTorrent.|
+sfp_instagram.py|Instagram|Gather information from Instagram profiles.|
+sfp_intelx.py|IntelligenceX|Obtain information from IntelligenceX about identified IP addresses, domains, e-mail addresses and phone numbers.|
+sfp_intfiles.py|Interesting Files|Identifies potential files of interest, e.g. office documents, zip files.|
+sfp_ipinfo.py|IPInfo.io|Identifies the physical location of IP addresses identified using ipinfo.io.|
+sfp_ipstack.py|ipstack|Identifies the physical location of IP addresses identified using ipstack.com.|
+sfp_isc.py|Internet Storm Center|Check if an IP is malicious according to SANS ISC.|
+sfp_junkfiles.py|Junk Files|Looks for old/temporary and other similar files.|
+sfp_malwaredomainlist.py|malwaredomainlist.com|Check if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.|
+sfp_malwaredomains.py|malwaredomains.com|Check if a host/domain is malicious according to malwaredomains.com.|
+sfp_malwarepatrol.py|MalwarePatrol|Searches malwarepatrol.net's database of malicious URLs/IPs.|
+sfp_metadefender.py|MetaDefender|Search MetaDefender API for IP address and domain IP reputation.|
+sfp_mnemonic.py|Mnemonic PassiveDNS|Obtain Passive DNS information from PassiveDNS.mnemonic.no.|
+sfp_multiproxy.py|multiproxy.org Open Proxies|Check if an IP is an open proxy according to multiproxy.org' open proxy list.|
+sfp_myspace.py|MySpace|Gather username and location from MySpace.com profiles.|
+sfp_names.py|Name Extractor|Attempt to identify human names in fetched content.|
+sfp_neutrinoapi.py|NeutrinoAPI|Search NeutrinoAPI for IP address info and check IP reputation.|
+sfp_norton.py|Norton ConnectSafe|Check if a host would be blocked by Norton ConnectSafe DNS|
+sfp_nothink.py|Nothink.org|Check if a host/domain, netblock or IP is malicious according to Nothink.org.|
+sfp_numpi.py|numpi|Lookup USA/Canada phone number location and carrier information from numpi.com.|
+sfp_numverify.py|numverify|Lookup phone number location and carrier information from numverify.com.|
+sfp_onioncity.py|Onion.link|Search Tor 'Onion City' search engine for mentions of the target domain.|
+sfp_onionsearchengine.py|Onionsearchengine.com|Search Tor onionsearchengine.com for mentions of the target domain.|
+sfp_openbugbounty.py|Open Bug Bounty|Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.|
+sfp_opencorporates.py|OpenCorporates|Look up company information from OpenCorporates.|
+sfp_opendns.py|OpenDNS|Check if a host would be blocked by OpenDNS DNS|
+sfp_openphish.py|OpenPhish|Check if a host/domain is malicious according to OpenPhish.com.|
+sfp_openstreetmap.py|OpenStreetMap|Retrieves latitude/longitude coordinates for physical addresses from OpenStreetMap API.|
+sfp_pageinfo.py|Page Info|Obtain information about web pages (do they take passwords, do they contain forms, etc.)|
+sfp_pastebin.py|PasteBin|PasteBin scraping (via Google) to identify related content.|
+sfp_pgp.py|PGP Key Look-up|Look up e-mail addresses in PGP public key servers.|
+sfp_phishtank.py|PhishTank|Check if a host/domain is malicious according to PhishTank.|
+sfp_phone.py|Phone Numbers|Identify phone numbers in scraped webpages.|
+sfp_portscan_tcp.py|Port Scanner - TCP|Scans for commonly open TCP ports on Internet-facing systems.|
+sfp_psbdmp.py|Psbdmp.com|Check psbdmp.cc (PasteBin Dump) for potentially hacked e-mails and domains.|
+sfp_pulsedive.py|Pulsedive|Obtain information from Pulsedive's API.|
+sfp_quad9.py|Quad9|Check if a host would be blocked by Quad9|
+sfp_ripe.py|RIPE|Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.|
+sfp_riskiq.py|RiskIQ|Obtain information from RiskIQ's (formerly PassiveTotal) Passive DNS and Passive SSL databases.|
+sfp_robtex.py|Robtex|Search Robtex.com for hosts sharing the same IP.|
+sfp_s3bucket.py|Amazon S3 Bucket Finder|Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.|
+sfp_scylla.py|Scylla|Gather breach data from Scylla API.|
+sfp_securitytrails.py|SecurityTrails|Obtain Passive DNS and other information from SecurityTrails|
+sfp_shodan.py|SHODAN|Obtain information from SHODAN about identified IP addresses.|
+sfp_similar.py|Similar Domains|Search various sources to identify similar looking domain names, for instance squatted domains.|
+sfp_skymem.py|Skymem|Look up e-mail addresses on Skymem.|
+sfp_slideshare.py|SlideShare|Gather name and location from SlideShare profiles.|
+sfp_socialprofiles.py|Social Media Profiles|Tries to discover the social media profiles for human names identified.|
+sfp_social.py|Social Networks|Identify presence on social media networks such as LinkedIn, Twitter and others.|
+sfp_sorbs.py|SORBS|Query the SORBS database for open relays, open proxies, vulnerable servers, etc.|
+sfp_spamcop.py|SpamCop|Query various spamcop databases for open relays, open proxies, vulnerable servers, etc.|
+sfp_spamhaus.py|Spamhaus|Query the Spamhaus databases for open relays, open proxies, vulnerable servers, etc.|
+sfp_spider.py|Spider|Spidering of web-pages to extract content for searching.|
+sfp_spyonweb.py|SpyOnWeb|Search SpyOnWeb for hosts sharing the same IP address, Google Analytics code, or Google Adsense code.|
+sfp_sslcert.py|SSL Certificates|Gather information about SSL certificates used by the target's HTTPS sites.|
+sfp_ssltools.py|SSL Tools|Gather information about SSL certificates from SSLTools.com.|
+sfp__stor_db.py|Storage|Stores scan results into the back-end SpiderFoot database. You will need this.|
+sfp__stor_stdout.py|Command-line output|Dumps output to standard out. Used for when a SpiderFoot scan is run via the command-line.|
+sfp_strangeheaders.py|Strange Headers|Obtain non-standard HTTP headers returned by web servers.|
+sfp_talosintel.py|Talos Intelligence|Check if a netblock or IP is malicious according to talosintelligence.com.|
+sfp_threatcrowd.py|ThreatCrowd|Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.|
+sfp_threatexpert.py|ThreatExpert.com|Check if a host/domain or IP is malicious according to ThreatExpert.com.|
+sfp_threatminer.py|ThreatMiner|Obtain information from ThreatMiner's database for passive DNS and threat intelligence.|
+sfp_tldsearch.py|TLD Search|Search all Internet TLDs for domains with the same name as the target (this can be very slow.)|
+sfp_tool_cmseek.py|Tool - CMSeeK|Identify what Content Management System (CMS) might be used.|
+sfp_tool_dnstwist.py|Tool - DNSTwist|Identify bit-squatting, typo and other similar domains to the target using a local DNSTwist installation.|
+sfp_tool_whatweb.py|Tool - WhatWeb|Identify what software is in use on the specified website.|
+sfp_torch.py|TORCH|Search Tor 'TORCH' search engine for mentions of the target domain.|
+sfp_torexits.py|TOR Exit Nodes|Check if an IP or netblock appears on the torproject.org exit node list.|
+sfp_totalhash.py|TotalHash.com|Check if a host/domain or IP is malicious according to TotalHash.com.|
+sfp_twitter.py|Twitter|Gather name and location from Twitter profiles.|
+sfp_uceprotect.py|UCEPROTECT|Query the UCEPROTECT databases for open relays, open proxies, vulnerable servers, etc.|
+sfp_urlscan.py|URLScan.io|Search URLScan.io cache for domain information.|
+sfp_venmo.py|Venmo|Gather user information from Venmo API.|
+sfp_viewdns.py|ViewDNS.info|Reverse Whois lookups using ViewDNS.info.|
+sfp_virustotal.py|VirusTotal|Obtain information from VirusTotal about identified IP addresses.|
+sfp_voipbl.py|VoIPBL OpenPBX IPs|Check if an IP or netblock is an open PBX according to VoIPBL OpenPBX IPs.|
+sfp_vxvault.py|VXVault.net|Check if a domain or IP is malicious according to VXVault.net.|
+sfp_watchguard.py|Watchguard|Check if an IP is malicious according to Watchguard's reputationauthority.org.|
+sfp_webanalytics.py|Web Analytics|Identify web analytics IDs in scraped webpages and DNS TXT records.|
+sfp_webframework.py|Web Framework|Identify the usage of popular web frameworks like jQuery, YUI and others.|
+sfp_webserver.py|Web Server|Obtain web server banners to identify versions of web servers being used.|
+sfp_whatcms.py|WhatCMS|Check web technology using WhatCMS.org API.|
+sfp_whoisology.py|Whoisology|Reverse Whois lookups using Whoisology.com.|
+sfp_whois.py|Whois|Perform a WHOIS look-up on domain names and owned netblocks.|
+sfp_whoxy.py|Whoxy|Reverse Whois lookups using Whoxy.com.|
+sfp_wigle.py|Wigle.net|Query wigle.net to identify nearby WiFi access points.|
+sfp_wikileaks.py|Wikileaks|Search Wikileaks for mentions of domain names and e-mail addresses.|
+sfp_wikipediaedits.py|Wikipedia Edits|Identify edits to Wikipedia articles made from a given IP address or username.|
+sfp_xforce.py|XForce Exchange|Obtain information from IBM X-Force Exchange|
+sfp_yandexdns.py|Yandex DNS|Check if a host would be blocked by Yandex DNS|
+sfp_zoneh.py|Zone-H Defacement Check|Check if a hostname/domain appears on the zone-h.org 'special defacements' RSS feed.|
+
+### DOCUMENTATION
+
+Read more at the [project website](https://www.spiderfoot.net), including more complete documentation, blog posts with tutorials/guides, plus information about [SpiderFoot HX](https://www.spiderfoot.net/hx).
Latest updates announced on [Twitter](https://twitter.com/spiderfoot).
+
diff --git a/test/unit/conftest.py b/test/unit/conftest.py
deleted file mode 100644
index 7c736634..00000000
--- a/test/unit/conftest.py
+++ /dev/null
@@ -1,7 +0,0 @@
-import pytest
-import responses
-
-def mocked_responses():
- with responses.RequestsMock() as rsps:
- yield rsps
\ No newline at end of file
diff --git a/test/unit/test_sflib.py b/test/unit/test_sflib.py
deleted file mode 100644
index a0934cc5..00000000
--- a/test/unit/test_sflib.py
+++ /dev/null
@@ -1,231 +0,0 @@
-import pytest
-import requests
-
-from sflib import SpiderFoot, SpiderFootPlugin, SpiderFootEvent
-
-
-class TestSflib:
- @pytest.mark.parametrize(
- "test_string,expected_emails",
- [
- ("[email protected]", ["[email protected]"]), # valid email
- ("b@f.", []), # too short (less than 5 charachters), should be ignored
- ("better%[email protected]", []), # contains '%' should be ignored
- ("amr@wh...", []), # truncated, should be ignored
- ],
- )
- def test_parse_emails(self, test_string, expected_emails):
- sf = SpiderFoot(options={"_debug": False})
-
- assert sf.parseEmails(test_string) == expected_emails
-
- @pytest.mark.parametrize(
- "test_string,expected_links",
- [
- ( # valid full link
- "<a href='https://www.google.com/'></a>",
- {
- "https://www.google.com/": {
- "original": "https://www.google.com/",
- "source": "https://www.example.com/",
- }
- },
- ),
- ( # for a partial link, it's added to the source url
- "<a href='/partial'></a>",
- {
- "https://www.example.com/partial": {
- "original": "/partial",
- "source": "https://www.example.com/",
- }
- },
- ),
- ("<a href='#in-page-id'></a>", {},), # In page ids are ignored
- ("<a href='mailto:[email protected]'></a>", {},), # mailto: are ignored
- ],
- )
- def test_parse_links(self, test_string, expected_links):
- sf = SpiderFoot(options={"_debug": False})
-
- parsed_links = sf.parseLinks(
- url="https://www.example.com/",
- data=test_string,
- domains=["testdomain.org"],
- )
-
- assert parsed_links == expected_links
-
- @pytest.mark.parametrize(
- "input_value,expected_data",
- [
- (None, None), # None returns None
- ("simple-string", "simple-string"), # a string returns the same string
- ],
- )
- def test_opt_value_to_data(self, mocker, input_value, expected_data):
- sf = SpiderFoot(options={"_debug": False, "__logging": False})
-
- assert sf.optValueToData(val=input_value) == expected_data
-
- def test_opt_value_to_data_file_input_file_doesnt_exist(self, mocker):
- file_name = "test.txt"
-
- sf = SpiderFoot(options={"_debug": False, "__logging": False})
- mock_error = mocker.patch.object(SpiderFoot, "error")
-
- # if fatal=False, return 'None', no errors are logged
- assert sf.optValueToData(val=f"@{file_name}", fatal=False) is None
- mock_error.assert_not_called()
-
- # if fatal=True, log an error and return the file name
- assert sf.optValueToData(val=f"@{file_name}", fatal=True) == f"@{file_name}"
- mock_error.assert_called_once()
-
- def test_opt_value_to_data_file_input_file_exists(self, mocker):
- """if input start with '@', return the contents of that file"""
- test_content = "file-content"
- file_name = "test.txt"
-
- sf = SpiderFoot(options={"_debug": False, "__logging": False})
- mocker.patch("sflib.open", mocker.mock_open(read_data=test_content))
-
- assert sf.optValueToData(val=f"@{file_name}", splitLines=False) == test_content
- assert sf.optValueToData(val=f"@{file_name}", splitLines=True) == [test_content]
-
- def test_opt_value_to_data_url_input_url_exists(self, mocked_responses):
- """if input starts with 'http://' or 'https://', fetch the url content"""
- test_content = "url-content"
- test_url = "http://example.com"
-
- sf = SpiderFoot(options={"_debug": False, "__logging": False})
- mocked_responses.add(
- mocked_responses.GET, test_url, body=test_content, status=200
- )
- mocked_responses.add(
- mocked_responses.GET, test_url, body=test_content, status=200
- )
-
- # TODO: this function possibly returns unicode, will keep for now
- # to avoid breaking changes, but this should be fixed
- assert sf.optValueToData(val=test_url, splitLines=False) == test_content.encode(
- "utf-8"
- )
- assert sf.optValueToData(val=test_url, splitLines=True) == [
- test_content.encode("utf-8")
- ]
-
- # TODO: Make this test case pass
- @pytest.mark.xfail(
- reason=(
- "error responses are handled similarly to 200 for now "
- "to avoid breaking changes"
- )
- )
- def test_opt_value_to_data_url_input_404(self, mocked_responses, mocker):
- test_url = "http://example.com"
-
- sf = SpiderFoot(options={"_debug": False, "__logging": False})
- mocked_responses.add(
- mocked_responses.GET, test_url, json={"error": "not found"}, status=404
- )
- mocked_responses.add(
- mocked_responses.GET, test_url, json={"error": "not found"}, status=404
- )
- mock_error = mocker.patch.object(SpiderFoot, "error")
-
- # If fatal=False, return 'None', no errors are logged
- assert sf.optValueToData(val=test_url, fatal=False) is None
- mock_error.assert_not_called()
-
- # if fatal=True, log an error and return the url
- assert sf.optValueToData(val=test_url, fatal=True) == test_url
- mock_error.assert_called_once()
-
-
-class TestFetchUrl:
- def test_successful_requests(self, mocked_responses):
- test_url = "http://example.com"
- test_content = "url-content"
-
- mocked_responses.add(
- mocked_responses.GET, test_url, body=test_content, status=200
- )
-
- sf = SpiderFoot(
- options={"_debug": False, "__logging": False, "_socks1type": ""}
- )
- result = sf.fetchUrl(url=test_url)
-
- expected_response = {
- # TODO: shouldn't 'code' be a number rather than a string?
- "code": "200",
- "content": "url-content",
- "headers": {"content-type": "text/plain"},
- "realurl": "http://example.com/",
- "status": None,
- }
-
- assert result == expected_response
-
- def test_error_handling(self, mocked_responses, mocker):
- test_url = "http://example.com"
- test_content = "url-content"
- headers = {
- "api_key": "samplekey",
- "Content-Type": "application/json",
- }
-
- mocked_responses.add(
- mocked_responses.GET,
- test_url,
- body=requests.exceptions.HTTPError("Timeout"),
- )
-
- sf = SpiderFoot(options={"_socks1type": "", "__logging": False,})
-
- # Assert error function is called
- mock_error = mocker.patch.object(SpiderFoot, "error")
- result = sf.fetchUrl(url=test_url, headers=headers)
- mock_error.assert_called()
-
- def test_404_requests(self, mocked_responses):
- test_url = "http://example.com"
- test_content = "not-found"
-
- mocked_responses.add(
- mocked_responses.GET, test_url, json={"error": test_content}, status=404
- )
-
- sf = SpiderFoot(
- options={"_debug": False, "__logging": False, "_socks1type": ""}
- )
- result = sf.fetchUrl(url=test_url)
-
- expected_response = {
- "code": "404",
- "content": '{"error": "not-found"}',
- "headers": {"content-type": "application/json"},
- "realurl": "http://example.com/",
- "status": None,
- }
- assert result == expected_response
-
-
-class TestSfPlugin:
- def test_notify_listeners_works(self, mocker):
- """Tests when a module calls notifyListeners with an event,
- all listener modules have 'handleEvent' called"""
-
- mocked_fn = mocker.patch("sflib.SpiderFootPlugin.handleEvent")
-
- sf_plugin = SpiderFootPlugin()
- sf_plugin_2 = SpiderFootPlugin()
- sf_plugin.registerListener(listener=sf_plugin_2)
-
- root_event = SpiderFootEvent(
- eventType="ROOT", data="binarypool.com", module="", sourceEvent=None
- )
-
- sf_plugin.notifyListeners(sfEvent=root_event)
- mocked_fn.assert_called_once()
- mocked_fn.assert_called_with(root_event)
diff --git a/tests/test_sflib.py b/tests/test_sflib.py
deleted file mode 100644
index 71051fab..00000000
--- a/tests/test_sflib.py
+++ /dev/null
@@ -1,77 +0,0 @@
-import pytest
-from sflib import SpiderFoot, SpiderFootPlugin, SpiderFootEvent
-
-
-class TestSflib:
- @pytest.mark.parametrize(
- "test_string,expected_emails",
- [
- ("[email protected]", ["[email protected]"]), # valid email
- ("b@f.", []), # too short (less than 5 charachters), should be ignored
- ("better%[email protected]", []), # contains '%' should be ignored
- ("amr@wh...", []), # truncated, should be ignored
- ],
- )
- def test_parse_emails(self, test_string, expected_emails):
- sf = SpiderFoot(options={"_debug": False})
-
- assert sf.parseEmails(test_string) == expected_emails
-
- @pytest.mark.parametrize(
- "test_string,expected_links",
- [
- ( # valid full link
- "<a href='https://www.google.com/'></a>",
- {
- "https://www.google.com/": {
- "original": "https://www.google.com/",
- "source": "https://www.example.com/",
- }
- },
- ),
- ( # for a partial link, it's added to the source url
- "<a href='/partial'></a>",
- {
- "https://www.example.com/partial": {
- "original": "/partial",
- "source": "https://www.example.com/",
- }
- },
- ),
- ("<a href='#in-page-id'></a>", {},), # In page ids are ignored
- ("<a href='mailto:[email protected]'></a>", {},), # mailto: are ignored
- ],
- )
- def test_parse_links(self, test_string, expected_links):
- sf = SpiderFoot(options={"_debug": False})
-
- parsed_links = sf.parseLinks(
- url="https://www.example.com/",
- data=test_string,
- domains=["testdomain.org"],
- )
-
- assert parsed_links == expected_links
-
-
-class TestSfPlugin:
- def test_notify_listeners_works(self, mocker):
- """Tests when a module calls notifyListeners with an event,
- all listener modules have 'handleEvent' called"""
-
- mocked_fn = mocker.patch("sflib.SpiderFootPlugin.handleEvent")
-
- sf_plugin = SpiderFootPlugin()
- sf_plugin_2 = SpiderFootPlugin()
- sf_plugin.registerListener(listener=sf_plugin_2)
-
- root_event = SpiderFootEvent(
- eventType="ROOT",
- data=u'binarypool.com',
- module="",
- sourceEvent=None
- )
-
- sf_plugin.notifyListeners(sfEvent=root_event)
- mocked_fn.assert_called_once()
- mocked_fn.assert_called_with(root_event)
\ No newline at end of file