aljaz/recon-pipeline
1
0
Fork 0
mirror of https://github.com/aljazceru/recon-pipeline.git synced 2025-12-23 17:24:21 +01:00
Code Issues Packages Projects Releases Wiki Activity

cleaned up code blocks; scans conform to 1 template for sections

Browse Source 
* initial work on sphinx docs; much left to do

* first pass at docs complete; still has some warts

* added requirements for readthedocs

* added requirements for readthedocs

* added requirements for readthedocs

* cleaned up code blocks; scans conform to 1 template for sections

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs
This commit is contained in:
epi052
2020-01-27 20:02:09 -06:00
committed by GitHub
parent 8923d17d2e
commit 827e7ed6cc
14 changed files with 253 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/conf.py
View File

@@ -12,6 +12,7 @@
# #
import os import os
import sys import sys
import sphinx_rtd_theme
sys.path.insert(0, os.path.abspath("..")) sys.path.insert(0, os.path.abspath(".."))
@@ -58,7 +59,10 @@ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
# The theme to use for HTML and HTML Help pages. See the documentation for # The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes. # a list of builtin themes.
# #
pygments_style = "sphinx"
html_theme = "sphinx_rtd_theme" html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Add any paths that contain custom static files (such as style sheets) here, # Add any paths that contain custom static files (such as style sheets) here,

8
docs/modifications/new_wrapper.rst
View File

@@ -4,9 +4,9 @@ Creating a New Wrapper Scan
If for whatever reason you want something other than FullScan, the process for defining a new scan is relatively simple. If for whatever reason you want something other than FullScan, the process for defining a new scan is relatively simple.
The ``HTBScan`` is a good example. The ``HTBScan`` is a good example.
1. Define your new class, inheriting from :class:`luigi.WrapperTask` and use the ``inherits`` decorator to include any scan you want to utilize 1. Define your new class, inheriting from **luigi.WrapperTask** and use the ``inherits`` decorator to include any scan you want to utilize
.. code-block:: .. code-block:: python
@inherits(SearchsploitScan, AquatoneScan, GobusterScan, WebanalyzeScan) @inherits(SearchsploitScan, AquatoneScan, GobusterScan, WebanalyzeScan)
class HTBScan(luigi.WrapperTask): class HTBScan(luigi.WrapperTask):
@@ -14,7 +14,7 @@ The ``HTBScan`` is a good example.
2. Include all parameters needed by any of the scans passed to ``inherits`` 2. Include all parameters needed by any of the scans passed to ``inherits``
.. code-block:: .. code-block:: python
def requires(self): def requires(self):
""" HTBScan is a wrapper, as such it requires any Tasks that it wraps. """ """ HTBScan is a wrapper, as such it requires any Tasks that it wraps. """
@@ -36,7 +36,7 @@ The ``HTBScan`` is a good example.
3. ``yield`` from each scan, keeping in mind that some of the parameters won't be universal (i.e. need to be removed/added) 3. ``yield`` from each scan, keeping in mind that some of the parameters won't be universal (i.e. need to be removed/added)
.. code-block:: .. code-block:: python
def requires(self): def requires(self):
""" HTBScan is a wrapper, as such it requires any Tasks that it wraps. """ """ HTBScan is a wrapper, as such it requires any Tasks that it wraps. """

16
docs/overview/installation.rst
View File

@@ -13,7 +13,7 @@ Manual Steps
First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below) First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below)
.. code-block:: .. code-block:: console
apt install pipenv apt install pipenv
git clone https://github.com/epi052/recon-pipeline.git git clone https://github.com/epi052/recon-pipeline.git
@@ -52,12 +52,12 @@ for the auto installer to function:
- derivative of debian (some tools are installed using apt) - derivative of debian (some tools are installed using apt)
The alternatives would be to manually install each tool or to modify the distro-specific portions of the commands The alternatives would be to manually install each tool or to modify the distro-specific portions of the commands
laid out in ``recon.__init__``. For example, on Fedora, you could change the package manager from ``apt-get`` to laid out in ``recon.__init__.py``. For example, on Fedora, you could change the package manager from ``apt-get`` to
``dnf`` and remove any ``apt-get`` specific options. ``dnf`` and remove any ``apt-get`` specific options.
Example from ``recon-pipeline/recon/__init__.py`` Example from ``recon.__init__.py``
.. code-block:: .. code-block:: python
"pipenv": { "pipenv": {
"installed": False, "installed": False,
@@ -67,16 +67,10 @@ Example from ``recon-pipeline/recon/__init__.py``
would become would become
.. code-block:: .. code-block:: python
"pipenv": { "pipenv": {
"installed": False, "installed": False,
"dependencies": None, "dependencies": None,
"commands": ["sudo dnf install -y pipenv"], "commands": ["sudo dnf install -y pipenv"],
}, },

4
docs/overview/scope.rst
View File

@@ -7,7 +7,7 @@ The pipeline expects a file that describes the target's scope to be provided as
``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line. Ip ``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line. Ip
addresses and ip ranges can be mixed/matched, but domains cannot. addresses and ip ranges can be mixed/matched, but domains cannot.
.. code-block:: .. code-block:: console
tesla.com tesla.com
tesla.cn tesla.cn
@@ -17,7 +17,7 @@ addresses and ip ranges can be mixed/matched, but domains cannot.
Some bug bounty scopes have expressly verboten subdomains and/or top-level domains, for that there is the Some bug bounty scopes have expressly verboten subdomains and/or top-level domains, for that there is the
``--exempt-list`` option. The exempt list follows the same rules as the target file. ``--exempt-list`` option. The exempt list follows the same rules as the target file.
.. code-block:: .. code-block:: console
shop.eu.teslamotors.com shop.eu.teslamotors.com
energysupport.tesla.com energysupport.tesla.com

30
recon/amass.py
View File

@@ -10,18 +10,30 @@ from recon.targets import TargetList
@inherits(TargetList) @inherits(TargetList)
class AmassScan(ExternalProgramTask): class AmassScan(ExternalProgramTask):
""" Run amass scan to perform subdomain enumeration of given domain(s). """ Run ``amass`` scan to perform subdomain enumeration of given domain(s).
Expects TARGET_FILE.domains file to be a text file with one top-level domain per line. Note:
Expects **TARGET_FILE.domains** file to be a text file with one top-level domain per line.
Commands are similar to the following Install:
.. code-block:: console
amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json sudo apt-get install -y -q amass
Basic Example:
.. code-block:: console
amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.amass AmassScan --target-file tesla
Args: Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. exempt_list: Path to a file providing blacklisted subdomains, one per line.
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
exempt_list = luigi.Parameter(default="") exempt_list = luigi.Parameter(default="")
@@ -83,9 +95,9 @@ class ParseAmassOutput(luigi.Task):
""" Read amass JSON results and create categorized entries into ip|subdomain files. """ Read amass JSON results and create categorized entries into ip|subdomain files.
Args: Args:
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
def requires(self): def requires(self):

42
recon/masscan.py
View File

@@ -14,25 +14,37 @@ from recon.config import top_tcp_ports, top_udp_ports, defaults
@inherits(TargetList, ParseAmassOutput) @inherits(TargetList, ParseAmassOutput)
class MasscanScan(luigi.Task): class MasscanScan(luigi.Task):
""" Run masscan against a target specified via the TargetList Task. """ Run ``masscan`` against a target specified via the TargetList Task.
Masscan commands are structured like the example below. When specified, --top_ports is processed and Note:
then ultimately passed to --ports. When specified, ``--top_ports`` is processed and then ultimately passed to ``--ports``.
masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips Install:
.. code-block:: console
The corresponding luigi command is shown below. git clone https://github.com/robertdavidgraham/masscan /tmp/masscan
make -s -j -C /tmp/masscan
sudo mv /tmp/masscan/bin/masscan /usr/local/bin/masscan
rm -rf /tmp/masscan
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21 Basic Example:
.. code-block:: console
masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21
Args: Args:
rate: desired rate for transmitting packets (packets per second) rate: desired rate for transmitting packets (packets per second)
interface: use the named raw network interface, such as "eth0" interface: use the named raw network interface, such as "eth0"
top_ports: Scan top N most popular ports top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned ports: specifies the port(s) to be scanned
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
results_dir: specifies the directory on disk to which all Task results are written *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
""" """
rate = luigi.Parameter(default=defaults.get("masscan-rate", "")) rate = luigi.Parameter(default=defaults.get("masscan-rate", ""))
@@ -114,12 +126,12 @@ class ParseMasscanOutput(luigi.Task):
""" Read masscan JSON results and create a pickled dictionary of pertinent information for processing. """ Read masscan JSON results and create a pickled dictionary of pertinent information for processing.
Args: Args:
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
def requires(self): def requires(self):

58
recon/nmap.py
View File

@@ -13,24 +13,30 @@ from recon.masscan import ParseMasscanOutput
@inherits(ParseMasscanOutput) @inherits(ParseMasscanOutput)
class ThreadedNmapScan(luigi.Task): class ThreadedNmapScan(luigi.Task):
""" Run nmap against specific targets and ports gained from the ParseMasscanOutput Task. """ Run ``nmap`` against specific targets and ports gained from the ParseMasscanOutput Task.
nmap commands are structured like the example below. Install:
``nmap`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``nmap``.
nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155 Basic Example:
.. code-block:: console
The corresponding luigi command is shown below. nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000 Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
Args: Args:
threads: number of threads for parallel nmap command execution threads: number of threads for parallel nmap command execution
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
threads = luigi.Parameter(default=defaults.get("threads", "")) threads = luigi.Parameter(default=defaults.get("threads", ""))
@@ -127,24 +133,30 @@ class ThreadedNmapScan(luigi.Task):
@inherits(ThreadedNmapScan) @inherits(ThreadedNmapScan)
class SearchsploitScan(luigi.Task): class SearchsploitScan(luigi.Task):
""" Run searchcploit against each nmap*.xml file in the TARGET-nmap-results directory and write results to disk. """ Run ``searchcploit`` against each ``nmap*.xml`` file in the **TARGET-nmap-results** directory and write results to disk.
searchsploit commands are structured like the example below. Install:
``searchcploit`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``searchcploit``.
searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml Basic Example:
.. code-block:: console
The corresponding luigi command is shown below. searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000 Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
Args: Args:
threads: number of threads for parallel nmap command execution *--* Required by upstream Task threads: number of threads for parallel nmap command execution *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifies the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifies the directory on disk to which all Task results are written *Required by upstream Task*
""" """
def requires(self): def requires(self):

33
recon/web/aquatone.py
View File

@@ -12,24 +12,35 @@ from recon.web.targets import GatherWebTargets
class AquatoneScan(luigi.Task): class AquatoneScan(luigi.Task):
""" Screenshot all web targets and generate HTML report. """ Screenshot all web targets and generate HTML report.
aquatone commands are structured like the example below. Install:
.. code-block:: console
cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20 mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone
An example of the corresponding luigi command is shown below. Basic Example:
``aquatone`` commands are structured like the example below.
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000 ``cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20``
Luigi Example:
.. code-block:: python
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
Args: Args:
threads: number of threads for parallel aquatone command execution threads: number of threads for parallel aquatone command execution
scan_timeout: timeout in miliseconds for aquatone port scans scan_timeout: timeout in miliseconds for aquatone port scans
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
threads = luigi.Parameter(default=defaults.get("threads", "")) threads = luigi.Parameter(default=defaults.get("threads", ""))

44
recon/web/corscanner.py
View File

@@ -8,31 +8,35 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets) @inherits(GatherWebTargets)
class CORScannerScan(ExternalProgramTask): class CORScannerScan(ExternalProgramTask):
""" Use CORScanner to scan for potential CORS misconfigurations. """ Use ``CORScanner`` to scan for potential CORS misconfigurations.
CORScanner commands are structured like the example below.
python cors_scan.py -i webtargets.tesla.txt -t 100
An example of the corresponding luigi command is shown below.
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.corscanner CORScannerScan --target-file tesla --top-ports 1000 --interface eth0
Install: Install:
git clone https://github.com/chenjj/CORScanner.git .. code-block:: console
cd CORScanner
pip install -r requirements.txt git clone https://github.com/chenjj/CORScanner.git
pip install future cd CORScanner
pip install -r requirements.txt
pip install future
Basic Example:
.. code-block:: console
python cors_scan.py -i webtargets.tesla.txt -t 100
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.corscanner CORScannerScan --target-file tesla --top-ports 1000 --interface eth0
Args: Args:
threads: number of threads for parallel subjack command execution threads: number of threads for parallel subjack command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
threads = luigi.Parameter(default=defaults.get("threads", "")) threads = luigi.Parameter(default=defaults.get("threads", ""))

41
recon/web/gobuster.py
View File

@@ -14,40 +14,37 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets) @inherits(GatherWebTargets)
class GobusterScan(luigi.Task): class GobusterScan(luigi.Task):
""" Use gobuster to perform forced browsing. """ Use ``gobuster`` to perform forced browsing.
gobuster commands are structured like the example below.
.. code-block::
gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html
An example of the corresponding luigi command is shown below.
Example:
.. code-block::
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20
Install: Install:
.. code-block:: .. code-block:: console
go get github.com/OJ/gobuster go get github.com/OJ/gobuster
git clone https://github.com/epi052/recursive-gobuster.git git clone https://github.com/epi052/recursive-gobuster.git
Basic Example:
.. code-block:: console
gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20
Args: Args:
threads: number of threads for parallel gobuster command execution threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly) recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
proxy = luigi.Parameter(default=defaults.get("proxy", "")) proxy = luigi.Parameter(default=defaults.get("proxy", ""))

68
recon/web/subdomain_takeover.py
View File

@@ -8,24 +8,34 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets) @inherits(GatherWebTargets)
class TKOSubsScan(ExternalProgramTask): class TKOSubsScan(ExternalProgramTask):
""" Use tko-subs to scan for potential subdomain takeovers. """ Use ``tko-subs`` to scan for potential subdomain takeovers.
tko-subs commands are structured like the example below. Install:
.. code-block:: console
tko-subs -domains=tesla.subdomains -data=/root/go/src/github.com/anshumanbh/tko-subs/providers-data.csv -output=tkosubs.tesla.csv go get github.com/anshumanbh/tko-subs
cd ~/go/src/github.com/anshumanbh/tko-subs
go build
go install
An example of the corresponding luigi command is shown below. Basic Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0 tko-subs -domains=tesla.subdomains -data=/root/go/src/github.com/anshumanbh/tko-subs/providers-data.csv -output=tkosubs.tesla.csv
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0
Args: Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
def requires(self): def requires(self):
@@ -77,25 +87,35 @@ class TKOSubsScan(ExternalProgramTask):
@inherits(GatherWebTargets) @inherits(GatherWebTargets)
class SubjackScan(ExternalProgramTask): class SubjackScan(ExternalProgramTask):
""" Use subjack to scan for potential subdomain takeovers. """ Use ``subjack`` to scan for potential subdomain takeovers.
subjack commands are structured like the example below. Install:
.. code-block:: console
subjack -w webtargets.tesla.txt -t 100 -timeout 30 -o subjack.tesla.txt -ssl go get github.com/haccer/subjack
cd ~/go/src/github.com/haccer/subjack
go build
go install
An example of the corresponding luigi command is shown below. Basic Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover SubjackScan --target-file tesla --top-ports 1000 --interface eth0 subjack -w webtargets.tesla.txt -t 100 -timeout 30 -o subjack.tesla.txt -ssl
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover SubjackScan --target-file tesla --top-ports 1000 --interface eth0
Args: Args:
threads: number of threads for parallel subjack command execution threads: number of threads for parallel subjack command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
threads = luigi.Parameter(default=defaults.get("threads", "")) threads = luigi.Parameter(default=defaults.get("threads", ""))

14
recon/web/targets.py
View File

@@ -13,13 +13,13 @@ class GatherWebTargets(luigi.Task):
""" Gather all subdomains as well as any ip addresses known to have a configured web port open. """ Gather all subdomains as well as any ip addresses known to have a configured web port open.
Args: Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
""" """
def requires(self): def requires(self):

31
recon/web/webanalyze.py
View File

@@ -16,20 +16,23 @@ from recon.web.targets import GatherWebTargets
class WebanalyzeScan(luigi.Task): class WebanalyzeScan(luigi.Task):
""" Use webanalyze to determine the technology stack on the given target(s). """ Use webanalyze to determine the technology stack on the given target(s).
webanalyze commands are structured like the example below.
webanalyze -host www.tesla.com -output json
An example of the corresponding luigi command is shown below.
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Install: Install:
.. code-block:: console
go get -u github.com/rverton/webanalyze go get -u github.com/rverton/webanalyze
# loads new apps.json file from wappalyzer project # loads new apps.json file from wappalyzer project
webanalyze -update webanalyze -update
Basic Example:
.. code-block:: console
webanalyze -host www.tesla.com -output json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Args: Args:
threads: number of threads for parallel webanalyze command execution threads: number of threads for parallel webanalyze command execution
@@ -105,7 +108,11 @@ class WebanalyzeScan(luigi.Task):
pass pass
for url_scheme in ("https://", "http://"): for url_scheme in ("https://", "http://"):
command = [tool_paths.get("webanalyze"), "-host", f"{url_scheme}{target}"] command = [
tool_paths.get("webanalyze"),
"-host",
f"{url_scheme}{target}",
]
commands.append(command) commands.append(command)
Path(self.output().path).mkdir(parents=True, exist_ok=True) Path(self.output().path).mkdir(parents=True, exist_ok=True)

34
recon/wrappers.py
View File

@@ -21,19 +21,22 @@ from recon.web.webanalyze import WebanalyzeScan
class FullScan(luigi.WrapperTask): class FullScan(luigi.WrapperTask):
""" Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time. """ Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.
Note:
Because FullScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.
Args: Args:
threads: number of threads for parallel gobuster command execution threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly) recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line.
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0"
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second)
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written
""" """
def requires(self): def requires(self):
@@ -80,19 +83,22 @@ class FullScan(luigi.WrapperTask):
class HTBScan(luigi.WrapperTask): class HTBScan(luigi.WrapperTask):
""" Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time. """ Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.
Note:
Because HTBScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.
Args: Args:
threads: number of threads for parallel gobuster command execution threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly) recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task exempt_list: Path to a file providing blacklisted subdomains, one per line.
top_ports: Scan top N most popular ports *--* Required by upstream Task top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned *--* Required by upstream Task ports: specifies the port(s) to be scanned
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task interface: use the named raw network interface, such as "eth0"
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task rate: desired rate for transmitting packets (packets per second)
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task target_file: specifies the file on disk containing a list of ips or domains
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task results_dir: specifes the directory on disk to which all Task results are written
""" """
def requires(self): def requires(self):