aljaz/recon-pipeline
1
0
Fork 0
mirror of https://github.com/aljazceru/recon-pipeline.git synced 2025-12-23 09:14:24 +01:00
Code Issues Packages Projects Releases Wiki Activity

cleaned up code blocks; scans conform to 1 template for sections

Browse Source 
* initial work on sphinx docs; much left to do

* first pass at docs complete; still has some warts

* added requirements for readthedocs

* added requirements for readthedocs

* added requirements for readthedocs

* cleaned up code blocks; scans conform to 1 template for sections

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs
This commit is contained in:
epi052
2020-01-27 20:02:09 -06:00
committed by GitHub
parent 8923d17d2e
commit 827e7ed6cc
14 changed files with 253 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/conf.py
View File

@@ -12,6 +12,7 @@
#
import os
import sys
import sphinx_rtd_theme
sys.path.insert(0, os.path.abspath(".."))
@@ -58,7 +59,10 @@ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
pygments_style = "sphinx"
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Add any paths that contain custom static files (such as style sheets) here,

8
docs/modifications/new_wrapper.rst
View File

@@ -4,9 +4,9 @@ Creating a New Wrapper Scan
If for whatever reason you want something other than FullScan, the process for defining a new scan is relatively simple.
The ``HTBScan`` is a good example.
1. Define your new class, inheriting from :class:`luigi.WrapperTask` and use the ``inherits`` decorator to include any scan you want to utilize
1. Define your new class, inheriting from **luigi.WrapperTask** and use the ``inherits`` decorator to include any scan you want to utilize
.. code-block::
.. code-block:: python
@inherits(SearchsploitScan, AquatoneScan, GobusterScan, WebanalyzeScan)
class HTBScan(luigi.WrapperTask):
@@ -14,7 +14,7 @@ The ``HTBScan`` is a good example.
2. Include all parameters needed by any of the scans passed to ``inherits``
.. code-block::
.. code-block:: python
def requires(self):
""" HTBScan is a wrapper, as such it requires any Tasks that it wraps. """
@@ -36,7 +36,7 @@ The ``HTBScan`` is a good example.
3. ``yield`` from each scan, keeping in mind that some of the parameters won't be universal (i.e. need to be removed/added)
.. code-block::
.. code-block:: python
def requires(self):
""" HTBScan is a wrapper, as such it requires any Tasks that it wraps. """

16
docs/overview/installation.rst
View File

@@ -13,7 +13,7 @@ Manual Steps
First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below)
.. code-block::
.. code-block:: console
apt install pipenv
git clone https://github.com/epi052/recon-pipeline.git
@@ -52,12 +52,12 @@ for the auto installer to function:
- derivative of debian (some tools are installed using apt)
The alternatives would be to manually install each tool or to modify the distro-specific portions of the commands
laid out in ``recon.__init__``. For example, on Fedora, you could change the package manager from ``apt-get`` to
laid out in ``recon.__init__.py``. For example, on Fedora, you could change the package manager from ``apt-get`` to
``dnf`` and remove any ``apt-get`` specific options.
Example from ``recon-pipeline/recon/__init__.py``
Example from ``recon.__init__.py``
.. code-block::
.. code-block:: python
"pipenv": {
"installed": False,
@@ -67,16 +67,10 @@ Example from ``recon-pipeline/recon/__init__.py``
would become
.. code-block::
.. code-block:: python
"pipenv": {
"installed": False,
"dependencies": None,
"commands": ["sudo dnf install -y pipenv"],
},

4
docs/overview/scope.rst
View File

@@ -7,7 +7,7 @@ The pipeline expects a file that describes the target's scope to be provided as
``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line. Ip
addresses and ip ranges can be mixed/matched, but domains cannot.
.. code-block::
.. code-block:: console
tesla.com
tesla.cn
@@ -17,7 +17,7 @@ addresses and ip ranges can be mixed/matched, but domains cannot.
Some bug bounty scopes have expressly verboten subdomains and/or top-level domains, for that there is the
``--exempt-list`` option. The exempt list follows the same rules as the target file.
.. code-block::
.. code-block:: console
shop.eu.teslamotors.com
energysupport.tesla.com

28
recon/amass.py
View File

@@ -10,18 +10,30 @@ from recon.targets import TargetList
@inherits(TargetList)
class AmassScan(ExternalProgramTask):
""" Run amass scan to perform subdomain enumeration of given domain(s).
""" Run ``amass`` scan to perform subdomain enumeration of given domain(s).
Expects TARGET_FILE.domains file to be a text file with one top-level domain per line.
Note:
Expects **TARGET_FILE.domains** file to be a text file with one top-level domain per line.
Commands are similar to the following
Install:
.. code-block:: console
sudo apt-get install -y -q amass
Basic Example:
.. code-block:: console
amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.amass AmassScan --target-file tesla
Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line.
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
exempt_list = luigi.Parameter(default="")
@@ -83,9 +95,9 @@ class ParseAmassOutput(luigi.Task):
""" Read amass JSON results and create categorized entries into ip|subdomain files.
Args:
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):

38
recon/masscan.py
View File

@@ -14,14 +14,26 @@ from recon.config import top_tcp_ports, top_udp_ports, defaults
@inherits(TargetList, ParseAmassOutput)
class MasscanScan(luigi.Task):
""" Run masscan against a target specified via the TargetList Task.
""" Run ``masscan`` against a target specified via the TargetList Task.
Masscan commands are structured like the example below. When specified, --top_ports is processed and
then ultimately passed to --ports.
Note:
When specified, ``--top_ports`` is processed and then ultimately passed to ``--ports``.
Install:
.. code-block:: console
git clone https://github.com/robertdavidgraham/masscan /tmp/masscan
make -s -j -C /tmp/masscan
sudo mv /tmp/masscan/bin/masscan /usr/local/bin/masscan
rm -rf /tmp/masscan
Basic Example:
.. code-block:: console
masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips
The corresponding luigi command is shown below.
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21
@@ -30,9 +42,9 @@ class MasscanScan(luigi.Task):
interface: use the named raw network interface, such as "eth0"
top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
results_dir: specifies the directory on disk to which all Task results are written *--* Optional for upstream Task
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
"""
rate = luigi.Parameter(default=defaults.get("masscan-rate", ""))
@@ -114,12 +126,12 @@ class ParseMasscanOutput(luigi.Task):
""" Read masscan JSON results and create a pickled dictionary of pertinent information for processing.
Args:
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):

50
recon/nmap.py
View File

@@ -13,24 +13,30 @@ from recon.masscan import ParseMasscanOutput
@inherits(ParseMasscanOutput)
class ThreadedNmapScan(luigi.Task):
""" Run nmap against specific targets and ports gained from the ParseMasscanOutput Task.
""" Run ``nmap`` against specific targets and ports gained from the ParseMasscanOutput Task.
nmap commands are structured like the example below.
Install:
``nmap`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``nmap``.
Basic Example:
.. code-block:: console
nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
The corresponding luigi command is shown below.
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
@@ -127,24 +133,30 @@ class ThreadedNmapScan(luigi.Task):
@inherits(ThreadedNmapScan)
class SearchsploitScan(luigi.Task):
""" Run searchcploit against each nmap*.xml file in the TARGET-nmap-results directory and write results to disk.
""" Run ``searchcploit`` against each ``nmap*.xml`` file in the **TARGET-nmap-results** directory and write results to disk.
searchsploit commands are structured like the example below.
Install:
``searchcploit`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``searchcploit``.
Basic Example:
.. code-block:: console
searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
The corresponding luigi command is shown below.
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifies the directory on disk to which all Task results are written *--* Required by upstream Task
threads: number of threads for parallel nmap command execution *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifies the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):

31
recon/web/aquatone.py
View File

@@ -12,24 +12,35 @@ from recon.web.targets import GatherWebTargets
class AquatoneScan(luigi.Task):
""" Screenshot all web targets and generate HTML report.
aquatone commands are structured like the example below.
Install:
.. code-block:: console
cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20
mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone
An example of the corresponding luigi command is shown below.
Basic Example:
``aquatone`` commands are structured like the example below.
``cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20``
Luigi Example:
.. code-block:: python
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
Args:
threads: number of threads for parallel aquatone command execution
scan_timeout: timeout in miliseconds for aquatone port scans
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))

36
recon/web/corscanner.py
View File

@@ -8,31 +8,35 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class CORScannerScan(ExternalProgramTask):
""" Use CORScanner to scan for potential CORS misconfigurations.
CORScanner commands are structured like the example below.
python cors_scan.py -i webtargets.tesla.txt -t 100
An example of the corresponding luigi command is shown below.
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.corscanner CORScannerScan --target-file tesla --top-ports 1000 --interface eth0
""" Use ``CORScanner`` to scan for potential CORS misconfigurations.
Install:
.. code-block:: console
git clone https://github.com/chenjj/CORScanner.git
cd CORScanner
pip install -r requirements.txt
pip install future
Basic Example:
.. code-block:: console
python cors_scan.py -i webtargets.tesla.txt -t 100
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.corscanner CORScannerScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel subjack command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))

41
recon/web/gobuster.py
View File

@@ -14,40 +14,37 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class GobusterScan(luigi.Task):
""" Use gobuster to perform forced browsing.
gobuster commands are structured like the example below.
.. code-block::
gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html
An example of the corresponding luigi command is shown below.
Example:
.. code-block::
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20
""" Use ``gobuster`` to perform forced browsing.
Install:
.. code-block::
.. code-block:: console
go get github.com/OJ/gobuster
git clone https://github.com/epi052/recursive-gobuster.git
Basic Example:
.. code-block:: console
gobuster dir -q -e -k -t 20 -u www.tesla.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -p http://127.0.0.1:8080 -o gobuster.tesla.txt -x php,html
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.gobuster GobusterScan --target-file tesla --top-ports 1000 --interface eth0 --proxy http://127.0.0.1:8080 --extensions php,html --wordlist /usr/share/seclists/Discovery/Web-Content/common.txt --threads 20
Args:
threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
proxy = luigi.Parameter(default=defaults.get("proxy", ""))

60
recon/web/subdomain_takeover.py
View File

@@ -8,24 +8,34 @@ from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class TKOSubsScan(ExternalProgramTask):
""" Use tko-subs to scan for potential subdomain takeovers.
""" Use ``tko-subs`` to scan for potential subdomain takeovers.
tko-subs commands are structured like the example below.
Install:
.. code-block:: console
go get github.com/anshumanbh/tko-subs
cd ~/go/src/github.com/anshumanbh/tko-subs
go build
go install
Basic Example:
.. code-block:: console
tko-subs -domains=tesla.subdomains -data=/root/go/src/github.com/anshumanbh/tko-subs/providers-data.csv -output=tkosubs.tesla.csv
An example of the corresponding luigi command is shown below.
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0
Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):
@@ -77,25 +87,35 @@ class TKOSubsScan(ExternalProgramTask):
@inherits(GatherWebTargets)
class SubjackScan(ExternalProgramTask):
""" Use subjack to scan for potential subdomain takeovers.
""" Use ``subjack`` to scan for potential subdomain takeovers.
subjack commands are structured like the example below.
Install:
.. code-block:: console
go get github.com/haccer/subjack
cd ~/go/src/github.com/haccer/subjack
go build
go install
Basic Example:
.. code-block:: console
subjack -w webtargets.tesla.txt -t 100 -timeout 30 -o subjack.tesla.txt -ssl
An example of the corresponding luigi command is shown below.
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover SubjackScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel subjack command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))

14
recon/web/targets.py
View File

@@ -13,13 +13,13 @@ class GatherWebTargets(luigi.Task):
""" Gather all subdomains as well as any ip addresses known to have a configured web port open.
Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):

25
recon/web/webanalyze.py
View File

@@ -16,21 +16,24 @@ from recon.web.targets import GatherWebTargets
class WebanalyzeScan(luigi.Task):
""" Use webanalyze to determine the technology stack on the given target(s).
webanalyze commands are structured like the example below.
webanalyze -host www.tesla.com -output json
An example of the corresponding luigi command is shown below.
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Install:
.. code-block:: console
go get -u github.com/rverton/webanalyze
# loads new apps.json file from wappalyzer project
webanalyze -update
Basic Example:
.. code-block:: console
webanalyze -host www.tesla.com -output json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel webanalyze command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
@@ -105,7 +108,11 @@ class WebanalyzeScan(luigi.Task):
pass
for url_scheme in ("https://", "http://"):
command = [tool_paths.get("webanalyze"), "-host", f"{url_scheme}{target}"]
command = [
tool_paths.get("webanalyze"),
"-host",
f"{url_scheme}{target}",
]
commands.append(command)
Path(self.output().path).mkdir(parents=True, exist_ok=True)

34
recon/wrappers.py
View File

@@ -21,19 +21,22 @@ from recon.web.webanalyze import WebanalyzeScan
class FullScan(luigi.WrapperTask):
""" Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.
Note:
Because FullScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.
Args:
threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line.
top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned
interface: use the named raw network interface, such as "eth0"
rate: desired rate for transmitting packets (packets per second)
target_file: specifies the file on disk containing a list of ips or domains
results_dir: specifes the directory on disk to which all Task results are written
"""
def requires(self):
@@ -80,19 +83,22 @@ class FullScan(luigi.WrapperTask):
class HTBScan(luigi.WrapperTask):
""" Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.
Note:
Because HTBScan is a wrapper, it requires all Parameters for any of the Scans that it wraps.
Args:
threads: number of threads for parallel gobuster command execution
wordlist: wordlist used for forced browsing
extensions: additional extensions to apply to each item in the wordlist
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
exempt_list: Path to a file providing blacklisted subdomains, one per line.
top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned
interface: use the named raw network interface, such as "eth0"
rate: desired rate for transmitting packets (packets per second)
target_file: specifies the file on disk containing a list of ips or domains
results_dir: specifes the directory on disk to which all Task results are written
"""
def requires(self):