aljaz/recon-pipeline
1
0
Fork 0
mirror of https://github.com/aljazceru/recon-pipeline.git synced 2025-12-20 15:54:25 +01:00
Code Issues Packages Projects Releases Wiki Activity

Completed store results in a database project (#32)

Browse Source 
Co-authored-by: Ryan Good <usafaryangood@gmail.com>

* added initial skeleton; restructured project directories

* removed workers directive from luigi; changed input to tko-subs

* changed masscan command to use config.tool_paths

* linted __init__ files and updated docstring for get_scans

* added per-file-ignores for linting

* recon-pipeline linted

* PoC working for amass results -> db; rudimentary db mgmt commands also

* more linting

* added database management commands to the shell

* db_location passes through to all tasks; masscan results added to db

* removed unused imports from masscan.py

* added ParseNmapOutput class to handle parsing for database storage

* cleaned up repeat code

* searchsploit results stored in db

* lint/format

* gobuster scans now stored in database

* fixed test_recon tests to use db_location

* fixed web tests

* tkosub entries recorded in db

* subjack scan results stored in database

* webanalyze results stored in db

* refactored older commits to use newer helper functions

* refactored older commits to use newer helper functions

* aquatone results stored in database

refactored a few scans to use dbmanager helper functions
refactored db structure wrt headers/screenshots
added 80/443 to web_ports in config.py

* fixed a few queries and re-added webanalyze to FullScan

* view targets/endpoints done

* overhauled nmap parsing

* print all nmap_results good, next to focus on filtering

* complex nmap filters complete

* nmap printing done

* updated pipfile

* view web-technologies complete

* view searchsploit results complete

* removed filesystem code from amass

* targetlist moved to db only

* targets,amass,masscan all cutover to full database; added view ports

* nmap fully db compliant

* aquatone and webtargets db compliant

* gobuster uses db now

* webanalyze db compliant

* all scans except corscanner are db compliant

* recon tests passing

* web tests passing

* linted files

* added tests for helpers.py and parsers.py

* refactored some redundant code

* added tests to pre-commit

* updated amass tests and pre-commit version

* updated recon.targets tests

* updated nmap tests

* updated masscan tests

* updated config tests

* updated web targets tests

* added gobuster tests

* added aquatone tests

* added subdomain takeover and webanalyze tests; updated test data

* removed homegrown sqlite target in favor of the sqla implementation

* added tests for recon-pipeline.py

* fixed cluge function to set __package__ globally

* updated amass tests

* updated targets tests

* updated nmap tests

* updated masscan tests

* updated aquatone tests

* updated nmap tests to account for no searchsploit

* updated nmap tests to account for no searchsploit

* updated masscan tests

* updated subjack/tkosub tests

* updated web targets tests

* updated webanalyze tests

* added corscanner tests

* linted DBManager a bit

* fixed weird cyclic import issue that only happened during docs build; housekeeping

* added models tests, removed test_install dir

* updated docs a bit; sidenav is wonky

* fixed readthedocs requirements.txt

* fixed issue where view results werent populated directly after scan

* added new tests to pipeline; working on docs

* updated a few overlooked view command items

* updated tests to reflect changes to shell

* incremental push of docs update

* documentation done

* updated exploitdb install

* updated exploitdb install

* updated seclists install

* parseamass updates db in the event of no amass output

* removed corscanner

* added pipenv shell to install instructions per @GreaterGoodest

* added pipenv shell to install instructions per @GreaterGoodest

* added check for chromium-browser during aquatone install; closes #26

* added check for old recon-tools dir; updated Path.resolve calls to Path.expanduser.resolve; fixed very specific import bug due to filesystem location

* added CONTIBUTING.md; updated pre-commit hooks/README

* added .gitattributes for linguist reporting

* updated tests

* fixed a few weird bugs found during test

* updated README

* updated asciinema links in README

* updated README with view command video

* updated other location for url scheme /status

* add ability to specify single target using --target (#31)

* updated a few items in docs and moved tool-dict to tools-dir

* fixed issue where removing tempfile without --verbose caused scan to fail
This commit is contained in:
epi052
2020-04-17 10:29:16 -05:00
committed by GitHub
parent ff801dfc6b
commit 6eb3bd8cb0
4682 changed files with 133470 additions and 7368 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.flake8
View File

@@ -3,3 +3,9 @@ max-line-length = 120
select = C,E,F,W,B,B950
ignore = E203, E501, W503
max-complexity = 13
per-file-ignores =
pipeline/recon/__init__.py:F401
pipeline/models/__init__.py:F401
pipeline/recon/web/__init__.py:F401
pipeline/luigi_targets/__init__.py:F401
tests/test_recon/test_parsers.py:F405

1
.gitattributes vendored Normal file
View File

@@ -0,0 +1 @@
docs/* linguist-documentation

32
.github/workflows/pythonapp.yml vendored
View File

@@ -28,14 +28,12 @@ jobs:
with:
args: ". --check"
test-install:
test-shell:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Set up Golang
uses: actions/setup-go@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
@@ -47,8 +45,8 @@ jobs:
pipenv install -d
- name: Test with pytest
run: |
pipenv install pytest cmd2 luigi
pipenv run python -m pytest tests/test_install
pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
pipenv run python -m pytest tests/test_shell
test-recon:
@@ -67,7 +65,7 @@ jobs:
pipenv install -d
- name: Test with pytest
run: |
pipenv install pytest cmd2 luigi
pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
pipenv run python -m pytest tests/test_recon
test-web:
@@ -87,5 +85,25 @@ jobs:
pipenv install -d
- name: Test with pytest
run: |
pipenv install pytest cmd2 luigi
pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
pipenv run python -m pytest tests/test_web
test-models:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Set up pipenv
run: |
python -m pip install --upgrade pip
pip install pipenv
pipenv install -d
- name: Test with pytest
run: |
pipenv install pytest cmd2 luigi sqlalchemy python-libnmap
pipenv run python -m pytest tests/test_models

20
.pre-commit-config.yaml
View File

@@ -4,9 +4,21 @@ repos:
hooks:
- id: black
language_version: python3.7
args: ['.']
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v1.2.3
args: ['pipeline', 'tests/test_web', 'tests/test_recon', 'tests/test_shell', 'tests/test_models']
- repo: https://gitlab.com/pycqa/flake8
rev: 3.7.9
hooks:
- id: flake8
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.5.0 # Use the ref you want to point at
hooks:
- id: trailing-whitespace
- id: debug-statements
- repo: local
hooks:
- id: tests
name: run tests
entry: pytest
language: system
types: [python]
args: ['tests/test_web', 'tests/test_recon', 'tests/test_shell', 'tests/test_models']

402
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,402 @@
# Contributor's guide
<!-- this guide is a modified version of the guide used by the awesome guys that wrote cmd2 -->
First of all, thank you for contributing! Please follow these steps to contribute:
1. Find an issue that needs assistance by searching for the [Help Wanted](https://github.com/epi052/recon-pipeline/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) tag
2. Let us know you're working on it by posting a comment on the issue
3. Follow the [Contribution guidelines](#contribution-guidelines) to start working on the issue
Remember to feel free to ask for help by leaving a comment within the Issue.
Working on your first pull request? You can learn how from this *free* series
[How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github).
###### If you've found a bug that is not on the board, [follow these steps](README.md#found-a-bug).
---
## Contribution guidelines
- [Prerequisites](#prerequisites)
- [Forking the project](#forking-the-project)
- [Creating a branch](#creating-a-branch)
- [Setting up for recon-pipeline development](#setting-up-for-recon-pipeline-development)
- [Making changes](#making-changes)
- [Static code analysis](#static-code-analysis)
- [Running the test suite](#running-the-test-suite)
- [Squashing your commits](#squashing-your-commits)
- [Creating a pull request](#creating-a-pull-request)
- [How we review and merge pull requests](#how-we-review-and-merge-pull-requests)
- [Next steps](#next-steps)
- [Other resources](#other-resources)
- [Advice](#advice)
### Forking the project
#### Setting up your system
1. Install your favorite `git` client
2. Create a parent projects directory on your system. For this guide, it will be assumed that it is `~/projects`.
#### Forking recon-pipeline
1. Go to the top-level recon-pipeline repository: <https://github.com/epi052/recon-pipeline>
2. Click the "Fork" button in the upper right hand corner of the interface
([more details here](https://help.github.com/articles/fork-a-repo/))
3. After the repository has been forked, you will be taken to your copy of the recon-pipeline repo at `your_username/recon-pipeline`
#### Cloning your fork
1. Open a terminal / command line / Bash shell in your projects directory (_e.g.: `~/projects/`_)
2. Clone your fork of recon-pipeline, making sure to replace `your_username` with your GitHub username. This will download the
entire recon-pipeline repo to your projects directory.
```sh
$ git clone https://github.com/your_username/recon-pipeline.git
```
#### Set up your upstream
1. Change directory to the new recon-pipeline directory (`cd recon-pipeline`)
2. Add a remote to the official recon-pipeline repo:
```sh
$ git remote add upstream https://github.com/epi052/recon-pipeline.git
```
Now you have a local copy of the recon-pipeline repo!
#### Maintaining your fork
Now that you have a copy of your fork, there is work you will need to do to keep it current.
##### **Rebasing from upstream**
Do this prior to every time you create a branch for a PR:
1. Make sure you are on the `master` branch
> ```sh
> $ git status
> On branch master
> Your branch is up-to-date with 'origin/master'.
> ```
> If your aren't on `master`, resolve outstanding files and commits and checkout the `master` branch
> ```sh
> $ git checkout master
> ```
2. Do a pull with rebase against `upstream`
> ```sh
> $ git pull --rebase upstream master
> ```
> This will pull down all of the changes to the official master branch, without making an additional commit in your local repo.
3. (_Optional_) Force push your updated master branch to your GitHub fork
> ```sh
> $ git push origin master --force
> ```
> This will overwrite the master branch of your fork.
### Creating a branch
Before you start working, you will need to create a separate branch specific to the issue or feature you're working on.
You will push your work to this branch.
#### Naming your branch
Name the branch something like `23-xxx` where `xxx` is a short description of the changes or feature
you are attempting to add and 23 corresponds to the Issue you're working on.
#### Adding your branch
To create a branch on your local machine (and switch to this branch):
```sh
$ git checkout -b [name_of_your_new_branch]
```
and to push to GitHub:
```sh
$ git push origin [name_of_your_new_branch]
```
##### If you need more help with branching, take a look at _[this](https://github.com/Kunena/Kunena-Forum/wiki/Create-a-new-branch-with-git-and-manage-branches)_.
### Setting up for recon-pipeline development
For doing recon-pipeline development, it is recommended you create a virtual environment and install both the project
dependencies as well as the development dependencies.
#### Create a new environment for recon-pipeline using Pipenv
`recon-pipeline` has support for using [Pipenv](https://docs.pipenv.org/en/latest/) for development.
`Pipenv` essentially combines the features of `pip` and `virtualenv` into a single tool. `recon-pipeline` contains a Pipfile which
makes it extremely easy to setup a `recon-pipeline` development environment using `pipenv`.
To create a virtual environment and install everything needed for `recon-pipeline` development using `pipenv`, do the following
from a GitHub checkout:
```sh
pipenv install --dev
```
To create a new virtualenv, using a specific version of Python you have installed (and on your PATH), use the
--python VERSION flag, like so:
```sh
pipenv install --dev --python 3.7
```
### Making changes
It's your time to shine!
#### How to find code in the recon-pipeline codebase to fix/edit
The recon-pipeline project directory structure is pretty simple and straightforward. All
actual code for recon-pipeline is located underneath the `pipeline` directory. The code to
generate the documentation is in the `docs` directory. Unit tests are in the
`tests` directory. There are various other files in the root directory, but these are
primarily related to continuous integration and release deployment.
#### Changes to the documentation files
If you made changes to any file in the `/docs` directory, you need to build the
Sphinx documentation and make sure your changes look good:
```sh
$ sphinx-build docs/ docs/_build/
```
In order to see the changes, use your web browser of choice to open `docs/_build/index.html`.
### Static code analysis
recon-pipeline uses two code checking tools:
1. [black](https://github.com/psf/black)
2. [flake8](https://github.com/PyCQA/flake8)
#### pre-commit setup
recon-pipeline uses [pre-commit](https://github.com/pre-commit/pre-commit) to automatically run both of its static code analysis tools. From within your
virtual environment's shell, run the following command:
```sh
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
```
With that complete, you should be able to run all the pre-commit hooks.
```text
❯ pre-commit run --all-files
black....................................................................Passed
flake8...................................................................Passed
Trim Trailing Whitespace.................................................Passed
Debug Statements (Python)................................................Passed
run tests................................................................Passed
```
> Please do not ignore any linting errors in code you write or modify, as they are meant to **help** you and to ensure a clean and simple code base.
### Running the test suite
When you're ready to share your code, run the test suite:
```sh
$ cd ~/projects/recon-pipeline
$ python -m pytest tests
```
and ensure all tests pass.
Test coverage can be checked using `coverage`:
```sh
coverage run --source=pipeline -m pytest tests/test_recon/ tests/test_shell/ tests/test_web/ tests/test_models && coverage report -m
=====================================================================================================================================
platform linux -- Python 3.7.5, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/epi/PycharmProjects/recon-pipeline, inifile: pytest.ini
collected 225 items
tests/test_recon/test_amass.py ......... [ 4%]
tests/test_recon/test_config.py ........... [ 8%]
tests/test_recon/test_helpers.py ............. [ 14%]
tests/test_recon/test_masscan.py ....... [ 18%]
tests/test_recon/test_nmap.py ........... [ 23%]
tests/test_recon/test_parsers.py ............................................................. [ 50%]
tests/test_recon/test_targets.py .. [ 51%]
tests/test_shell/test_recon_pipeline_shell.py ................................................................ [ 79%]
tests/test_web/test_aquatone.py ...... [ 82%]
tests/test_web/test_gobuster.py ....... [ 85%]
tests/test_web/test_subdomain_takeover.py ................ [ 92%]
tests/test_web/test_targets.py ... [ 93%]
tests/test_web/test_webanalyze.py ....... [ 96%]
tests/test_models/test_db_manager.py .... [ 98%]
tests/test_models/test_pretty_prints.py ... [100%]
============================================================================================ 225 passed in 20.35s ============================================================================================
Name Stmts Miss Cover Missing
------------------------------------------------------------------------
pipeline/__init__.py 0 0 100%
pipeline/models/__init__.py 0 0 100%
pipeline/models/base_model.py 2 0 100%
pipeline/models/db_manager.py 123 0 100%
pipeline/models/endpoint_model.py 12 0 100%
pipeline/models/header_model.py 12 0 100%
pipeline/models/ip_address_model.py 10 0 100%
pipeline/models/nmap_model.py 47 0 100%
pipeline/models/nse_model.py 12 0 100%
pipeline/models/port_model.py 12 0 100%
pipeline/models/screenshot_model.py 16 0 100%
pipeline/models/searchsploit_model.py 34 0 100%
pipeline/models/target_model.py 18 0 100%
pipeline/models/technology_model.py 28 0 100%
pipeline/recon-pipeline.py 388 5 99% 94, 104-105, 356-358
pipeline/recon/__init__.py 9 0 100%
pipeline/recon/amass.py 66 2 97% 186-187
pipeline/recon/config.py 7 0 100%
pipeline/recon/helpers.py 36 0 100%
pipeline/recon/masscan.py 82 24 71% 83-143
pipeline/recon/nmap.py 120 0 100%
pipeline/recon/parsers.py 68 0 100%
pipeline/recon/targets.py 27 0 100%
pipeline/recon/tool_definitions.py 3 0 100%
pipeline/recon/web/__init__.py 5 0 100%
pipeline/recon/web/aquatone.py 93 0 100%
pipeline/recon/web/gobuster.py 72 0 100%
pipeline/recon/web/subdomain_takeover.py 87 0 100%
pipeline/recon/web/targets.py 27 0 100%
pipeline/recon/web/webanalyze.py 70 0 100%
pipeline/recon/wrappers.py 34 21 38% 35-70, 97-127
------------------------------------------------------------------------
TOTAL 1520 52 97%
```
### Squashing your commits
When you make a pull request, it is preferable for all of your changes to be in one commit. Github has made it very
simple to squash commits now as it's [available through the web interface](https://stackoverflow.com/a/43858707) at
pull request submission time.
### Creating a pull request
#### What is a pull request?
A pull request (PR) is a method of submitting proposed changes to the recon-pipeline
repo (or any repo, for that matter). You will make changes to copies of the
files which make up recon-pipeline in a personal fork, then apply to have them
accepted by the recon-pipeline team.
#### Need help?
GitHub has a good guide on how to contribute to open source [here](https://opensource.guide/how-to-contribute/).
##### Editing via your local fork
1. Perform the maintenance step of rebasing `master`
2. Ensure you're on the `master` branch using `git status`:
```sh
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
```
1. If you're not on master or your working directory is not clean, resolve
any outstanding files/commits and checkout master `git checkout master`
2. Create a branch off of `master` with git: `git checkout -B
branch/name-here`
3. Edit your file(s) locally with the editor of your choice
4. Check your `git status` to see unstaged files
5. Add your edited files: `git add path/to/filename.ext` You can also do: `git
add .` to add all unstaged files. Take care, though, because you can
accidentally add files you don't want added. Review your `git status` first.
6. Commit your edits: `git commit -m "Brief description of commit"`.
7. Squash your commits, if there are more than one
8. Push your commits to your GitHub Fork: `git push -u origin branch/name-here`
9. Once the edits have been committed, you will be prompted to create a pull
request on your fork's GitHub page
10. By default, all pull requests should be against the `master` branch
11. Submit a pull request from your branch to recon-pipeline's `master` branch
12. The title (also called the subject) of your PR should be descriptive of your
changes and succinctly indicate what is being fixed
- Examples: `Add test cases for Unicode support`; `Correct typo in overview documentation`
13. In the body of your PR include a more detailed summary of the changes you
made and why
- If the PR is meant to fix an existing bug/issue, then, at the end of
your PR's description, append the keyword `closes` and #xxxx (where xxxx
is the issue number). Example: `closes #1337`. This tells GitHub to
close the existing issue if the PR is merged.
14. Indicate what local testing you have done (e.g. what OS and version(s) of Python did you run the
unit test suite with)
15. Creating the PR causes our continuous integration (CI) systems to automatically run all of the
unit tests on all supported OSes and all supported versions of Python. You should watch your PR
to make sure that all unit tests pass.
16. If any unit tests fail, you should look at the details and fix the failures. You can then push
the fix to the same branch in your fork. The PR will automatically get updated and the CI system
will automatically run all of the unit tests again.
### How we review and merge pull requests
1. If your changes can merge without conflicts and all unit tests pass, then your pull request (PR) will have a big
green checkbox which says something like "All Checks Passed" next to it. If this is not the case, there will be a
link you can click on to get details regarding what the problem is. It is your responsibility to make sure all unit
tests are passing. Generally a Maintainer will not QA a pull request unless it can merge without conflicts and all
unit tests pass.
2. If a Maintainer reviews a pull request and confirms that the new code does what it is supposed to do without
seeming to introduce any new bugs, and doesn't present any backward compatibility issues, they will merge the pull request.
### Next steps
#### If your PR is accepted
Once your PR is accepted, you may delete the branch you created to submit it.
This keeps your working fork clean.
You can do this with a press of a button on the GitHub PR interface. You can
delete the local copy of the branch with: `git branch -D branch/to-delete-name`
#### If your PR is rejected
Don't worry! You will receive solid feedback from the Maintainers as to
why it was rejected and what changes are needed.
Many pull requests, especially first pull requests, require correction or
updating.
If you have a local copy of the repo, you can make the requested changes and
amend your commit with: `git commit --amend` This will update your existing
commit. When you push it to your fork you will need to do a force push to
overwrite your old commit: `git push --force`
Be sure to post in the PR conversation that you have made the requested changes.
### Other resources
- [PEP 8 Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/)
- [Searching for your issue on GitHub](https://help.github.com/articles/searching-issues/)
- [Creating a new GitHub issue](https://help.github.com/articles/creating-an-issue/)
### Advice
Here is some advice regarding what makes a good pull request (PR) from our perspective:
- Multiple smaller PRs divided by topic are better than a single large PR containing a bunch of unrelated changes
- Good unit/functional tests are very important
- Accurate documentation is also important
- It's best to create a dedicated branch for a PR, use it only for that PR, and delete it once the PR has been merged
- It's good if the branch name is related to the PR contents, even if it's just "fix123" or "add_more_tests"
- Code coverage of the unit tests matters, so try not to decrease it
- Think twice before adding dependencies to third-party libraries (outside of the Python standard library) because it could affect a lot of users
## Acknowledgement
Thanks to the awesome guys at [cmd2](https://github.com/python-cmd2/cmd2) for their fantastic `CONTRIBUTING` file from
which we have borrowed heavily.

17
Pipfile
View File

@@ -4,17 +4,20 @@ url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pytest = "*"
flake8 = "*"
pre-commit = "*"
coverage = "*"
sphinx = "*"
sphinx-argparse = "*"
sphinx-rtd-theme = "*"
sphinxcontrib-napoleon = "*"
[packages]
cmd2 = "*"
flake8 = "*"
luigi = "*"
requests = "*"
gevent = "*"
tldextract = "*"
argparse = "*"
colorama = "*"
future = "*"
sqlalchemy = "*"
python-libnmap = "*"
[requires]
python_version = "3.7"

610
Pipfile.lock generated
View File

@@ -1,7 +1,7 @@
{
"_meta": {
"hash": {
"sha256": "0ec8caf476955e2bb5faa79127061d3f9f7bc50a3fd1e18c9d39871a93335655"
"sha256": "e46cfefa532750883fcc0ace0bcc14795a987589ee6a7db482d67a03310f59ac"
},
"pipfile-spec": 6,
"requires": {
@@ -16,14 +16,6 @@
]
},
"default": {
"argparse": {
"hashes": [
"sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4",
"sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314"
],
"index": "pypi",
"version": "==1.4.0"
},
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
@@ -31,34 +23,19 @@
],
"version": "==19.3.0"
},
"certifi": {
"hashes": [
"sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3",
"sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f"
],
"version": "==2019.11.28"
},
"chardet": {
"hashes": [
"sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
"sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
],
"version": "==3.0.4"
},
"cmd2": {
"hashes": [
"sha256:8872ba0bc73d4678026c2f6d336fed92d1349052cbafabf676beb4245704eba8",
"sha256:ac417a4e9770ebc915c8d43858bfd98497b66a3de0a6ec92daba63aba880b270"
"sha256:377fd4ad2249189865f908f5e59feb4922b98b877cdb6c90ae16516444304572",
"sha256:d339166d8f65d342f37df01b7fb4820f9618209937d12e8f1af6245f12605c3a"
],
"index": "pypi",
"version": "==0.9.25"
"version": "==1.0.1"
},
"colorama": {
"hashes": [
"sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff",
"sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1"
],
"index": "pypi",
"version": "==0.4.3"
},
"docutils": {
@@ -68,92 +45,6 @@
],
"version": "==0.16"
},
"entrypoints": {
"hashes": [
"sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19",
"sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451"
],
"version": "==0.3"
},
"flake8": {
"hashes": [
"sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb",
"sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca"
],
"index": "pypi",
"version": "==3.7.9"
},
"future": {
"hashes": [
"sha256:b1bead90b70cf6ec3f0710ae53a525360fa360d306a86583adc6bf83a4db537d"
],
"index": "pypi",
"version": "==0.18.2"
},
"gevent": {
"hashes": [
"sha256:0774babec518a24d9a7231d4e689931f31b332c4517a771e532002614e270a64",
"sha256:0e1e5b73a445fe82d40907322e1e0eec6a6745ca3cea19291c6f9f50117bb7ea",
"sha256:0ff2b70e8e338cf13bedf146b8c29d475e2a544b5d1fe14045aee827c073842c",
"sha256:107f4232db2172f7e8429ed7779c10f2ed16616d75ffbe77e0e0c3fcdeb51a51",
"sha256:14b4d06d19d39a440e72253f77067d27209c67e7611e352f79fe69e0f618f76e",
"sha256:1b7d3a285978b27b469c0ff5fb5a72bcd69f4306dbbf22d7997d83209a8ba917",
"sha256:1eb7fa3b9bd9174dfe9c3b59b7a09b768ecd496debfc4976a9530a3e15c990d1",
"sha256:2711e69788ddb34c059a30186e05c55a6b611cb9e34ac343e69cf3264d42fe1c",
"sha256:28a0c5417b464562ab9842dd1fb0cc1524e60494641d973206ec24d6ec5f6909",
"sha256:3249011d13d0c63bea72d91cec23a9cf18c25f91d1f115121e5c9113d753fa12",
"sha256:44089ed06a962a3a70e96353c981d628b2d4a2f2a75ea5d90f916a62d22af2e8",
"sha256:4bfa291e3c931ff3c99a349d8857605dca029de61d74c6bb82bd46373959c942",
"sha256:50024a1ee2cf04645535c5ebaeaa0a60c5ef32e262da981f4be0546b26791950",
"sha256:53b72385857e04e7faca13c613c07cab411480822ac658d97fd8a4ddbaf715c8",
"sha256:74b7528f901f39c39cdbb50cdf08f1a2351725d9aebaef212a29abfbb06895ee",
"sha256:7d0809e2991c9784eceeadef01c27ee6a33ca09ebba6154317a257353e3af922",
"sha256:896b2b80931d6b13b5d9feba3d4eebc67d5e6ec54f0cf3339d08487d55d93b0e",
"sha256:8d9ec51cc06580f8c21b41fd3f2b3465197ba5b23c00eb7d422b7ae0380510b0",
"sha256:9f7a1e96fec45f70ad364e46de32ccacab4d80de238bd3c2edd036867ccd48ad",
"sha256:ab4dc33ef0e26dc627559786a4fba0c2227f125db85d970abbf85b77506b3f51",
"sha256:d1e6d1f156e999edab069d79d890859806b555ce4e4da5b6418616322f0a3df1",
"sha256:d752bcf1b98174780e2317ada12013d612f05116456133a6acf3e17d43b71f05",
"sha256:e5bcc4270671936349249d26140c267397b7b4b1381f5ec8b13c53c5b53ab6e1"
],
"index": "pypi",
"version": "==1.4.0"
},
"greenlet": {
"hashes": [
"sha256:000546ad01e6389e98626c1367be58efa613fa82a1be98b0c6fc24b563acc6d0",
"sha256:0d48200bc50cbf498716712129eef819b1729339e34c3ae71656964dac907c28",
"sha256:23d12eacffa9d0f290c0fe0c4e81ba6d5f3a5b7ac3c30a5eaf0126bf4deda5c8",
"sha256:37c9ba82bd82eb6a23c2e5acc03055c0e45697253b2393c9a50cef76a3985304",
"sha256:51155342eb4d6058a0ffcd98a798fe6ba21195517da97e15fca3db12ab201e6e",
"sha256:51503524dd6f152ab4ad1fbd168fc6c30b5795e8c70be4410a64940b3abb55c0",
"sha256:7457d685158522df483196b16ec648b28f8e847861adb01a55d41134e7734122",
"sha256:8041e2de00e745c0e05a502d6e6db310db7faa7c979b3a5877123548a4c0b214",
"sha256:81fcd96a275209ef117e9ec91f75c731fa18dcfd9ffaa1c0adbdaa3616a86043",
"sha256:853da4f9563d982e4121fed8c92eea1a4594a2299037b3034c3c898cb8e933d6",
"sha256:8b4572c334593d449113f9dc8d19b93b7b271bdbe90ba7509eb178923327b625",
"sha256:9416443e219356e3c31f1f918a91badf2e37acf297e2fa13d24d1cc2380f8fbc",
"sha256:9854f612e1b59ec66804931df5add3b2d5ef0067748ea29dc60f0efdcda9a638",
"sha256:99a26afdb82ea83a265137a398f570402aa1f2b5dfb4ac3300c026931817b163",
"sha256:a19bf883b3384957e4a4a13e6bd1ae3d85ae87f4beb5957e35b0be287f12f4e4",
"sha256:a9f145660588187ff835c55a7d2ddf6abfc570c2651c276d3d4be8a2766db490",
"sha256:ac57fcdcfb0b73bb3203b58a14501abb7e5ff9ea5e2edfa06bb03035f0cff248",
"sha256:bcb530089ff24f6458a81ac3fa699e8c00194208a724b644ecc68422e1111939",
"sha256:beeabe25c3b704f7d56b573f7d2ff88fc99f0138e43480cecdfcaa3b87fe4f87",
"sha256:d634a7ea1fc3380ff96f9e44d8d22f38418c1c381d5fac680b272d7d90883720",
"sha256:d97b0661e1aead761f0ded3b769044bb00ed5d33e1ec865e891a8b128bf7c656",
"sha256:e538b8dae561080b542b0f5af64d47ef859f22517f7eca617bb314e0e03fd7ef"
],
"markers": "platform_python_implementation == 'CPython'",
"version": "==0.4.15"
},
"idna": {
"hashes": [
"sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
"sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
],
"version": "==2.8"
},
"lockfile": {
"hashes": [
"sha256:6aed02de03cba24efabcd600b30540140634fc06cfa603822d508d5361e9f799",
@@ -163,37 +54,16 @@
},
"luigi": {
"hashes": [
"sha256:c2b3dcecc565fe77920553434ed475fa21f562d4b76da6bd1a179a8b732fcc9e"
"sha256:b6dfef1b4cfb821e9bcd7ecdcb8cf9ced35aa2e4475f54d411bbf0a371af03dd"
],
"index": "pypi",
"version": "==2.8.11"
},
"mccabe": {
"hashes": [
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
],
"version": "==0.6.1"
},
"pycodestyle": {
"hashes": [
"sha256:95a2219d12372f05704562a14ec30bc76b05a5b297b21a5dfe3f6fac3491ae56",
"sha256:e40a936c9a450ad81df37f549d676d127b1b66000a6c500caa2b085bc0ca976c"
],
"version": "==2.5.0"
},
"pyflakes": {
"hashes": [
"sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
"sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
],
"version": "==2.1.1"
"version": "==2.8.12"
},
"pyperclip": {
"hashes": [
"sha256:979325468ccf682104d5dcaf753f869868100631301d3e72f47babdea5700d1c"
"sha256:b75b975160428d84608c26edba2dec146e7799566aea42c1fe1b32e72b6028f2"
],
"version": "==1.7.0"
"version": "==1.8.0"
},
"python-daemon": {
"hashes": [
@@ -209,20 +79,12 @@
],
"version": "==2.8.1"
},
"requests": {
"python-libnmap": {
"hashes": [
"sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
"sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
"sha256:9d14919142395aaca952e129398f0c7371c0f0a034c63de6dad99cd7050177ad"
],
"index": "pypi",
"version": "==2.22.0"
},
"requests-file": {
"hashes": [
"sha256:75c175eed739270aec3c5279ffd74e6527dada275c5c0d76b5817e9c86bb7dea",
"sha256:8f04aa6201bacda0567e7ac7f677f1499b0fc76b22140c54bc06edf1ba92e2fa"
],
"version": "==1.4.3"
"version": "==0.7.0"
},
"six": {
"hashes": [
@@ -231,13 +93,12 @@
],
"version": "==1.14.0"
},
"tldextract": {
"sqlalchemy": {
"hashes": [
"sha256:16b2f7e81d89c2a5a914d25bdbddd3932c31a6b510db886c3ce0764a195c0ee7",
"sha256:9aa21a1f7827df4209e242ec4fc2293af5940ec730cde46ea80f66ed97bfc808"
"sha256:c4cca4aed606297afbe90d4306b49ad3a4cd36feb3f87e4bfd655c57fd9ef445"
],
"index": "pypi",
"version": "==2.2.2"
"version": "==1.3.15"
},
"tornado": {
"hashes": [
@@ -251,6 +112,426 @@
],
"version": "==5.1.1"
},
"wcwidth": {
"hashes": [
"sha256:cafe2186b3c009a04067022ce1dcd79cb38d8d65ee4f4791b8888d6599d1bbe1",
"sha256:ee73862862a156bf77ff92b09034fc4825dd3af9cf81bc5b360668d425f3c5f1"
],
"version": "==0.1.9"
}
},
"develop": {
"alabaster": {
"hashes": [
"sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359",
"sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02"
],
"version": "==0.7.12"
},
"appdirs": {
"hashes": [
"sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92",
"sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"
],
"version": "==1.4.3"
},
"attrs": {
"hashes": [
"sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
"sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
],
"version": "==19.3.0"
},
"babel": {
"hashes": [
"sha256:1aac2ae2d0d8ea368fa90906567f5c08463d98ade155c0c4bfedd6a0f7160e38",
"sha256:d670ea0b10f8b723672d3a6abeb87b565b244da220d76b4dba1b66269ec152d4"
],
"version": "==2.8.0"
},
"certifi": {
"hashes": [
"sha256:1d987a998c75633c40847cc966fcf5904906c920a7f17ef374f5aa4282abd304",
"sha256:51fcb31174be6e6664c5f69e3e1691a2d72a1a12e90f872cbdb1567eb47b6519"
],
"version": "==2020.4.5.1"
},
"cfgv": {
"hashes": [
"sha256:1ccf53320421aeeb915275a196e23b3b8ae87dea8ac6698b1638001d4a486d53",
"sha256:c8e8f552ffcc6194f4e18dd4f68d9aef0c0d58ae7e7be8c82bee3c5e9edfa513"
],
"version": "==3.1.0"
},
"chardet": {
"hashes": [
"sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
"sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
],
"version": "==3.0.4"
},
"coverage": {
"hashes": [
"sha256:03f630aba2b9b0d69871c2e8d23a69b7fe94a1e2f5f10df5049c0df99db639a0",
"sha256:046a1a742e66d065d16fb564a26c2a15867f17695e7f3d358d7b1ad8a61bca30",
"sha256:0a907199566269e1cfa304325cc3b45c72ae341fbb3253ddde19fa820ded7a8b",
"sha256:165a48268bfb5a77e2d9dbb80de7ea917332a79c7adb747bd005b3a07ff8caf0",
"sha256:1b60a95fc995649464e0cd48cecc8288bac5f4198f21d04b8229dc4097d76823",
"sha256:1f66cf263ec77af5b8fe14ef14c5e46e2eb4a795ac495ad7c03adc72ae43fafe",
"sha256:2e08c32cbede4a29e2a701822291ae2bc9b5220a971bba9d1e7615312efd3037",
"sha256:3844c3dab800ca8536f75ae89f3cf566848a3eb2af4d9f7b1103b4f4f7a5dad6",
"sha256:408ce64078398b2ee2ec08199ea3fcf382828d2f8a19c5a5ba2946fe5ddc6c31",
"sha256:443be7602c790960b9514567917af538cac7807a7c0c0727c4d2bbd4014920fd",
"sha256:4482f69e0701139d0f2c44f3c395d1d1d37abd81bfafbf9b6efbe2542679d892",
"sha256:4a8a259bf990044351baf69d3b23e575699dd60b18460c71e81dc565f5819ac1",
"sha256:513e6526e0082c59a984448f4104c9bf346c2da9961779ede1fc458e8e8a1f78",
"sha256:5f587dfd83cb669933186661a351ad6fc7166273bc3e3a1531ec5c783d997aac",
"sha256:62061e87071497951155cbccee487980524d7abea647a1b2a6eb6b9647df9006",
"sha256:641e329e7f2c01531c45c687efcec8aeca2a78a4ff26d49184dce3d53fc35014",
"sha256:65a7e00c00472cd0f59ae09d2fb8a8aaae7f4a0cf54b2b74f3138d9f9ceb9cb2",
"sha256:6ad6ca45e9e92c05295f638e78cd42bfaaf8ee07878c9ed73e93190b26c125f7",
"sha256:73aa6e86034dad9f00f4bbf5a666a889d17d79db73bc5af04abd6c20a014d9c8",
"sha256:7c9762f80a25d8d0e4ab3cb1af5d9dffbddb3ee5d21c43e3474c84bf5ff941f7",
"sha256:85596aa5d9aac1bf39fe39d9fa1051b0f00823982a1de5766e35d495b4a36ca9",
"sha256:86a0ea78fd851b313b2e712266f663e13b6bc78c2fb260b079e8b67d970474b1",
"sha256:8a620767b8209f3446197c0e29ba895d75a1e272a36af0786ec70fe7834e4307",
"sha256:922fb9ef2c67c3ab20e22948dcfd783397e4c043a5c5fa5ff5e9df5529074b0a",
"sha256:9fad78c13e71546a76c2f8789623eec8e499f8d2d799f4b4547162ce0a4df435",
"sha256:a37c6233b28e5bc340054cf6170e7090a4e85069513320275a4dc929144dccf0",
"sha256:c3fc325ce4cbf902d05a80daa47b645d07e796a80682c1c5800d6ac5045193e5",
"sha256:cda33311cb9fb9323958a69499a667bd728a39a7aa4718d7622597a44c4f1441",
"sha256:db1d4e38c9b15be1521722e946ee24f6db95b189d1447fa9ff18dd16ba89f732",
"sha256:eda55e6e9ea258f5e4add23bcf33dc53b2c319e70806e180aecbff8d90ea24de",
"sha256:f372cdbb240e09ee855735b9d85e7f50730dcfb6296b74b95a3e5dea0615c4c1"
],
"index": "pypi",
"version": "==5.0.4"
},
"distlib": {
"hashes": [
"sha256:2e166e231a26b36d6dfe35a48c4464346620f8645ed0ace01ee31822b288de21"
],
"version": "==0.3.0"
},
"docutils": {
"hashes": [
"sha256:0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af",
"sha256:c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc"
],
"version": "==0.16"
},
"entrypoints": {
"hashes": [
"sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19",
"sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451"
],
"version": "==0.3"
},
"filelock": {
"hashes": [
"sha256:18d82244ee114f543149c66a6e0c14e9c4f8a1044b5cdaadd0f82159d6a6ff59",
"sha256:929b7d63ec5b7d6b71b0fa5ac14e030b3f70b75747cef1b10da9b879fef15836"
],
"version": "==3.0.12"
},
"flake8": {
"hashes": [
"sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb",
"sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca"
],
"index": "pypi",
"version": "==3.7.9"
},
"identify": {
"hashes": [
"sha256:2bb8760d97d8df4408f4e805883dad26a2d076f04be92a10a3e43f09c6060742",
"sha256:faffea0fd8ec86bb146ac538ac350ed0c73908326426d387eded0bcc9d077522"
],
"version": "==1.4.14"
},
"idna": {
"hashes": [
"sha256:7588d1c14ae4c77d74036e8c22ff447b26d0fde8f007354fd48a7814db15b7cb",
"sha256:a068a21ceac8a4d63dbfd964670474107f541babbd2250d61922f029858365fa"
],
"version": "==2.9"
},
"imagesize": {
"hashes": [
"sha256:6965f19a6a2039c7d48bca7dba2473069ff854c36ae6f19d2cde309d998228a1",
"sha256:b1f6b5a4eab1f73479a50fb79fcf729514a900c341d8503d62a62dbc4127a2b1"
],
"version": "==1.2.0"
},
"importlib-metadata": {
"hashes": [
"sha256:2a688cbaa90e0cc587f1df48bdc97a6eadccdcd9c35fb3f976a09e3b5016d90f",
"sha256:34513a8a0c4962bc66d35b359558fd8a5e10cd472d37aec5f66858addef32c1e"
],
"markers": "python_version < '3.8'",
"version": "==1.6.0"
},
"jinja2": {
"hashes": [
"sha256:93187ffbc7808079673ef52771baa950426fd664d3aad1d0fa3e95644360e250",
"sha256:b0eaf100007721b5c16c1fc1eecb87409464edc10469ddc9a22a27a99123be49"
],
"version": "==2.11.1"
},
"markupsafe": {
"hashes": [
"sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473",
"sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161",
"sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235",
"sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5",
"sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42",
"sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff",
"sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b",
"sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1",
"sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e",
"sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183",
"sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66",
"sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b",
"sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1",
"sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15",
"sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1",
"sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e",
"sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b",
"sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905",
"sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735",
"sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d",
"sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e",
"sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d",
"sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c",
"sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21",
"sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2",
"sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5",
"sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b",
"sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6",
"sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f",
"sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f",
"sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2",
"sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7",
"sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be"
],
"version": "==1.1.1"
},
"mccabe": {
"hashes": [
"sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
],
"version": "==0.6.1"
},
"more-itertools": {
"hashes": [
"sha256:5dd8bcf33e5f9513ffa06d5ad33d78f31e1931ac9a18f33d37e77a180d393a7c",
"sha256:b1ddb932186d8a6ac451e1d95844b382f55e12686d51ca0c68b6f61f2ab7a507"
],
"version": "==8.2.0"
},
"nodeenv": {
"hashes": [
"sha256:5b2438f2e42af54ca968dd1b374d14a1194848955187b0e5e4be1f73813a5212"
],
"version": "==1.3.5"
},
"packaging": {
"hashes": [
"sha256:3c292b474fda1671ec57d46d739d072bfd495a4f51ad01a055121d81e952b7a3",
"sha256:82f77b9bee21c1bafbf35a84905d604d5d1223801d639cf3ed140bd651c08752"
],
"version": "==20.3"
},
"pluggy": {
"hashes": [
"sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
"sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
],
"version": "==0.13.1"
},
"pockets": {
"hashes": [
"sha256:68597934193c08a08eb2bf6a1d85593f627c22f9b065cc727a4f03f669d96d86",
"sha256:9320f1a3c6f7a9133fe3b571f283bcf3353cd70249025ae8d618e40e9f7e92b3"
],
"version": "==0.9.1"
},
"pre-commit": {
"hashes": [
"sha256:487c675916e6f99d355ec5595ad77b325689d423ef4839db1ed2f02f639c9522",
"sha256:c0aa11bce04a7b46c5544723aedf4e81a4d5f64ad1205a30a9ea12d5e81969e1"
],
"index": "pypi",
"version": "==2.2.0"
},
"py": {
"hashes": [
"sha256:5e27081401262157467ad6e7f851b7aa402c5852dbcb3dae06768434de5752aa",
"sha256:c20fdd83a5dbc0af9efd622bee9a5564e278f6380fffcacc43ba6f43db2813b0"
],
"version": "==1.8.1"
},
"pycodestyle": {
"hashes": [
"sha256:95a2219d12372f05704562a14ec30bc76b05a5b297b21a5dfe3f6fac3491ae56",
"sha256:e40a936c9a450ad81df37f549d676d127b1b66000a6c500caa2b085bc0ca976c"
],
"version": "==2.5.0"
},
"pyflakes": {
"hashes": [
"sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
"sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
],
"version": "==2.1.1"
},
"pygments": {
"hashes": [
"sha256:647344a061c249a3b74e230c739f434d7ea4d8b1d5f3721bc0f3558049b38f44",
"sha256:ff7a40b4860b727ab48fad6360eb351cc1b33cbf9b15a0f689ca5353e9463324"
],
"version": "==2.6.1"
},
"pyparsing": {
"hashes": [
"sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
"sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
],
"version": "==2.4.7"
},
"pytest": {
"hashes": [
"sha256:0e5b30f5cb04e887b91b1ee519fa3d89049595f428c1db76e73bd7f17b09b172",
"sha256:84dde37075b8805f3d1f392cc47e38a0e59518fb46a431cfdaf7cf1ce805f970"
],
"index": "pypi",
"version": "==5.4.1"
},
"pytz": {
"hashes": [
"sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
"sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
],
"version": "==2019.3"
},
"pyyaml": {
"hashes": [
"sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97",
"sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76",
"sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2",
"sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648",
"sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf",
"sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f",
"sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2",
"sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee",
"sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d",
"sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c",
"sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a"
],
"version": "==5.3.1"
},
"requests": {
"hashes": [
"sha256:43999036bfa82904b6af1d99e4882b560e5e2c68e5c4b0aa03b655f3d7d73fee",
"sha256:b3f43d496c6daba4493e7c431722aeb7dbc6288f52a6e04e7b6023b0247817e6"
],
"version": "==2.23.0"
},
"six": {
"hashes": [
"sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
"sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
],
"version": "==1.14.0"
},
"snowballstemmer": {
"hashes": [
"sha256:209f257d7533fdb3cb73bdbd24f436239ca3b2fa67d56f6ff88e86be08cc5ef0",
"sha256:df3bac3df4c2c01363f3dd2cfa78cce2840a79b9f1c2d2de9ce8d31683992f52"
],
"version": "==2.0.0"
},
"sphinx": {
"hashes": [
"sha256:6a099e6faffdc3ceba99ca8c2d09982d43022245e409249375edf111caf79ed3",
"sha256:b63a0c879c4ff9a4dffcb05217fa55672ce07abdeb81e33c73303a563f8d8901"
],
"index": "pypi",
"version": "==3.0.0"
},
"sphinx-argparse": {
"hashes": [
"sha256:60ab98f80ffd38731d62e267171388a421abbc96c74901b7785a8e058b438c17"
],
"index": "pypi",
"version": "==0.2.5"
},
"sphinx-rtd-theme": {
"hashes": [
"sha256:00cf895504a7895ee433807c62094cf1e95f065843bf3acd17037c3e9a2becd4",
"sha256:728607e34d60456d736cc7991fd236afb828b21b82f956c5ea75f94c8414040a"
],
"index": "pypi",
"version": "==0.4.3"
},
"sphinxcontrib-applehelp": {
"hashes": [
"sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a",
"sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58"
],
"version": "==1.0.2"
},
"sphinxcontrib-devhelp": {
"hashes": [
"sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e",
"sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4"
],
"version": "==1.0.2"
},
"sphinxcontrib-htmlhelp": {
"hashes": [
"sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f",
"sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b"
],
"version": "==1.0.3"
},
"sphinxcontrib-jsmath": {
"hashes": [
"sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178",
"sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8"
],
"version": "==1.0.1"
},
"sphinxcontrib-napoleon": {
"hashes": [
"sha256:407382beed396e9f2d7f3043fad6afda95719204a1e1a231ac865f40abcbfcf8",
"sha256:711e41a3974bdf110a484aec4c1a556799eb0b3f3b897521a018ad7e2db13fef"
],
"index": "pypi",
"version": "==0.7"
},
"sphinxcontrib-qthelp": {
"hashes": [
"sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72",
"sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6"
],
"version": "==1.0.3"
},
"sphinxcontrib-serializinghtml": {
"hashes": [
"sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc",
"sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a"
],
"version": "==1.1.4"
},
"toml": {
"hashes": [
"sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c",
"sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e"
],
"version": "==0.10.0"
},
"urllib3": {
"hashes": [
"sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc",
@@ -258,13 +539,26 @@
],
"version": "==1.25.8"
},
"virtualenv": {
"hashes": [
"sha256:6ea131d41c477f6c4b7863948a9a54f7fa196854dbef73efbdff32b509f4d8bf",
"sha256:94f647e12d1e6ced2541b93215e51752aecbd1bbb18eb1816e2867f7532b1fe1"
],
"version": "==20.0.16"
},
"wcwidth": {
"hashes": [
"sha256:8fd29383f539be45b20bd4df0dc29c20ba48654a41e661925e612311e9f3c603",
"sha256:f28b3e8a6483e5d49e7f8949ac1a78314e740333ae305b4ba5defd3e74fb37a8"
"sha256:cafe2186b3c009a04067022ce1dcd79cb38d8d65ee4f4791b8888d6599d1bbe1",
"sha256:ee73862862a156bf77ff92b09034fc4825dd3af9cf81bc5b360668d425f3c5f1"
],
"version": "==0.1.8"
}
"version": "==0.1.9"
},
"develop": {}
"zipp": {
"hashes": [
"sha256:aa36550ff0c0b7ef7fa639055d797116ee891440eac1a56f378e2d3179e0320b",
"sha256:c599e4d75c98f6798c509911d08a22e6c021d074469042177c8c86fb92eefd96"
],
"version": "==3.1.0"
}
}
}

171
README.md
View File

@@ -2,25 +2,40 @@
![version](https://img.shields.io/github/v/release/epi052/recon-pipeline?style=for-the-badge)
![Python application](https://img.shields.io/github/workflow/status/epi052/recon-pipeline/recon-pipeline%20build?style=for-the-badge)
![code coverage](https://img.shields.io/badge/coverage-97%25-blue?style=for-the-badge)
![python](https://img.shields.io/badge/python-3.7-informational?style=for-the-badge)
![luigi](https://img.shields.io/github/pipenv/locked/dependency-version/epi052/recon-pipeline/luigi?style=for-the-badge)
![cmd2](https://img.shields.io/github/pipenv/locked/dependency-version/epi052/recon-pipeline/cmd2?style=for-the-badge)
![SQLAlchemy](https://img.shields.io/github/pipenv/locked/dependency-version/epi052/recon-pipeline/SQLAlchemy?style=for-the-badge)
![python-libnmap](https://img.shields.io/github/pipenv/locked/dependency-version/epi052/recon-pipeline/python-libnmap?style=for-the-badge)
![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge)
There are an [accompanying set of blog posts](https://epi052.gitlab.io/notes-to-self/blog/2019-09-01-how-to-build-an-automated-recon-pipeline-with-python-and-luigi/) detailing the development process and underpinnings of the pipeline. Feel free to check them out if you're so inclined, but they're in no way required reading to use the tool.
Check out [recon-pipeline's readthedocs entry](https://recon-pipeline.readthedocs.io/) for some more in depth information than what this README provides.
Table of Contents
-----------------
- [Installation](#installation)
- [Defining Scope](#defining-a-scans-scope)
- [Example Scan](#example-scan)
- [Viewing Results](#viewing-results)
- [Chaining Results w/ Commands](#chaining-results-w-commands)
- [Choosing a Scheduler](#choosing-a-scheduler)
- [Found a Bug?](#found-a-bug)
- [Special Thanks](#special-thanks)
## Installation
> Automatic installation tested on kali 2019.4 and Ubuntu 18.04
There are two primary phases for installation:
1. prior to [cmd2](https://github.com/python-cmd2/cmd2) being installed
1. prior to the python dependencies being installed
2. everything else
First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below), starting with [pipenv](https://github.com/pypa/pipenv)
First, the manual steps to get dependencies installed in a virtual environment are as follows, starting with [pipenv](https://github.com/pypa/pipenv)
### Kali
```bash
@@ -40,27 +55,30 @@ bash
```bash
git clone https://github.com/epi052/recon-pipeline.git
cd recon-pipeline
pipenv install cmd2
pipenv install
pipenv shell
```
After installing the python dependencies, the `recon-pipeline` shell provides its own [install](https://recon-pipeline.readthedocs.io/en/latest/api/commands.html#install) command (seen below). A simple `install all` will handle all additional installation steps.
[![asciicast](https://asciinema.org/a/AxFd1SaLVx7mQdxqQBLfh6aqj.svg)](https://asciinema.org/a/AxFd1SaLVx7mQdxqQBLfh6aqj)
> Ubuntu-18.04 Note (and newer kali versions): You may consider running `sudo -v` prior to running `./recon-pipeline.py`. `sudo -v` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password. It'll work either way though.
Once manual installation of [cmd2](https://github.com/python-cmd2/cmd2) is complete, the `recon-pipeline` shell provides its own `install` command (seen below). A simple `install all` will handle all installation steps.
Individual tools may be installed by running `install TOOLNAME` where `TOOLNAME` is one of the known tools that make
up the pipeline.
> Ubuntu-18.04 Note: You may consider running `sudo -v` prior to running `./recon-pipeline.py`. `sudo -v` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password. It'll work either way though.
The installer maintains a (naive) list of installed tools at `~/.local/recon-pipeline/tools/.tool-dict.pkl`. The installer in no way attempts to be a package manager. It knows how to execute the steps necessary to install its tools. Beyond that, it's like Jon Snow, **it knows nothing**.
[![asciicast](https://asciinema.org/a/294414.svg)](https://asciinema.org/a/294414)
[![asciicast](https://asciinema.org/a/318395.svg)](https://asciinema.org/a/318395)
## Command Execution
## Defining a Scan's Scope
Command execution is handled through the `recon-pipeline` shell (seen below).
**New in v0.9.0**: In the event you're scanning a single ip address or host, simply use `--target`. It accepts a single target and works in conjunction with `--exempt-list` if specified.
[![asciicast](https://asciinema.org/a/293302.svg)](https://asciinema.org/a/293302)
```text
scan HTBScan --target 10.10.10.183 --top-ports 1000
```
### Target File and Exempt List File (defining scope)
The pipeline expects a file that describes the target's scope to be provided as an argument to the `--target-file` option. The target file can consist of domains, ip addresses, and ip ranges, one per line.
In order to scan more than one host at a time, the pipeline needs a file that describes the target's scope to be provided as an argument to the `--target-file` option. The target file can consist of domains, ip addresses, and ip ranges, one per line.
```text
tesla.com
@@ -78,7 +96,119 @@ feedback.tesla.com
...
```
### Using a Scheduler
## Example Scan
Here are the steps the video below takes to scan tesla.com.
Create a targetfile
```bash
# use virtual environment
pipenv shell
# create targetfile; a targetfile is required for all scans
mkdir /root/bugcrowd/tesla
cd /root/bugcrowd/tesla
echo tesla.com > tesla-targetfile
# create a blacklist (if necessary based on target's scope)
echo energysupport.tesla.com > tesla-blacklist
echo feedback.tesla.com >> tesla-blacklist
echo employeefeedback.tesla.com >> tesla-blacklist
echo ir.tesla.com >> tesla-blacklist
# drop into the interactive shell
/root/PycharmProjects/recon-pipeline/pipeline/recon-pipeline.py
recon-pipeline>
```
Create a new database to store scan results
```bash
recon-pipeline> database attach
1. create new database
Your choice? 1
new database name? (recommend something unique for this target)
-> tesla-scan
[*] created database @ /home/epi/.local/recon-pipeline/databases/tesla-scan
[+] attached to sqlite database @ /home/epi/.local/recon-pipeline/databases/tesla-scan
[db-1] recon-pipeline>
```
Scan the target
```bash
[db-1] recon-pipeline> scan FullScan --exempt-list tesla-blacklist --target-file tesla-targetfile --interface eno1 --top-ports 2000 --rate 1200
[-] FullScan queued
[-] TKOSubsScan queued
[-] GatherWebTargets queued
[-] ParseAmassOutput queued
[-] AmassScan queued
[-] ParseMasscanOutput queued
[-] MasscanScan queued
[-] WebanalyzeScan queued
[-] SearchsploitScan queued
[-] ThreadedNmapScan queued
[-] SubjackScan queued
[-] AquatoneScan queued
[-] GobusterScan queued
[db-1] recon-pipeline>
```
The same steps can be seen in realtime in the linked video below.
[![asciicast](https://asciinema.org/a/318397.svg)](https://asciinema.org/a/318397)
## Viewing Results
As of version 0.9.0, scan results are stored in a database located (by default) at `~/.local/recon-pipeline/databases`. Databases themselves are managed through the [database command](https://recon-pipeline.readthedocs.io/en/latest/api/commands.html#database) while viewing their contents is done via [view command](https://recon-pipeline.readthedocs.io/en/latest/api/commands.html#view-command).
The view command allows one to inspect different pieces of scan information via the following sub-commands
- endpoints (gobuster results)
- nmap-scans
- ports
- searchsploit-results
- targets
- web-technologies (webanalyze results)
Each of the sub-commands has a list of tab-completable options and values that can help drilling down to the data you care about.
All of the subcommands offer a `--paged` option for dealing with large amounts of output. `--paged` will show you one page of output at a time (using `less` under the hood).
A few examples of different view commands are shown below.
[![asciicast](https://asciinema.org/a/KtiV1ihl16DLyYpapyrmjIplk.svg)](https://asciinema.org/a/KtiV1ihl16DLyYpapyrmjIplk)
## Chaining Results w/ Commands
All of the results can be **piped out to other commands**. Let’s say you want to feed some results from recon-pipeline into another tool that isn’t part of the pipeline. Simply using a normal unix pipe `|` followed by the next command will get that done for you. Below is an example of piping targets into [gau](https://github.com/lc/gau)
```text
[db-2] recon-pipeline> view targets --paged
3.tesla.cn
3.tesla.com
api-internal.sn.tesla.services
api-toolbox.tesla.com
api.mp.tesla.services
api.sn.tesla.services
api.tesla.cn
api.toolbox.tb.tesla.services
...
[db-2] recon-pipeline> view targets | gau
https://3.tesla.com/pt_PT/model3/design
https://3.tesla.com/pt_PT/model3/design?redirect=no
https://3.tesla.com/robots.txt
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-160x160.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-16x16.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-196x196.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-32x32.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-96x96.png?2
https://3.tesla.com/sv_SE/model3/design
...
```
For more examples of view, please see the [documentation](https://recon-pipeline.readthedocs.io/en/latest/overview/viewing_results.html#).
## Choosing a Scheduler
The backbone of this pipeline is spotify's [luigi](https://github.com/spotify/luigi) batch process management framework. Luigi uses the concept of a scheduler in order to manage task execution. Two types of scheduler are available, a local scheduler and a central scheduler. The local scheduler is useful for development and debugging while the central scheduler provides the following two benefits:
@@ -91,11 +221,24 @@ and running easily.
The other option is to add `--local-scheduler` to your `scan` command from within the `recon-pipeline` shell.
## Found a bug?
<!-- this section is a modified version of what's used by the awesome guys that wrote cmd2 -->
If you think you've found a bug, please first read through the open [Issues](https://github.com/epi052/recon-pipeline/issues). If you're confident it's a new bug, go ahead and create a new GitHub issue. Be sure to include as much information as possible so we can reproduce the bug. At a minimum, please state the following:
* ``recon-pipeline`` version
* Python version
* OS name and version
* How to reproduce the bug
* Include any traceback or error message associated with the bug
## Special Thanks
- [@aringo](https://github.com/aringo) for his help on the precursor to this tool
- [@kernelsndrs](https://github.com/kernelsndrs) for identifying a few bugs after initial launch
- [@GreaterGoodest](https://github.com/GreaterGoodest) for identifying bugs and the project's first PR!
- The [cmd2](https://github.com/python-cmd2/cmd2) team for a lot of inspiration for project layout and documentation

44
docs/api/commands.rst
View File

@@ -1,8 +1,21 @@
.. toctree::
:maxdepth: 1
:hidden:
.. _commands-ref-label:
Commands
========
``recon-pipeline`` provides three commands ``install``, ``scan``, and ``status``. All other commands are inherited
from `cmd2 <https://github.com/python-cmd2/cmd2>`_.
``recon-pipeline`` provides a handful of commands:
- ``install``
- ``scan``
- ``status``
- ``database``
- ``view``
All other available commands are inherited from `cmd2 <https://github.com/python-cmd2/cmd2>`_.
.. _install_command:
@@ -10,17 +23,28 @@ install
#######
.. argparse::
:module: recon
:module: pipeline.recon
:func: install_parser
:prog: install
.. _database_command:
database
########
.. argparse::
:module: pipeline.recon
:func: database_parser
:prog: database
.. _scan_command:
scan
####
.. argparse::
:module: recon
:module: pipeline.recon
:func: scan_parser
:prog: scan
@@ -30,6 +54,16 @@ status
######
.. argparse::
:module: recon
:module: pipeline.recon
:func: status_parser
:prog: status
.. _view_command:
view
#######
.. argparse::
:module: pipeline.recon
:func: view_parser
:prog: view

1
docs/api/index.rst
View File

@@ -7,3 +7,4 @@ API Reference
scanners
parsers
commands
models

11
docs/api/manager.rst Normal file
View File

@@ -0,0 +1,11 @@
.. toctree::
:maxdepth: 1
:hidden:
.. _db_manager_label:
Database Manager
================
.. autoclass:: pipeline.models.db_manager.DBManager
:members:

66
docs/api/models.rst Normal file
View File

@@ -0,0 +1,66 @@
.. toctree::
:maxdepth: 1
:hidden:
.. _models-ref-label:
Database Models
===============
.. image:: ../img/database-design.png
Target Model
############
.. autoclass:: pipeline.models.target_model.Target
Endpoint Model
##############
.. autoclass:: pipeline.models.endpoint_model.Endpoint
Header Model
############
.. autoclass:: pipeline.models.header_model.Header
IP Address Model
################
.. autoclass:: pipeline.models.ip_address_model.IPAddress
Nmap Model
##########
.. autoclass:: pipeline.models.nmap_model.NmapResult
Nmap Scripting Engine Model
###########################
.. autoclass:: pipeline.models.nse_model.NSEResult
Port Model
##########
.. autoclass:: pipeline.models.port_model.Port
Screenshot Model
################
.. autoclass:: pipeline.models.screenshot_model.Screenshot
Searchsploit Model
##################
.. autoclass:: pipeline.models.searchsploit_model.SearchsploitResult
Technology Model
################
.. autoclass:: pipeline.models.technology_model.Technology

15
docs/api/parsers.rst
View File

@@ -1,17 +1,26 @@
.. toctree::
:maxdepth: 1
:hidden:
.. _parsers-ref-label:
Parsers
=======
Amass Parser
############
.. autoclass:: recon.amass.ParseAmassOutput
.. autoclass:: pipeline.recon.amass.ParseAmassOutput
:members:
Web Targets Parser
##################
.. autoclass:: recon.web.targets.GatherWebTargets
.. autoclass:: pipeline.recon.web.targets.GatherWebTargets
:members:
Masscan Parser
##############
.. autoclass:: recon.masscan.ParseMasscanOutput
.. autoclass:: pipeline.recon.masscan.ParseMasscanOutput
:members:

44
docs/api/scanners.rst
View File

@@ -1,62 +1,74 @@
.. toctree::
:maxdepth: 1
:hidden:
.. _scanners-ref-label:
Scanners
================
Amass Scanner
#############
.. autoclass:: recon.amass.AmassScan
.. autoclass:: pipeline.recon.amass.AmassScan
:members:
Aquatone Scanner
################
.. autoclass:: recon.web.aquatone.AquatoneScan
CORS Scanner
############
.. autoclass:: recon.web.corscanner.CORScannerScan
.. autoclass:: pipeline.recon.web.aquatone.AquatoneScan
:members:
Full Scanner
############
.. autoclass:: recon.wrappers.FullScan
.. autoclass:: pipeline.recon.wrappers.FullScan
:members:
Gobuster Scanner
################
.. autoclass:: recon.web.gobuster.GobusterScan
.. autoclass:: pipeline.recon.web.gobuster.GobusterScan
:members:
Hackthebox Scanner
##################
.. autoclass:: recon.wrappers.HTBScan
.. autoclass:: pipeline.recon.wrappers.HTBScan
:members:
Masscan Scanner
###############
.. autoclass:: recon.masscan.MasscanScan
.. autoclass:: pipeline.recon.masscan.MasscanScan
:members:
Searchsploit Scanner
####################
.. autoclass:: recon.nmap.SearchsploitScan
.. autoclass:: pipeline.recon.nmap.SearchsploitScan
:members:
Subjack Scanner
###############
.. autoclass:: recon.web.subdomain_takeover.SubjackScan
.. autoclass:: pipeline.recon.web.subdomain_takeover.SubjackScan
:members:
ThreadedNmap Scanner
####################
.. autoclass:: recon.nmap.ThreadedNmapScan
.. autoclass:: pipeline.recon.nmap.ThreadedNmapScan
:members:
TKOSubs Scanner
###############
.. autoclass:: recon.web.subdomain_takeover.TKOSubsScan
.. autoclass:: pipeline.recon.web.subdomain_takeover.TKOSubsScan
:members:
Webanalyze Scanner
##################
.. autoclass:: recon.web.webanalyze.WebanalyzeScan
.. autoclass:: pipeline.recon.web.webanalyze.WebanalyzeScan
:members:

BIN
docs/img/database-design.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

13
docs/index.rst
View File

@@ -17,13 +17,13 @@ Getting Started
overview/index
Changing the Code
=================
Personalization
===============
.. toctree::
:maxdepth: 1
modifications/index
Creating a New Wrapper Scan <modifications/index>
API Reference
=============
@@ -31,11 +31,14 @@ API Reference
.. toctree::
:maxdepth: 2
api/index
api/commands
api/manager
api/models
api/parsers
api/scanners
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

9
docs/modifications/index.rst
View File

@@ -1,7 +1,2 @@
Making Modifications
====================
.. toctree::
:maxdepth: 1
new_wrapper
.. include::
new_wrapper.rst

4
docs/overview/index.rst
View File

@@ -6,9 +6,11 @@ Getting Started
:hidden:
installation
scope
running_scans
viewing_results
scheduler
visualization
scope
.. include:: summary.rst

25
docs/overview/installation.rst
View File

@@ -5,20 +5,20 @@ Installation Instructions
There are two primary phases for installation:
* prior to `cmd2 <https://github.com/python-cmd2/cmd2>`_ being installed
* prior to the python dependencies being installed
* everything else
Manual Steps
############
First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below)
First, the steps to get python dependencies installed in a virtual environment are as follows (and shown below)
Kali
----
.. code-block:: console
apt install pipenv
sudo apt install pipenv
Ubuntu 18.04
@@ -38,27 +38,27 @@ Both OSs After ``pipenv`` Install
git clone https://github.com/epi052/recon-pipeline.git
cd recon-pipeline
pipenv install cmd2
pipenv install
pipenv shell
.. raw:: html
<script id="asciicast-293306" src="https://asciinema.org/a/293306.js" async></script>
<script id="asciicast-318395" src="https://asciinema.org/a/318395.js" async></script>
Everything Else
###############
After manual installation of cmd2_ is complete, the recon-pipeline shell provides its own :ref:`install_command` command (seen below).
A simple ``install all`` will handle all installation steps. Installation has **only** been tested on **Kali 2019.4**.
After installing the python dependencies, the recon-pipeline shell provides its own :ref:`install_command` command (seen below).
A simple ``install all`` will handle all installation steps. Installation has **only** been tested on **Kali 2019.4 and Ubuntu 18.04**.
**Ubuntu-18.04 Note**: You may consider running ``sudo -v`` prior to running ``./recon-pipeline.py``. ``sudo -v`` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password. It'll work either way though.
**Ubuntu-18.04 Note (and newer kali versions)**: You may consider running ``sudo -v`` prior to running ``./recon-pipeline.py``. ``sudo -v`` will refresh your creds, and the underlying subprocess calls during installation won't prompt you for your password. It'll work either way though.
Individual tools may be installed by running ``install TOOLNAME`` where ``TOOLNAME`` is one of the known tools that make
up the pipeline.
The installer maintains a (naive) list of installed tools at ``~/.cache/.tool-dict.pkl``. The installer in no way
attempts to be a package manager. It knows how to execute the steps necessary to install its tools. Beyond that, it's
like Jon Snow, it knows nothing.
The installer maintains a (naive) list of installed tools at ``~/.local/recon-pipeline/tools/.tool-dict.pkl``. The installer in no way attempts to be a package manager. It knows how to execute the steps necessary to install its tools. Beyond that, it's
like Jon Snow, **it knows nothing**.
.. raw:: html
@@ -73,8 +73,7 @@ for the auto installer to function:
- systemd-based system (``luigid`` is installed as a systemd service)
- python3.6+ installed
- ``pip`` installed
With the above requirements met, following the installation steps above starting with ``pipenv`` install should be sufficient.
With the above requirements met, following the installation steps above starting with ``pipenv install`` should be sufficient.
The alternative would be to manually install each tool.

91
docs/overview/running_scans.rst
View File

@@ -3,29 +3,88 @@
Running Scans
=============
All scans are ran from within ``recon-pipeline``'s shell. There are a number of individual scans, however to execute
All scans are run from within ``recon-pipeline``'s shell. There are a number of individual scans, however to execute
multiple scans at once, ``recon-pipeline`` includes wrappers around multiple commands. As of version |version|, the
following individual scans are available
- :class:`recon.amass.AmassScan`
- :class:`recon.web.aquatone.AquatoneScan`
- :class:`recon.web.corscanner.CORScannerScan`
- :class:`recon.web.gobuster.GobusterScan`
- :class:`recon.masscan.MasscanScan`
- :class:`recon.nmap.SearchsploitScan`
- :class:`recon.web.subdomain_takeover.SubjackScan`
- :class:`recon.nmap.ThreadedNmapScan`
- :class:`recon.web.subdomain_takeover.TKOSubsScan`
- :class:`recon.web.webanalyze.WebanalyzeScan`
- :class:`pipeline.recon.amass.AmassScan`
- :class:`pipeline.recon.web.aquatone.AquatoneScan`
- :class:`pipeline.recon.web.gobuster.GobusterScan`
- :class:`pipeline.recon.masscan.MasscanScan`
- :class:`pipeline.recon.nmap.SearchsploitScan`
- :class:`pipeline.recon.web.subdomain_takeover.SubjackScan`
- :class:`pipeline.recon.nmap.ThreadedNmapScan`
- :class:`pipeline.recon.web.subdomain_takeover.TKOSubsScan`
- :class:`pipeline.recon.web.webanalyze.WebanalyzeScan`
Additionally, two wrapper scans are made available as well.
Additionally, two wrapper scans are made available. These execute multiple scans in a pipeline.
- :class:`recon.wrappers.FullScan` - runs the entire pipeline
- :class:`recon.wrappers.HTBScan` - nicety for hackthebox players (myself included) that omits the scans in FullScan that don't make sense for HTB
- :class:`pipeline.recon.wrappers.FullScan` - runs the entire pipeline
- :class:`pipeline.recon.wrappers.HTBScan` - nicety for hackthebox players (myself included) that omits the scans in FullScan that don't make sense for HTB
Example Scan
============
############
Here are the steps the video below takes to scan tesla[.]com.
Create a targetfile
.. code-block:: console
# use virtual environment
pipenv shell
# create targetfile; a targetfile is required for all scans
mkdir /root/bugcrowd/tesla
cd /root/bugcrowd/tesla
echo tesla.com > tesla-targetfile
# create a blacklist (if necessary based on target's scope)
echo energysupport.tesla.com > tesla-blacklist
echo feedback.tesla.com >> tesla-blacklist
echo employeefeedback.tesla.com >> tesla-blacklist
echo ir.tesla.com >> tesla-blacklist
# drop into the interactive shell
/root/PycharmProjects/recon-pipeline/pipeline/recon-pipeline.py
recon-pipeline>
Create a new database to store scan results
.. code-block:: console
recon-pipeline> database attach
1. create new database
Your choice? 1
new database name? (recommend something unique for this target)
-> tesla-scan
[*] created database @ /home/epi/.local/recon-pipeline/databases/tesla-scan
[+] attached to sqlite database @ /home/epi/.local/recon-pipeline/databases/tesla-scan
[db-1] recon-pipeline>
Scan the target
.. code-block:: console
[db-1] recon-pipeline> scan FullScan --exempt-list tesla-blacklist --target-file tesla-targetfile --interface eno1 --top-ports 2000 --rate 1200
[-] FullScan queued
[-] TKOSubsScan queued
[-] GatherWebTargets queued
[-] ParseAmassOutput queued
[-] AmassScan queued
[-] ParseMasscanOutput queued
[-] MasscanScan queued
[-] WebanalyzeScan queued
[-] SearchsploitScan queued
[-] ThreadedNmapScan queued
[-] SubjackScan queued
[-] AquatoneScan queued
[-] GobusterScan queued
[db-1] recon-pipeline>
.. raw:: html
<script id="asciicast-293302" src="https://asciinema.org/a/293302.js" async></script>
<script id="asciicast-318397" src="https://asciinema.org/a/318397.js" async></script>

14
docs/overview/scope.rst
View File

@@ -3,9 +3,17 @@
Defining Target Scope
=====================
The pipeline expects a file that describes the target's scope to be provided as an argument to the
``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line. Ip
addresses and ip ranges can be mixed/matched, but domains cannot.
**New in v0.9.0**: In the event you're scanning a single ip address or host, simply use ``--target``. It accepts a single target and works in conjunction with ``--exempt-list`` if specified.
.. code-block:: console
[db-1] recon-pipeline> scan HTBScan --target 10.10.10.183 --top-ports 1000
...
In order to scan more than one host at a time, the pipeline needs a file that describes the target's scope to be provided as an argument to the `--target-file` option. The target file can consist of domains, ip addresses, and ip ranges, one per line.
In order to scan more than one host at a time, the pipeline expects a file that describes the target's scope to be provided as an argument to the ``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line. Domains, ip addresses and ip ranges can be mixed/matched within the scope file.
.. code-block:: console

5
docs/overview/summary.rst
View File

@@ -3,7 +3,8 @@ detailing the development process and underpinnings of the pipeline. Feel free t
you're so inclined, but they're in no way required reading to use the tool.
* :ref:`install-ref-label` - How to install ``recon-pipeline`` and associated dependencies
* :ref:`scan-ref-label` - Example scan of **tesla.com** using ``recon-pipeline``
* :ref:`scope-ref-label` - How to define the scope of your scans (list of targets and a blacklist)
* :ref:`scan-ref-label` - Example scan of **tesla.com** using ``recon-pipeline``
* :ref:`view-scan-label` - How to view scan results
* :ref:`scheduler-ref-label` - The Luigi schedulers and which to choose
* :ref:`visualization-ref-label` - How to check on active tasks
* :ref:`visualization-ref-label` - How to check on active tasks once they're running

611
docs/overview/viewing_results.rst Normal file
View File

@@ -0,0 +1,611 @@
.. toctree::
:hidden:
:maxdepth: 1
.. _view-scan-label:
Viewing Scan Results
====================
As of version 0.9.0, scan results are stored in a database located (by default) at ``~/.local/recon-pipeline/databases``. Databases themselves are managed through the :ref:`database_command` command while viewing their contents is done via :ref:`view_command`.
The view command allows one to inspect different pieces of scan information via the following sub-commands
- endpoints (gobuster results)
- nmap-scans
- ports
- searchsploit-results
- targets
- web-technologies (webanalyze results)
Each of the sub-commands has a list of tab-completable options and values that can help drilling down to the data you care about.
All of the subcommands offer a ``--paged`` option for dealing with large amounts of output. ``--paged`` will show you one page of output at a time (using ``less`` under the hood).
Chaining Results w/ Commands
############################
All of the results can be **piped out to other commands**. Let's say you want to feed some results from ``recon-pipeline`` into another tool that isn't part of the pipeline. Simply using a normal unix pipe ``|`` followed by the next command will get that done for you. Below is an example of piping targets into `gau <https://github.com/lc/gau>`_
.. code-block:: console
[db-2] recon-pipeline> view targets --paged
3.tesla.cn
3.tesla.com
api-internal.sn.tesla.services
api-toolbox.tesla.com
api.mp.tesla.services
api.sn.tesla.services
api.tesla.cn
api.toolbox.tb.tesla.services
...
[db-2] recon-pipeline> view targets | gau
https://3.tesla.com/pt_PT/model3/design
https://3.tesla.com/pt_PT/model3/design?redirect=no
https://3.tesla.com/robots.txt
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-160x160.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-16x16.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-196x196.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-32x32.png?2
https://3.tesla.com/sites/all/themes/custom/tesla_theme/assets/img/icons/favicon-96x96.png?2
https://3.tesla.com/sv_SE/model3/design
...
view endpoints
##############
An endpoint consists of a status code and the scanned URL. Endpoints are populated via gobuster.
Show All Endpoints
------------------
.. code-block:: console
[db-2] recon-pipeline> view endpoints --paged
[200] http://westream.teslamotors.com/y
[301] https://mobileapps.teslamotors.com/aspnet_client
[403] https://209.133.79.49/analog.html
[302] https://209.133.79.49/api
[403] https://209.133.79.49/cgi-bin/
[200] https://209.133.79.49/client
...
Filter by Host
--------------
.. code-block:: console
[db-2] recon-pipeline> view endpoints --host shop.uk.teslamotors.com
[402] http://shop.uk.teslamotors.com/
[403] https://shop.uk.teslamotors.com:8443/
[301] http://shop.uk.teslamotors.com/assets
[302] http://shop.uk.teslamotors.com/admin.cgi
[200] http://shop.uk.teslamotors.com/.well-known/apple-developer-merchantid-domain-association
[302] http://shop.uk.teslamotors.com/admin
[403] http://shop.uk.teslamotors.com:8080/
[302] http://shop.uk.teslamotors.com/admin.php
[302] http://shop.uk.teslamotors.com/admin.pl
[200] http://shop.uk.teslamotors.com/crossdomain.xml
[403] https://shop.uk.teslamotors.com/
[db-2] recon-pipeline>
Filter by Host and Status Code
------------------------------
.. code-block:: console
[db-2] recon-pipeline> view endpoints --host shop.uk.teslamotors.com --status-code 200
[200] http://shop.uk.teslamotors.com/crossdomain.xml
[200] http://shop.uk.teslamotors.com/.well-known/apple-developer-merchantid-domain-association
[db-2] recon-pipeline>
Remove Status Code from Output
------------------------------
Using ``--plain`` will remove the status-code prefix, allowing for easy piping of results into other commands.
.. code-block:: console
[db-2] recon-pipeline> view endpoints --host shop.uk.teslamotors.com --plain
http://shop.uk.teslamotors.com/admin.pl
http://shop.uk.teslamotors.com/admin
http://shop.uk.teslamotors.com/
http://shop.uk.teslamotors.com/admin.cgi
http://shop.uk.teslamotors.com/.well-known/apple-developer-merchantid-domain-association
http://shop.uk.teslamotors.com:8080/
http://shop.uk.teslamotors.com/crossdomain.xml
https://shop.uk.teslamotors.com:8443/
https://shop.uk.teslamotors.com/
http://shop.uk.teslamotors.com/admin.php
http://shop.uk.teslamotors.com/assets
[db-2] recon-pipeline>
Include Headers
---------------
If you'd like to include any headers found during scanning, ``--headers`` will do that for you.
.. code-block:: console
[db-2] recon-pipeline> view endpoints --host shop.uk.teslamotors.com --headers
[302] http://shop.uk.teslamotors.com/admin.php
[302] http://shop.uk.teslamotors.com/admin.cgi
[302] http://shop.uk.teslamotors.com/admin
[200] http://shop.uk.teslamotors.com/crossdomain.xml
[403] https://shop.uk.teslamotors.com/
Server: cloudflare
Date: Mon, 06 Apr 2020 13:56:12 GMT
Content-Type: text/html
Content-Length: 553
Retry-Count: 0
Cf-Ray: 57fc02c788f7e03f-DFW
[403] https://shop.uk.teslamotors.com:8443/
Content-Type: text/html
Content-Length: 553
Retry-Count: 0
Cf-Ray: 57fc06e5fcbfd266-DFW
Server: cloudflare
Date: Mon, 06 Apr 2020 13:59:00 GMT
[302] http://shop.uk.teslamotors.com/admin.pl
[200] http://shop.uk.teslamotors.com/.well-known/apple-developer-merchantid-domain-association
[403] http://shop.uk.teslamotors.com:8080/
Server: cloudflare
Date: Mon, 06 Apr 2020 13:58:50 GMT
Content-Type: text/html; charset=UTF-8
Set-Cookie: __cfduid=dfbf45a8565fda1325b8c1482961518511586181530; expires=Wed, 06-May-20 13:58:50 GMT; path=/; domain=.shop.uk.teslamotors.com; HttpOnly; SameSite=Lax
Cache-Control: max-age=15
X-Frame-Options: SAMEORIGIN
Alt-Svc: h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
Expires: Mon, 06 Apr 2020 13:59:05 GMT
Cf-Ray: 57fc06a53887d286-DFW
Retry-Count: 0
[402] http://shop.uk.teslamotors.com/
Cf-Cache-Status: DYNAMIC
X-Dc: gcp-us-central1,gcp-us-central1
Date: Mon, 06 Apr 2020 13:54:49 GMT
Cf-Ray: 57fc00c39c0b581d-DFW
X-Request-Id: 79146367-4c68-4e1b-9784-31f76d51b60b
Set-Cookie: __cfduid=d94fad82fbdc0c110cb03cbcf58d097e21586181289; expires=Wed, 06-May-20 13:54:49 GMT; path=/; domain=.shop.uk.teslamotors.com; HttpOnly; SameSite=Lax _shopify_y=e3f19482-99e9-46cd-af8d-89fb8557fd28; path=/; expires=Thu, 07 Apr 2022 01:33:13 GMT
X-Shopid: 4232821
Content-Language: en
Alt-Svc: h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none
X-Xss-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fshop&source%5Bsection%5D=storefront&source%5Buuid%5D=79146367-4c68-4e1b-9784-31f76d51b60b
Server: cloudflare
Content-Type: text/html; charset=utf-8
X-Sorting-Hat-Shopid: 4232821
X-Shardid: 78
Content-Security-Policy: frame-ancestors *; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=storefront_section%2Fshop&source%5Bsection%5D=storefront&source%5Buuid%5D=79146367-4c68-4e1b-9784-31f76d51b60b
Retry-Count: 0
X-Sorting-Hat-Podid: 78
X-Shopify-Stage: production
X-Download-Options: noopen
[301] http://shop.uk.teslamotors.com/assets
[db-2] recon-pipeline>
view nmap-scans
###############
Nmap results can be filtered by host, NSE script type, scanned port, and product.
Show All Results
----------------
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --paged
2600:9000:21d4:7800:c:d401:5a80:93a1 - http
===========================================
tcp port: 80 - open - syn-ack
product: Amazon CloudFront httpd :: None
nse script(s) output:
http-server-header
CloudFront
http-title
ERROR: The request could not be satisfied
...
Filter by product
-----------------
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --product "Splunkd httpd"
209.133.79.101 - http
=====================
tcp port: 443 - open - syn-ack
product: Splunkd httpd :: None
nse script(s) output:
http-robots.txt
1 disallowed entry
/
http-server-header
Splunkd
http-title
404 Not Found
ssl-cert
Subject: commonName=*.teslamotors.com/organizationName=Tesla Motors, Inc./stateOrProvinceName=California/countryName=US
Subject Alternative Name: DNS:*.teslamotors.com, DNS:teslamotors.com
Not valid before: 2019-01-17T00:00:00
Not valid after: 2021-02-03T12:00:00
ssl-date
TLS randomness does not represent time
Filter by NSE Script
--------------------
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --nse-script ssl-cert --paged
199.66.9.47 - http-proxy
========================
tcp port: 443 - open - syn-ack
product: Varnish http accelerator :: None
nse script(s) output:
ssl-cert
Subject: commonName=*.tesla.com/organizationName=Tesla, Inc./stateOrProvinceName=California/countryName=US
Subject Alternative Name: DNS:*.tesla.com, DNS:tesla.com
Not valid before: 2020-02-07T00:00:00
Not valid after: 2022-04-08T12:00:00
...
Filter by NSE Script and Port Number
------------------------------------
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --nse-script ssl-cert --port 8443
104.22.11.42 - https-alt
========================
tcp port: 8443 - open - syn-ack
product: cloudflare :: None
nse script(s) output:
ssl-cert
Subject: commonName=sni.cloudflaressl.com/organizationName=Cloudflare, Inc./stateOrProvinceName=CA/countryName=US
Subject Alternative Name: DNS:*.tesla.services, DNS:tesla.services, DNS:sni.cloudflaressl.com
Not valid before: 2020-02-13T00:00:00
Not valid after: 2020-10-09T12:00:00
[db-2] recon-pipeline>
Filter by Host (ipv4/6 or domain name)
--------------------------------------
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --host 2600:9000:21d4:3000:c:d401:5a80:93a1
2600:9000:21d4:3000:c:d401:5a80:93a1 - http
===========================================
tcp port: 80 - open - syn-ack
product: Amazon CloudFront httpd :: None
nse script(s) output:
http-server-header
CloudFront
http-title
ERROR: The request could not be satisfied
[db-2] recon-pipeline>
Include Command Used to Scan
----------------------------
The ``--commandline`` option will append the command used to scan the target to the results.
.. code-block:: console
[db-2] recon-pipeline> view nmap-scans --host 2600:9000:21d4:3000:c:d401:5a80:93a1 --commandline
2600:9000:21d4:3000:c:d401:5a80:93a1 - http
===========================================
tcp port: 80 - open - syn-ack
product: Amazon CloudFront httpd :: None
nse script(s) output:
http-server-header
CloudFront
http-title
ERROR: The request could not be satisfied
command used:
nmap --open -sT -n -sC -T 4 -sV -Pn -p 80 -6 -oA /home/epi/PycharmProjects/recon-pipeline/tests/data/tesla-results/nmap-results/nmap.2600:9000:21d4:3000:c:d401:5a80:93a1-tcp 2600:9000:21d4:3000:c:d401:5a80:93a1
[db-2] recon-pipeline>
view ports
##########
Port results are populated via masscan. Ports can be filtered by host and port number.
Show All Results
----------------
.. code-block:: console
[db-2] recon-pipeline> view ports --paged
apmv3.go.tesla.services: 80
autodiscover.teslamotors.com: 80
csp.teslamotors.com: 443
image.emails.tesla.com: 443
marketing.teslamotors.com: 443
partnerleadsharing.tesla.com: 443
service.tesla.cn: 80
shop.uk.teslamotors.com: 8080
sip.tesla.cn: 5061
...
Filter by Host
--------------
.. code-block:: console
[db-2] recon-pipeline> view ports --host tesla.services
tesla.services: 8443,8080
[db-2] recon-pipeline>
Filter by Port Number
---------------------
.. code-block:: console
[db-2] recon-pipeline> view ports --port-number 8443
tesla.services: 8443,8080
104.22.10.42: 8443,8080
104.22.11.42: 8443,8080
2606:4700:10::6816:a2a: 8443,8080
2606:4700:10::6816:b2a: 8443,8080
[db-2] recon-pipeline>
view searchsploit-results
#########################
Searchsploit results can be filtered by host and type, the full path to any relevant exploit code can be shown as well.
Show All Results
----------------
.. code-block:: console
[db-2] recon-pipeline> view searchsploit-results --paged
52.209.48.104, 34.252.120.214, 52.48.121.107, telemetry-eng.vn.tesla.services
=============================================================================
local | 40768.sh | Nginx (Debian Based Distros + Gentoo) - 'logrotate' Local Privilege
| Escalation
remote | 12804.txt| Nginx 0.6.36 - Directory Traversal
local | 14830.py | Nginx 0.6.38 - Heap Corruption
webapps | 24967.txt| Nginx 0.6.x - Arbitrary Code Execution NullByte Injection
dos | 9901.txt | Nginx 0.7.0 < 0.7.61 / 0.6.0 < 0.6.38 / 0.5.0 < 0.5.37 / 0.4.0 <
| 0.4.14 - Denial of Service (PoC)
remote | 9829.txt | Nginx 0.7.61 - WebDAV Directory Traversal
remote | 33490.txt| Nginx 0.7.64 - Terminal Escape Sequence in Logs Command Injection
remote | 13822.txt| Nginx 0.7.65/0.8.39 (dev) - Source Disclosure / Download
remote | 13818.txt| Nginx 0.8.36 - Source Disclosure / Denial of Service
remote | 38846.txt| Nginx 1.1.17 - URI Processing SecURIty Bypass
remote | 25775.rb | Nginx 1.3.9 < 1.4.0 - Chuncked Encoding Stack Buffer Overflow
| (Metasploit)
dos | 25499.py | Nginx 1.3.9 < 1.4.0 - Denial of Service (PoC)
remote | 26737.pl | Nginx 1.3.9/1.4.0 (x86) - Brute Force
remote | 32277.txt| Nginx 1.4.0 (Generic Linux x64) - Remote Overflow
webapps | 47553.md | PHP-FPM + Nginx - Remote Code Execution
...
Filter by Host
--------------
.. code-block:: console
[db-2] recon-pipeline> view searchsploit-results --paged --host telemetry-eng.vn.tesla.services
52.209.48.104, 34.252.120.214, 52.48.121.107, telemetry-eng.vn.tesla.services
=============================================================================
local | 40768.sh | Nginx (Debian Based Distros + Gentoo) - 'logrotate' Local Privilege
| Escalation
remote | 12804.txt| Nginx 0.6.36 - Directory Traversal
local | 14830.py | Nginx 0.6.38 - Heap Corruption
webapps | 24967.txt| Nginx 0.6.x - Arbitrary Code Execution NullByte Injection
dos | 9901.txt | Nginx 0.7.0 < 0.7.61 / 0.6.0 < 0.6.38 / 0.5.0 < 0.5.37 / 0.4.0 <
| 0.4.14 - Denial of Service (PoC)
remote | 9829.txt | Nginx 0.7.61 - WebDAV Directory Traversal
remote | 33490.txt| Nginx 0.7.64 - Terminal Escape Sequence in Logs Command Injection
remote | 13822.txt| Nginx 0.7.65/0.8.39 (dev) - Source Disclosure / Download
remote | 13818.txt| Nginx 0.8.36 - Source Disclosure / Denial of Service
remote | 38846.txt| Nginx 1.1.17 - URI Processing SecURIty Bypass
remote | 25775.rb | Nginx 1.3.9 < 1.4.0 - Chuncked Encoding Stack Buffer Overflow
| (Metasploit)
dos | 25499.py | Nginx 1.3.9 < 1.4.0 - Denial of Service (PoC)
remote | 26737.pl | Nginx 1.3.9/1.4.0 (x86) - Brute Force
remote | 32277.txt| Nginx 1.4.0 (Generic Linux x64) - Remote Overflow
webapps | 47553.md | PHP-FPM + Nginx - Remote Code Execution
[db-2] recon-pipeline>
Filter by Type
--------------
.. code-block:: console
[db-2] recon-pipeline> view searchsploit-results --paged --type webapps
52.209.48.104, 34.252.120.214, 52.48.121.107, telemetry-eng.vn.tesla.services
=============================================================================
webapps | 24967.txt| Nginx 0.6.x - Arbitrary Code Execution NullByte Injection
webapps | 47553.md | PHP-FPM + Nginx - Remote Code Execution
...
Include Full Path to Exploit Code
---------------------------------
.. code-block:: console
52.209.48.104, 34.252.120.214, 52.48.121.107, telemetry-eng.vn.tesla.services
=============================================================================
webapps | Nginx 0.6.x - Arbitrary Code Execution NullByte Injection
| /home/epi/.recon-tools/exploitdb/exploits/multiple/webapps/24967.txt
webapps | PHP-FPM + Nginx - Remote Code Execution
| /home/epi/.recon-tools/exploitdb/exploits/php/webapps/47553.md
...
view targets
############
Target results can be filtered by type and whether or not they've been reported as vulnerable to subdomain takeover.
Show All Results
----------------
.. code-block:: console
[db-2] recon-pipeline> view targets --paged
3.tesla.com
api-internal.sn.tesla.services
api-toolbox.tesla.com
api.mp.tesla.services
api.sn.tesla.services
api.tesla.cn
...
Filter by Target Type
---------------------
.. code-block:: console
[db-2] recon-pipeline> view targets --type ipv6 --paged
2600:1404:23:183::358f
2600:1404:23:188::3fe7
2600:1404:23:18f::700
2600:1404:23:190::700
2600:1404:23:194::16cf
...
Filter by Possibility of Subdomain Takeover
-------------------------------------------
.. code-block:: console
[db-2] recon-pipeline> view targets --paged --vuln-to-subdomain-takeover
[vulnerable] api-internal.sn.tesla.services
...
view web-technologies
#####################
Web technology results are produced by webanalyze. Web technology results can be filtered by host, type, and product.
Show All Results
----------------
.. code-block:: console
[db-2] recon-pipeline> view web-technologies --paged
Varnish (Caching)
=================
- inventory-assets.tesla.com
- www.tesla.com
- errlog.tesla.com
- static-assets.tesla.com
- partnerleadsharing.tesla.com
- 199.66.9.47
- onboarding-pre-delivery-prod.teslamotors.com
- 2600:1404:23:194::16cf
- 2600:1404:23:196::16cf
...
Filter by Technology Type
-------------------------
.. code-block:: console
[db-2] recon-pipeline> view web-technologies --type "Programming languages"
PHP (Programming languages)
===========================
- www.tesla.com
- dummy.teslamotors.com
- 209.10.208.20
- 211.147.80.206
- trt.tesla.com
- trt.teslamotors.com
- cn-origin.teslamotors.com
- www.tesla.cn
- events.tesla.cn
- 23.67.209.106
- service.teslamotors.com
Python (Programming languages)
==============================
- api-toolbox.tesla.com
- 52.26.53.228
- 34.214.187.20
- 35.166.29.132
- api.toolbox.tb.tesla.services
- toolbox.teslamotors.com
- 209.133.79.93
Ruby (Programming languages)
============================
- storagesim.teslamotors.com
- 209.10.208.39
...
Filter by Product
-----------------
.. code-block:: console
[db-2] recon-pipeline> view web-technologies --product OpenResty-1.15.8.2
OpenResty-1.15.8.2 (Web servers)
================================
- links.tesla.com
[db-2] recon-pipeline>
Filter by Host
--------------
.. code-block:: console
[db-2] recon-pipeline> view web-technologies --host api-toolbox.tesla.com
api-toolbox.tesla.com
=====================
- gunicorn-19.4.5 (Web servers)
- Python (Programming languages)
[db-2] recon-pipeline>
Manually interacting with the Database
======================================
If for whatever reason you'd like to query the database manually, from within the recon-pipeline shell, you can use the ``py`` command to drop into a python REPL with your current ReconShell instance available as ``self``.
.. code-block:: console
./pipeline/recon-pipeline.py
recon-pipeline> py
Python 3.7.5 (default, Nov 20 2019, 09:21:52)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, `exit()`.
Non-Python commands can be issued with: app("your command")
>>> self
<__main__.ReconShell object at 0x7f69f457f790>
Once in the REPL, the currently connected database is available as ``self.db_mgr``. The database is an instance of :ref:`db_manager_label` and has a ``session`` attribute which can be used to issue manual SQLAlchemy style queries.
.. code-block:: console
>>> from pipeline.models.port_model import Port
>>> self.db_mgr.session.query(Port).filter_by(port_number=443)
<sqlalchemy.orm.query.Query object at 0x7f8cef804250>
>>>

2
docs/overview/visualization.rst
View File

@@ -7,7 +7,7 @@ Setup
#####
To use the web console, you'll need to :ref:`install the luigid service<install-ref-label>`. Assuming you've already
installed ``pipenv`` and created a virtual environment with ``cmd2``, you can simple run the ``install luigi-service``
installed ``pipenv`` and created a virtual environment, you can simply run the ``install luigi-service``
from within the pipeline.
Dashboard

8
docs/requirements.txt
View File

@@ -1,5 +1,7 @@
sphinx-argparse==0.2.5
sphinxcontrib-napoleon==0.7
sphinx-rtd-theme==0.4.3
cmd2==0.9.24
luigi==2.8.11
sphinxcontrib-napoleon==0.7
cmd2==1.0.1
luigi==2.8.12
python-libnmap==0.7.0
SQLAlchemy==1.3.15

0
recon/web/__init__.py → pipeline/__init__.py
View File

0
tests/test_install/__init__.py → pipeline/models/__init__.py
View File

3
pipeline/models/base_model.py Normal file
View File

@@ -0,0 +1,3 @@
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

211
pipeline/models/db_manager.py Normal file
View File

@@ -0,0 +1,211 @@
import sqlite3
from pathlib import Path
from urllib.parse import urlparse
from cmd2 import ansi
from sqlalchemy.orm import sessionmaker
from sqlalchemy import exc, or_, create_engine
from sqlalchemy.sql.expression import ClauseElement
from .base_model import Base
from .port_model import Port
from .nse_model import NSEResult
from .target_model import Target
from .nmap_model import NmapResult
from .endpoint_model import Endpoint
from .ip_address_model import IPAddress
from .technology_model import Technology
from .searchsploit_model import SearchsploitResult
from ..recon.helpers import get_ip_address_version, is_ip_address
class DBManager:
""" Class that encapsulates database transactions and queries """
def __init__(self, db_location):
self.location = Path(db_location).expanduser().resolve()
self.connection_string = f"sqlite:///{self.location}"
engine = create_engine(self.connection_string)
Base.metadata.create_all(engine) # noqa: F405
session_factory = sessionmaker(bind=engine)
self.session = session_factory()
def get_or_create(self, model, **kwargs):
""" Simple helper to either get an existing record if it exists otherwise create and return a new instance """
instance = self.session.query(model).filter_by(**kwargs).first()
if instance:
return instance
else:
params = dict((k, v) for k, v in kwargs.items() if not isinstance(v, ClauseElement))
instance = model(**params)
return instance
def add(self, item):
""" Simple helper to add a record to the database """
try:
self.session.add(item)
self.session.commit()
except (sqlite3.IntegrityError, exc.IntegrityError):
print(ansi.style(f"[-] unique key constraint handled, moving on...", fg="bright_white"))
self.session.rollback()
def get_or_create_target_by_ip_or_hostname(self, ip_or_host):
""" Simple helper to query a Target record by either hostname or ip address, whichever works """
# get existing instance
instance = (
self.session.query(Target)
.filter(
or_(
Target.ip_addresses.any(
or_(IPAddress.ipv4_address.in_([ip_or_host]), IPAddress.ipv6_address.in_([ip_or_host]))
),
Target.hostname == ip_or_host,
)
)
.first()
)
if instance:
return instance
else:
# create new entry
tgt = self.get_or_create(Target)
if get_ip_address_version(ip_or_host) == "4":
tgt.ip_addresses.append(IPAddress(ipv4_address=ip_or_host))
elif get_ip_address_version(ip_or_host) == "6":
tgt.ip_addresses.append(IPAddress(ipv6_address=ip_or_host))
else:
# we've already determined it's not an IP, only other possibility is a hostname
tgt.hostname = ip_or_host
return tgt
def get_all_hostnames(self) -> list:
""" Simple helper to return all hostnames from Target records """
return [x[0] for x in self.session.query(Target.hostname).filter(Target.hostname != None)] # noqa: E711
def get_all_ipv4_addresses(self) -> list:
""" Simple helper to return all ipv4 addresses from Target records """
return [
x[0] for x in self.session.query(IPAddress.ipv4_address).filter(IPAddress.ipv4_address != None)
] # noqa: E711
def get_all_ipv6_addresses(self) -> list:
""" Simple helper to return all ipv6 addresses from Target records """
return [
x[0] for x in self.session.query(IPAddress.ipv6_address).filter(IPAddress.ipv6_address != None)
] # noqa: E711
def close(self):
""" Simple helper to close the database session """
self.session.close()
def get_all_targets(self):
""" Simple helper to return all ipv4/6 and hostnames produced by running amass """
return self.get_all_hostnames() + self.get_all_ipv4_addresses() + self.get_all_ipv6_addresses()
def get_all_endpoints(self):
""" Simple helper that returns all Endpoints from the database """
return self.session.query(Endpoint).all()
def get_all_port_numbers(self):
""" Simple helper that returns all Port.port_numbers from the database """
return set(str(x[0]) for x in self.session.query(Port.port_number).all())
def get_endpoint_by_status_code(self, code):
""" Simple helper that returns all Endpoints filtered by status code """
return self.session.query(Endpoint).filter(Endpoint.status_code == code).all()
def get_endpoints_by_ip_or_hostname(self, ip_or_host):
""" Simple helper that returns all Endpoints filtered by ip or hostname """
endpoints = list()
tmp_endpoints = self.session.query(Endpoint).filter(Endpoint.url.contains(ip_or_host)).all()
for ep in tmp_endpoints:
parsed_url = urlparse(ep.url)
if parsed_url.hostname == ip_or_host:
endpoints.append(ep)
return endpoints
def get_nmap_scans_by_ip_or_hostname(self, ip_or_host):
""" Simple helper that returns all Endpoints filtered by ip or hostname """
scans = list()
for result in self.session.query(NmapResult).filter(NmapResult.commandline.contains(ip_or_host)).all():
if result.commandline.split()[-1] == ip_or_host:
scans.append(result)
return scans
def get_status_codes(self):
""" Simple helper that returns all status codes found during scanning """
return set(str(x[0]) for x in self.session.query(Endpoint.status_code).all())
def get_and_filter(self, model, defaults=None, **kwargs):
""" Simple helper to either get an existing record if it exists otherwise create and return a new instance """
return self.session.query(model).filter_by(**kwargs).all()
def get_all_nse_script_types(self):
""" Simple helper that returns all NSE Script types from the database """
return set(str(x[0]) for x in self.session.query(NSEResult.script_id).all())
def get_all_nmap_reported_products(self):
""" Simple helper that returns all products reported by nmap """
return set(str(x[0]) for x in self.session.query(NmapResult.product).all())
def get_all_exploit_types(self):
""" Simple helper that returns all exploit types reported by searchsploit """
return set(str(x[0]) for x in self.session.query(SearchsploitResult.type).all())
def add_ipv4_or_v6_address_to_target(self, tgt, ipaddr):
""" Simple helper that adds an appropriate IPAddress to the given target """
if not is_ip_address(ipaddr):
return
if get_ip_address_version(ipaddr) == "4":
ip_address = self.get_or_create(IPAddress, ipv4_address=ipaddr)
else:
ip_address = self.get_or_create(IPAddress, ipv6_address=ipaddr)
tgt.ip_addresses.append(ip_address)
return tgt
def get_all_web_targets(self):
""" Simple helper that returns all Targets tagged as having an open web port """
# return set(str(x[0]) for x in self.session.query(Target).all())
web_targets = list()
targets = self.get_and_filter(Target, is_web=True)
for target in targets:
if target.hostname:
web_targets.append(target.hostname)
for ipaddr in target.ip_addresses:
if ipaddr.ipv4_address:
web_targets.append(ipaddr.ipv4_address)
if ipaddr.ipv6_address:
web_targets.append(ipaddr.ipv6_address)
return web_targets
def get_ports_by_ip_or_host_and_protocol(self, ip_or_host, protocol):
""" Simple helper that returns all ports based on the given protocol and host """
tgt = self.get_or_create_target_by_ip_or_hostname(ip_or_host)
ports = list()
for port in tgt.open_ports:
if port.protocol == protocol:
ports.append(str(port.port_number))
return ports
def get_all_searchsploit_results(self):
return self.get_and_filter(SearchsploitResult)
def get_all_web_technology_types(self):
return set(str(x[0]) for x in self.session.query(Technology.type).all())
def get_all_web_technology_products(self):
return set(str(x[0]) for x in self.session.query(Technology.text).all())

26
pipeline/models/endpoint_model.py Normal file
View File

@@ -0,0 +1,26 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String
from .base_model import Base
from .header_model import header_association_table
class Endpoint(Base):
""" Database model that describes a URL/endpoint.
Represents gobuster data.
Relationships:
``target``: many to one -> :class:`pipeline.models.target_model.Target`
``headers``: many to many -> :class:`pipeline.models.header_model.Header`
"""
__tablename__ = "endpoint"
id = Column(Integer, primary_key=True)
url = Column(String, unique=True)
status_code = Column(Integer)
target_id = Column(Integer, ForeignKey("target.id"))
target = relationship("Target", back_populates="endpoints")
headers = relationship("Header", secondary=header_association_table, back_populates="endpoints")

28
pipeline/models/header_model.py Normal file
View File

@@ -0,0 +1,28 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String, UniqueConstraint, Table
from .base_model import Base
header_association_table = Table(
"header_association",
Base.metadata,
Column("header_id", Integer, ForeignKey("header.id")),
Column("endpoint_id", Integer, ForeignKey("endpoint.id")),
)
class Header(Base):
""" Database model that describes an http header (i.e. Server=cloudflare).
Relationships:
``endpoints``: many to many -> :class:`pipeline.models.target_model.Endpoint`
"""
__tablename__ = "header"
__table_args__ = (UniqueConstraint("name", "value"),) # combination of name/value == unique
id = Column(Integer, primary_key=True)
name = Column(String)
value = Column(String)
endpoint_id = Column(Integer, ForeignKey("endpoint.id"))
endpoints = relationship("Endpoint", secondary=header_association_table, back_populates="headers")

22
pipeline/models/ip_address_model.py Normal file
View File

@@ -0,0 +1,22 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String
from .base_model import Base
class IPAddress(Base):
""" Database model that describes an ip address (ipv4 or ipv6).
Represents amass data or targets specified manually as part of the ``target-file``.
Relationships:
``target``: many to one -> :class:`pipeline.models.target_model.Target`
"""
__tablename__ = "ip_address"
id = Column(Integer, primary_key=True)
ipv4_address = Column(String, unique=True)
ipv6_address = Column(String, unique=True)
target_id = Column(Integer, ForeignKey("target.id"))
target = relationship("Target", back_populates="ip_addresses")

77
pipeline/models/nmap_model.py Normal file
View File

@@ -0,0 +1,77 @@
import textwrap
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String, Boolean
from .base_model import Base
from .port_model import Port
from .ip_address_model import IPAddress
from .nse_model import nse_result_association_table
class NmapResult(Base):
""" Database model that describes the TARGET.nmap scan results.
Represents nmap data.
Relationships:
``target``: many to one -> :class:`pipeline.models.target_model.Target`
``ip_address``: one to one -> :class:`pipeline.models.ip_address_model.IPAddress`
``port``: one to one -> :class:`pipeline.models.port_model.Port`
``nse_results``: one to many -> :class:`pipeline.models.nse_model.NSEResult`
"""
def __str__(self):
return self.pretty()
def pretty(self, commandline=False, nse_results=None):
pad = " "
ip_address = self.ip_address.ipv4_address or self.ip_address.ipv6_address
msg = f"{ip_address} - {self.service}\n"
msg += f"{'=' * (len(ip_address) + len(self.service) + 3)}\n\n"
msg += f"{self.port.protocol} port: {self.port.port_number} - {'open' if self.open else 'closed'} - {self.reason}\n"
msg += f"product: {self.product} :: {self.product_version}\n"
msg += f"nse script(s) output:\n"
if nse_results is None:
# add all nse scripts
for nse_result in self.nse_results:
msg += f"{pad}{nse_result.script_id}\n"
msg += textwrap.indent(nse_result.script_output, pad * 2)
msg += "\n"
else:
# filter used, only return those specified
for nse_result in nse_results:
if nse_result in self.nse_results:
msg += f"{pad}{nse_result.script_id}\n"
msg += textwrap.indent(nse_result.script_output, pad * 2)
msg += "\n"
if commandline:
msg += f"command used:\n"
msg += f"{pad}{self.commandline}\n"
return msg
__tablename__ = "nmap_result"
id = Column(Integer, primary_key=True)
open = Column(Boolean)
reason = Column(String)
service = Column(String)
product = Column(String)
commandline = Column(String)
product_version = Column(String)
port = relationship(Port)
port_id = Column(Integer, ForeignKey("port.id"))
ip_address = relationship(IPAddress)
ip_address_id = Column(Integer, ForeignKey("ip_address.id"))
target_id = Column(Integer, ForeignKey("target.id"))
target = relationship("Target", back_populates="nmap_results")
nse_results = relationship("NSEResult", secondary=nse_result_association_table, back_populates="nmap_results")

31
pipeline/models/nse_model.py Normal file
View File

@@ -0,0 +1,31 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String, UniqueConstraint, Table
from .base_model import Base
nse_result_association_table = Table(
"nse_result_association",
Base.metadata,
Column("nse_result_id", Integer, ForeignKey("nse_result.id")),
Column("nmap_result_id", Integer, ForeignKey("nmap_result.id")),
)
class NSEResult(Base):
""" Database model that describes the NSE script executions as part of an nmap scan.
Represents NSE script data.
Relationships:
``NmapResult``: many to many -> :class:`pipeline.models.nmap_model.NmapResult`
"""
__tablename__ = "nse_result"
__table_args__ = (UniqueConstraint("script_id", "script_output"),) # combination of proto/port == unique
id = Column(Integer, primary_key=True)
script_id = Column(String)
script_output = Column(String)
nmap_result_id = Column(Integer, ForeignKey("nmap_result.id"))
nmap_results = relationship("NmapResult", secondary=nse_result_association_table, back_populates="nse_results")

30
pipeline/models/port_model.py Normal file
View File

@@ -0,0 +1,30 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String, Table, UniqueConstraint
from .base_model import Base
port_association_table = Table(
"port_association",
Base.metadata,
Column("port_id", Integer, ForeignKey("port.id")),
Column("target_id", Integer, ForeignKey("target.id")),
)
class Port(Base):
""" Database model that describes a port (tcp or udp).
Relationships:
``targets``: many to many -> :class:`pipeline.models.target_model.Target`
"""
__tablename__ = "port"
__table_args__ = (UniqueConstraint("protocol", "port_number"),) # combination of proto/port == unique
id = Column(Integer, primary_key=True)
protocol = Column("protocol", String)
port_number = Column("port_number", Integer)
target_id = Column(Integer, ForeignKey("target.id"))
targets = relationship("Target", secondary=port_association_table, back_populates="open_ports")

49
pipeline/models/screenshot_model.py Normal file
View File

@@ -0,0 +1,49 @@
from sqlalchemy.orm import relationship, relation
from sqlalchemy import Column, Integer, ForeignKey, LargeBinary, Table, String
from .base_model import Base
screenshot_association_table = Table(
"screenshot_association",
Base.metadata,
Column("screenshot_id", Integer, ForeignKey("screenshot.id")),
Column("similar_page_id", Integer, ForeignKey("screenshot.id")),
)
class Screenshot(Base):
""" Database model that describes a screenshot of a given webpage hosted on a ``Target``.
Represents aquatone data.
Relationships:
``port``: one to one -> :class:`pipeline.models.port_model.Port`
``target``: many to one -> :class:`pipeline.models.target_model.Target`
``endpoint``: one to one -> :class:`pipeline.models.endpoint_model.Endpoint`
``similar_pages``: black magic -> :class:`pipeline.models.screenshot_model.Screenshot`
"""
__tablename__ = "screenshot"
id = Column(Integer, primary_key=True)
image = Column(LargeBinary)
url = Column(String, unique=True)
port = relationship("Port")
port_id = Column(Integer, ForeignKey("port.id"))
target_id = Column(Integer, ForeignKey("target.id"))
target = relationship("Target", back_populates="screenshots")
endpoint = relationship("Endpoint")
endpoint_id = Column(Integer, ForeignKey("endpoint.id"))
similar_pages = relation(
"Screenshot",
secondary=screenshot_association_table,
primaryjoin=screenshot_association_table.c.screenshot_id == id,
secondaryjoin=screenshot_association_table.c.similar_page_id == id,
backref="similar_to",
)

59
pipeline/models/searchsploit_model.py Normal file
View File

@@ -0,0 +1,59 @@
import textwrap
from pathlib import Path
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String
from .base_model import Base
class SearchsploitResult(Base):
""" Database model that describes results from running searchsploit --nmap TARGET.xml.
Represents searchsploit data.
Relationships:
``target``: many to one -> :class:`pipeline.models.target_model.Target`
"""
__tablename__ = "searchsploit_result"
def __str__(self):
return self.pretty()
def pretty(self, fullpath=False):
pad = " "
type_padlen = 8
filename_padlen = 9
if not fullpath:
filename = Path(self.path).name
msg = f"{pad}{self.type:<{type_padlen}} | {filename:<{filename_padlen}}"
for i, line in enumerate(textwrap.wrap(self.title)):
if i > 0:
msg += f"{' ' * (type_padlen + filename_padlen + 5)}|{pad * 2}{line}\n"
else:
msg += f"|{pad}{line}\n"
msg = msg[:-1] # remove last newline
else:
msg = f"{pad}{self.type:<{type_padlen}}"
for i, line in enumerate(textwrap.wrap(self.title)):
if i > 0:
msg += f"{' ' * (type_padlen + 2)}|{pad * 2}{line}\n"
else:
msg += f"|{pad}{line}\n"
msg += f"{' ' * (type_padlen + 2)}|{pad}{self.path}"
return msg
id = Column(Integer, primary_key=True)
title = Column(String, unique=True)
path = Column(String)
type = Column(String)
target_id = Column(Integer, ForeignKey("target.id"))
target = relationship("Target", back_populates="searchsploit_results")

42
pipeline/models/target_model.py Normal file
View File

@@ -0,0 +1,42 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, String, Boolean
from .base_model import Base
from .port_model import port_association_table
from .technology_model import technology_association_table
class Target(Base):
""" Database model that describes a target; This is the model that functions as the "top" model.
Relationships:
``ip_addresses``: one to many -> :class:`pipeline.models.ip_address_model.IPAddress`
``open_ports``: many to many -> :class:`pipeline.models.port_model.Port`
``nmap_results``: one to many -> :class:`pipeline.models.nmap_model.NmapResult`
``searchsploit_results``: one to many -> :class:`pipeline.models.searchsploit_model.SearchsploitResult`
``endpoints``: one to many -> :class:`pipeline.models.endpoint_model.Endpoint`
``technologies``: many to many -> :class:`pipeline.models.technology_model.Technology`
``screenshots``: one to many -> :class:`pipeline.models.screenshot_model.Screenshot`
"""
__tablename__ = "target"
id = Column(Integer, primary_key=True)
hostname = Column(String, unique=True)
is_web = Column(Boolean, default=False)
vuln_to_sub_takeover = Column(Boolean, default=False)
endpoints = relationship("Endpoint", back_populates="target")
ip_addresses = relationship("IPAddress", back_populates="target")
screenshots = relationship("Screenshot", back_populates="target")
nmap_results = relationship("NmapResult", back_populates="target")
searchsploit_results = relationship("SearchsploitResult", back_populates="target")
open_ports = relationship("Port", secondary=port_association_table, back_populates="targets")
technologies = relationship("Technology", secondary=technology_association_table, back_populates="targets")

52
pipeline/models/technology_model.py Normal file
View File

@@ -0,0 +1,52 @@
from sqlalchemy.orm import relationship
from sqlalchemy import Column, Integer, ForeignKey, String, Table, UniqueConstraint
from .base_model import Base
technology_association_table = Table(
"technology_association",
Base.metadata,
Column("technology_id", Integer, ForeignKey("technology.id")),
Column("target_id", Integer, ForeignKey("target.id")),
)
class Technology(Base):
""" Database model that describes a web technology (i.e. Nginx 1.14).
Represents webanalyze data.
Relationships:
``targets``: many to many -> :class:`pipeline.models.target_model.Target`
"""
__tablename__ = "technology"
__table_args__ = (UniqueConstraint("type", "text"),) # combination of type/text == unique
def __str__(self):
return self.pretty()
def pretty(self, padlen=0):
pad = " "
msg = f"{self.text} ({self.type})\n"
msg += "=" * len(f"{self.text} ({self.type})")
msg += "\n\n"
for target in self.targets:
if target.hostname:
msg += f"{pad * padlen} - {target.hostname}\n"
for ipaddr in target.ip_addresses:
if ipaddr.ipv4_address:
msg += f"{pad * padlen} - {ipaddr.ipv4_address}\n"
elif ipaddr.ipv6_address:
msg += f"{pad * padlen} - {ipaddr.ipv6_address}\n"
return msg
id = Column(Integer, primary_key=True)
type = Column(String)
text = Column(String)
target_id = Column(Integer, ForeignKey("target.id"))
targets = relationship("Target", secondary=technology_association_table, back_populates="technologies")

767
pipeline/recon-pipeline.py Executable file
View File

@@ -0,0 +1,767 @@
#!/usr/bin/env python
# stdlib imports
import os
import sys
import shlex
import shutil
import pickle
import selectors
import tempfile
import threading
import subprocess
import webbrowser
from pathlib import Path
DEFAULT_PROMPT = "recon-pipeline> "
# fix up the PYTHONPATH so we can simply execute the shell from wherever in the filesystem
os.environ["PYTHONPATH"] = f"{os.environ.get('PYTHONPATH')}:{str(Path(__file__).expanduser().resolve().parents[1])}"
# suppress "You should consider upgrading via the 'pip install --upgrade pip' command." warning
os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# in case we need pipenv, add its default --user installed directory to the PATH
sys.path.append(str(Path.home() / ".local" / "bin"))
# third party imports
import cmd2 # noqa: E402
from cmd2.ansi import style # noqa: E402
def cluge_package_imports(name, package):
""" project's module imports; need to cluge the package to handle relative imports at this level
putting into a function for testability
"""
if name == "__main__" and package is None:
file = Path(__file__).expanduser().resolve()
parent, top = file.parent, file.parents[1]
sys.path.append(str(top))
try:
sys.path.remove(str(parent))
except ValueError: # already gone
pass
import pipeline # noqa: F401
sys.modules[name].__package__ = "pipeline"
cluge_package_imports(name=__name__, package=__package__)
from .recon.config import defaults # noqa: F401,E402
from .models.nse_model import NSEResult # noqa: F401,E402
from .models.db_manager import DBManager # noqa: F401,E402
from .models.nmap_model import NmapResult # noqa: F401,E402
from .models.technology_model import Technology # noqa: F401,E402
from .models.searchsploit_model import SearchsploitResult # noqa: F401,E402
from .recon import ( # noqa: F401,E402
get_scans,
tools,
scan_parser,
install_parser,
status_parser,
database_parser,
db_detach_parser,
db_list_parser,
db_attach_parser,
db_delete_parser,
view_parser,
target_results_parser,
endpoint_results_parser,
nmap_results_parser,
technology_results_parser,
searchsploit_results_parser,
port_results_parser,
)
# select loop, handles async stdout/stderr processing of subprocesses
selector = selectors.DefaultSelector()
class SelectorThread(threading.Thread):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._stop_event = threading.Event()
def stop(self):
""" Helper to set the SelectorThread's Event and cleanup the selector's fds """
self._stop_event.set()
# close any fds that were registered and still haven't been unregistered
for key in selector.get_map():
selector.get_key(key).fileobj.close()
def stopped(self):
""" Helper to determine whether the SelectorThread's Event is set or not. """
return self._stop_event.is_set()
def run(self):
""" Run thread that executes a select loop; handles async stdout/stderr processing of subprocesses. """
while not self.stopped():
for k, mask in selector.select():
callback = k.data
callback(k.fileobj)
class ReconShell(cmd2.Cmd):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = None
self.sentry = False
self.self_in_py = True
self.selectorloop = None
self.continue_install = True
self.prompt = DEFAULT_PROMPT
self._initialize_parsers()
Path(defaults.get("tools-dir")).mkdir(parents=True, exist_ok=True)
Path(defaults.get("database-dir")).mkdir(parents=True, exist_ok=True)
# register hooks to handle selector loop start and cleanup
self.register_preloop_hook(self._preloop_hook)
self.register_postloop_hook(self._postloop_hook)
def _initialize_parsers(self):
""" Internal helper to associate methods with the subparsers that use them """
db_list_parser.set_defaults(func=self.database_list)
db_attach_parser.set_defaults(func=self.database_attach)
db_detach_parser.set_defaults(func=self.database_detach)
db_delete_parser.set_defaults(func=self.database_delete)
endpoint_results_parser.set_defaults(func=self.print_endpoint_results)
target_results_parser.set_defaults(func=self.print_target_results)
nmap_results_parser.set_defaults(func=self.print_nmap_results)
technology_results_parser.set_defaults(func=self.print_webanalyze_results)
searchsploit_results_parser.set_defaults(func=self.print_searchsploit_results)
port_results_parser.set_defaults(func=self.print_port_results)
def _preloop_hook(self) -> None:
""" Hook function that runs prior to the cmdloop function starting; starts the selector loop. """
self.selectorloop = SelectorThread(daemon=True)
self.selectorloop.start()
def _postloop_hook(self) -> None:
""" Hook function that runs after the cmdloop function stops; stops the selector loop. """
if self.selectorloop.is_alive():
self.selectorloop.stop()
selector.close()
def _install_error_reporter(self, stderr):
""" Helper to print errors that crop up during any tool installation commands. """
output = stderr.readline()
if not output:
return
output = output.decode().strip()
self.async_alert(style(f"[!] {output}", fg="bright_red"))
def _luigi_pretty_printer(self, stderr):
""" Helper to clean up the VERY verbose luigi log messages that are normally spewed to the terminal. """
output = stderr.readline()
if not output:
return
output = output.decode()
if "===== Luigi Execution Summary =====" in output:
# header that begins the summary of all luigi tasks that have executed/failed
self.async_alert("")
self.sentry = True
# block below used for printing status messages
if self.sentry:
# only set once the Luigi Execution Summary is seen
self.async_alert(style(output.strip(), fg="bright_blue"))
elif output.startswith("INFO: Informed") and output.strip().endswith("PENDING"):
# luigi Task has been queued for execution
words = output.split()
self.async_alert(style(f"[-] {words[5].split('_')[0]} queued", fg="bright_white"))
elif output.startswith("INFO: ") and "running" in output:
# luigi Task is currently running
words = output.split()
# output looks similar to , pid=3938074) running MasscanScan(
# want to grab the index of the luigi task running and use it to find the name of the scan (i.e. MassScan)
scantypeidx = words.index("running") + 1
scantype = words[scantypeidx].split("(", 1)[0]
self.async_alert(style(f"[*] {scantype} running...", fg="bright_yellow"))
elif output.startswith("INFO: Informed") and output.strip().endswith("DONE"):
# luigi Task has completed
words = output.split()
self.async_alert(style(f"[+] {words[5].split('_')[0]} complete!", fg="bright_green"))
@cmd2.with_argparser(scan_parser)
def do_scan(self, args):
""" Scan something.
Possible scans include
AmassScan GobusterScan SearchsploitScan
ThreadedNmapScan WebanalyzeScan AquatoneScan FullScan
MasscanScan SubjackScan TKOSubsScan HTBScan
"""
if self.db_mgr is None:
return self.poutput(
style(f"[!] You are not connected to a database; run database attach before scanning", fg="bright_red")
)
self.poutput(
style(
"If anything goes wrong, rerun your command with --verbose to enable debug statements.",
fg="cyan",
dim=True,
)
)
# get_scans() returns mapping of {classname: [modulename, ...]} in the recon module
# each classname corresponds to a potential recon-pipeline command, i.e. AmassScan, GobusterScan ...
scans = get_scans()
# command is a list that will end up looking something like what's below
# luigi --module pipeline.recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
command = ["luigi", "--module", scans.get(args.scantype)[0]]
tgt_file_path = None
if args.target:
tgt_file_fd, tgt_file_path = tempfile.mkstemp() # temp file to hold target for later parsing
tgt_file_path = Path(tgt_file_path)
tgt_idx = args.__statement__.arg_list.index("--target")
tgt_file_path.write_text(args.target)
args.__statement__.arg_list[tgt_idx + 1] = str(tgt_file_path)
args.__statement__.arg_list[tgt_idx] = "--target-file"
command.extend(args.__statement__.arg_list)
command.extend(["--db-location", str(self.db_mgr.location)])
if args.sausage:
# sausage is not a luigi option, need to remove it
# name for the option came from @AlphaRingo
command.pop(command.index("--sausage"))
webbrowser.open("http://127.0.0.1:8082") # hard-coded here, can specify different with the status command
if args.verbose:
# verbose is not a luigi option, need to remove it
command.pop(command.index("--verbose"))
subprocess.run(command)
else:
# suppress luigi messages in favor of less verbose/cleaner output
proc = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
# add stderr to the selector loop for processing when there's something to read from the fd
selector.register(proc.stderr, selectors.EVENT_READ, self._luigi_pretty_printer)
self.add_dynamic_parser_arguments()
@cmd2.with_argparser(install_parser)
def do_install(self, args):
""" Install any/all of the libraries/tools necessary to make the recon-pipeline function. """
# imported tools variable is in global scope, and we reassign over it later
global tools
persistent_tool_dict = Path(defaults.get("tools-dir")) / ".tool-dict.pkl"
if args.tool == "all":
# show all tools have been queued for installation
[
self.poutput(style(f"[-] {x} queued", fg="bright_white"))
for x in tools.keys()
if not tools.get(x).get("installed")
]
for tool in tools.keys():
self.do_install(tool)
return
if persistent_tool_dict.exists():
tools = pickle.loads(persistent_tool_dict.read_bytes())
if tools.get(args.tool).get("dependencies"):
# get all of the requested tools dependencies
for dependency in tools.get(args.tool).get("dependencies"):
if tools.get(dependency).get("installed"):
# already installed, skip it
continue
self.poutput(
style(f"[!] {args.tool} has an unmet dependency; installing {dependency}", fg="yellow", bold=True)
)
# install the dependency before continuing with installation
self.do_install(dependency)
if tools.get(args.tool).get("installed"):
return self.poutput(style(f"[!] {args.tool} is already installed.", fg="yellow"))
else:
# list of return values from commands run during each tool installation
# used to determine whether the tool installed correctly or not
retvals = list()
self.poutput(style(f"[*] Installing {args.tool}...", fg="bright_yellow"))
addl_env_vars = tools.get(args.tool).get("environ")
if addl_env_vars is not None:
addl_env_vars.update(dict(os.environ))
for command in tools.get(args.tool).get("commands"):
# run all commands required to install the tool
# print each command being run
self.poutput(style(f"[=] {command}", fg="cyan"))
if tools.get(args.tool).get("shell"):
# go tools use subshells (cmd1 && cmd2 && cmd3 ...) during install, so need shell=True
proc = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=addl_env_vars
)
else:
# "normal" command, split up the string as usual and run it
proc = subprocess.Popen(
shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=addl_env_vars
)
out, err = proc.communicate()
if err:
self.poutput(style(f"[!] {err.decode().strip()}", fg="bright_red"))
retvals.append(proc.returncode)
if all(x == 0 for x in retvals):
# all return values in retvals are 0, i.e. all exec'd successfully; tool has been installed
self.poutput(style(f"[+] {args.tool} installed!", fg="bright_green"))
tools[args.tool]["installed"] = True
else:
# unsuccessful tool install
tools[args.tool]["installed"] = False
self.poutput(
style(
f"[!!] one (or more) of {args.tool}'s commands failed and may have not installed properly; check output from the offending command above...",
fg="bright_red",
bold=True,
)
)
# store any tool installs/failures (back) to disk
pickle.dump(tools, persistent_tool_dict.open("wb"))
@cmd2.with_argparser(status_parser)
def do_status(self, args):
""" Open a web browser to Luigi's central scheduler's visualization site """
webbrowser.open(f"http://{args.host}:{args.port}")
@staticmethod
def get_databases():
""" Simple helper to list all known databases from default directory """
dbdir = defaults.get("database-dir")
for db in sorted(Path(dbdir).iterdir()):
yield db
def database_list(self, args):
""" List all known databases """
try:
next(self.get_databases())
except StopIteration:
return self.poutput(style(f"[-] There are no databases.", fg="bright_white"))
for i, location in enumerate(self.get_databases(), start=1):
self.poutput(style(f" {i}. {location}"))
def database_attach(self, args):
""" Attach to the selected database """
locations = [str(x) for x in self.get_databases()] + ["create new database"]
location = self.select(locations)
if location == "create new database":
location = self.read_input(
style("new database name? (recommend something unique for this target)\n-> ", fg="bright_white")
)
new_location = str(Path(defaults.get("database-dir")) / location)
index = sorted([new_location] + locations[:-1]).index(new_location) + 1
self.db_mgr = DBManager(db_location=new_location)
self.poutput(style(f"[*] created database @ {new_location}", fg="bright_yellow"))
location = new_location
else:
index = locations.index(location) + 1
self.db_mgr = DBManager(db_location=location)
self.add_dynamic_parser_arguments()
self.poutput(
style(f"[+] attached to sqlite database @ {Path(location).expanduser().resolve()}", fg="bright_green")
)
self.prompt = f"[db-{index}] {DEFAULT_PROMPT}"
def add_dynamic_parser_arguments(self):
""" Populate command parsers with information from the currently attached database """
port_results_parser.add_argument("--host", choices=self.db_mgr.get_all_targets(), help="filter results by host")
port_results_parser.add_argument(
"--port-number", choices=self.db_mgr.get_all_port_numbers(), help="filter results by port number"
)
endpoint_results_parser.add_argument(
"--status-code", choices=self.db_mgr.get_status_codes(), help="filter results by status code"
)
endpoint_results_parser.add_argument(
"--host", choices=self.db_mgr.get_all_targets(), help="filter results by host"
)
nmap_results_parser.add_argument("--host", choices=self.db_mgr.get_all_targets(), help="filter results by host")
nmap_results_parser.add_argument(
"--nse-script", choices=self.db_mgr.get_all_nse_script_types(), help="filter results by nse script type ran"
)
nmap_results_parser.add_argument(
"--port", choices=self.db_mgr.get_all_port_numbers(), help="filter results by port scanned"
)
nmap_results_parser.add_argument(
"--product", help="filter results by reported product", choices=self.db_mgr.get_all_nmap_reported_products()
)
technology_results_parser.add_argument(
"--host", choices=self.db_mgr.get_all_targets(), help="filter results by host"
)
technology_results_parser.add_argument(
"--type", choices=self.db_mgr.get_all_web_technology_types(), help="filter results by type"
)
technology_results_parser.add_argument(
"--product", choices=self.db_mgr.get_all_web_technology_products(), help="filter results by product"
)
searchsploit_results_parser.add_argument(
"--host", choices=self.db_mgr.get_all_targets(), help="filter results by host"
)
searchsploit_results_parser.add_argument(
"--type", choices=self.db_mgr.get_all_exploit_types(), help="filter results by exploit type"
)
def database_detach(self, args):
""" Detach from the currently attached database """
if self.db_mgr is None:
return self.poutput(style(f"[!] you are not connected to a database", fg="magenta"))
self.db_mgr.close()
self.poutput(style(f"[*] detached from sqlite database @ {self.db_mgr.location}", fg="bright_yellow"))
self.db_mgr = None
self.prompt = DEFAULT_PROMPT
def database_delete(self, args):
""" Delete the selected database """
locations = [str(x) for x in self.get_databases()]
to_delete = self.select(locations)
index = locations.index(to_delete) + 1
Path(to_delete).unlink()
if f"[db-{index}]" in self.prompt:
self.poutput(style(f"[*] detached from sqlite database at {self.db_mgr.location}", fg="bright_yellow"))
self.prompt = DEFAULT_PROMPT
self.db_mgr.close()
self.db_mgr = None
self.poutput(
style(f"[+] deleted sqlite database @ {Path(to_delete).expanduser().resolve()}", fg="bright_green")
)
@cmd2.with_argparser(database_parser)
def do_database(self, args):
""" Manage database connections (list/attach/detach/delete) """
func = getattr(args, "func", None)
if func is not None:
func(args)
else:
self.do_help("database")
def print_target_results(self, args):
""" Display all Targets from the database, ipv4/6 and hostname """
results = list()
printer = self.ppaged if args.paged else self.poutput
if args.type == "ipv4":
targets = self.db_mgr.get_all_ipv4_addresses()
elif args.type == "ipv6":
targets = self.db_mgr.get_all_ipv6_addresses()
elif args.type == "domain-name":
targets = self.db_mgr.get_all_hostnames()
else:
targets = self.db_mgr.get_all_targets()
for target in targets:
if args.vuln_to_subdomain_takeover:
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(target)
if not tgt.vuln_to_sub_takeover:
# skip targets that aren't vulnerable
continue
vulnstring = style("vulnerable", fg="green")
vulnstring = f"[{vulnstring}] {target}"
results.append(vulnstring)
else:
results.append(target)
if results:
printer("\n".join(results))
def print_endpoint_results(self, args):
""" Display all Endpoints from the database """
host_endpoints = status_endpoints = None
printer = self.ppaged if args.paged else self.poutput
color_map = {"2": "green", "3": "blue", "4": "bright_red", "5": "bright_magenta"}
if args.status_code is not None:
status_endpoints = self.db_mgr.get_endpoint_by_status_code(args.status_code)
if args.host is not None:
host_endpoints = self.db_mgr.get_endpoints_by_ip_or_hostname(args.host)
endpoints = self.db_mgr.get_all_endpoints()
for subset in [status_endpoints, host_endpoints]:
if subset is not None:
endpoints = set(endpoints).intersection(set(subset))
results = list()
for endpoint in endpoints:
color = color_map.get(str(endpoint.status_code)[0])
if args.plain:
results.append(endpoint.url)
else:
results.append(f"[{style(endpoint.status_code, fg=color)}] {endpoint.url}")
if not args.headers:
continue
for header in endpoint.headers:
if args.plain:
results.append(f" {header.name}: {header.value}")
else:
results.append(style(f" {header.name}:", fg="cyan") + f" {header.value}")
if results:
printer("\n".join(results))
def print_nmap_results(self, args):
""" Display all NmapResults from the database """
results = list()
printer = self.ppaged if args.paged else self.poutput
if args.host is not None:
# limit by host, if necessary
scans = self.db_mgr.get_nmap_scans_by_ip_or_hostname(args.host)
else:
scans = self.db_mgr.get_and_filter(NmapResult)
if args.port is not None or args.product is not None:
# limit by port, if necessary
tmpscans = scans[:]
for scan in scans:
if args.port is not None and scan.port.port_number != int(args.port) and scan in tmpscans:
del tmpscans[tmpscans.index(scan)]
if args.product is not None and scan.product != args.product and scan in tmpscans:
del tmpscans[tmpscans.index(scan)]
scans = tmpscans
if args.nse_script:
# grab the specific nse-script, check that the corresponding nmap result is one we care about, and print
for nse_scan in self.db_mgr.get_and_filter(NSEResult, script_id=args.nse_script):
for nmap_result in nse_scan.nmap_results:
if nmap_result not in scans:
continue
results.append(nmap_result.pretty(nse_results=[nse_scan], commandline=args.commandline))
else:
# done filtering, grab w/e is left
for scan in scans:
results.append(scan.pretty(commandline=args.commandline))
if results:
printer("\n".join(results))
def print_webanalyze_results(self, args):
""" Display all NmapResults from the database """
results = list()
printer = self.ppaged if args.paged else self.poutput
filters = dict()
if args.type is not None:
filters["type"] = args.type
if args.product is not None:
filters["text"] = args.product
if args.host:
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(args.host)
printer(args.host)
printer("=" * len(args.host))
for tech in tgt.technologies:
if args.product is not None and args.product != tech.text:
continue
if args.type is not None and args.type != tech.type:
continue
printer(f" - {tech.text} ({tech.type})")
else:
for scan in self.db_mgr.get_and_filter(Technology, **filters):
results.append(scan.pretty(padlen=1))
if results:
printer("\n".join(results))
def print_searchsploit_results(self, args):
""" Display all NmapResults from the database """
results = list()
targets = self.db_mgr.get_all_targets()
printer = self.ppaged if args.paged else self.poutput
for ss_scan in self.db_mgr.get_and_filter(SearchsploitResult):
tmp_targets = set()
if (
args.host is not None
and self.db_mgr.get_or_create_target_by_ip_or_hostname(args.host) != ss_scan.target
):
continue
if ss_scan.target.hostname in targets:
# hostname is in targets, so hasn't been reported yet
tmp_targets.add(ss_scan.target.hostname) # add to this report
targets.remove(ss_scan.target.hostname) # remove from targets list, having been reported
for ipaddr in ss_scan.target.ip_addresses:
address = ipaddr.ipv4_address or ipaddr.ipv6_address
if address is not None and address in targets:
tmp_targets.add(ipaddr.ipv4_address)
targets.remove(ipaddr.ipv4_address)
if tmp_targets:
header = ", ".join(tmp_targets)
results.append(header)
results.append("=" * len(header))
for scan in ss_scan.target.searchsploit_results:
if args.type is not None and scan.type != args.type:
continue
results.append(scan.pretty(fullpath=args.fullpath))
if results:
printer("\n".join(results))
def print_port_results(self, args):
""" Display all Ports from the database """
results = list()
targets = self.db_mgr.get_all_targets()
printer = self.ppaged if args.paged else self.poutput
for target in targets:
if args.host is not None and target != args.host:
# host specified, but it's not this particular target
continue
ports = [
str(port.port_number) for port in self.db_mgr.get_or_create_target_by_ip_or_hostname(target).open_ports
]
if args.port_number and args.port_number not in ports:
continue
if ports:
results.append(f"{target}: {','.join(ports)}")
if results:
printer("\n".join(results))
@cmd2.with_argparser(view_parser)
def do_view(self, args):
""" View results of completed scans """
if self.db_mgr is None:
return self.poutput(style(f"[!] you are not connected to a database", fg="magenta"))
func = getattr(args, "func", None)
if func is not None:
func(args)
else:
self.do_help("view")
def main(
name,
old_tools_dir=Path().home() / ".recon-tools",
old_tools_dict=Path().home() / ".cache" / ".tool-dict.pkl",
old_searchsploit_rc=Path().home() / ".searchsploit_rc",
):
""" Functionified for testability """
if name == "__main__":
if old_tools_dir.exists() and old_tools_dir.is_dir():
# want to try and ensure a smooth transition for folks who have used the pipeline before from
# v0.8.4 and below to v0.9.0+
print(style(f"[*] Found remnants of an older version of recon-pipeline.", fg="bright_yellow"))
print(
style(
f"[*] It's {style('strongly', fg='red')} advised that you allow us to remove them.",
fg="bright_white",
)
)
print(
style(
f"[*] Do you want to remove {old_tools_dir}/*, {old_searchsploit_rc}, and {old_tools_dict}?",
fg="bright_white",
)
)
answer = cmd2.Cmd().select(["Yes", "No"])
print(style(f"[+] You chose {answer}", fg="bright_green"))
if answer == "Yes":
shutil.rmtree(old_tools_dir)
print(style(f"[+] {old_tools_dir} removed", fg="bright_green"))
if old_tools_dict.exists():
old_tools_dict.unlink()
print(style(f"[+] {old_tools_dict} removed", fg="bright_green"))
if old_searchsploit_rc.exists():
old_searchsploit_rc.unlink()
print(style(f"[+] {old_searchsploit_rc} removed", fg="bright_green"))
print(style(f"[=] Please run the install all command to complete setup", fg="bright_blue"))
rs = ReconShell(persistent_history_file="~/.reconshell_history", persistent_history_length=10000)
sys.exit(rs.cmdloop())
main(name=__name__)

25
pipeline/recon/__init__.py Normal file
View File

@@ -0,0 +1,25 @@
from .helpers import get_scans
from .targets import TargetList
from .tool_definitions import tools
from .wrappers import FullScan, HTBScan
from .amass import AmassScan, ParseAmassOutput
from .masscan import MasscanScan, ParseMasscanOutput
from .nmap import ThreadedNmapScan, SearchsploitScan
from .config import tool_paths, top_udp_ports, top_tcp_ports, defaults, web_ports
from .parsers import (
install_parser,
scan_parser,
status_parser,
database_parser,
db_attach_parser,
db_delete_parser,
db_detach_parser,
db_list_parser,
view_parser,
target_results_parser,
endpoint_results_parser,
nmap_results_parser,
technology_results_parser,
searchsploit_results_parser,
port_results_parser,
)

111
recon/amass.py → pipeline/recon/amass.py
View File

@@ -1,17 +1,19 @@
import json
import ipaddress
import subprocess
from pathlib import Path
import luigi
from luigi.util import inherits
from luigi.contrib.external_program import ExternalProgramTask
from luigi.contrib.sqla import SQLAlchemyTarget
from recon.config import tool_paths
from recon.targets import TargetList
import pipeline.models.db_manager
from .config import tool_paths
from .targets import TargetList
from ..models.target_model import Target
@inherits(TargetList)
class AmassScan(ExternalProgramTask):
class AmassScan(luigi.Task):
""" Run ``amass`` scan to perform subdomain enumeration of given domain(s).
Note:
@@ -34,12 +36,18 @@ class AmassScan(ExternalProgramTask):
Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line.
db_location: specifies the path to the database used for storing results *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
exempt_list = luigi.Parameter(default="")
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "amass-results").expanduser().resolve()
def requires(self):
""" AmassScan depends on TargetList to run.
@@ -48,7 +56,7 @@ class AmassScan(ExternalProgramTask):
Returns:
luigi.ExternalTask - TargetList
"""
args = {"target_file": self.target_file, "results_dir": self.results_dir}
args = {"target_file": self.target_file, "results_dir": self.results_dir, "db_location": self.db_location}
return TargetList(**args)
def output(self):
@@ -63,19 +71,27 @@ class AmassScan(ExternalProgramTask):
new_path = results_subfolder / "amass.json"
return luigi.LocalTarget(new_path.resolve())
return luigi.LocalTarget(new_path.expanduser().resolve())
def program_args(self):
def run(self):
""" Defines the options/arguments sent to amass after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
if not self.input().path.endswith("domains"):
return f"touch {self.output().path}".split()
hostnames = self.db_mgr.get_all_hostnames()
if hostnames:
# TargetList generated some domains for us to scan with amass
amass_input_file = self.results_subfolder / "input-from-targetlist"
with open(amass_input_file, "w") as f:
for hostname in hostnames:
f.write(f"{hostname}\n")
else:
return subprocess.run(f"touch {self.output().path}".split())
command = [
f"{tool_paths.get('amass')}",
@@ -86,7 +102,7 @@ class AmassScan(ExternalProgramTask):
"-min-for-recursive",
"3",
"-df",
self.input().path,
str(amass_input_file),
"-json",
self.output().path,
]
@@ -95,7 +111,9 @@ class AmassScan(ExternalProgramTask):
command.append("-blf") # Path to a file providing blacklisted subdomains
command.append(self.exempt_list)
return command
subprocess.run(command)
amass_input_file.unlink()
@inherits(AmassScan)
@@ -103,11 +121,17 @@ class ParseAmassOutput(luigi.Task):
""" Read amass JSON results and create categorized entries into ip|subdomain files.
Args:
db_location: specifies the path to the database used for storing results *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "amass-results").expanduser().resolve()
def requires(self):
""" ParseAmassOutput depends on AmassScan to run.
@@ -118,31 +142,23 @@ class ParseAmassOutput(luigi.Task):
luigi.ExternalTask - TargetList
"""
args = {"target_file": self.target_file, "exempt_list": self.exempt_list, "results_dir": self.results_dir}
args = {
"target_file": self.target_file,
"exempt_list": self.exempt_list,
"results_dir": self.results_dir,
"db_location": self.db_location,
}
return AmassScan(**args)
def output(self):
""" Returns the target output files for this task.
Naming conventions for the output files are:
TARGET_FILE.ips
TARGET_FILE.ip6s
TARGET_FILE.subdomains
Returns:
dict(str: luigi.local_target.LocalTarget)
luigi.contrib.sqla.SQLAlchemyTarget
"""
results_subfolder = Path(self.results_dir) / "target-results"
ips = (results_subfolder / "ipv4_addresses").resolve()
ip6s = ips.with_name("ipv6_addresses").resolve()
subdomains = ips.with_name("subdomains").resolve()
return {
"target-ips": luigi.LocalTarget(ips),
"target-ip6s": luigi.LocalTarget(ip6s),
"target-subdomains": luigi.LocalTarget(subdomains),
}
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="target", update_id=self.task_id
)
def run(self):
""" Parse the json file produced by AmassScan and categorize the results into ip|subdomain files.
@@ -164,35 +180,26 @@ class ParseAmassOutput(luigi.Task):
"source": "Previous Enum"
}
"""
unique_ips = set()
unique_ip6s = set()
unique_subs = set()
self.results_subfolder.mkdir(parents=True, exist_ok=True)
Path(self.output().get("target-ips").path).parent.mkdir(parents=True, exist_ok=True)
if Path(self.input().path).stat().st_size == 0:
self.output().touch()
return
amass_json = self.input().open()
ip_file = self.output().get("target-ips").open("w")
ip6_file = self.output().get("target-ip6s").open("w")
subdomain_file = self.output().get("target-subdomains").open("w")
with amass_json as aj, ip_file as ip_out, ip6_file as ip6_out, subdomain_file as subdomain_out:
for line in aj:
with amass_json as amass_json_file:
for line in amass_json_file:
entry = json.loads(line)
unique_subs.add(entry.get("name"))
tgt = self.db_mgr.get_or_create(Target, hostname=entry.get("name"), is_web=True)
for address in entry.get("addresses"):
ipaddr = address.get("ip")
if isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv4Address): # ipv4 addr
unique_ips.add(ipaddr)
elif isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv6Address): # ipv6
unique_ip6s.add(ipaddr)
# send gathered results to their appropriate destination
for ip in unique_ips:
print(ip, file=ip_out)
tgt = self.db_mgr.add_ipv4_or_v6_address_to_target(tgt, ipaddr)
for sub in unique_subs:
print(sub, file=subdomain_out)
self.db_mgr.add(tgt)
self.output().touch()
for ip6 in unique_ip6s:
print(ip6, file=ip6_out)
self.db_mgr.close()

59
recon/config.py → pipeline/recon/config.py
View File

@@ -7,15 +7,15 @@ defaults = {
"threads": "10",
"masscan-rate": "1000",
"masscan-iface": "tun0",
"tools-dir": f"{Path.home()}/.recon-tools",
"gobuster-extensions": "",
"results-dir": "recon-results",
"aquatone-scan-timeout": "900",
"gobuster-extensions": "",
"tools-dir": f"{Path.home()}/.local/recon-pipeline/tools",
"database-dir": f"{Path.home()}/.local/recon-pipeline/databases",
}
defaults["gobuster-wordlist"] = f"{defaults.get('tools-dir')}/seclists/Discovery/Web-Content/common.txt"
web_ports = {"80", "443", "8080", "8000", "8443"}
tool_paths = {
"aquatone": f"{defaults.get('tools-dir')}/aquatone",
@@ -23,14 +23,63 @@ tool_paths = {
"tko-subs-dir": f"{Path.home()}/go/src/github.com/anshumanbh/tko-subs",
"subjack": f"{Path.home()}/go/bin/subjack",
"subjack-fingerprints": f"{Path.home()}/go/src/github.com/haccer/subjack/fingerprints.json",
"CORScanner": f"{defaults.get('tools-dir')}/CORScanner/cors_scan.py",
"gobuster": f"{Path.home()}/go/bin/gobuster",
"recursive-gobuster": f"{defaults.get('tools-dir')}/recursive-gobuster/recursive-gobuster.pyz",
"webanalyze": f"{Path.home()}/go/bin/webanalyze",
"masscan": f"{defaults.get('tools-dir')}/masscan",
"amass": f"{defaults.get('tools-dir')}/amass",
"go": "/usr/local/go/bin/go",
"searchsploit": f"{defaults.get('tools-dir')}/exploitdb/searchsploit"
"searchsploit": f"{defaults.get('tools-dir')}/exploitdb/searchsploit",
}
web_ports = {
"80",
"81",
"280",
"443",
"591",
"593",
"2080",
"2480",
"3080",
"4080",
"4567",
"5080",
"5104",
"5800",
"6080",
"7001",
"7080",
"7777",
"8000",
"8008",
"8042",
"8080",
"8081",
"8082",
"8088",
"8180",
"8222",
"8280",
"8281",
"8443",
"8530",
"8887",
"9000",
"9080",
"9090",
"16080",
"832",
"981",
"1311",
"7002",
"7021",
"7023",
"7025",
"7777",
"8333",
"8531",
"8888",
}
top_tcp_ports = [

68
pipeline/recon/helpers.py Normal file
View File

@@ -0,0 +1,68 @@
import sys
import inspect
import pkgutil
import importlib
import ipaddress
from pathlib import Path
from collections import defaultdict
def get_scans():
""" Iterates over the recon package and its modules to find all of the classes that end in [Ss]can.
**A contract exists here that says any scans need to end with the word scan in order to be found by this function.**
Example:
``defaultdict(<class 'list'>, {'AmassScan': ['pipeline.recon.amass'], 'MasscanScan': ['pipeline.recon.masscan'], ... })``
Returns:
dict containing mapping of ``classname -> [modulename, ...]`` for all potential recon-pipeline commands
"""
scans = defaultdict(list)
file = Path(__file__).expanduser().resolve()
web = file.parent / "web"
recon = file.parents[1] / "recon"
lib_paths = [str(web), str(recon)]
# recursively walk packages; import each module in each package
# walk_packages yields ModuleInfo objects for all modules recursively on path
# prefix is a string to output on the front of every module name on output.
for loader, module_name, is_pkg in pkgutil.walk_packages(path=lib_paths, prefix=f"{__package__}."):
try:
importlib.import_module(module_name)
except ModuleNotFoundError:
# skipping things like recon.aquatone, not entirely sure why they're showing up...
pass
# walk all modules, grabbing classes that we've written and add them to the classlist defaultdict
# getmembers returns all members of an object in a list of tuples (name, value)
for name, obj in inspect.getmembers(sys.modules[__package__]):
if inspect.ismodule(obj) and not name.startswith("_"):
# we're only interested in modules that don't begin with _ i.e. magic methods __len__ etc...
for subname, subobj in inspect.getmembers(obj):
if inspect.isclass(subobj) and subname.lower().endswith("scan"):
# now we only care about classes that end in [Ss]can
scans[subname].append(f"{__package__}.{name}")
return scans
def is_ip_address(ipaddr):
""" Simple helper to determine if given string is an ip address or subnet """
try:
ipaddress.ip_interface(ipaddr)
return True
except ValueError:
return False
def get_ip_address_version(ipaddr):
""" Simple helper to determine whether a given ip address is ipv4 or ipv6 """
if is_ip_address(ipaddr):
if isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv4Address): # ipv4 addr
return "4"
elif isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv6Address): # ipv6
return "6"

129
recon/masscan.py → pipeline/recon/masscan.py
View File

@@ -1,16 +1,19 @@
import json
import pickle
import logging
import subprocess
from pathlib import Path
from collections import defaultdict
import luigi
from luigi.util import inherits
from luigi.contrib.sqla import SQLAlchemyTarget
from recon.targets import TargetList
from recon.amass import ParseAmassOutput
from recon.config import top_tcp_ports, top_udp_ports, defaults
import pipeline.models.db_manager
from .targets import TargetList
from .amass import ParseAmassOutput
from ..models.port_model import Port
from ..models.ip_address_model import IPAddress
from .config import top_tcp_ports, top_udp_ports, defaults, tool_paths, web_ports
@inherits(TargetList, ParseAmassOutput)
@@ -43,16 +46,22 @@ class MasscanScan(luigi.Task):
interface: use the named raw network interface, such as "eth0"
top_ports: Scan top N most popular ports
ports: specifies the port(s) to be scanned
db_location: specifies the path to the database used for storing results *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
"""
rate = luigi.Parameter(default=defaults.get("masscan-rate", ""))
interface = luigi.Parameter(default=defaults.get("masscan-iface", ""))
rate = luigi.Parameter(default=defaults.get("masscan-rate"))
interface = luigi.Parameter(default=defaults.get("masscan-iface"))
top_ports = luigi.IntParameter(default=0) # IntParameter -> top_ports expected as int
ports = luigi.Parameter(default="")
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "masscan-results").expanduser().resolve()
def output(self):
""" Returns the target output for this task.
@@ -61,11 +70,9 @@ class MasscanScan(luigi.Task):
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "masscan-results"
new_path = self.results_subfolder / "masscan.json"
new_path = results_subfolder / "masscan.json"
return luigi.LocalTarget(new_path.resolve())
return luigi.LocalTarget(new_path.expanduser().resolve())
def run(self):
""" Defines the options/arguments sent to masscan after processing.
@@ -73,22 +80,11 @@ class MasscanScan(luigi.Task):
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
if self.ports and self.top_ports:
# can't have both
logging.error("Only --ports or --top-ports is permitted, not both.")
exit(1)
if not self.ports and not self.top_ports:
# need at least one
# need at least one, can't be put into argparse scanner because things like amass don't require ports option
logging.error("Must specify either --top-ports or --ports.")
exit(2)
if self.top_ports < 0:
# sanity check
logging.error("--top-ports must be greater than 0")
exit(3)
if self.top_ports:
# if --top-ports used, format the top_*_ports lists as strings and then into a proper masscan --ports option
top_tcp_ports_str = ",".join(str(x) for x in top_tcp_ports[: self.top_ports])
@@ -97,19 +93,22 @@ class MasscanScan(luigi.Task):
self.ports = f"{top_tcp_ports_str},U:{top_udp_ports_str}"
self.top_ports = 0
target_list = yield TargetList(target_file=self.target_file, results_dir=self.results_dir)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
yield TargetList(target_file=self.target_file, results_dir=self.results_dir, db_location=self.db_location)
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
if self.db_mgr.get_all_hostnames():
# TargetList generated some domains for us to scan with amass
if target_list.path.endswith("domains"):
yield ParseAmassOutput(
target_file=self.target_file, exempt_list=self.exempt_list, results_dir=self.results_dir
target_file=self.target_file,
exempt_list=self.exempt_list,
results_dir=self.results_dir,
db_location=self.db_location,
)
command = [
"masscan",
tool_paths.get("masscan"),
"-v",
"--open",
"--banners",
@@ -124,12 +123,24 @@ class MasscanScan(luigi.Task):
"-iL",
]
if target_list.path.endswith("domains"):
command.append(target_list.path.replace("domains", "ipv4_addresses"))
else:
command.append(target_list.path.replace("domains", "ip_addresses"))
# masscan only understands how to scan ipv4
ip_addresses = self.db_mgr.get_all_ipv4_addresses()
masscan_input_file = None
subprocess.run(command)
if ip_addresses:
# TargetList generated ip addresses for us to scan with masscan
masscan_input_file = self.results_subfolder / "input-from-amass"
with open(masscan_input_file, "w") as f:
for ip_address in ip_addresses:
f.write(f"{ip_address}\n")
command.append(str(masscan_input_file))
subprocess.run(command) # will fail if no ipv4 addresses were found
if masscan_input_file is not None:
masscan_input_file.unlink()
@inherits(MasscanScan)
@@ -141,10 +152,16 @@ class ParseMasscanOutput(luigi.Task):
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
db_location: specifies the path to the database used for storing results *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "masscan-results").expanduser().resolve()
def requires(self):
""" ParseMasscanOutput depends on Masscan to run.
@@ -160,6 +177,7 @@ class ParseMasscanOutput(luigi.Task):
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"db_location": self.db_location,
}
return MasscanScan(**args)
@@ -171,16 +189,12 @@ class ParseMasscanOutput(luigi.Task):
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "masscan-results"
new_path = results_subfolder / "masscan.parsed.pickle"
return luigi.LocalTarget(new_path.resolve())
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="port", update_id=self.task_id
)
def run(self):
""" Reads masscan JSON results and creates a pickled dictionary of pertinent information for processing. """
ip_dict = defaultdict(lambda: defaultdict(set)) # nested defaultdict
try:
# load masscan results from Masscan Task
entries = json.load(self.input().open())
@@ -189,10 +203,10 @@ class ParseMasscanOutput(luigi.Task):
# this task if restarted because we never hit pickle.dump
return print(e)
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
"""
build out ip_dictionary from the loaded JSON
populate database from the loaded JSON
masscan JSON structure over which we're looping
[
@@ -200,20 +214,27 @@ class ParseMasscanOutput(luigi.Task):
,
{ "ip": "10.10.10.146", "timestamp": "1567856130", "ports": [ {"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 63} ] }
]
ip_dictionary structure that is built out from each JSON entry
{
"IP_ADDRESS":
{'udp': {"161", "5000", ... },
...
i.e. {protocol: set(ports) }
}
"""
for entry in entries:
single_target_ip = entry.get("ip")
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(single_target_ip)
if single_target_ip not in tgt.ip_addresses:
tgt.ip_addresses.append(self.db_mgr.get_or_create(IPAddress, ipv4_address=single_target_ip))
for port_entry in entry.get("ports"):
protocol = port_entry.get("proto")
ip_dict[single_target_ip][protocol].add(str(port_entry.get("port")))
with open(self.output().path, "wb") as f:
pickle.dump(dict(ip_dict), f)
port = self.db_mgr.get_or_create(Port, protocol=protocol, port_number=port_entry.get("port"))
if str(port.port_number) in web_ports:
tgt.is_web = True
tgt.open_ports.append(port)
self.db_mgr.add(tgt)
self.output().touch()
self.db_mgr.close()

324
pipeline/recon/nmap.py Normal file
View File

@@ -0,0 +1,324 @@
import ast
import logging
import subprocess
import concurrent.futures
from pathlib import Path
import luigi
import sqlalchemy
from luigi.util import inherits
from libnmap.parser import NmapParser
from luigi.contrib.sqla import SQLAlchemyTarget
import pipeline.models.db_manager
from .masscan import ParseMasscanOutput
from .config import defaults, tool_paths
from .helpers import get_ip_address_version, is_ip_address
from ..models.port_model import Port
from ..models.nse_model import NSEResult
from ..models.target_model import Target
from ..models.nmap_model import NmapResult
from ..models.ip_address_model import IPAddress
from ..models.searchsploit_model import SearchsploitResult
@inherits(ParseMasscanOutput)
class ThreadedNmapScan(luigi.Task):
""" Run ``nmap`` against specific targets and ports gained from the ParseMasscanOutput Task.
Install:
``nmap`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``nmap``.
Basic Example:
.. code-block:: console
nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution
db_location: specifies the path to the database used for storing results *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads"))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "nmap-results").expanduser().resolve()
def requires(self):
""" ThreadedNmap depends on ParseMasscanOutput to run.
TargetList expects target_file, results_dir, and db_location as parameters.
Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.
Returns:
luigi.Task - ParseMasscanOutput
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"db_location": self.db_location,
}
return ParseMasscanOutput(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output folder is TARGET_FILE-nmap-results.
The output folder will be populated with all of the output files generated by
any nmap commands run. Because the nmap command uses -oA, there will be three
files per target scanned: .xml, .nmap, .gnmap.
Returns:
luigi.local_target.LocalTarget
"""
return {
"sqltarget": SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="nmap_result", update_id=self.task_id
),
"localtarget": luigi.LocalTarget(str(self.results_subfolder)),
}
def parse_nmap_output(self):
""" Read nmap .xml results and add entries into specified database """
for entry in self.results_subfolder.glob("nmap*.xml"):
# relying on python-libnmap here
report = NmapParser.parse_fromfile(entry)
for host in report.hosts:
for service in host.services:
port = self.db_mgr.get_or_create(Port, protocol=service.protocol, port_number=service.port)
if is_ip_address(host.address) and get_ip_address_version(host.address) == "4":
ip_address = self.db_mgr.get_or_create(IPAddress, ipv4_address=host.address)
else:
ip_address = self.db_mgr.get_or_create(IPAddress, ipv6_address=host.address)
if ip_address.target is None:
# account for ip addresses identified that aren't already tied to a target
# almost certainly ipv6 addresses
tgt = self.db_mgr.get_or_create(Target)
tgt.ip_addresses.append(ip_address)
else:
tgt = ip_address.target
try:
nmap_result = self.db_mgr.get_or_create(
NmapResult, port=port, ip_address=ip_address, target=tgt
)
except sqlalchemy.exc.StatementError:
# one of the three (port/ip/tgt) didn't exist and we're querying on ids that the db doesn't know
self.db_mgr.add(port)
self.db_mgr.add(ip_address)
self.db_mgr.add(tgt)
nmap_result = self.db_mgr.get_or_create(
NmapResult, port=port, ip_address=ip_address, target=tgt
)
for nse_result in service.scripts_results:
script_id = nse_result.get("id")
script_output = nse_result.get("output")
nse_obj = self.db_mgr.get_or_create(NSEResult, script_id=script_id, script_output=script_output)
nmap_result.nse_results.append(nse_obj)
nmap_result.open = service.open()
nmap_result.reason = service.reason
nmap_result.service = service.service
nmap_result.commandline = report.commandline
nmap_result.product = service.service_dict.get("product")
nmap_result.product_version = service.service_dict.get("version")
nmap_result.target.nmap_results.append(nmap_result)
self.db_mgr.add(nmap_result)
self.output().get("sqltarget").touch()
self.db_mgr.close()
def run(self):
""" Parses pickled target info dictionary and runs targeted nmap scans against only open ports. """
try:
self.threads = abs(int(self.threads))
except (TypeError, ValueError):
return logging.error("The value supplied to --threads must be a non-negative integer.")
nmap_command = [ # placeholders will be overwritten with appropriate info in loop below
"nmap",
"--open",
"PLACEHOLDER-IDX-2",
"-n",
"-sC",
"-T",
"4",
"-sV",
"-Pn",
"-p",
"PLACEHOLDER-IDX-10",
"-oA",
]
commands = list()
for target in self.db_mgr.get_all_targets():
for protocol in ("tcp", "udp"):
ports = self.db_mgr.get_ports_by_ip_or_host_and_protocol(target, protocol)
if ports:
tmp_cmd = nmap_command[:]
tmp_cmd[2] = "-sT" if protocol == "tcp" else "-sU"
# arg to -oA, will drop into subdir off curdir
tmp_cmd[10] = ",".join(ports)
tmp_cmd.append(str(Path(self.output().get("localtarget").path) / f"nmap.{target}-{protocol}"))
if is_ip_address(target) and get_ip_address_version(target) == "6":
# got an ipv6 address
tmp_cmd.insert(-2, "-6")
tmp_cmd.append(target) # target as final arg to nmap
commands.append(tmp_cmd)
# basically mkdir -p, won't error out if already there
self.results_subfolder.mkdir(parents=True, exist_ok=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.map(subprocess.run, commands)
self.parse_nmap_output()
@inherits(ThreadedNmapScan)
class SearchsploitScan(luigi.Task):
""" Run ``searchcploit`` against each ``nmap*.xml`` file in the **TARGET-nmap-results** directory and write results to disk.
Install:
``searchcploit`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``searchcploit``.
Basic Example:
.. code-block:: console
searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution *Required by upstream Task*
db_location: specifies the path to the database used for storing results *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifies the directory on disk to which all Task results are written *Required by upstream Task*
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
def requires(self):
""" Searchsploit depends on ThreadedNmap to run.
TargetList expects target_file, results_dir, and db_location as parameters.
Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.
ThreadedNmap expects threads
Returns:
luigi.Task - ThreadedNmap
"""
args = {
"rate": self.rate,
"ports": self.ports,
"threads": self.threads,
"top_ports": self.top_ports,
"interface": self.interface,
"target_file": self.target_file,
"results_dir": self.results_dir,
"db_location": self.db_location,
}
return ThreadedNmapScan(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output folder is TARGET_FILE-searchsploit-results.
The output folder will be populated with all of the output files generated by
any searchsploit commands run.
Returns:
luigi.local_target.LocalTarget
"""
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="searchsploit_result", update_id=self.task_id
)
def run(self):
""" Grabs the xml files created by ThreadedNmap and runs searchsploit --nmap on each one, saving the output. """
for entry in Path(self.input().get("localtarget").path).glob("nmap*.xml"):
proc = subprocess.run(
[tool_paths.get("searchsploit"), "-j", "-v", "--nmap", str(entry)], stdout=subprocess.PIPE
)
if proc.stdout:
# change wall-searchsploit-results/nmap.10.10.10.157-tcp to 10.10.10.157
ipaddr = entry.stem.replace("nmap.", "").replace("-tcp", "").replace("-udp", "")
contents = proc.stdout.decode()
for line in contents.splitlines():
if "Title" in line:
# {'Title': "Nginx (Debian Based Distros + Gentoo) ... }
# oddity introduced on 15 Apr 2020 from an exploitdb update
# entries have two double quotes in a row for no apparent reason
# {"Title":"PHP-FPM + Nginx - Remote Code Execution"", ...
# seems to affect all entries at the moment. will remove this line if it
# ever returns to normal
line = line.replace('""', '"')
if line.endswith(","):
# result would be a tuple if the comma is left on the line; remove it
tmp_result = ast.literal_eval(line.strip()[:-1])
else:
# normal dict
tmp_result = ast.literal_eval(line.strip())
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(ipaddr)
ssr_type = tmp_result.get("Type")
ssr_title = tmp_result.get("Title")
ssr_path = tmp_result.get("Path")
ssr = self.db_mgr.get_or_create(
SearchsploitResult, type=ssr_type, title=ssr_title, path=ssr_path
)
tgt.searchsploit_results.append(ssr)
self.db_mgr.add(tgt)
self.output().touch()
self.db_mgr.close()

193
pipeline/recon/parsers.py Normal file
View File

@@ -0,0 +1,193 @@
import socket
import cmd2
from .config import defaults
from .helpers import get_scans
from .tool_definitions import tools
# options for ReconShell's 'install' command
install_parser = cmd2.Cmd2ArgumentParser()
install_parser.add_argument("tool", help="which tool to install", choices=list(tools.keys()) + ["all"])
# options for ReconShell's 'status' command
status_parser = cmd2.Cmd2ArgumentParser()
status_parser.add_argument(
"--port",
help="port on which the luigi central scheduler's visualization site is running (default: 8082)",
default="8082",
)
status_parser.add_argument(
"--host",
help="host on which the luigi central scheduler's visualization site is running (default: localhost)",
default="127.0.0.1",
)
# options for ReconShell's 'scan' command
scan_parser = cmd2.Cmd2ArgumentParser()
scan_parser.add_argument("scantype", choices_function=get_scans, help="which type of scan to run")
target_group = scan_parser.add_mutually_exclusive_group(required=True)
target_group.add_argument(
"--target-file",
completer_method=cmd2.Cmd.path_complete,
help="file created by the user that defines the target's scope; list of ips/domains",
)
target_group.add_argument("--target", help="ip or domain to target")
scan_parser.add_argument(
"--exempt-list", completer_method=cmd2.Cmd.path_complete, help="list of blacklisted ips/domains"
)
scan_parser.add_argument(
"--results-dir",
completer_method=cmd2.Cmd.path_complete,
help=f"directory in which to save scan results (default: {defaults.get('results-dir')})",
)
scan_parser.add_argument(
"--wordlist",
completer_method=cmd2.Cmd.path_complete,
help=f"path to wordlist used by gobuster (default: {defaults.get('gobuster-wordlist')})",
)
scan_parser.add_argument(
"--interface",
choices_function=lambda: [x[1] for x in socket.if_nameindex()],
help=f"which interface masscan should use (default: {defaults.get('masscan-iface')})",
)
scan_parser.add_argument(
"--recursive", action="store_true", help="whether or not to recursively gobust (default: False)", default=False
)
scan_parser.add_argument("--rate", help=f"rate at which masscan should scan (default: {defaults.get('masscan-rate')})")
port_group = scan_parser.add_mutually_exclusive_group()
port_group.add_argument(
"--top-ports",
help="ports to scan as specified by nmap's list of top-ports (only meaningful to around 5000)",
type=int,
)
port_group.add_argument("--ports", help="port specification for masscan (all ports example: 1-65535,U:1-65535)")
scan_parser.add_argument(
"--threads",
help=f"number of threads for all of the threaded applications to use (default: {defaults.get('threads')})",
)
scan_parser.add_argument(
"--scan-timeout", help=f"scan timeout for aquatone (default: {defaults.get('aquatone-scan-timeout')})"
)
scan_parser.add_argument("--proxy", help="proxy for gobuster if desired (ex. 127.0.0.1:8080)")
scan_parser.add_argument("--extensions", help="list of extensions for gobuster (ex. asp,html,aspx)")
scan_parser.add_argument(
"--sausage",
action="store_true",
default=False,
help="open a web browser to Luigi's central scheduler's visualization site (see how the sausage is made!)",
)
scan_parser.add_argument(
"--local-scheduler",
action="store_true",
help="use the local scheduler instead of the central scheduler (luigid) (default: False)",
default=False,
)
scan_parser.add_argument(
"--verbose",
action="store_true",
help="shows debug messages from luigi, useful for troubleshooting (default: False)",
)
# top level and subparsers for ReconShell's database command
database_parser = cmd2.Cmd2ArgumentParser()
database_subparsers = database_parser.add_subparsers(
title="subcommands", help="Manage database connections (list/attach/detach/delete)"
)
db_list_parser = database_subparsers.add_parser("list", help="List all known databases")
db_delete_parser = database_subparsers.add_parser("delete", help="Delete the selected database")
db_attach_parser = database_subparsers.add_parser("attach", help="Attach to the selected database")
db_detach_parser = database_subparsers.add_parser("detach", help="Detach from the currently attached database")
# ReconShell's view command
view_parser = cmd2.Cmd2ArgumentParser()
view_subparsers = view_parser.add_subparsers(title="result types")
target_results_parser = view_subparsers.add_parser(
"targets", help="List all known targets (ipv4/6 & domain names); produced by amass", conflict_handler="resolve"
)
target_results_parser.add_argument(
"--vuln-to-subdomain-takeover",
action="store_true",
default=False,
help="show targets identified as vulnerable to subdomain takeover",
)
target_results_parser.add_argument("--type", choices=["ipv4", "ipv6", "domain-name"], help="filter by target type")
target_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
technology_results_parser = view_subparsers.add_parser(
"web-technologies",
help="List all known web technologies identified; produced by webanalyze",
conflict_handler="resolve",
)
technology_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
endpoint_results_parser = view_subparsers.add_parser(
"endpoints", help="List all known endpoints; produced by gobuster", conflict_handler="resolve"
)
endpoint_results_parser.add_argument(
"--headers", action="store_true", default=False, help="include headers found at each endpoint (default: False)"
)
endpoint_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
endpoint_results_parser.add_argument(
"--plain", action="store_true", default=False, help="display without status-codes/color (default: False)"
)
nmap_results_parser = view_subparsers.add_parser(
"nmap-scans", help="List all known nmap scan results; produced by nmap", conflict_handler="resolve"
)
nmap_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
nmap_results_parser.add_argument(
"--commandline", action="store_true", default=False, help="display command used to scan (default: False)"
)
searchsploit_results_parser = view_subparsers.add_parser(
"searchsploit-results",
help="List all known searchsploit hits; produced by searchsploit",
conflict_handler="resolve",
)
searchsploit_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
searchsploit_results_parser.add_argument(
"--fullpath", action="store_true", default=False, help="display full path to exploit PoC (default: False)"
)
port_results_parser = view_subparsers.add_parser(
"ports", help="List all known open ports; produced by masscan", conflict_handler="resolve"
)
port_results_parser.add_argument(
"--paged", action="store_true", default=False, help="display output page-by-page (default: False)"
)
# all options below this line will be updated with a choices option in recon-pipeline.py's
# add_dynamic_parser_arguments function. They're included here primarily to ease auto documentation of the commands
port_results_parser.add_argument("--host", help="filter results by host")
port_results_parser.add_argument("--port-number", help="filter results by port number")
endpoint_results_parser.add_argument("--status-code", help="filter results by status code")
endpoint_results_parser.add_argument("--host", help="filter results by host")
nmap_results_parser.add_argument("--host", help="filter results by host")
nmap_results_parser.add_argument("--nse-script", help="filter results by nse script type ran")
nmap_results_parser.add_argument("--port", help="filter results by port scanned")
nmap_results_parser.add_argument("--product", help="filter results by reported product")
technology_results_parser.add_argument("--host", help="filter results by host")
technology_results_parser.add_argument("--type", help="filter results by type")
technology_results_parser.add_argument("--product", help="filter results by product")
searchsploit_results_parser.add_argument("--host", help="filter results by host")
searchsploit_results_parser.add_argument("--type", help="filter results by exploit type")

64
pipeline/recon/targets.py Normal file
View File

@@ -0,0 +1,64 @@
from pathlib import Path
import luigi
from luigi.contrib.sqla import SQLAlchemyTarget
import pipeline.models.db_manager
from .config import defaults
from .helpers import is_ip_address
from ..models.target_model import Target
class TargetList(luigi.ExternalTask):
""" External task. ``TARGET_FILE`` is generated manually by the user from target's scope.
Args:
results_dir: specifies the directory on disk to which all Task results are written
db_location: specifies the path to the database used for storing results
"""
target_file = luigi.Parameter()
db_location = luigi.Parameter()
results_dir = luigi.Parameter(default=defaults.get("results-dir"))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
def output(self):
""" Returns the target output for this task. target_file.ips || target_file.domains
In this case, it expects a file to be present in the local filesystem.
By convention, TARGET_NAME should be something like tesla or some other
target identifier. The returned target output will either be target_file.ips
or target_file.domains, depending on what is found on the first line of the file.
Example: Given a TARGET_FILE of tesla where the first line is tesla.com; tesla.domains
is written to disk.
Returns:
luigi.local_target.LocalTarget
"""
# normally the call is self.output().touch(), however, that causes recursion here, so we grab the target now
# in order to call .touch() on it later and eventually return it
db_target = SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="target", update_id=self.task_id
)
with open(Path(self.target_file).expanduser().resolve()) as f:
for line in f.readlines():
line = line.strip()
if is_ip_address(line):
tgt = self.db_mgr.get_or_create(Target)
tgt = self.db_mgr.add_ipv4_or_v6_address_to_target(tgt, line)
else:
# domain name assumed if not ip address
tgt = self.db_mgr.get_or_create(Target, hostname=line, is_web=True)
self.db_mgr.add(tgt)
db_target.touch()
self.db_mgr.close()
return db_target

125
pipeline/recon/tool_definitions.py Normal file
View File

@@ -0,0 +1,125 @@
from pathlib import Path
from .config import tool_paths, defaults
# tool definitions for recon-pipeline's auto-installer
tools = {
"luigi-service": {
"installed": False,
"dependencies": None,
"commands": [
f"sudo cp {str(Path(__file__).parents[2] / 'luigid.service')} /lib/systemd/system/luigid.service",
f"sudo cp $(which luigid) /usr/local/bin",
"sudo systemctl daemon-reload",
"sudo systemctl start luigid.service",
"sudo systemctl enable luigid.service",
],
"shell": True,
},
"seclists": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
f"bash -c 'if [[ -d /usr/share/seclists ]]; then ln -s /usr/share/seclists {defaults.get('tools-dir')}/seclists; elif [[ -d {defaults.get('tools-dir')}/seclists ]] ; then cd {defaults.get('tools-dir')}/seclists && git fetch --all && git pull; else git clone https://github.com/danielmiessler/SecLists.git {defaults.get('tools-dir')}/seclists; fi'"
],
},
"searchsploit": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
f"bash -c 'if [[ -d /usr/share/exploitdb ]]; then ln -s /usr/share/exploitdb {defaults.get('tools-dir')}/exploitdb && sudo ln -s $(which searchsploit) {defaults.get('tools-dir')}/exploitdb/searchsploit; elif [[ -d {Path(tool_paths.get('searchsploit')).parent} ]]; then cd {Path(tool_paths.get('searchsploit')).parent} && git fetch --all && git pull; else git clone https://github.com/offensive-security/exploitdb.git {defaults.get('tools-dir')}/exploitdb; fi'",
f"bash -c 'if [[ -f {Path(tool_paths.get('searchsploit')).parent}/.searchsploit_rc ]]; then cp -n {Path(tool_paths.get('searchsploit')).parent}/.searchsploit_rc {Path.home().expanduser().resolve()}; fi'",
f"bash -c 'if [[ -f {Path.home().resolve()}/.searchsploit_rc ]]; then sed -i 's#/opt#{defaults.get('tools-dir')}#g' {Path.home().resolve()}/.searchsploit_rc; fi'",
],
},
"masscan": {
"installed": False,
"dependencies": None,
"commands": [
"git clone https://github.com/robertdavidgraham/masscan /tmp/masscan",
"make -s -j -C /tmp/masscan",
f"mv /tmp/masscan/bin/masscan {tool_paths.get('masscan')}",
"rm -rf /tmp/masscan",
f"sudo setcap CAP_NET_RAW+ep {tool_paths.get('masscan')}",
],
},
"amass": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get -u github.com/OWASP/Amass/v3/...",
f"cp ~/go/bin/amass {tool_paths.get('amass')}",
],
"shell": True,
"environ": {"GO111MODULE": "on"},
},
"aquatone": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
"mkdir /tmp/aquatone",
"wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip",
"bash -c 'if [[ ! $(which unzip) ]]; then sudo apt install -y zip; fi'",
"unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone",
f"mv /tmp/aquatone/aquatone {tool_paths.get('aquatone')}",
"rm -rf /tmp/aquatone",
"bash -c 'found=false; for loc in {/usr/bin/google-chrome,/usr/bin/google-chrome-beta,/usr/bin/google-chrome-unstable,/usr/bin/chromium-browser,/usr/bin/chromium}; do if [[ $(which $loc) ]]; then found=true; break; fi ; done; if [[ $found = false ]]; then sudo apt install -y chromium-browser ; fi'",
],
},
"gobuster": {
"installed": False,
"dependencies": ["go", "seclists"],
"commands": [
f"{tool_paths.get('go')} get github.com/OJ/gobuster",
f"(cd ~/go/src/github.com/OJ/gobuster && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"tko-subs": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/anshumanbh/tko-subs",
f"(cd ~/go/src/github.com/anshumanbh/tko-subs && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"subjack": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/haccer/subjack",
f"(cd ~/go/src/github.com/haccer/subjack && {tool_paths.get('go')} install)",
],
"shell": True,
},
"webanalyze": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/rverton/webanalyze/...",
f"(cd ~/go/src/github.com/rverton/webanalyze && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"recursive-gobuster": {
"installed": False,
"dependencies": ["gobuster", "seclists"],
"shell": True,
"commands": [
f"bash -c 'if [[ -d {Path(tool_paths.get('recursive-gobuster')).parent} ]] ; then cd {Path(tool_paths.get('recursive-gobuster')).parent} && git fetch --all && git pull; else git clone https://github.com/epi052/recursive-gobuster.git {Path(tool_paths.get('recursive-gobuster')).parent}; fi'"
],
},
"go": {
"installed": False,
"dependencies": None,
"commands": [
"wget -q https://dl.google.com/go/go1.13.7.linux-amd64.tar.gz -O /tmp/go.tar.gz",
"sudo tar -C /usr/local -xvf /tmp/go.tar.gz",
f'bash -c \'if [[ ! $(echo "${{PATH}}" | grep $(dirname {tool_paths.get("go")})) ]]; then echo "PATH=${{PATH}}:/usr/local/go/bin" >> ~/.bashrc; fi\'',
],
},
}

5
pipeline/recon/web/__init__.py Normal file
View File

@@ -0,0 +1,5 @@
from .aquatone import AquatoneScan
from .gobuster import GobusterScan
from .targets import GatherWebTargets
from .webanalyze import WebanalyzeScan
from .subdomain_takeover import SubjackScan, TKOSubsScan

274
pipeline/recon/web/aquatone.py Normal file
View File

@@ -0,0 +1,274 @@
import json
import logging
import subprocess
from pathlib import Path
from urllib.parse import urlparse
import luigi
from luigi.util import inherits
from luigi.contrib.sqla import SQLAlchemyTarget
from .targets import GatherWebTargets
from ..config import tool_paths, defaults
import pipeline.models.db_manager
from ...models.port_model import Port
from ...models.header_model import Header
from ...models.endpoint_model import Endpoint
from ...models.screenshot_model import Screenshot
@inherits(GatherWebTargets)
class AquatoneScan(luigi.Task):
""" Screenshot all web targets and generate HTML report.
Install:
.. code-block:: console
mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone
Basic Example:
``aquatone`` commands are structured like the example below.
``cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20``
Luigi Example:
.. code-block:: python
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
Args:
threads: number of threads for parallel aquatone command execution
scan_timeout: timeout in miliseconds for aquatone port scans
db_location: specifies the path to the database used for storing results *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
scan_timeout = luigi.Parameter(default=defaults.get("aquatone-scan-timeout", ""))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = Path(self.results_dir) / "aquatone-results"
def requires(self):
""" AquatoneScan depends on GatherWebTargets to run.
GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:
luigi.Task - GatherWebTargets
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
"db_location": self.db_location,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Returns:
luigi.contrib.sqla.SQLAlchemyTarget
"""
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="screenshot", update_id=self.task_id
)
def _get_similar_pages(self, url, results):
# populate similar pages if any exist
similar_pages = None
for cluster_id, cluster in results.get("pageSimilarityClusters").items():
if url not in cluster:
continue
similar_pages = list()
for similar_url in cluster:
if similar_url == url:
continue
similar_pages.append(self.db_mgr.get_or_create(Screenshot, url=similar_url))
return similar_pages
def parse_results(self):
""" Read in aquatone's .json file and update the associated Target record """
""" Example data
"https://email.assetinventory.bugcrowd.com:8443/": {
"uuid": "679b0dc7-02ea-483f-9e0a-3a5e6cdea4b6",
"url": "https://email.assetinventory.bugcrowd.com:8443/",
"hostname": "email.assetinventory.bugcrowd.com",
"addrs": [
"104.20.60.51",
"104.20.61.51",
"2606:4700:10::6814:3d33",
"2606:4700:10::6814:3c33"
],
"status": "403 Forbidden",
"pageTitle": "403 Forbidden",
"headersPath": "headers/https__email_assetinventory_bugcrowd_com__8443__42099b4af021e53f.txt",
"bodyPath": "html/https__email_assetinventory_bugcrowd_com__8443__42099b4af021e53f.html",
"screenshotPath": "screenshots/https__email_assetinventory_bugcrowd_com__8443__42099b4af021e53f.png",
"hasScreenshot": true,
"headers": [
{
"name": "Cf-Ray",
"value": "55d396727981d25a-DFW",
"decreasesSecurity": false,
"increasesSecurity": false
},
...
],
"tags": [
{
"text": "CloudFlare",
"type": "info",
"link": "http://www.cloudflare.com",
"hash": "9ea92fc7dce5e740ccc8e64d8f9e3336a96efc2a"
}
],
"notes": null
},
...
"pageSimilarityClusters": {
"11905c72-fd18-43de-9133-99ba2a480e2b": [
"http://52.53.92.161/",
"https://staging.bitdiscovery.com/",
"https://52.53.92.161/",
...
],
"139fc2c4-0faa-4ae3-a6e4-0a1abe2418fa": [
"https://104.20.60.51:8443/",
"https://email.assetinventory.bugcrowd.com:8443/",
...
],
"""
try:
with open(self.results_subfolder / "aquatone_session.json") as f:
# results.keys -> dict_keys(['version', 'stats', 'pages', 'pageSimilarityClusters'])
results = json.load(f)
except FileNotFoundError as e:
logging.error(e)
return
for page, page_dict in results.get("pages").items():
headers = list()
url = page_dict.get("url") # one url to one screenshot, unique key
# build out the endpoint's data to include headers, this has value whether or not there's a screenshot
endpoint = self.db_mgr.get_or_create(Endpoint, url=url)
if not endpoint.status_code:
status = page_dict.get("status").split(maxsplit=1)
if len(status) > 1:
endpoint.status_code, _ = status
else:
endpoint.status_code = status[0]
for header_dict in page_dict.get("headers"):
header = self.db_mgr.get_or_create(Header, name=header_dict.get("name"), value=header_dict.get("value"))
if endpoint not in header.endpoints:
header.endpoints.append(endpoint)
headers.append(header)
endpoint.headers = headers
parsed_url = urlparse(url)
ip_or_hostname = parsed_url.hostname
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(ip_or_hostname)
endpoint.target = tgt
if not page_dict.get("hasScreenshot"):
# if there isn't a screenshot, save the endpoint data and move along
self.db_mgr.add(endpoint)
# This causes an integrity error on insertion due to the task_id being the same for two
# different target tables. Could subclass SQLAlchemyTarget and set the unique-ness to be the
# combination of update_id + target_table. The question is, does it matter?
# TODO: assess above and act
# SQLAlchemyTarget(
# connection_string=self.db_mgr.connection_string, target_table="endpoint", update_id=self.task_id
# ).touch()
continue
# build out screenshot data
port = parsed_url.port if parsed_url.port else 80
port = self.db_mgr.get_or_create(Port, protocol="tcp", port_number=port)
image = (self.results_subfolder / page_dict.get("screenshotPath")).read_bytes()
screenshot = self.db_mgr.get_or_create(Screenshot, url=url)
screenshot.port = port
screenshot.endpoint = endpoint
screenshot.target = screenshot.endpoint.target
screenshot.image = image
similar_pages = self._get_similar_pages(url, results)
if similar_pages is not None:
screenshot.similar_pages = similar_pages
self.db_mgr.add(screenshot)
self.output().touch()
self.db_mgr.close()
def run(self):
""" Defines the options/arguments sent to aquatone after processing.
cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
self.results_subfolder.mkdir(parents=True, exist_ok=True)
command = [
tool_paths.get("aquatone"),
"-scan-timeout",
self.scan_timeout,
"-threads",
self.threads,
"-silent",
"-out",
self.results_subfolder,
]
aquatone_input_file = self.results_subfolder / "input-from-webtargets"
with open(aquatone_input_file, "w") as f:
for target in self.db_mgr.get_all_web_targets():
f.write(f"{target}\n")
with open(aquatone_input_file) as target_list:
subprocess.run(command, stdin=target_list)
aquatone_input_file.unlink()
self.parse_results()

73
recon/web/gobuster.py → pipeline/recon/web/gobuster.py
View File

@@ -1,15 +1,19 @@
import os
import logging
import ipaddress
import subprocess
from pathlib import Path
from urllib.parse import urlparse
from concurrent.futures import ThreadPoolExecutor
import luigi
from luigi.util import inherits
from luigi.contrib.sqla import SQLAlchemyTarget
from recon.config import tool_paths, defaults
from recon.web.targets import GatherWebTargets
import pipeline.models.db_manager
from .targets import GatherWebTargets
from ..config import tool_paths, defaults
from ...models.endpoint_model import Endpoint
from ..helpers import get_ip_address_version, is_ip_address
@inherits(GatherWebTargets)
@@ -39,6 +43,7 @@ class GobusterScan(luigi.Task):
recursive: whether or not to recursively gobust the target (may produce a LOT of traffic... quickly)
proxy: protocol://ip:port proxy specification for gobuster
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
db_location: specifies the path to the database used for storing results *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
@@ -47,11 +52,16 @@ class GobusterScan(luigi.Task):
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
proxy = luigi.Parameter(default=defaults.get("proxy", ""))
threads = luigi.Parameter(default=defaults.get("threads", ""))
wordlist = luigi.Parameter(default=defaults.get("gobuster-wordlist", ""))
extensions = luigi.Parameter(default=defaults.get("gobuster-extensions", ""))
recursive = luigi.BoolParameter(default=False)
proxy = luigi.Parameter(default=defaults.get("proxy"))
threads = luigi.Parameter(default=defaults.get("threads"))
wordlist = luigi.Parameter(default=defaults.get("gobuster-wordlist"))
extensions = luigi.Parameter(default=defaults.get("gobuster-extensions"))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = Path(self.results_dir) / "gobuster-results"
def requires(self):
""" GobusterScan depends on GatherWebTargets to run.
@@ -70,6 +80,7 @@ class GobusterScan(luigi.Task):
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
"db_location": self.db_location,
}
return GatherWebTargets(**args)
@@ -84,9 +95,30 @@ class GobusterScan(luigi.Task):
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "gobuster-results"
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="endpoint", update_id=self.task_id
)
return luigi.LocalTarget(results_subfolder.resolve())
def parse_results(self):
""" Reads in each individual gobuster file and adds each line to the database as an Endpoint """
for file in self.results_subfolder.iterdir():
tgt = None
for i, line in enumerate(file.read_text().splitlines()):
url, status = line.split(maxsplit=1) # http://somewhere/path (Status:200)
if i == 0:
# parse first entry to determine ip address -> target relationship
parsed_url = urlparse(url)
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(parsed_url.hostname)
if tgt is not None:
status_code = status.split(maxsplit=1)[1]
ep = self.db_mgr.get_or_create(Endpoint, url=url, status_code=status_code.replace(")", ""))
if ep not in tgt.endpoints:
tgt.endpoints.append(ep)
self.db_mgr.add(tgt)
self.output().touch()
def run(self):
""" Defines the options/arguments sent to gobuster after processing.
@@ -96,25 +128,18 @@ class GobusterScan(luigi.Task):
"""
try:
self.threads = abs(int(self.threads))
except TypeError:
except (TypeError, ValueError):
return logging.error("The value supplied to --threads must be a non-negative integer.")
commands = list()
with self.input().open() as f:
for target in f:
target = target.strip()
try:
if isinstance(ipaddress.ip_address(target), ipaddress.IPv6Address): # ipv6
for target in self.db_mgr.get_all_web_targets():
if is_ip_address(target) and get_ip_address_version(target) == "6":
target = f"[{target}]"
except ValueError:
# domain names raise ValueErrors, just assume we have a domain and keep on keepin on
pass
for url_scheme in ("https://", "http://"):
if self.recursive:
command = [tool_paths.get("recursive-gobuster"), "-w", self.wordlist, f"{url_scheme}{target}"]
command = [tool_paths.get("recursive-gobuster"), "-s", "-w", self.wordlist, f"{url_scheme}{target}"]
else:
command = [
tool_paths.get("gobuster"),
@@ -127,7 +152,7 @@ class GobusterScan(luigi.Task):
"-w",
self.wordlist,
"-o",
Path(self.output().path).joinpath(
self.results_subfolder.joinpath(
f"gobuster.{url_scheme.replace('//', '_').replace(':', '')}{target}.txt"
),
]
@@ -140,15 +165,17 @@ class GobusterScan(luigi.Task):
commands.append(command)
Path(self.output().path).mkdir(parents=True, exist_ok=True)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
if self.recursive:
# workaround for recursive gobuster not accepting output directory
cwd = Path().cwd()
os.chdir(self.output().path)
os.chdir(self.results_subfolder)
with ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.map(subprocess.run, commands)
if self.recursive:
os.chdir(str(cwd))
self.parse_results()

151
recon/web/subdomain_takeover.py → pipeline/recon/web/subdomain_takeover.py
View File

@@ -1,15 +1,19 @@
import re
import csv
import subprocess
from pathlib import Path
import luigi
from luigi.util import inherits
from luigi.contrib.external_program import ExternalProgramTask
from luigi.contrib.sqla import SQLAlchemyTarget
from recon.config import tool_paths, defaults
from recon.web.targets import GatherWebTargets
import pipeline.models.db_manager
from .targets import GatherWebTargets
from ..config import tool_paths, defaults
@inherits(GatherWebTargets)
class TKOSubsScan(ExternalProgramTask):
class TKOSubsScan(luigi.Task):
""" Use ``tko-subs`` to scan for potential subdomain takeovers.
Install:
@@ -31,6 +35,7 @@ class TKOSubsScan(ExternalProgramTask):
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.subdomain_takeover TKOSubsScan --target-file tesla --top-ports 1000 --interface eth0
Args:
db_location: specifies the path to the database used for storing results *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
@@ -40,6 +45,12 @@ class TKOSubsScan(ExternalProgramTask):
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "tkosubs-results").expanduser().resolve()
self.output_file = self.results_subfolder / "tkosubs.csv"
def requires(self):
""" TKOSubsScan depends on GatherWebTargets to run.
@@ -57,43 +68,71 @@ class TKOSubsScan(ExternalProgramTask):
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
"db_location": self.db_location,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output file is tkosubs.TARGET_FILE.csv.
Returns:
luigi.local_target.LocalTarget
luigi.contrib.sqla.SQLAlchemyTarget
"""
results_subfolder = Path(self.results_dir) / "tkosubs-results"
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="target", update_id=self.task_id
)
new_path = results_subfolder / "tkosubs.csv"
def parse_results(self):
""" Reads in the tkosubs .csv file and updates the associated Target record. """
with open(self.output_file, newline="") as f:
reader = csv.reader(f)
return luigi.LocalTarget(new_path.resolve())
next(reader, None) # skip the headers
def program_args(self):
for row in reader:
domain = row[0]
is_vulnerable = row[3]
if "true" in is_vulnerable.lower():
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(domain)
tgt.vuln_to_sub_takeover = True
self.db_mgr.add(tgt)
self.output().touch()
self.db_mgr.close()
# make sure task doesn't fail due to no results, it's the last in its chain, so doesn't
# affect any downstream tasks
self.output().touch()
def run(self):
""" Defines the options/arguments sent to tko-subs after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
domains = self.db_mgr.get_all_hostnames()
if not domains:
return
command = [
tool_paths.get("tko-subs"),
f"-domains={self.input().path}",
f"-domain={','.join(domains)}",
f"-data={tool_paths.get('tko-subs-dir')}/providers-data.csv",
f"-output={self.output().path}",
f"-output={self.output_file}",
]
return command
subprocess.run(command)
self.parse_results()
@inherits(GatherWebTargets)
class SubjackScan(ExternalProgramTask):
class SubjackScan(luigi.Task):
""" Use ``subjack`` to scan for potential subdomain takeovers.
Install:
@@ -116,6 +155,7 @@ class SubjackScan(ExternalProgramTask):
Args:
threads: number of threads for parallel subjack command execution
db_location: specifies the path to the database used for storing results *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
@@ -125,7 +165,13 @@ class SubjackScan(ExternalProgramTask):
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
threads = luigi.Parameter(default=defaults.get("threads"))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = (Path(self.results_dir) / "subjack-results").expanduser().resolve()
self.output_file = self.results_subfolder / "subjack.txt"
def requires(self):
""" SubjackScan depends on GatherWebTargets to run.
@@ -144,46 +190,95 @@ class SubjackScan(ExternalProgramTask):
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
"db_location": self.db_location,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output file is subjack.TARGET_FILE.txt.
Returns:
luigi.local_target.LocalTarget
luigi.contrib.sqla.SQLAlchemyTarget
"""
results_subfolder = Path(self.results_dir) / "subjack-results"
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="target", update_id=self.task_id
)
new_path = results_subfolder / "subjack.txt"
def parse_results(self):
""" Reads in the subjack's subjack.txt file and updates the associated Target record. """
return luigi.LocalTarget(new_path.resolve())
with open(self.output_file) as f:
""" example data
def program_args(self):
[Not Vulnerable] 52.53.92.161:443
[Not Vulnerable] 13.57.162.100
[Not Vulnerable] 2606:4700:10::6814:3d33
[Not Vulnerable] assetinventory.bugcrowd.com
"""
for line in f:
match = re.match(r"\[(?P<vuln_status>.+)] (?P<ip_or_hostname>.*)", line)
if not match:
continue
if match.group("vuln_status") == "Not Vulnerable":
continue
ip_or_host = match.group("ip_or_hostname")
if ip_or_host.count(":") == 1: # ip or host/port
ip_or_host, port = ip_or_host.split(":", maxsplit=1)
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(ip_or_host)
tgt.vuln_to_sub_takeover = True
self.db_mgr.add(tgt)
self.output().touch()
self.db_mgr.close()
# make sure task doesn't fail due to no results, it's the last in its chain, so doesn't
# affect any downstream tasks
self.output().touch()
def run(self):
""" Defines the options/arguments sent to subjack after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
hostnames = self.db_mgr.get_all_hostnames()
if not hostnames:
return
subjack_input_file = self.results_subfolder / "input-from-webtargets"
with open(subjack_input_file, "w") as f:
for hostname in hostnames:
f.write(f"{hostname}\n")
command = [
tool_paths.get("subjack"),
"-w",
self.input().path,
str(subjack_input_file),
"-t",
self.threads,
"-a",
"-timeout",
"30",
"-o",
self.output().path,
str(self.output_file),
"-v",
"-ssl",
"-c",
tool_paths.get("subjack-fingerprints"),
]
return command
subprocess.run(command)
self.parse_results()
subjack_input_file.unlink()

84
pipeline/recon/web/targets.py Normal file
View File

@@ -0,0 +1,84 @@
import luigi
from luigi.util import inherits
from luigi.contrib.sqla import SQLAlchemyTarget
import pipeline.models.db_manager
from ..config import web_ports
from ..amass import ParseAmassOutput
from ..masscan import ParseMasscanOutput
@inherits(ParseMasscanOutput)
class GatherWebTargets(luigi.Task):
""" Gather all subdomains as well as any ip addresses known to have a configured web port open.
Args:
db_location: specifies the path to the database used for storing results *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
def requires(self):
""" GatherWebTargets depends on ParseMasscanOutput and ParseAmassOutput to run.
ParseMasscanOutput expects rate, target_file, interface, and either ports or top_ports as parameters.
ParseAmassOutput accepts exempt_list and expects target_file
Returns:
dict(str: ParseMasscanOutput, str: ParseAmassOutput)
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"db_location": self.db_location,
}
return {
"masscan-output": ParseMasscanOutput(**args),
"amass-output": ParseAmassOutput(
exempt_list=self.exempt_list,
target_file=self.target_file,
results_dir=self.results_dir,
db_location=self.db_location,
),
}
def output(self):
""" Returns the target output for this task.
Returns:
luigi.contrib.sqla.SQLAlchemyTarget
"""
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="target", update_id=self.task_id
)
def run(self):
""" Gather all potential web targets and tag them as web in the database. """
for target in self.db_mgr.get_all_targets():
ports = self.db_mgr.get_ports_by_ip_or_host_and_protocol(target, "tcp")
if any(port in web_ports for port in ports):
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(target)
tgt.is_web = True
self.db_mgr.add(tgt)
self.output().touch()
# in the event that there are no web ports for any target, we still want to be able to mark the
# task complete successfully. we accomplish this by calling .touch() even though a database entry
# may not have happened
self.output().touch()
self.db_mgr.close()

172
pipeline/recon/web/webanalyze.py Normal file
View File

@@ -0,0 +1,172 @@
import os
import csv
import logging
import subprocess
from pathlib import Path
from urllib.parse import urlparse
from concurrent.futures import ThreadPoolExecutor
import luigi
from luigi.util import inherits
from luigi.contrib.sqla import SQLAlchemyTarget
import pipeline.models.db_manager
from .targets import GatherWebTargets
from ..config import tool_paths, defaults
from ...models.technology_model import Technology
from ..helpers import get_ip_address_version, is_ip_address
@inherits(GatherWebTargets)
class WebanalyzeScan(luigi.Task):
""" Use webanalyze to determine the technology stack on the given target(s).
Install:
.. code-block:: console
go get -u github.com/rverton/webanalyze
# loads new apps.json file from wappalyzer project
webanalyze -update
Basic Example:
.. code-block:: console
webanalyze -host www.tesla.com -output json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel webanalyze command execution
db_location: specifies the path to the database used for storing results *Required by upstream Task*
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional for upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads"))
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.db_mgr = pipeline.models.db_manager.DBManager(db_location=self.db_location)
self.results_subfolder = Path(self.results_dir) / "webanalyze-results"
def requires(self):
""" WebanalyzeScan depends on GatherWebTargets to run.
GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:
luigi.Task - GatherWebTargets
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
"db_location": self.db_location,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Returns:
luigi.contrib.sqla.SQLAlchemyTarget
"""
return SQLAlchemyTarget(
connection_string=self.db_mgr.connection_string, target_table="technology", update_id=self.task_id
)
def parse_results(self):
""" Reads in the webanalyze's .csv files and updates the associated Target record. """
for entry in self.results_subfolder.glob("webanalyze*.csv"):
""" example data
http://13.57.162.100,Font scripts,Google Font API,
http://13.57.162.100,"Web servers,Reverse proxies",Nginx,1.16.1
http://13.57.162.100,Font scripts,Font Awesome,
"""
with open(entry, newline="") as f:
reader = csv.reader(f)
# skip the empty line at the start; webanalyze places an empty line at the top of the file
# need to skip that. remove this line if the files have no empty lines at the top
next(reader, None)
next(reader, None) # skip the headers; keep this one forever and always
tgt = None
for row in reader:
# each row in a file is a technology specific to that target
host, category, app, version = row
parsed_url = urlparse(host)
text = f"{app}-{version}" if version else app
technology = self.db_mgr.get_or_create(Technology, type=category, text=text)
if tgt is None:
# should only hit the first line of each file
tgt = self.db_mgr.get_or_create_target_by_ip_or_hostname(parsed_url.hostname)
tgt.technologies.append(technology)
if tgt is not None:
self.db_mgr.add(tgt)
self.output().touch()
self.db_mgr.close()
def _wrapped_subprocess(self, cmd):
with open(f"webanalyze-{cmd[2].replace('//', '_').replace(':', '')}.csv", "wb") as f:
subprocess.run(cmd, stdout=f)
def run(self):
""" Defines the options/arguments sent to webanalyze after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
try:
self.threads = abs(int(self.threads))
except (TypeError, ValueError):
return logging.error("The value supplied to --threads must be a non-negative integer.")
commands = list()
for target in self.db_mgr.get_all_web_targets():
if is_ip_address(target) and get_ip_address_version(target) == "6":
target = f"[{target}]"
for url_scheme in ("https://", "http://"):
command = [tool_paths.get("webanalyze"), "-host", f"{url_scheme}{target}", "-output", "csv"]
commands.append(command)
self.results_subfolder.mkdir(parents=True, exist_ok=True)
cwd = Path().cwd()
os.chdir(self.results_subfolder)
if not Path("apps.json").exists():
subprocess.run(f"{tool_paths.get('webanalyze')} -update".split())
with ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.map(self._wrapped_subprocess, commands)
os.chdir(str(cwd))
self.parse_results()

16
recon/wrappers.py → pipeline/recon/wrappers.py
View File

@@ -1,15 +1,14 @@
import luigi
from luigi.util import inherits
from recon.nmap import SearchsploitScan
from recon.web.aquatone import AquatoneScan
from recon.web.corscanner import CORScannerScan
from recon.web.subdomain_takeover import TKOSubsScan, SubjackScan
from recon.web.gobuster import GobusterScan
from recon.web.webanalyze import WebanalyzeScan
from .nmap import SearchsploitScan
from .web import AquatoneScan
from .web import GobusterScan
from .web import WebanalyzeScan
from .web import TKOSubsScan, SubjackScan
@inherits(SearchsploitScan, AquatoneScan, TKOSubsScan, SubjackScan, CORScannerScan, GobusterScan, WebanalyzeScan)
@inherits(SearchsploitScan, AquatoneScan, TKOSubsScan, SubjackScan, GobusterScan, WebanalyzeScan)
class FullScan(luigi.WrapperTask):
""" Wraps multiple scan types in order to run tasks on the same hierarchical level at the same time.
@@ -46,6 +45,7 @@ class FullScan(luigi.WrapperTask):
"wordlist": self.wordlist,
"extensions": self.extensions,
"recursive": self.recursive,
"db_location": self.db_location,
}
yield GobusterScan(**args)
@@ -63,7 +63,6 @@ class FullScan(luigi.WrapperTask):
yield SubjackScan(**args)
yield SearchsploitScan(**args)
yield CORScannerScan(**args)
yield WebanalyzeScan(**args)
del args["threads"]
@@ -105,6 +104,7 @@ class HTBScan(luigi.WrapperTask):
"exempt_list": self.exempt_list,
"threads": self.threads,
"proxy": self.proxy,
"db_location": self.db_location,
"wordlist": self.wordlist,
"extensions": self.extensions,
"recursive": self.recursive,

3
pytest.ini Normal file
View File

@@ -0,0 +1,3 @@
[pytest]
filterwarnings =
ignore::DeprecationWarning:luigi.*:

301
recon-pipeline.py
View File

@@ -1,301 +0,0 @@
#!/usr/bin/env python
# stdlib imports
import os
import sys
import shlex
import pickle
import selectors
import threading
import subprocess
import webbrowser
from pathlib import Path
# fix up the PYTHONPATH so we can simply execute the shell from wherever in the filesystem
os.environ["PYTHONPATH"] = f"{os.environ.get('PYTHONPATH')}:{str(Path(__file__).parent.resolve())}"
# suppress "You should consider upgrading via the 'pip install --upgrade pip' command." warning
os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
# in case we need pipenv, add its default --user installed directory to the PATH
sys.path.append(str(Path.home() / ".local" / "bin"))
# third party imports
import cmd2 # noqa: E402
from cmd2.ansi import style # noqa: E402
# project's module imports
from recon import get_scans, tools, scan_parser, install_parser, status_parser # noqa: F401,E402
from recon.config import defaults # noqa: F401,E402
# select loop, handles async stdout/stderr processing of subprocesses
selector = selectors.DefaultSelector()
class SelectorThread(threading.Thread):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._stop_event = threading.Event()
def stop(self):
""" Helper to set the SelectorThread's Event and cleanup the selector's fds """
self._stop_event.set()
# close any fds that were registered and still haven't been unregistered
for key in selector.get_map():
selector.get_key(key).fileobj.close()
def stopped(self):
""" Helper to determine whether the SelectorThread's Event is set or not. """
return self._stop_event.is_set()
def run(self):
""" Run thread that executes a select loop; handles async stdout/stderr processing of subprocesses. """
while not self.stopped():
for k, mask in selector.select():
callback = k.data
callback(k.fileobj)
class ReconShell(cmd2.Cmd):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.sentry = False
self.prompt = "recon-pipeline> "
self.selectorloop = None
self.continue_install = True
Path(defaults.get("tools-dir")).mkdir(parents=True, exist_ok=True)
# register hooks to handle selector loop start and cleanup
self.register_preloop_hook(self._preloop_hook)
self.register_postloop_hook(self._postloop_hook)
def _preloop_hook(self) -> None:
""" Hook function that runs prior to the cmdloop function starting; starts the selector loop. """
self.selectorloop = SelectorThread(daemon=True)
self.selectorloop.start()
def _postloop_hook(self) -> None:
""" Hook function that runs after the cmdloop function stops; stops the selector loop. """
if self.selectorloop.is_alive():
self.selectorloop.stop()
selector.close()
def _install_error_reporter(self, stderr):
""" Helper to print errors that crop up during any tool installation commands. """
output = stderr.readline()
if not output:
return
output = output.decode().strip()
self.async_alert(style(f"[!] {output}", fg="bright_red"))
def _luigi_pretty_printer(self, stderr):
""" Helper to clean up the VERY verbose luigi log messages that are normally spewed to the terminal. """
output = stderr.readline()
if not output:
return
output = output.decode()
if "===== Luigi Execution Summary =====" in output:
# header that begins the summary of all luigi tasks that have executed/failed
self.async_alert("")
self.sentry = True
# block below used for printing status messages
if self.sentry:
# only set once the Luigi Execution Summary is seen
self.async_alert(style(output.strip(), fg="bright_blue"))
elif output.startswith("INFO: Informed") and output.strip().endswith("PENDING"):
# luigi Task has been queued for execution
words = output.split()
self.async_alert(style(f"[-] {words[5].split('_')[0]} queued", fg="bright_white"))
elif output.startswith("INFO: ") and "running" in output:
# luigi Task is currently running
words = output.split()
# output looks similar to , pid=3938074) running MasscanScan(
# want to grab the index of the luigi task running and use it to find the name of the scan (i.e. MassScan)
scantypeidx = words.index("running") + 1
scantype = words[scantypeidx].split("(", 1)[0]
self.async_alert(style(f"[*] {scantype} running...", fg="bright_yellow"))
elif output.startswith("INFO: Informed") and output.strip().endswith("DONE"):
# luigi Task has completed
words = output.split()
self.async_alert(style(f"[+] {words[5].split('_')[0]} complete!", fg="bright_green"))
@cmd2.with_argparser(scan_parser)
def do_scan(self, args):
""" Scan something.
Possible scans include
AmassScan CORScannerScan GobusterScan SearchsploitScan
ThreadedNmapScan WebanalyzeScan AquatoneScan FullScan
MasscanScan SubjackScan TKOSubsScan HTBScan
"""
self.async_alert(
style(
"If anything goes wrong, rerun your command with --verbose to enable debug statements.",
fg="cyan",
dim=True,
)
)
# get_scans() returns mapping of {classname: [modulename, ...]} in the recon module
# each classname corresponds to a potential recon-pipeline command, i.e. AmassScan, CORScannerScan ...
scans = get_scans()
# command is a list that will end up looking something like what's below
# luigi --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
command = ["luigi", "--module", scans.get(args.scantype)[0]]
command.extend(args.__statement__.arg_list)
if args.sausage:
# sausage is not a luigi option, need to remove it
# name for the option came from @AlphaRingo
command.pop(command.index("--sausage"))
webbrowser.open("127.0.0.1:8082") # hard-coded here, can specify different with the status command
if args.verbose:
# verbose is not a luigi option, need to remove it
command.pop(command.index("--verbose"))
subprocess.run(command)
else:
# suppress luigi messages in favor of less verbose/cleaner output
proc = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
# add stderr to the selector loop for processing when there's something to read from the fd
selector.register(proc.stderr, selectors.EVENT_READ, self._luigi_pretty_printer)
@cmd2.with_argparser(install_parser)
def do_install(self, args):
""" Install any/all of the libraries/tools necessary to make the recon-pipeline function. """
# imported tools variable is in global scope, and we reassign over it later
global tools
# create .cache dir in the home directory, on the off chance it doesn't exist
cachedir = Path.home() / ".cache/"
cachedir.mkdir(parents=True, exist_ok=True)
persistent_tool_dict = cachedir / ".tool-dict.pkl"
if args.tool == "all":
# show all tools have been queued for installation
[
self.async_alert(style(f"[-] {x} queued", fg="bright_white"))
for x in tools.keys()
if not tools.get(x).get("installed")
]
for tool in tools.keys():
self.do_install(tool)
return
if persistent_tool_dict.exists():
tools = pickle.loads(persistent_tool_dict.read_bytes())
if tools.get(args.tool).get("dependencies"):
# get all of the requested tools dependencies
for dependency in tools.get(args.tool).get("dependencies"):
if tools.get(dependency).get("installed"):
# already installed, skip it
continue
self.async_alert(
style(f"[!] {args.tool} has an unmet dependency; installing {dependency}", fg="yellow", bold=True)
)
# install the dependency before continuing with installation
self.do_install(dependency)
if tools.get(args.tool).get("installed"):
return self.async_alert(style(f"[!] {args.tool} is already installed.", fg="yellow"))
else:
# list of return values from commands run during each tool installation
# used to determine whether the tool installed correctly or not
retvals = list()
self.async_alert(style(f"[*] Installing {args.tool}...", fg="bright_yellow"))
addl_env_vars = tools.get(args.tool).get("environ")
if addl_env_vars is not None:
addl_env_vars.update(dict(os.environ))
for command in tools.get(args.tool).get("commands"):
# run all commands required to install the tool
# print each command being run
self.async_alert(style(f"[=] {command}", fg="cyan"))
if tools.get(args.tool).get("shell"):
# go tools use subshells (cmd1 && cmd2 && cmd3 ...) during install, so need shell=True
proc = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=addl_env_vars
)
else:
# "normal" command, split up the string as usual and run it
proc = subprocess.Popen(
shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=addl_env_vars
)
out, err = proc.communicate()
if err:
self.poutput(style(f"[!] {err.decode().strip()}", fg="bright_red"))
retvals.append(proc.returncode)
if all(x == 0 for x in retvals):
# all return values in retvals are 0, i.e. all exec'd successfully; tool has been installed
self.async_alert(style(f"[+] {args.tool} installed!", fg="bright_green"))
tools[args.tool]["installed"] = True
else:
# unsuccessful tool install
tools[args.tool]["installed"] = False
self.async_alert(
style(
f"[!!] one (or more) of {args.tool}'s commands failed and may have not installed properly; check output from the offending command above...",
fg="bright_red",
bold=True,
)
)
# store any tool installs/failures (back) to disk
pickle.dump(tools, persistent_tool_dict.open("wb"))
@cmd2.with_argparser(status_parser)
def do_status(self, args):
""" Open a web browser to Luigi's central scheduler's visualization site """
webbrowser.open(f"{args.host}:{args.port}")
if __name__ == "__main__":
rs = ReconShell(persistent_history_file="~/.reconshell_history", persistent_history_length=10000)
sys.exit(rs.cmdloop())

241
recon/__init__.py
View File

@@ -1,241 +0,0 @@
# flake8: noqa E231
import sys
import socket
import inspect
import pkgutil
import importlib
from pathlib import Path
from collections import defaultdict
import cmd2
import recon
from recon.config import tool_paths, defaults
# tool definitions for recon-pipeline's auto-installer
tools = {
"luigi-service": {
"installed": False,
"dependencies": ["luigi"],
"commands": [
f"sudo cp {str(Path(__file__).parent.parent / 'luigid.service')} /lib/systemd/system/luigid.service",
f"sudo cp $(which luigid) /usr/local/bin",
"sudo systemctl daemon-reload",
"sudo systemctl start luigid.service",
"sudo systemctl enable luigid.service",
],
"shell": True,
},
"luigi": {"installed": False, "dependencies": None, "commands": ["pip install luigi"]},
"seclists": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
f"bash -c 'if [[ -d {defaults.get('tools-dir')}/seclists ]] ; then cd {defaults.get('tools-dir')}/seclists && git fetch --all && git pull; else git clone https://github.com/danielmiessler/SecLists.git {defaults.get('tools-dir')}/seclists; fi'"
],
},
"searchsploit": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
f"bash -c 'if [[ -d {Path(tool_paths.get('searchsploit')).parent} ]] ; then cd {Path(tool_paths.get('searchsploit')).parent} && git fetch --all && git pull; else git clone https://github.com/offensive-security/exploitdb.git {defaults.get('tools-dir')}/exploitdb; fi'",
f"cp -n {Path(tool_paths.get('searchsploit')).parent}/.searchsploit_rc {Path.home().resolve()}",
f"sed -i 's#/opt#{defaults.get('tools-dir')}#g' {Path.home().resolve()}/.searchsploit_rc",
],
},
"masscan": {
"installed": False,
"dependencies": None,
"commands": [
"git clone https://github.com/robertdavidgraham/masscan /tmp/masscan",
"make -s -j -C /tmp/masscan",
f"mv /tmp/masscan/bin/masscan {tool_paths.get('masscan')}",
"rm -rf /tmp/masscan",
f"sudo setcap CAP_NET_RAW+ep {tool_paths.get('masscan')}",
],
},
"amass": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get -u github.com/OWASP/Amass/v3/...",
f"cp ~/go/bin/amass {tool_paths.get('amass')}",
],
"shell": True,
"environ": {"GO111MODULE": "on"},
},
"aquatone": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
"mkdir /tmp/aquatone",
"wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip",
"unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone",
f"mv /tmp/aquatone/aquatone {tool_paths.get('aquatone')}",
"rm -rf /tmp/aquatone",
],
},
"corscanner": {
"installed": False,
"dependencies": None,
"shell": True,
"commands": [
f"bash -c 'if [[ -d {Path(tool_paths.get('CORScanner')).parent} ]] ; then cd {Path(tool_paths.get('CORScanner')).parent} && git fetch --all && git pull; else git clone https://github.com/chenjj/CORScanner.git {Path(tool_paths.get('CORScanner')).parent}; fi'",
f"pip install -r {Path(tool_paths.get('CORScanner')).parent / 'requirements.txt'}",
"pip install future",
],
},
"gobuster": {
"installed": False,
"dependencies": ["go", "seclists"],
"commands": [
f"{tool_paths.get('go')} get github.com/OJ/gobuster",
f"(cd ~/go/src/github.com/OJ/gobuster && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"tko-subs": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/anshumanbh/tko-subs",
f"(cd ~/go/src/github.com/anshumanbh/tko-subs && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"subjack": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/haccer/subjack",
f"(cd ~/go/src/github.com/haccer/subjack && {tool_paths.get('go')} install)",
],
"shell": True,
},
"webanalyze": {
"installed": False,
"dependencies": ["go"],
"commands": [
f"{tool_paths.get('go')} get github.com/rverton/webanalyze/...",
f"(cd ~/go/src/github.com/rverton/webanalyze && {tool_paths.get('go')} build && {tool_paths.get('go')} install)",
],
"shell": True,
},
"recursive-gobuster": {
"installed": False,
"dependencies": ["gobuster", "seclists"],
"shell": True,
"commands": [
f"bash -c 'if [[ -d {Path(tool_paths.get('recursive-gobuster')).parent} ]] ; then cd {Path(tool_paths.get('recursive-gobuster')).parent} && git fetch --all && git pull; else git clone https://github.com/epi052/recursive-gobuster.git {Path(tool_paths.get('recursive-gobuster')).parent}; fi'"
],
},
"go": {
"installed": False,
"dependencies": None,
"commands": [
"wget -q https://dl.google.com/go/go1.13.7.linux-amd64.tar.gz -O /tmp/go.tar.gz",
"sudo tar -C /usr/local -xvf /tmp/go.tar.gz",
f'bash -c \'if [[ ! $(echo "${{PATH}}" | grep $(dirname {tool_paths.get("go")})) ]]; then echo "PATH=${{PATH}}:/usr/local/go/bin" >> ~/.bashrc; fi\'',
],
},
}
def get_scans():
""" Iterates over the recon package and its modules to find all of the \*Scan classes.
**A contract exists here that says any scans need to end with the word scan in order to be found by this function.**
Example:
``defaultdict(<class 'list'>, {'AmassScan': ['recon.amass'], 'MasscanScan': ['recon.masscan'], ... })``
Returns:
dict containing mapping of ``classname -> [modulename, ...]`` for all potential recon-pipeline commands
"""
scans = defaultdict(list)
# recursively walk packages; import each module in each package
# walk_packages yields ModuleInfo objects for all modules recursively on path
# prefix is a string to output on the front of every module name on output.
for loader, module_name, is_pkg in pkgutil.walk_packages(path=recon.__path__, prefix="recon."):
importlib.import_module(module_name)
# walk all modules, grabbing classes that we've written and add them to the classlist defaultdict
# getmembers returns all members of an object in a list of tuples (name, value)
for name, obj in inspect.getmembers(sys.modules[__name__]):
if inspect.ismodule(obj) and not name.startswith("_"):
# we're only interested in modules that don't begin with _ i.e. magic methods __len__ etc...
for subname, subobj in inspect.getmembers(obj):
if inspect.isclass(subobj) and subname.lower().endswith("scan"):
# now we only care about classes that end in [Ss]can
scans[subname].append(f"recon.{name}")
return scans
# options for ReconShell's 'install' command
install_parser = cmd2.Cmd2ArgumentParser()
install_parser.add_argument("tool", help="which tool to install", choices=list(tools.keys()) + ["all"])
# options for ReconShell's 'status' command
status_parser = cmd2.Cmd2ArgumentParser()
status_parser.add_argument(
"--port",
help="port on which the luigi central scheduler's visualization site is running (default: 8082)",
default="8082",
)
status_parser.add_argument(
"--host",
help="host on which the luigi central scheduler's visualization site is running (default: localhost)",
default="127.0.0.1",
)
# options for ReconShell's 'scan' command
scan_parser = cmd2.Cmd2ArgumentParser()
scan_parser.add_argument("scantype", choices_function=get_scans)
scan_parser.add_argument(
"--target-file",
completer_method=cmd2.Cmd.path_complete,
help="file created by the user that defines the target's scope; list of ips/domains",
)
scan_parser.add_argument(
"--exempt-list", completer_method=cmd2.Cmd.path_complete, help="list of blacklisted ips/domains"
)
scan_parser.add_argument(
"--results-dir", completer_method=cmd2.Cmd.path_complete, help="directory in which to save scan results"
)
scan_parser.add_argument(
"--wordlist", completer_method=cmd2.Cmd.path_complete, help="path to wordlist used by gobuster"
)
scan_parser.add_argument(
"--interface",
choices_function=lambda: [x[1] for x in socket.if_nameindex()],
help="which interface masscan should use",
)
scan_parser.add_argument("--recursive", action="store_true", help="whether or not to recursively gobust")
scan_parser.add_argument("--rate", help="rate at which masscan should scan")
scan_parser.add_argument(
"--top-ports", help="ports to scan as specified by nmap's list of top-ports (only meaningful to around 5000)"
)
scan_parser.add_argument("--ports", help="port specification for masscan (all ports example: 1-65535,U:1-65535)")
scan_parser.add_argument("--threads", help="number of threads for all of the threaded applications to use")
scan_parser.add_argument("--scan-timeout", help="scan timeout for aquatone")
scan_parser.add_argument("--proxy", help="proxy for gobuster if desired (ex. 127.0.0.1:8080)")
scan_parser.add_argument("--extensions", help="list of extensions for gobuster (ex. asp,html,aspx)")
scan_parser.add_argument(
"--sausage",
action="store_true",
help="open a web browser to Luigi's central scheduler's visualization site (see how the sausage is made!)",
)
scan_parser.add_argument(
"--local-scheduler", action="store_true", help="use the local scheduler instead of the central scheduler (luigid)"
)
scan_parser.add_argument(
"--verbose", action="store_true", help="shows debug messages from luigi, useful for troubleshooting"
)

210
recon/nmap.py
View File

@@ -1,210 +0,0 @@
import pickle
import logging
import subprocess
import concurrent.futures
from pathlib import Path
import luigi
from luigi.util import inherits
from recon.masscan import ParseMasscanOutput
from recon.config import defaults, tool_paths
@inherits(ParseMasscanOutput)
class ThreadedNmapScan(luigi.Task):
""" Run ``nmap`` against specific targets and ports gained from the ParseMasscanOutput Task.
Install:
``nmap`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``nmap``.
Basic Example:
.. code-block:: console
nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
def requires(self):
""" ThreadedNmap depends on ParseMasscanOutput to run.
TargetList expects target_file as a parameter.
Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.
Returns:
luigi.Task - ParseMasscanOutput
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
}
return ParseMasscanOutput(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output folder is TARGET_FILE-nmap-results.
The output folder will be populated with all of the output files generated by
any nmap commands run. Because the nmap command uses -oA, there will be three
files per target scanned: .xml, .nmap, .gnmap.
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "nmap-results"
return luigi.LocalTarget(results_subfolder.resolve())
def run(self):
""" Parses pickled target info dictionary and runs targeted nmap scans against only open ports. """
try:
self.threads = abs(int(self.threads))
except TypeError:
return logging.error("The value supplied to --threads must be a non-negative integer.")
ip_dict = pickle.load(open(self.input().path, "rb"))
nmap_command = [ # placeholders will be overwritten with appropriate info in loop below
"nmap",
"--open",
"PLACEHOLDER-IDX-2",
"-n",
"-sC",
"-T",
"4",
"-sV",
"-Pn",
"-p",
"PLACEHOLDER-IDX-10",
"-oA",
]
commands = list()
"""
ip_dict structure
{
"IP_ADDRESS":
{'udp': {"161", "5000", ... },
...
i.e. {protocol: set(ports) }
}
"""
for target, protocol_dict in ip_dict.items():
for protocol, ports in protocol_dict.items():
tmp_cmd = nmap_command[:]
tmp_cmd[2] = "-sT" if protocol == "tcp" else "-sU"
# arg to -oA, will drop into subdir off curdir
tmp_cmd[10] = ",".join(ports)
tmp_cmd.append(str(Path(self.output().path) / f"nmap.{target}-{protocol}"))
tmp_cmd.append(target) # target as final arg to nmap
commands.append(tmp_cmd)
# basically mkdir -p, won't error out if already there
Path(self.output().path).mkdir(parents=True, exist_ok=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.map(subprocess.run, commands)
@inherits(ThreadedNmapScan)
class SearchsploitScan(luigi.Task):
""" Run ``searchcploit`` against each ``nmap*.xml`` file in the **TARGET-nmap-results** directory and write results to disk.
Install:
``searchcploit`` is already on your system if you're using kali. If you're not using kali, refer to your own
distributions instructions for installing ``searchcploit``.
Basic Example:
.. code-block:: console
searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
Args:
threads: number of threads for parallel nmap command execution *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifies the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):
""" Searchsploit depends on ThreadedNmap to run.
TargetList expects target_file as a parameter.
Masscan expects rate, target_file, interface, and either ports or top_ports as parameters.
ThreadedNmap expects threads
Returns:
luigi.Task - ThreadedNmap
"""
args = {
"rate": self.rate,
"ports": self.ports,
"threads": self.threads,
"top_ports": self.top_ports,
"interface": self.interface,
"target_file": self.target_file,
"results_dir": self.results_dir,
}
return ThreadedNmapScan(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output folder is TARGET_FILE-searchsploit-results.
The output folder will be populated with all of the output files generated by
any searchsploit commands run.
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "searchsploit-results"
return luigi.LocalTarget(results_subfolder.resolve())
def run(self):
""" Grabs the xml files created by ThreadedNmap and runs searchsploit --nmap on each one, saving the output. """
for entry in Path(self.input().path).glob("nmap*.xml"):
proc = subprocess.run([tool_paths.get("searchsploit"), "--nmap", str(entry)], stderr=subprocess.PIPE)
if proc.stderr:
Path(self.output().path).mkdir(parents=True, exist_ok=True)
# change wall-searchsploit-results/nmap.10.10.10.157-tcp to 10.10.10.157
target = entry.stem.replace("nmap.", "").replace("-tcp", "").replace("-udp", "")
Path(f"{self.output().path}/searchsploit.{target}-{entry.stem[-3:]}.txt").write_bytes(proc.stderr)

61
recon/targets.py
View File

@@ -1,61 +0,0 @@
import shutil
import logging
import ipaddress
from pathlib import Path
import luigi
from recon.config import defaults
class TargetList(luigi.ExternalTask):
""" External task. ``TARGET_FILE`` is generated manually by the user from target's scope.
Args:
results_dir: specifies the directory on disk to which all Task results are written
"""
target_file = luigi.Parameter()
results_dir = luigi.Parameter(default=defaults.get("results-dir", ""))
def output(self):
""" Returns the target output for this task. target_file.ips || target_file.domains
In this case, it expects a file to be present in the local filesystem.
By convention, TARGET_NAME should be something like tesla or some other
target identifier. The returned target output will either be target_file.ips
or target_file.domains, depending on what is found on the first line of the file.
Example: Given a TARGET_FILE of tesla where the first line is tesla.com; tesla.domains
is written to disk.
Returns:
luigi.local_target.LocalTarget
"""
self.results_dir = Path(self.results_dir)
self.target_file = Path(self.target_file)
try:
with open(self.target_file) as f:
first_line = f.readline()
ipaddress.ip_interface(first_line.strip()) # is it a valid ip/network?
except OSError as e:
# can't open file; log error / return nothing
return logging.error(f"opening {self.target_file}: {e.strerror}")
except ValueError as e:
# exception thrown by ip_interface; domain name assumed
logging.debug(e)
new_target = "domains"
else:
# no exception thrown; ip address found
new_target = "ip_addresses"
results_subfolder = self.results_dir / "target-results"
results_subfolder.mkdir(parents=True, exist_ok=True)
new_path = results_subfolder / new_target
shutil.copy(self.target_file, new_path.resolve())
return luigi.LocalTarget(new_path.resolve())

103
recon/web/aquatone.py
View File

@@ -1,103 +0,0 @@
import subprocess
from pathlib import Path
import luigi
from luigi.util import inherits
from recon.config import tool_paths, defaults
from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class AquatoneScan(luigi.Task):
""" Screenshot all web targets and generate HTML report.
Install:
.. code-block:: console
mkdir /tmp/aquatone
wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
rm -rf /tmp/aquatone
Basic Example:
``aquatone`` commands are structured like the example below.
``cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20``
Luigi Example:
.. code-block:: python
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
Args:
threads: number of threads for parallel aquatone command execution
scan_timeout: timeout in miliseconds for aquatone port scans
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
scan_timeout = luigi.Parameter(default=defaults.get("aquatone-scan-timeout", ""))
def requires(self):
""" AquatoneScan depends on GatherWebTargets to run.
GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:
luigi.Task - GatherWebTargets
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output file is amass.TARGET_FILE.json.
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "aquatone-results"
return luigi.LocalTarget(results_subfolder.resolve())
def run(self):
""" Defines the options/arguments sent to aquatone after processing.
cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
Path(self.output().path).mkdir(parents=True, exist_ok=True)
command = [
tool_paths.get("aquatone"),
"-scan-timeout",
self.scan_timeout,
"-threads",
self.threads,
"-silent",
"-out",
self.output().path,
]
with self.input().open() as target_list:
subprocess.run(command, stdin=target_list)

99
recon/web/corscanner.py
View File

@@ -1,99 +0,0 @@
from pathlib import Path
import luigi
from luigi.util import inherits
from luigi.contrib.external_program import ExternalProgramTask
from recon.config import tool_paths, defaults
from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class CORScannerScan(ExternalProgramTask):
""" Use ``CORScanner`` to scan for potential CORS misconfigurations.
Install:
.. code-block:: console
git clone https://github.com/chenjj/CORScanner.git
cd CORScanner
pip install -r requirements.txt
pip install future
Basic Example:
.. code-block:: console
python cors_scan.py -i webtargets.tesla.txt -t 100
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.corscanner CORScannerScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel subjack command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
def requires(self):
""" CORScannerScan depends on GatherWebTargets to run.
GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:
luigi.Task - GatherWebTargets
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
Naming convention for the output file is corscanner.TARGET_FILE.json.
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "corscanner-results"
new_path = results_subfolder / "corscanner.json"
return luigi.LocalTarget(new_path.resolve())
def program_args(self):
""" Defines the options/arguments sent to tko-subs after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
command = [
"python3",
tool_paths.get("CORScanner"),
"-i",
self.input().path,
"-t",
self.threads,
"-o",
self.output().path,
]
return command

99
recon/web/targets.py
View File

@@ -1,99 +0,0 @@
import pickle
from pathlib import Path
import luigi
from luigi.util import inherits
from recon.amass import ParseAmassOutput
from recon.masscan import ParseMasscanOutput
from recon.config import web_ports
@inherits(ParseMasscanOutput, ParseAmassOutput)
class GatherWebTargets(luigi.Task):
""" Gather all subdomains as well as any ip addresses known to have a configured web port open.
Args:
exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
top_ports: Scan top N most popular ports *Required by upstream Task*
ports: specifies the port(s) to be scanned *Required by upstream Task*
interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
"""
def requires(self):
""" GatherWebTargets depends on ParseMasscanOutput and ParseAmassOutput to run.
ParseMasscanOutput expects rate, target_file, interface, and either ports or top_ports as parameters.
ParseAmassOutput accepts exempt_list and expects target_file
Returns:
dict(str: ParseMasscanOutput, str: ParseAmassOutput)
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
}
return {
"masscan-output": ParseMasscanOutput(**args),
"amass-output": ParseAmassOutput(
exempt_list=self.exempt_list, target_file=self.target_file, results_dir=self.results_dir
),
}
def output(self):
""" Returns the target output for this task.
Naming convention for the output file is webtargets.TARGET_FILE.txt.
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "target-results"
new_path = results_subfolder / "webtargets.txt"
return luigi.LocalTarget(new_path.resolve())
def run(self):
""" Gather all potential web targets into a single file to pass farther down the pipeline. """
Path(self.output().path).parent.mkdir(parents=True, exist_ok=True)
targets = set()
ip_dict = pickle.load(open(self.input().get("masscan-output").path, "rb"))
"""
structure over which we're looping
{
"IP_ADDRESS":
{'udp': {"161", "5000", ... },
...
i.e. {protocol: set(ports) }
}
"""
for target, protocol_dict in ip_dict.items():
for protocol, ports in protocol_dict.items():
for port in ports:
if protocol == "udp":
continue
if port == "80":
targets.add(target)
elif port in web_ports:
targets.add(f"{target}:{port}")
for amass_result in self.input().get("amass-output").values():
with amass_result.open() as f:
for target in f:
# we care about all results returned from amass
targets.add(target.strip())
with self.output().open("w") as f:
for target in targets:
f.write(f"{target}\n")

127
recon/web/webanalyze.py
View File

@@ -1,127 +0,0 @@
import os
import logging
import ipaddress
import subprocess
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor
import luigi
from luigi.util import inherits
from recon.config import tool_paths, defaults
from recon.web.targets import GatherWebTargets
@inherits(GatherWebTargets)
class WebanalyzeScan(luigi.Task):
""" Use webanalyze to determine the technology stack on the given target(s).
Install:
.. code-block:: console
go get -u github.com/rverton/webanalyze
# loads new apps.json file from wappalyzer project
webanalyze -update
Basic Example:
.. code-block:: console
webanalyze -host www.tesla.com -output json
Luigi Example:
.. code-block:: console
PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.webanalyze WebanalyzeScan --target-file tesla --top-ports 1000 --interface eth0
Args:
threads: number of threads for parallel webanalyze command execution
exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
top_ports: Scan top N most popular ports *--* Required by upstream Task
ports: specifies the port(s) to be scanned *--* Required by upstream Task
interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
"""
threads = luigi.Parameter(default=defaults.get("threads", ""))
def requires(self):
""" WebanalyzeScan depends on GatherWebTargets to run.
GatherWebTargets accepts exempt_list and expects rate, target_file, interface,
and either ports or top_ports as parameters
Returns:
luigi.Task - GatherWebTargets
"""
args = {
"results_dir": self.results_dir,
"rate": self.rate,
"target_file": self.target_file,
"top_ports": self.top_ports,
"interface": self.interface,
"ports": self.ports,
"exempt_list": self.exempt_list,
}
return GatherWebTargets(**args)
def output(self):
""" Returns the target output for this task.
The naming convention for the output file is webanalyze.TARGET_FILE.txt
Results are stored in their own directory: webanalyze-TARGET_FILE-results
Returns:
luigi.local_target.LocalTarget
"""
results_subfolder = Path(self.results_dir) / "webanalyze-results"
return luigi.LocalTarget(results_subfolder.resolve())
def _wrapped_subprocess(self, cmd):
with open(f"webanalyze.{cmd[2].replace('//', '_').replace(':', '')}.txt", "wb") as f:
subprocess.run(cmd, stderr=f)
def run(self):
""" Defines the options/arguments sent to webanalyze after processing.
Returns:
list: list of options/arguments, beginning with the name of the executable to run
"""
try:
self.threads = abs(int(self.threads))
except TypeError:
return logging.error("The value supplied to --threads must be a non-negative integer.")
commands = list()
with self.input().open() as f:
for target in f:
target = target.strip()
try:
if isinstance(ipaddress.ip_address(target), ipaddress.IPv6Address): # ipv6
target = f"[{target}]"
except ValueError:
# domain names raise ValueErrors, just assume we have a domain and keep on keepin on
pass
for url_scheme in ("https://", "http://"):
command = [tool_paths.get("webanalyze"), "-host", f"{url_scheme}{target}"]
commands.append(command)
Path(self.output().path).mkdir(parents=True, exist_ok=True)
cwd = Path().cwd()
os.chdir(self.output().path)
if not Path("apps.json").exists():
subprocess.run(f"{tool_paths.get('webanalyze')} -update".split())
with ThreadPoolExecutor(max_workers=self.threads) as executor:
executor.map(self._wrapped_subprocess, commands)
os.chdir(str(cwd))

1
tests/data/bitdiscovery.small Normal file
View File

@@ -0,0 +1 @@
staging.bitdiscovery.com

BIN
tests/data/existing-database-test Normal file
View File

Binary file not shown.

1
tests/data/new-test/amass-results/amass.json Normal file
View File

@@ -0,0 +1 @@
{"name":"staging.bitdiscovery.com","domain":"staging.bitdiscovery.com","addresses":[{"ip":"184.72.14.217","cidr":"184.72.0.0/18","asn":16509,"desc":"AMAZON-02, US"},{"ip":"52.8.166.143","cidr":"52.8.0.0/16","asn":16509,"desc":"AMAZON-02, US"}],"tag":"api","source":"Robtex"}

739
tests/data/new-test/aquatone-results/aquatone_report.html Normal file
View File

File diff suppressed because one or more lines are too long

1
tests/data/new-test/aquatone-results/aquatone_session.json Normal file
View File

File diff suppressed because one or more lines are too long

6
tests/data/new-test/aquatone-results/aquatone_urls.txt Normal file
View File

@@ -0,0 +1,6 @@
http://52.8.166.143/
https://52.8.166.143/
https://184.72.14.217/
http://184.72.14.217/
https://staging.bitdiscovery.com/
http://staging.bitdiscovery.com/

22
tests/data/recon-results/aquatone-results/headers/http__52_9_23_177__42099b4af021e53f.txt → tests/data/new-test/aquatone-results/headers/http__184_72_14_217__42099b4af021e53f.txt
View File

@@ -1,17 +1,15 @@
200 OK
Date: Thu, 30 Jan 2020 12:52:51 GMT
Retry-Count: 0
500 Internal Server Error
Access-Control-Allow-Headers: X-Requested-With
Referrer-Policy: origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: nginx/1.16.1
Etag: W/"2c44-aSq52Bwr/hjAPBUWuI0BtbI4oiE"
Set-Cookie: connect.sid=s%3AaucVLezKKP7aSHdiTuIpbYR7SOt_37do.B90p3nel1BvebCtsAQY0rYmUD5iP%2Byi%2FEzQUyDAYxKU; Path=/; HttpOnly
Etag: W/"25-FJs8qf7Gj2hoxpvMuuUdWSjYlfg"
Content-Type: text/html; charset=utf-8
Content-Length: 11332
Vary: Accept-Encoding
X-Xss-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains
Retry-Count: 0
Date: Mon, 06 Apr 2020 14:58:13 GMT
X-Frame-Options: SAMEORIGIN
Server: nginx/1.16.1
Content-Length: 37
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
Referrer-Policy: origin-when-cross-origin
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

24
tests/data/recon-results/aquatone-results/headers/https__13_57_162_100__42099b4af021e53f.txt → tests/data/new-test/aquatone-results/headers/http__52_8_166_143__42099b4af021e53f.txt
View File

@@ -1,17 +1,15 @@
200 OK
Set-Cookie: connect.sid=s%3ANI8ilMOt3ciRZXAHjA9C3eqv1jr4A3Cb.7RHKG1mNUvn%2Fl9GS%2FMlsAuCf630pMwvG2lfUS0M%2BjDs; Path=/; HttpOnly
500 Internal Server Error
Etag: W/"25-FJs8qf7Gj2hoxpvMuuUdWSjYlfg"
Content-Length: 37
Retry-Count: 0
Access-Control-Allow-Headers: X-Requested-With
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
Date: Mon, 06 Apr 2020 14:58:12 GMT
Server: nginx/1.16.1
Access-Control-Allow-Origin: *
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubDomains
Content-Type: text/html; charset=utf-8
Referrer-Policy: origin-when-cross-origin
X-Xss-Protection: 1; mode=block
Etag: W/"2c44-aSq52Bwr/hjAPBUWuI0BtbI4oiE"
Retry-Count: 0
Date: Thu, 30 Jan 2020 12:52:51 GMT
Content-Length: 11332
Server: nginx/1.16.1
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding

20
tests/data/recon-results/aquatone-results/headers/http__13_57_162_100__42099b4af021e53f.txt → tests/data/new-test/aquatone-results/headers/http__staging_bitdiscovery_com__42099b4af021e53f.txt
View File

@@ -1,17 +1,17 @@
200 OK
Date: Thu, 30 Jan 2020 12:52:50 GMT
Date: Mon, 06 Apr 2020 14:58:13 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 11332
Strict-Transport-Security: max-age=31536000; includeSubDomains
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
X-Xss-Protection: 1; mode=block
Etag: W/"5f8a-vQBYKrRAfqI5iQ/GP2AM/fcHMKU"
Retry-Count: 0
Referrer-Policy: origin-when-cross-origin
Server: nginx/1.16.1
Access-Control-Allow-Headers: X-Requested-With
X-Frame-Options: SAMEORIGIN
Etag: W/"2c44-aSq52Bwr/hjAPBUWuI0BtbI4oiE"
Content-Length: 24458
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
X-Xss-Protection: 1; mode=block
Set-Cookie: connect.sid=s%3ACE4sYwUvG9fBBal0Ud-mp35YuejVagNj.VcA4FtYY05sXbWrrnAtvLnZ7WsHiRS5sByXtQGBT3%2B4; Path=/; HttpOnly
Referrer-Policy: origin-when-cross-origin
Set-Cookie: connect.sid=s%3APd8ugrHwMLERUtsvE40vqkfKaFmhEODl.FeZjIm7jJi3UOiqBKtpwb72iyQaaXs3hcZCAtNIICAM; Path=/; HttpOnly
Server: nginx/1.16.1
Access-Control-Allow-Headers: X-Requested-With
Strict-Transport-Security: max-age=31536000; includeSubDomains

15
tests/data/new-test/aquatone-results/headers/https__184_72_14_217__42099b4af021e53f.txt Normal file
View File

@@ -0,0 +1,15 @@
500 Internal Server Error
Date: Mon, 06 Apr 2020 14:58:13 GMT
Access-Control-Allow-Origin: *
Referrer-Policy: origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
Etag: W/"25-FJs8qf7Gj2hoxpvMuuUdWSjYlfg"
Retry-Count: 0
Content-Type: text/html; charset=utf-8
X-Content-Type-Options: nosniff
Content-Length: 37
Access-Control-Allow-Headers: X-Requested-With
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Server: nginx/1.16.1

15
tests/data/new-test/aquatone-results/headers/https__52_8_166_143__42099b4af021e53f.txt Normal file
View File

@@ -0,0 +1,15 @@
500 Internal Server Error
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: nginx/1.16.1
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
Etag: W/"25-FJs8qf7Gj2hoxpvMuuUdWSjYlfg"
Date: Mon, 06 Apr 2020 14:58:12 GMT
Content-Length: 37
Access-Control-Allow-Headers: X-Requested-With
X-Content-Type-Options: nosniff
Retry-Count: 0
Content-Type: text/html; charset=utf-8
Referrer-Policy: origin-when-cross-origin

24
tests/data/recon-results/aquatone-results/headers/https__52_9_23_177__42099b4af021e53f.txt → tests/data/new-test/aquatone-results/headers/https__staging_bitdiscovery_com__42099b4af021e53f.txt
View File

@@ -1,17 +1,17 @@
200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 11332
Vary: Accept-Encoding
Server: nginx/1.16.1
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
Set-Cookie: connect.sid=s%3AcIC-bK1KNp8Ek9tdyo6EunQHvrX46WqB.vyi5KrQ%2FMDTq3BnCPRgGoiZG890MgFU0WgZ7OrzBSck; Path=/; HttpOnly
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubDomains
Date: Thu, 30 Jan 2020 12:52:53 GMT
Retry-Count: 0
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Referrer-Policy: origin-when-cross-origin
X-Xss-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Etag: W/"5f8a-vQBYKrRAfqI5iQ/GP2AM/fcHMKU"
Date: Mon, 06 Apr 2020 14:58:13 GMT
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Server: nginx/1.16.1
Content-Type: text/html; charset=utf-8
Content-Length: 24458
Content-Security-Policy: default-src 'self' https: 'unsafe-inline' 'unsafe-eval' data: client-api.arkoselabs.com
Etag: W/"2c44-aSq52Bwr/hjAPBUWuI0BtbI4oiE"
Retry-Count: 0
Set-Cookie: connect.sid=s%3Af5wxC_BuTDH2KEoNQP0Nr3HsO1cggFxw.rG0gVIzq3x1lWDOWoPstFmGBYW7k6hsNzfyF7XLvsug; Path=/; HttpOnly

1
tests/data/new-test/aquatone-results/html/http__184_72_14_217__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1 @@
Invalid host. Please contact support.

1
tests/data/new-test/aquatone-results/html/http__52_8_166_143__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1 @@
Invalid host. Please contact support.

511
tests/data/new-test/aquatone-results/html/http__staging_bitdiscovery_com__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1,511 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="X-UA-Compatible" content="IE=9; IE=8; IE=7; IE=EDGE"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Bit Discovery - Secure everything.</title>
<!-- CSS -->
<link rel="stylesheet"
href="https://fonts.googleapis.com/css?family=Avenir:100,300,400|Century+Gothic:300,400|Roboto:300,400,500">
<link rel="stylesheet" href="homepage/assets/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="homepage/assets/font-awesome/css/font-awesome.min.css">
<link rel="stylesheet" href="homepage/assets/css/animate.css">
<link rel="stylesheet" href="homepage/assets/css/style.css">
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<script>
if (window.location.search.indexOf("?to=") !== -1) {
window.sessionStorage.setItem("to", window.location.search.substring(4));
}
</script>
<!-- Top content -->
<div class="top-content1">
<!-- Top menu -->
<nav class="navbar navbar-expand-md navbar-dark navbar-inverse" role="navigation">
<div class="container">
<a class="navbar-brand" href="/">
<img src="/images/bitdiscovery2020logowhite.svg" class="navbar-logo">
</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#top-navbar-1">
<span class="navbar-toggler-icon"></span>
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<div class="collapse navbar-collapse" id="top-navbar-1">
<ul class="nav navbar-nav ml-auto">
<li><a href="/login" class="nav-link" data-testid="signInButton"><b>Sign In</b></a></li>
</ul>
</div>
</div>
</nav>
<div class="top-content-container">
<div class="container">
<div class="row">
<div class="col-sm-7 text wow fadeInLeft">
<div class="main-heading">
<h1 class="top-content-heading">Secure everything.</h1>
<p class="medium-paragraph">Asset Inventory that discovers, learns, and (finally) lets you
secure everything.</p>
<div class="contact-us">
<a href="/contact" class="contact-us-bottom" data-testid="contactLink">Contact Us</a>
</div>
</div>
</div>
<div class="col-sm-5 text wow fadeInLeft">
<div class="main-check">
<img src="homepage/assets/img/check.png" class="header-check">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="use-cases-container">
<div class="container">
<div class="row">
<div class="col-md-12 wow fadeInLeft">
<div class="use-cases">Use Cases</div>
</div>
</div>
</div>
</div>
<!-- Features -->
<div class="container-fluid">
<div class="row features-row">
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/pie-chart.svg" class="features-icon1">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Competitive Analysis</b></h3>
<p class="features-desc">
Discover the asset inventory of all of your competitors to compare with your own.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/tag.svg" class="features-icon2">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Marketing & Brand Protection</b></h3>
<p class="features-desc">
Discover expired marketing campaigns, misspellings, SEO opportunities, and misuse of your
brand.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/sucess-check.svg" class="features-icon3">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>GDPR Compliance</b></h3>
<p class="features-desc">
Know the places where PII is captured and stored, or where customer data may get exposed.
</p>
</div>
</div>
</div>
</div>
<div class="row features-row">
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/balance-scale.svg" class="features-icon4">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Legal</b></h3>
<p class="features-desc">
Know which of your assets have out of compliance technology, missing legal disclaimers, and
expired copyright notices.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/handshake.svg" class="features-icon5">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Mergers & Acquisitions</b></h3>
<p class="features-desc">
Instantly see all of the internet-facing assets of your target company before acquisition to
identify risk
early.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/lock.svg" class="features-icon6">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Information Security</b></h3>
<p class="features-desc">
You need to know what you own before you can secure what you own. Accurate Asset Inventory is
foundational to your security strategy.
</p>
</div>
</div>
</div>
</div>
</div>
<!-- logo section -->
<div class="section-logos-main">
<div class="container section-logos">
<div class="row">
<div class="col-sm-12 text wow fadeInLeft">
<p class="section-logos-text">Trusted By</p>
</div>
</div>
<div class="row vertical-center-row">
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/redshield-logo.png" alt="" srcset="" class="trustedby-logo" >
</div>
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/bugcrowd-logo.png" alt="" srcset="" class="trustedby-logo">
</div>
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/tenable-logo.png" class="trustedby-logo">
</div>
</div>
<div class="row vertical-center-row">
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;">
<img src="homepage/assets/img/sponsors/securitycompass-logo.png" class="section-logos-image">
</div>
<div class="col-md-1 col-sm-1 col-1 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
<div class="col-md-2 col-sm-2 col-2 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;">
<img src="homepage/assets/img/sponsors/sectheory-logo.png" class="section-logos-image">
</div>
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
</div>
</div>
</div>
<!-- Logo Section -->
<div class="container-fluid section-industry ">
<h2 class="section-industry-heading text-center wow fadeInLeft">Created by industry experts</h2>
<div class="row align-items-center">
<div class="col-sm-5 section-industry-side1 wow fadeInLeft ">
<h1 class="section-industry-heading2 ">Jeremiah Grossman</h1>
<p class="section-industry-text">Founder of WhiteHat Security. World-Renowned Professional Hacker.
Jeremiah's career spans nearly 20 years and he has become one of the computer security industry's
biggest
names.</p>
<a href="https://www.jeremiahgrossman.com" class="section-industry-link">More about Jeremiah ></a>
</div>
<div class="col-sm-7 wow fadeInRight" style="padding-right: 0px;">
<img src="homepage/assets/img/grossman.png" class="img-fluid section-industry-image1">
</div>
</div>
<div class="row align-items-center">
<div class="col-sm-5 order-sm-7 section-industry-side2 wow fadeInRight">
<h2 class="section-industry-heading2" style="width: 85%;">Robert Hansen</h2>
<p class="section-industry-text">Robert is a quarter-century veteran of infosec, spanning a career of
penetration testing, security architecture, security product management, and security research.
</p>
<a href="https://www.smartphoneexec.com" class="section-industry-link">More about Robert ></a>
</div>
<div class="col-sm-7 order-sm-5 wow fadeInLeft" style="padding-left: 0px;">
<img src="homepage/assets/img/robert.png" class="section-industry-image2">
</div>
</div>
</div>
<div class="section-discover">
<div class="container section-discover-container">
<div class="row">
<div class="col-md-12">
<h2 class="text-left section-discover-heading">Discover <span class="type section-discover-span"></span>
</h2>
<p class="text-left section-discover-text">The world's top companies and agencies use Bit Discovery to
discover just about anything about their internet-facing assets.</p>
</div>
</div>
</div>
</div>
<div class="section-inventory">
<div class="container">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft">
<img src="homepage/assets/img/inventry.png" class="section-inventory-image">
</div>
</div>
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft section-inventory-mainpart">
<h2 class="section-inventory-main-heading">An inventory of everything</h2>
<p class="section-inventory-main-text">It's all there. Your domains, subdomains, exposed
technologies-organized into a speedy, searchable Inventory that stays updated automatically.</p>
</div>
</div>
</div>
<div class="container section-inventory-dualpart">
<div class="row">
<div class="col-md-6 text wow fadeInLeft section-inventory-dual-side1">
<h5 class="section-inventory-sec-heading">Go Beyond Asset Inventory</h5>
<p class="section-inventory-sec-text">Bit Discovery also inventories extensive technology information
about each asset in your inventory.</p>
</div>
<div class="col-md-6 text wow fadeInLeft section-inventory-dual-side2">
<h5 class="section-inventory-sec-heading">Know Instantly</h5>
<p class="section-inventory-sec-text">Important changes to any of your assets are quickly identified.
Receive daily email summaries of changes.</p>
</div>
</div>
</div>
</div>
<div class="section-simple">
<div class="container-fluid">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft ">
<h2 class="section-simple-heading">Simple.</h2>
<p class="section-simple-text">Bit Discovery is simple, fast, and easy to use.</p>
</div>
<div class="col-md-12 col-sm-12 text wow fadeInLeft ">
<img src="homepage/assets/img/simple.png">
</div>
</div>
</div>
</div>
<div class="section-technology">
<div class="container-fluid">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft">
<h2 class="section-technology-heading">Technology Fingerprinting</h2>
<p class="section-technology-text">Thousands of technologies. Hundreds of ports. Zero hassle. </p>
</div>
<div class="col-md-12 col-sm-12 text wow fadeInUp">
<img src="homepage/assets/img/image-logos.png">
</div>
</div>
</div>
</div>
<div class="section-gdpr">
<div class="section-gdpr-row">
<div class="container section-gdpr-container">
<div class="row">
<div class="col-md-2 col-sm-4 col-4 text wow fadeInLeft">
<img src="homepage/assets/img/icons/stars.png" class="section-gdpr-img">
</div>
<div class="col-md-6 col-sm-8 col-8 text wow fadeInLeft ">
<h3 class="section-gdpr-main-heading">GDPR Compliance</h3>
<p class="section-gdpr-main-text">GDPR is a prolific set of compliance mandates and has sweeping
implications for companies that serve customers in the European Union. A critical component of
compliance is starting off with a known set of assets and identifying the places where PII is
captured and stored, or where customer data may get exposed.</p>
</div>
</div>
</div>
</div>
<div class="container">
<div class="row">
<!-- Row 1 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon1.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">TLS Key Length & Protocol</h4>
<p class="section-gdpr-text">Identify weak and outdated cipher suites.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon2.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Hosting Country</h4>
<p class="section-gdpr-text">Locate the geographic region of your assets.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon3.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Outdated Copyright Notices</h4>
<p class="section-gdpr-text">See if your legal team is paying attention to the sites in question.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon4.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Advertising Networks</h4>
<p class="section-gdpr-text">Identify 3rd parties who may be capturing user traffic/sentiment.</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
<div class="container">
<div class="row">
<!-- Row 2 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon5.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Google Analytics</h4>
<p class="section-gdpr-text">Find where 3rd party analytics software may be gathering PII.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon6.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Facebook Widgets</h4>
<p class="section-gdpr-text">Capture the locations where social sites might correlate your users.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon7.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">CRM </h4>
<p class="section-gdpr-text">Focus on the dynamic sites that are most likely to have customer data.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon8.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Fraud Investigation</h4>
<p class="section-gdpr-text">Fast forward fraud investigations by locating sites of interest.</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
<div class="container">
<div class="row">
<!-- Row 3 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon9.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">SEO</h4>
<p class="section-gdpr-text">See where your marketing team might be retargeting your users.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon10.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Marketing Automation</h4>
<p class="section-gdpr-text">Find where your marketing team might be gathering contacts.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon11.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Outdated Services</h4>
<p class="section-gdpr-text">If you aren't patching you aren't compliant.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon12.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Forms</h4>
<p class="section-gdpr-text">Are you collecting customer data and on which websites?</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
</div>
<div class="footer">
<section class="container">
<div class="row ">
<div class="col-lg-2 col-md-2 col-sm-2 col-12 "><img src="homepage/assets/img/Asset51x.png" alt="Bit Discovery" class="footer-image" srcset=""></div>
<div class="col-lg-7 col-md-7 col-sm-7 col-12" style="padding-top: 1%;">
<div class="row" data-testid="footerSection">
<div class="footer-link">
<a href="/about" class="footer-a" data-testid="aboutLink">About</a>
</div>
<div class="footer-link">
<a href="https://blog.bitdiscovery.com" class="footer-a" data-testid="blogLink">Blog</a>
</div>
<div class="footer-link">
<a href="/faq" class="footer-a" data-testid="faqLink">FAQ</a>
</div>
<div class="footer-link">
<a href="/contact" class="footer-a" data-testid="contactLink">Contact</a>
</div>
<div class="footer-link">
<a href="/press" class="footer-a" data-testid="pressLink">Press</a>
</div>
<div class="footer-link">
<a href="/terms" class="footer-a" data-testid="termsLink">Terms</a>
</div>
<div class="footer-link">
<a href="/lexicon" class="footer-a" data-testid="lexiconLink">Lexicon</a>
</div>
<div class="footer-link">
<a href="/disclosure_policy" class="footer-a" data-testid="policyLink">Disclosure Policy</a>
</div>
</div>
</div>
<div class="col-lg-3 col-md-3 col-sm-3 col-12"style="padding-top: 1%;">
<div class="footer-link-copy">Copyright © 2020 Bit Discovery</div>
</div>
</div>
</section>
</div>
<!-- Javascript -->
<script src="homepage/assets/js/jquery-1.11.1.min.js"></script>
<script src="homepage/assets/bootstrap/js/bootstrap.min.js"></script>
<script src="homepage/assets/js/jquery.backstretch.min.js"></script>
<script src="homepage/assets/js/wow.min.js"></script>
<script src="homepage/assets/js/retina-1.1.0.min.js"></script>
<script src="homepage/assets/js/scripts.js"></script>
<script src="https://cdn.jsdelivr.net/npm/typed.js@2.0.11"></script>
<!--[if lt IE 10]>
<script src="homepage/assets/js/placeholder.js"></script>
<![endif]-->
<script>
var typed = new Typed('.type', {
strings: [
'your attack surface',
'forgotten assets',
'login forms',
'all of your ASNs',
'open ports',
'which assets set cookies',
'web servers',
'spelling errors',
'programming languages',
'by geographic location',
'all of your CMSs',
'cloud hosted assets',
'running services',
'expired TLS certificates',
'Javascript frameworks',
'more.'
],
typeSpeed: 38,
backSpeed: 55,
loop: true
});
</script>
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-114279261-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-114279261-1');
</script>
</body>
</html>

1
tests/data/new-test/aquatone-results/html/https__184_72_14_217__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1 @@
Invalid host. Please contact support.

1
tests/data/new-test/aquatone-results/html/https__52_8_166_143__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1 @@
Invalid host. Please contact support.

511
tests/data/new-test/aquatone-results/html/https__staging_bitdiscovery_com__42099b4af021e53f.html Normal file
View File

@@ -0,0 +1,511 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="X-UA-Compatible" content="IE=9; IE=8; IE=7; IE=EDGE"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Bit Discovery - Secure everything.</title>
<!-- CSS -->
<link rel="stylesheet"
href="https://fonts.googleapis.com/css?family=Avenir:100,300,400|Century+Gothic:300,400|Roboto:300,400,500">
<link rel="stylesheet" href="homepage/assets/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="homepage/assets/font-awesome/css/font-awesome.min.css">
<link rel="stylesheet" href="homepage/assets/css/animate.css">
<link rel="stylesheet" href="homepage/assets/css/style.css">
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<script>
if (window.location.search.indexOf("?to=") !== -1) {
window.sessionStorage.setItem("to", window.location.search.substring(4));
}
</script>
<!-- Top content -->
<div class="top-content1">
<!-- Top menu -->
<nav class="navbar navbar-expand-md navbar-dark navbar-inverse" role="navigation">
<div class="container">
<a class="navbar-brand" href="/">
<img src="/images/bitdiscovery2020logowhite.svg" class="navbar-logo">
</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#top-navbar-1">
<span class="navbar-toggler-icon"></span>
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<div class="collapse navbar-collapse" id="top-navbar-1">
<ul class="nav navbar-nav ml-auto">
<li><a href="/login" class="nav-link" data-testid="signInButton"><b>Sign In</b></a></li>
</ul>
</div>
</div>
</nav>
<div class="top-content-container">
<div class="container">
<div class="row">
<div class="col-sm-7 text wow fadeInLeft">
<div class="main-heading">
<h1 class="top-content-heading">Secure everything.</h1>
<p class="medium-paragraph">Asset Inventory that discovers, learns, and (finally) lets you
secure everything.</p>
<div class="contact-us">
<a href="/contact" class="contact-us-bottom" data-testid="contactLink">Contact Us</a>
</div>
</div>
</div>
<div class="col-sm-5 text wow fadeInLeft">
<div class="main-check">
<img src="homepage/assets/img/check.png" class="header-check">
</div>
</div>
</div>
</div>
</div>
</div>
<div class="use-cases-container">
<div class="container">
<div class="row">
<div class="col-md-12 wow fadeInLeft">
<div class="use-cases">Use Cases</div>
</div>
</div>
</div>
</div>
<!-- Features -->
<div class="container-fluid">
<div class="row features-row">
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/pie-chart.svg" class="features-icon1">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Competitive Analysis</b></h3>
<p class="features-desc">
Discover the asset inventory of all of your competitors to compare with your own.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/tag.svg" class="features-icon2">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Marketing & Brand Protection</b></h3>
<p class="features-desc">
Discover expired marketing campaigns, misspellings, SEO opportunities, and misuse of your
brand.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/sucess-check.svg" class="features-icon3">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>GDPR Compliance</b></h3>
<p class="features-desc">
Know the places where PII is captured and stored, or where customer data may get exposed.
</p>
</div>
</div>
</div>
</div>
<div class="row features-row">
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/balance-scale.svg" class="features-icon4">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Legal</b></h3>
<p class="features-desc">
Know which of your assets have out of compliance technology, missing legal disclaimers, and
expired copyright notices.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/handshake.svg" class="features-icon5">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Mergers & Acquisitions</b></h3>
<p class="features-desc">
Instantly see all of the internet-facing assets of your target company before acquisition to
identify risk
early.
</p>
</div>
</div>
</div>
<div class="col-md-4 col-sm-4 features-box wow fadeInLeft">
<div class="row">
<div class="col-md-3 features-box-icon">
<img src="homepage/assets/img/icons/lock.svg" class="features-icon6">
</div>
<div class="col-md-9 features-col">
<h3 class="features-heading"><b>Information Security</b></h3>
<p class="features-desc">
You need to know what you own before you can secure what you own. Accurate Asset Inventory is
foundational to your security strategy.
</p>
</div>
</div>
</div>
</div>
</div>
<!-- logo section -->
<div class="section-logos-main">
<div class="container section-logos">
<div class="row">
<div class="col-sm-12 text wow fadeInLeft">
<p class="section-logos-text">Trusted By</p>
</div>
</div>
<div class="row vertical-center-row">
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/redshield-logo.png" alt="" srcset="" class="trustedby-logo" >
</div>
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/bugcrowd-logo.png" alt="" srcset="" class="trustedby-logo">
</div>
<div class="col-md-4 col-sm-4 col-lg-4 col-4">
<img src="homepage/assets/img/sponsors/tenable-logo.png" class="trustedby-logo">
</div>
</div>
<div class="row vertical-center-row">
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;">
<img src="homepage/assets/img/sponsors/securitycompass-logo.png" class="section-logos-image">
</div>
<div class="col-md-1 col-sm-1 col-1 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
<div class="col-md-2 col-sm-2 col-2 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;">
<img src="homepage/assets/img/sponsors/sectheory-logo.png" class="section-logos-image">
</div>
<div class="col-md-3 col-sm-3 col-3 text wow fadeInRight animated" style="visibility: visible; animation-name: fadeInRight;"></div>
</div>
</div>
</div>
<!-- Logo Section -->
<div class="container-fluid section-industry ">
<h2 class="section-industry-heading text-center wow fadeInLeft">Created by industry experts</h2>
<div class="row align-items-center">
<div class="col-sm-5 section-industry-side1 wow fadeInLeft ">
<h1 class="section-industry-heading2 ">Jeremiah Grossman</h1>
<p class="section-industry-text">Founder of WhiteHat Security. World-Renowned Professional Hacker.
Jeremiah's career spans nearly 20 years and he has become one of the computer security industry's
biggest
names.</p>
<a href="https://www.jeremiahgrossman.com" class="section-industry-link">More about Jeremiah ></a>
</div>
<div class="col-sm-7 wow fadeInRight" style="padding-right: 0px;">
<img src="homepage/assets/img/grossman.png" class="img-fluid section-industry-image1">
</div>
</div>
<div class="row align-items-center">
<div class="col-sm-5 order-sm-7 section-industry-side2 wow fadeInRight">
<h2 class="section-industry-heading2" style="width: 85%;">Robert Hansen</h2>
<p class="section-industry-text">Robert is a quarter-century veteran of infosec, spanning a career of
penetration testing, security architecture, security product management, and security research.
</p>
<a href="https://www.smartphoneexec.com" class="section-industry-link">More about Robert ></a>
</div>
<div class="col-sm-7 order-sm-5 wow fadeInLeft" style="padding-left: 0px;">
<img src="homepage/assets/img/robert.png" class="section-industry-image2">
</div>
</div>
</div>
<div class="section-discover">
<div class="container section-discover-container">
<div class="row">
<div class="col-md-12">
<h2 class="text-left section-discover-heading">Discover <span class="type section-discover-span"></span>
</h2>
<p class="text-left section-discover-text">The world's top companies and agencies use Bit Discovery to
discover just about anything about their internet-facing assets.</p>
</div>
</div>
</div>
</div>
<div class="section-inventory">
<div class="container">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft">
<img src="homepage/assets/img/inventry.png" class="section-inventory-image">
</div>
</div>
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft section-inventory-mainpart">
<h2 class="section-inventory-main-heading">An inventory of everything</h2>
<p class="section-inventory-main-text">It's all there. Your domains, subdomains, exposed
technologies-organized into a speedy, searchable Inventory that stays updated automatically.</p>
</div>
</div>
</div>
<div class="container section-inventory-dualpart">
<div class="row">
<div class="col-md-6 text wow fadeInLeft section-inventory-dual-side1">
<h5 class="section-inventory-sec-heading">Go Beyond Asset Inventory</h5>
<p class="section-inventory-sec-text">Bit Discovery also inventories extensive technology information
about each asset in your inventory.</p>
</div>
<div class="col-md-6 text wow fadeInLeft section-inventory-dual-side2">
<h5 class="section-inventory-sec-heading">Know Instantly</h5>
<p class="section-inventory-sec-text">Important changes to any of your assets are quickly identified.
Receive daily email summaries of changes.</p>
</div>
</div>
</div>
</div>
<div class="section-simple">
<div class="container-fluid">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft ">
<h2 class="section-simple-heading">Simple.</h2>
<p class="section-simple-text">Bit Discovery is simple, fast, and easy to use.</p>
</div>
<div class="col-md-12 col-sm-12 text wow fadeInLeft ">
<img src="homepage/assets/img/simple.png">
</div>
</div>
</div>
</div>
<div class="section-technology">
<div class="container-fluid">
<div class="row">
<div class="col-md-12 col-sm-12 text wow fadeInLeft">
<h2 class="section-technology-heading">Technology Fingerprinting</h2>
<p class="section-technology-text">Thousands of technologies. Hundreds of ports. Zero hassle. </p>
</div>
<div class="col-md-12 col-sm-12 text wow fadeInUp">
<img src="homepage/assets/img/image-logos.png">
</div>
</div>
</div>
</div>
<div class="section-gdpr">
<div class="section-gdpr-row">
<div class="container section-gdpr-container">
<div class="row">
<div class="col-md-2 col-sm-4 col-4 text wow fadeInLeft">
<img src="homepage/assets/img/icons/stars.png" class="section-gdpr-img">
</div>
<div class="col-md-6 col-sm-8 col-8 text wow fadeInLeft ">
<h3 class="section-gdpr-main-heading">GDPR Compliance</h3>
<p class="section-gdpr-main-text">GDPR is a prolific set of compliance mandates and has sweeping
implications for companies that serve customers in the European Union. A critical component of
compliance is starting off with a known set of assets and identifying the places where PII is
captured and stored, or where customer data may get exposed.</p>
</div>
</div>
</div>
</div>
<div class="container">
<div class="row">
<!-- Row 1 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon1.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">TLS Key Length & Protocol</h4>
<p class="section-gdpr-text">Identify weak and outdated cipher suites.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon2.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Hosting Country</h4>
<p class="section-gdpr-text">Locate the geographic region of your assets.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon3.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Outdated Copyright Notices</h4>
<p class="section-gdpr-text">See if your legal team is paying attention to the sites in question.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon4.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Advertising Networks</h4>
<p class="section-gdpr-text">Identify 3rd parties who may be capturing user traffic/sentiment.</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
<div class="container">
<div class="row">
<!-- Row 2 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon5.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Google Analytics</h4>
<p class="section-gdpr-text">Find where 3rd party analytics software may be gathering PII.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon6.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Facebook Widgets</h4>
<p class="section-gdpr-text">Capture the locations where social sites might correlate your users.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon7.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">CRM </h4>
<p class="section-gdpr-text">Focus on the dynamic sites that are most likely to have customer data.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon8.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Fraud Investigation</h4>
<p class="section-gdpr-text">Fast forward fraud investigations by locating sites of interest.</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
<div class="container">
<div class="row">
<!-- Row 3 -->
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon9.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">SEO</h4>
<p class="section-gdpr-text">See where your marketing team might be retargeting your users.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon10.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Marketing Automation</h4>
<p class="section-gdpr-text">Find where your marketing team might be gathering contacts.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon11.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Outdated Services</h4>
<p class="section-gdpr-text">If you aren't patching you aren't compliant.</p>
</div>
<div class="col-lg-3 col-md-6 col-sm-6 text section-gdpr-single wow fadeInLeft ">
<img src="homepage/assets/img/icons/icon12.svg" class="section-gdpr-icon">
<h4 class="section-gdpr-heading2">Forms</h4>
<p class="section-gdpr-text">Are you collecting customer data and on which websites?</p>
</div>
</div>
</div>
<div class="section-gdrpr-border"></div>
</div>
<div class="footer">
<section class="container">
<div class="row ">
<div class="col-lg-2 col-md-2 col-sm-2 col-12 "><img src="homepage/assets/img/Asset51x.png" alt="Bit Discovery" class="footer-image" srcset=""></div>
<div class="col-lg-7 col-md-7 col-sm-7 col-12" style="padding-top: 1%;">
<div class="row" data-testid="footerSection">
<div class="footer-link">
<a href="/about" class="footer-a" data-testid="aboutLink">About</a>
</div>
<div class="footer-link">
<a href="https://blog.bitdiscovery.com" class="footer-a" data-testid="blogLink">Blog</a>
</div>
<div class="footer-link">
<a href="/faq" class="footer-a" data-testid="faqLink">FAQ</a>
</div>
<div class="footer-link">
<a href="/contact" class="footer-a" data-testid="contactLink">Contact</a>
</div>
<div class="footer-link">
<a href="/press" class="footer-a" data-testid="pressLink">Press</a>
</div>
<div class="footer-link">
<a href="/terms" class="footer-a" data-testid="termsLink">Terms</a>
</div>
<div class="footer-link">
<a href="/lexicon" class="footer-a" data-testid="lexiconLink">Lexicon</a>
</div>
<div class="footer-link">
<a href="/disclosure_policy" class="footer-a" data-testid="policyLink">Disclosure Policy</a>
</div>
</div>
</div>
<div class="col-lg-3 col-md-3 col-sm-3 col-12"style="padding-top: 1%;">
<div class="footer-link-copy">Copyright © 2020 Bit Discovery</div>
</div>
</div>
</section>
</div>
<!-- Javascript -->
<script src="homepage/assets/js/jquery-1.11.1.min.js"></script>
<script src="homepage/assets/bootstrap/js/bootstrap.min.js"></script>
<script src="homepage/assets/js/jquery.backstretch.min.js"></script>
<script src="homepage/assets/js/wow.min.js"></script>
<script src="homepage/assets/js/retina-1.1.0.min.js"></script>
<script src="homepage/assets/js/scripts.js"></script>
<script src="https://cdn.jsdelivr.net/npm/typed.js@2.0.11"></script>
<!--[if lt IE 10]>
<script src="homepage/assets/js/placeholder.js"></script>
<![endif]-->
<script>
var typed = new Typed('.type', {
strings: [
'your attack surface',
'forgotten assets',
'login forms',
'all of your ASNs',
'open ports',
'which assets set cookies',
'web servers',
'spelling errors',
'programming languages',
'by geographic location',
'all of your CMSs',
'cloud hosted assets',
'running services',
'expired TLS certificates',
'Javascript frameworks',
'more.'
],
typeSpeed: 38,
backSpeed: 55,
loop: true
});
</script>
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-114279261-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-114279261-1');
</script>
</body>
</html>

BIN
tests/data/new-test/aquatone-results/screenshots/http__184_72_14_217__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
tests/data/new-test/aquatone-results/screenshots/http__52_8_166_143__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
tests/data/new-test/aquatone-results/screenshots/http__staging_bitdiscovery_com__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 111 KiB

BIN
tests/data/new-test/aquatone-results/screenshots/https__184_72_14_217__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
tests/data/new-test/aquatone-results/screenshots/https__52_8_166_143__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
tests/data/new-test/aquatone-results/screenshots/https__staging_bitdiscovery_com__42099b4af021e53f.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 104 KiB

1
tests/data/new-test/corscanner-results/corscanner.json Normal file
View File

@@ -0,0 +1 @@
[]

0
tests/data/recon-results/gobuster-results/gobuster.http_104.20.60.51:443.txt → tests/data/new-test/gobuster-results/gobuster.http_184.72.14.217.txt
View File

0
tests/data/recon-results/gobuster-results/gobuster.http_104.20.60.51:8443.txt → tests/data/new-test/gobuster-results/gobuster.http_52.8.166.143.txt
View File

0
tests/data/recon-results/gobuster-results/gobuster.http_104.20.61.51:443.txt → tests/data/new-test/gobuster-results/gobuster.http_staging.bitdiscovery.com.txt
View File

7
tests/data/new-test/gobuster-results/gobuster.https_184.72.14.217.txt Normal file
View File

@@ -0,0 +1,7 @@
https://184.72.14.217/css (Status: 301)
https://184.72.14.217/dist (Status: 301)
https://184.72.14.217/fonts (Status: 301)
https://184.72.14.217/homepage (Status: 301)
https://184.72.14.217/images (Status: 301)
https://184.72.14.217/js (Status: 301)
https://184.72.14.217/robots.txt (Status: 200)

7
tests/data/new-test/gobuster-results/gobuster.https_52.8.166.143.txt Normal file
View File

@@ -0,0 +1,7 @@
https://52.8.166.143/css (Status: 301)
https://52.8.166.143/dist (Status: 301)
https://52.8.166.143/fonts (Status: 301)
https://52.8.166.143/homepage (Status: 301)
https://52.8.166.143/images (Status: 301)
https://52.8.166.143/js (Status: 301)
https://52.8.166.143/robots.txt (Status: 200)

31
tests/data/new-test/gobuster-results/gobuster.https_staging.bitdiscovery.com.txt Normal file
View File

@@ -0,0 +1,31 @@
https://staging.bitdiscovery.com/ADMIN (Status: 403)
https://staging.bitdiscovery.com/Admin (Status: 403)
https://staging.bitdiscovery.com/About (Status: 200)
https://staging.bitdiscovery.com/Contact (Status: 200)
https://staging.bitdiscovery.com/FAQ (Status: 200)
https://staging.bitdiscovery.com/Login (Status: 302)
https://staging.bitdiscovery.com/Press (Status: 200)
https://staging.bitdiscovery.com/Privacy (Status: 200)
https://staging.bitdiscovery.com/about (Status: 200)
https://staging.bitdiscovery.com/admin (Status: 403)
https://staging.bitdiscovery.com/api (Status: 301)
https://staging.bitdiscovery.com/api/experiments (Status: 200)
https://staging.bitdiscovery.com/api/experiments/configurations (Status: 200)
https://staging.bitdiscovery.com/contact (Status: 200)
https://staging.bitdiscovery.com/css (Status: 301)
https://staging.bitdiscovery.com/dist (Status: 301)
https://staging.bitdiscovery.com/docs (Status: 301)
https://staging.bitdiscovery.com/faq (Status: 200)
https://staging.bitdiscovery.com/fonts (Status: 301)
https://staging.bitdiscovery.com/homepage (Status: 301)
https://staging.bitdiscovery.com/images (Status: 301)
https://staging.bitdiscovery.com/js (Status: 301)
https://staging.bitdiscovery.com/login (Status: 302)
https://staging.bitdiscovery.com/press (Status: 200)
https://staging.bitdiscovery.com/pricing (Status: 200)
https://staging.bitdiscovery.com/privacy (Status: 200)
https://staging.bitdiscovery.com/register (Status: 200)
https://staging.bitdiscovery.com/robots.txt (Status: 200)
https://staging.bitdiscovery.com/terms (Status: 200)
https://staging.bitdiscovery.com/unsubscribe (Status: 200)
https://staging.bitdiscovery.com/user (Status: 302)

9
tests/data/new-test/masscan-results/masscan.json Normal file
View File

@@ -0,0 +1,9 @@
[
{ "ip": "184.72.14.217", "timestamp": "1586185051", "ports": [ {"port": 443, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 237} ] }
,
{ "ip": "52.8.166.143", "timestamp": "1586185054", "ports": [ {"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 237} ] }
,
{ "ip": "52.8.166.143", "timestamp": "1586185055", "ports": [ {"port": 443, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 237} ] }
,
{ "ip": "184.72.14.217", "timestamp": "1586185056", "ports": [ {"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 237} ] }
]

4
tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp.gnmap Normal file
View File

@@ -0,0 +1,4 @@
# Nmap 7.80 scan initiated Mon Apr 6 09:58:17 2020 as: nmap --open -sT -n -sC -T 4 -sV -Pn -p 443,80 -oA /home/epi/PycharmProjects/recon-pipeline/tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp 184.72.14.217
Host: 184.72.14.217 () Status: Up
Host: 184.72.14.217 () Ports: 80/open/tcp//http//nginx 1.16.1/, 443/open/tcp//ssl|http//nginx 1.16.1/
# Nmap done at Mon Apr 6 09:58:31 2020 -- 1 IP address (1 host up) scanned in 14.16 seconds

19
tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp.nmap Normal file
View File

@@ -0,0 +1,19 @@
# Nmap 7.80 scan initiated Mon Apr 6 09:58:17 2020 as: nmap --open -sT -n -sC -T 4 -sV -Pn -p 443,80 -oA /home/epi/PycharmProjects/recon-pipeline/tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp 184.72.14.217
Nmap scan report for 184.72.14.217
Host is up (0.041s latency).
PORT STATE SERVICE VERSION
80/tcp open http nginx 1.16.1
|_http-server-header: nginx/1.16.1
|_http-title: Did not follow redirect to https://184.72.14.217/
443/tcp open ssl/http nginx 1.16.1
|_http-server-header: nginx/1.16.1
|_http-title: Site doesn't have a title (text/html; charset=utf-8).
| ssl-cert: Subject: commonName=bitdiscovery.com
| Subject Alternative Name: DNS:bitdiscovery.com, DNS:*.bitdiscovery.com
| Not valid before: 2019-07-20T00:00:00
|_Not valid after: 2020-08-20T12:00:00
|_ssl-date: TLS randomness does not represent time
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
# Nmap done at Mon Apr 6 09:58:31 2020 -- 1 IP address (1 host up) scanned in 14.16 seconds

90
tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp.xml Normal file
View File

@@ -0,0 +1,90 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
<?xml-stylesheet href="file:///usr/bin/../share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 7.80 scan initiated Mon Apr 6 09:58:17 2020 as: nmap -&#45;open -sT -n -sC -T 4 -sV -Pn -p 443,80 -oA /home/epi/PycharmProjects/recon-pipeline/tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp 184.72.14.217 -->
<nmaprun scanner="nmap" args="nmap -&#45;open -sT -n -sC -T 4 -sV -Pn -p 443,80 -oA /home/epi/PycharmProjects/recon-pipeline/tests/data/new-test/nmap-results/nmap.184.72.14.217-tcp 184.72.14.217" start="1586185097" startstr="Mon Apr 6 09:58:17 2020" version="7.80" xmloutputversion="1.04">
<scaninfo type="connect" protocol="tcp" numservices="2" services="80,443"/>
<verbose level="0"/>
<debugging level="0"/>
<host starttime="1586185097" endtime="1586185111"><status state="up" reason="user-set" reason_ttl="0"/>
<address addr="184.72.14.217" addrtype="ipv4"/>
<hostnames>
</hostnames>
<ports><port protocol="tcp" portid="80"><state state="open" reason="syn-ack" reason_ttl="0"/><service name="http" product="nginx" version="1.16.1" method="probed" conf="10"><cpe>cpe:/a:igor_sysoev:nginx:1.16.1</cpe></service><script id="http-server-header" output="nginx/1.16.1"><elem>nginx/1.16.1</elem>
</script><script id="http-title" output="Did not follow redirect to https://184.72.14.217/"><elem key="redirect_url">https://184.72.14.217/</elem>
</script></port>
<port protocol="tcp" portid="443"><state state="open" reason="syn-ack" reason_ttl="0"/><service name="http" product="nginx" version="1.16.1" tunnel="ssl" method="probed" conf="10"><cpe>cpe:/a:igor_sysoev:nginx:1.16.1</cpe></service><script id="http-server-header" output="nginx/1.16.1"><elem>nginx/1.16.1</elem>
</script><script id="http-title" output="Site doesn&apos;t have a title (text/html; charset=utf-8)."></script><script id="ssl-cert" output="Subject: commonName=bitdiscovery.com&#xa;Subject Alternative Name: DNS:bitdiscovery.com, DNS:*.bitdiscovery.com&#xa;Not valid before: 2019-07-20T00:00:00&#xa;Not valid after: 2020-08-20T12:00:00"><table key="subject">
<elem key="commonName">bitdiscovery.com</elem>
</table>
<table key="issuer">
<elem key="organizationName">Amazon</elem>
<elem key="organizationalUnitName">Server CA 1B</elem>
<elem key="countryName">US</elem>
<elem key="commonName">Amazon</elem>
</table>
<table key="pubkey">
<elem key="modulus">userdata: 0x55d754669f38</elem>
<elem key="exponent">userdata: 0x55d754669ef8</elem>
<elem key="type">rsa</elem>
<elem key="bits">2048</elem>
</table>
<table key="extensions">
<table>
<elem key="value">keyid:59:A4:66:06:52:A0:7B:95:92:3C:A3:94:07:27:96:74:5B:F9:3D:D0&#xa;</elem>
<elem key="name">X509v3 Authority Key Identifier</elem>
</table>
<table>
<elem key="value">A0:5F:63:1A:0F:FD:66:25:49:F2:4B:FD:8A:41:14:95:B8:3F:7A:96</elem>
<elem key="name">X509v3 Subject Key Identifier</elem>
</table>
<table>
<elem key="value">DNS:bitdiscovery.com, DNS:*.bitdiscovery.com</elem>
<elem key="name">X509v3 Subject Alternative Name</elem>
</table>
<table>
<elem key="value">Digital Signature, Key Encipherment</elem>
<elem key="name">X509v3 Key Usage</elem>
<elem key="critical">true</elem>
</table>
<table>
<elem key="value">TLS Web Server Authentication, TLS Web Client Authentication</elem>
<elem key="name">X509v3 Extended Key Usage</elem>
</table>
<table>
<elem key="value">&#xa;Full Name:&#xa; URI:http://crl.sca1b.amazontrust.com/sca1b.crl&#xa;</elem>
<elem key="name">X509v3 CRL Distribution Points</elem>
</table>
<table>
<elem key="value">Policy: 2.16.840.1.114412.1.2&#xa;Policy: 2.23.140.1.2.1&#xa;</elem>
<elem key="name">X509v3 Certificate Policies</elem>
</table>
<table>
<elem key="value">OCSP - URI:http://ocsp.sca1b.amazontrust.com&#xa;CA Issuers - URI:http://crt.sca1b.amazontrust.com/sca1b.crt&#xa;</elem>
<elem key="name">Authority Information Access</elem>
</table>
<table>
<elem key="value">CA:FALSE</elem>
<elem key="name">X509v3 Basic Constraints</elem>
<elem key="critical">true</elem>
</table>
<table>
<elem key="value">Signed Certificate Timestamp:&#xa; Version : v1 (0x0)&#xa; Log ID : EE:4B:BD:B7:75:CE:60:BA:E1:42:69:1F:AB:E1:9E:66:&#xa; A3:0F:7E:5F:B0:72:D8:83:00:C4:7B:89:7A:A8:FD:CB&#xa; Timestamp : Jul 20 00:04:21.981 2019 GMT&#xa; Extensions: none&#xa; Signature : ecdsa-with-SHA256&#xa; 30:45:02:21:00:A6:C1:F0:66:CB:1D:04:A6:F5:15:7A:&#xa; F2:44:95:49:7C:C0:A9:03:B0:77:5B:D4:FE:75:63:89:&#xa; F3:0D:E2:67:8B:02:20:0E:1C:FF:AC:13:CC:5B:F4:F3:&#xa; B9:8C:67:0A:B2:CB:53:9B:0C:81:5D:55:EB:F2:98:95:&#xa; E4:BE:D1:36:03:CA:B0&#xa;Signed Certificate Timestamp:&#xa; Version : v1 (0x0)&#xa; Log ID : 87:75:BF:E7:59:7C:F8:8C:43:99:5F:BD:F3:6E:FF:56:&#xa; 8D:47:56:36:FF:4A:B5:60:C1:B4:EA:FF:5E:A0:83:0F&#xa; Timestamp : Jul 20 00:04:22.059 2019 GMT&#xa; Extensions: none&#xa; Signature : ecdsa-with-SHA256&#xa; 30:44:02:20:2B:09:09:AF:DA:0F:1B:C4:13:77:23:28:&#xa; 30:A9:E3:14:8A:24:BD:02:C1:A1:CE:2D:F3:D7:6F:D8:&#xa; F3:AA:24:DF:02:20:7B:09:12:A8:58:8E:89:49:98:7E:&#xa; D8:23:0C:44:77:FE:39:B1:87:B8:5B:6A:B7:12:CF:90:&#xa; FE:B2:3E:4C:C0:C0</elem>
<elem key="name">CT Precertificate SCTs</elem>
</table>
</table>
<elem key="sig_algo">sha256WithRSAEncryption</elem>
<table key="validity">
<elem key="notBefore">2019-07-20T00:00:00</elem>
<elem key="notAfter">2020-08-20T12:00:00</elem>
</table>
<elem key="md5">9572e725db7c5e017a162de8286f90d1</elem>
<elem key="sha1">08c6544e100adcda3171129aec035ffb64fdadcd</elem>
<elem key="pem">-&#45;&#45;&#45;&#45;BEGIN CERTIFICATE-&#45;&#45;&#45;&#45;&#xa;MIIFfDCCBGSgAwIBAgIQBEGHqnpM4WHyF80JPSOj9jANBgkqhkiG9w0BAQsFADBG&#xa;MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg&#xa;Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0xOTA3MjAwMDAwMDBaFw0yMDA4MjAx&#xa;MjAwMDBaMBsxGTAXBgNVBAMTEGJpdGRpc2NvdmVyeS5jb20wggEiMA0GCSqGSIb3&#xa;DQEBAQUAA4IBDwAwggEKAoIBAQDdaJv3t5RKD0o2SK6PTz6tfw2o32lUS/OsEnt1&#xa;tbwtO1SaKXLErBL2BVxtOI9uG7OOkuADxnCr9TmOXF7enxNyVo4MiSUQcCfPQHxq&#xa;WY5KOPpzySmM5cNhglqyMlVftpxMZlaeQEBr8YZqQRlA/d0ep7Dgz5TrP7apQwpX&#xa;dU1b2lVM5ul5+xVHRmz0sBpfjwVUqTQsiEw1A0meKW/HM/l11uxJV1ucMU1Ybqeh&#xa;GvvIM+Yu/kw5fxzOmnBuMOBrvYN9vOnfy1F+FOuVOnFsGH+Ko6V3lvgIi4HR5ZPT&#xa;UvtSXt51opqBgRMrWGb9IO9gYS1V7L2tlxNo6aMqevc3oWIzAgMBAAGjggKPMIIC&#xa;izAfBgNVHSMEGDAWgBRZpGYGUqB7lZI8o5QHJ5Z0W/k90DAdBgNVHQ4EFgQUoF9j&#xa;Gg/9ZiVJ8kv9ikEUlbg/epYwLwYDVR0RBCgwJoIQYml0ZGlzY292ZXJ5LmNvbYIS&#xa;Ki5iaXRkaXNjb3ZlcnkuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr&#xa;BgEFBQcDAQYIKwYBBQUHAwIwOwYDVR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5z&#xa;Y2ExYi5hbWF6b250cnVzdC5jb20vc2NhMWIuY3JsMCAGA1UdIAQZMBcwCwYJYIZI&#xa;AYb9bAECMAgGBmeBDAECATB1BggrBgEFBQcBAQRpMGcwLQYIKwYBBQUHMAGGIWh0&#xa;dHA6Ly9vY3NwLnNjYTFiLmFtYXpvbnRydXN0LmNvbTA2BggrBgEFBQcwAoYqaHR0&#xa;cDovL2NydC5zY2ExYi5hbWF6b250cnVzdC5jb20vc2NhMWIuY3J0MAwGA1UdEwEB&#xa;/wQCMAAwggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdgDuS723dc5guuFCaR+r4Z5m&#xa;ow9+X7By2IMAxHuJeqj9ywAAAWwMsZtdAAAEAwBHMEUCIQCmwfBmyx0EpvUVevJE&#xa;lUl8wKkDsHdb1P51Y4nzDeJniwIgDhz/rBPMW/TzuYxnCrLLU5sMgV1V6/KYleS+&#xa;0TYDyrAAdQCHdb/nWXz4jEOZX73zbv9WjUdWNv9KtWDBtOr/XqCDDwAAAWwMsZur&#xa;AAAEAwBGMEQCICsJCa/aDxvEE3cjKDCp4xSKJL0CwaHOLfPXb9jzqiTfAiB7CRKo&#xa;WI6JSZh+2CMMRHf+ObGHuFtqtxLPkP6yPkzAwDANBgkqhkiG9w0BAQsFAAOCAQEA&#xa;AGZNFA2FQpNDjFtg7K/kcEWNYUuRMqRUlo1lmYXrvTUqljjK2tPKP7h3lNQZjCn+&#xa;KQn3hKq9Yd8164OXi3uV2DmN02/I/woEY76aJphon1F/MZJvoXxzmbjyEg1xsAIH&#xa;2hBWVJYouRybc26Ved6KH/GoFOnxinNCQgMXaJkxKtINWKSRJ7ymUInp6fjiHVIg&#xa;utQ5pCH6RhlKVc1zbhY9Ro4HumYWkb83kV5H5BSqHp9160XkSN37d89Z/CPJ5in7&#xa;2mYXc4LHEoUA/o5NP1Usyrupx7/n5o9aNlxzPRG8gi/cYueJwWIBLCPdy2pJ3ypd&#xa;ioTUc9q3PkZADISX9z9BIg==&#xa;-&#45;&#45;&#45;&#45;END CERTIFICATE-&#45;&#45;&#45;&#45;&#xa;</elem>
</script><script id="ssl-date" output="TLS randomness does not represent time"></script></port>
</ports>
<times srtt="40578" rttvar="30440" to="162338"/>
</host>
<runstats><finished time="1586185111" timestr="Mon Apr 6 09:58:31 2020" elapsed="14.16" summary="Nmap done at Mon Apr 6 09:58:31 2020; 1 IP address (1 host up) scanned in 14.16 seconds" exit="success"/><hosts up="1" down="0" total="1"/>
</runstats>
</nmaprun>

Some files were not shown because too many files have changed in this diff Show More