Commit Graph

104 Commits

Author SHA1 Message Date
epi
973893ee42 removed dependency on tool-dict.pkl (#93)
* removed tool_dict dependency

* updated tests

* updated go version

* added defaults for failing iteration during tool installation

* Update pythonapp.yml

* updated docs
2020-08-27 20:40:15 -05:00
Ryan Good
d7dbd1e7b3 Dependency Checking (#75)
* Adds req testing methodology, needs fixes

* Improves dependency exception handling

* Better meets_requirements implementation

Still need to adjust tests to fake installation

* Changes to exception boolean to enable tool check

tests and class variables modified for new tool check

* Adjust test_get_scans to use appropriate variable

* Adds Go requirement where relevant

* Adds missing scan dependencies

* Add clarification to error message
v1.0.0
2020-08-07 08:48:49 -05:00
epi
d97315a2da update go version in tests 2020-08-07 06:25:27 -05:00
epi
582ec4f208 update go version 2020-08-06 18:53:04 -05:00
bamhm182
f4162631df Added Dockerfile. Updated docs accordingly. #62 (#78) v0.11.5 2020-07-27 09:45:12 -05:00
epi052
2f2f5e7a7f corrected searchsploit out of index list error (#77) v0.11.4 2020-07-06 20:08:33 -05:00
epi052
4432990966 added support for 32bit go installations (#76) v0.11.3 2020-07-06 18:18:33 -05:00
epi052
4d1aef2d34 Limit tab completion of scan command to scans with installed components (#74)
* scans without installed requirements dont tab-complete

* added meets_requirements functions to classes

* updated tests

* added check for None case
v0.11.2
2020-06-28 15:33:34 -05:00
epi052
6ed51a19be Added tools command asciicast 2020-06-28 07:54:28 -05:00
epi052
80ce9ebb27 Update tool install asciicast
shows new tools command with subcommands
2020-06-28 07:52:47 -05:00
epi052
9d5cac6b34 WIP: added tools command; closes #44 (#60)
* added tools command with placeholders for un/reinstall along with placeholder tests

* added missing docs build dependency

* updated documentation to reflect tools vs install

* refactored some code for DRY, fixed up prior merge with master

* fixed broken tests in test_recon_pipeline_shell

* existing tests all passing

* added tools list command

* added tools list command

* added tools reinstall

* removed lint

* fixed reinstall test

* fixed install go test

* fixed go install test again
v0.11.1
2020-06-27 21:23:16 -05:00
Ryan Good
1ad3adca82 Uninstall command (#66)
* Add do_uninstall function

* uninstall f/ amass/aqua/go

* uninstall functional on all tools

* Add in missed fixes from rebase

* solve permission issue

* Removes un-needed vars from yaml

* Resolves go test issues

* adds framework for uninstall tests

* Fixes uninstall tests for Go tools

* Adds uninstall testing for luigi and improves uninstall

* Adds uninstall testing for searchsploit

* Update installation documentation
v0.11.0
2020-06-26 19:04:43 -05:00
epi052
9e863a2260 Troublehoot pipeline tool tests (#71)
* changed go version
2020-06-13 18:55:42 -05:00
epi052
7e261a16e4 institued fix from discussion; closes #68 (#70) v0.10.4 2020-06-13 15:47:09 -05:00
epi052
fce38c22de added tests for new ci stage; restructured go tool installs; closes #63; closes #46 (#64) v0.10.3 2020-05-12 14:02:29 -05:00
epi
9a64f4c50b fixed lint getting caught on CI but not locally 2020-05-11 22:21:37 -05:00
Ryan Good
c8bb606ecc Modifies reference structure for tool paths (#61) v0.10.2 2020-05-11 20:07:15 -05:00
epi052
1448cd037f Update requirements.txt 2020-05-04 20:06:40 -05:00
epi
6b8d08ef61 updated install test case 2020-05-04 15:40:57 -05:00
epi
ff49162fa4 Merge branch 'master' of github.com:epi052/recon-pipeline v0.10.1 2020-05-04 13:13:19 -05:00
epi
08b3b59e50 added default value to --results-dir 2020-05-04 13:11:51 -05:00
epi052
a0fdbcd8e5 Update pull_request_template.md 2020-05-03 13:37:00 -05:00
epi052
e6997e17b6 Added docs on how to add a new scanner & tool definition; closes #43 (#59) 2020-05-03 12:18:16 -05:00
epi052
f556319453 WIP: add waybackurls scan (#56)
* fixed up config.defaults definition

tools-dir and database-dir now use defaults.home value

* added tool definition file; closes #54

* added basic PoC for waybackurls scanner; updated helpers.py test

* added Endpoint/Target parsing; updated existing tests to pass

* added tests for waybackurls

* added WaybackurlsScan to FullScan

* added documenation for WaybackurlsScan
v0.10.0
2020-05-02 20:06:44 -05:00
Ryan Good
078fdaada7 Changes commands to be more uniform/readable (#58) 2020-05-01 20:51:35 -05:00
epi052
2ecdf4319a Prompt user on existing directory (#57)
* added tests and check_scan_directory

* added documentation; closes #30
v0.9.3
2020-04-30 10:19:56 -05:00
epi052
f1c1868a6e correct minor oversight with tko-sub definition v0.9.2 2020-04-29 21:46:10 -05:00
epi
5b56035391 added PyYaml to Pipfile, was only present in --dev installs previously 2020-04-29 19:32:04 -05:00
epi052
aa7796f793 added black to dev reqs and pinned all reqs; closes #51 and #37 (#52) 2020-04-29 07:05:52 -05:00
epi052
08119e2079 Added apt update to installation instructions; closes #38 (#50) 2020-04-28 08:28:55 -05:00
epi
4a8b134f77 removed erroneous metadata from pull request template 2020-04-28 08:27:06 -05:00
epi
7732d5d9ce added pull request template 2020-04-28 08:25:29 -05:00
epi052
f9120796b0 updated docs/readme to show new tested OS for install; closes #40 (#49) 2020-04-28 08:16:20 -05:00
epi052
6a55cb169c Update issue templates 2020-04-28 06:55:16 -05:00
epi052
75a7ed0546 marked html files in tests/data as python for linguist; closes #39 (#48) 2020-04-27 10:28:47 -05:00
Ryan Good
db1dbb8e2b pivot to yaml structure for tool definitions (#33) v0.9.1 2020-04-25 15:00:02 -05:00
epi052
5e7053fd33 Update CI to run on pull requests 2020-04-25 14:53:59 -05:00
epi
8bd43d371a removed dependency badges; added view command video to docs 2020-04-17 11:07:17 -05:00
epi
af03de34dc updated sqlalchemy badge 2020-04-17 10:55:27 -05:00
epi052
6eb3bd8cb0 Completed store results in a database project (#32)
Co-authored-by: Ryan Good <usafaryangood@gmail.com>

* added initial skeleton; restructured project directories

* removed workers directive from luigi; changed input to tko-subs

* changed masscan command to use config.tool_paths

* linted __init__ files and updated docstring for get_scans

* added per-file-ignores for linting

* recon-pipeline linted

* PoC working for amass results -> db; rudimentary db mgmt commands also

* more linting

* added database management commands to the shell

* db_location passes through to all tasks; masscan results added to db

* removed unused imports from masscan.py

* added ParseNmapOutput class to handle parsing for database storage

* cleaned up repeat code

* searchsploit results stored in db

* lint/format

* gobuster scans now stored in database

* fixed test_recon tests to use db_location

* fixed web tests

* tkosub entries recorded in db

* subjack scan results stored in database

* webanalyze results stored in db

* refactored older commits to use newer helper functions

* refactored older commits to use newer helper functions

* aquatone results stored in database

refactored a few scans to use dbmanager helper functions
refactored db structure wrt headers/screenshots
added 80/443 to web_ports in config.py

* fixed a few queries and re-added webanalyze to FullScan

* view targets/endpoints done

* overhauled nmap parsing

* print all nmap_results good, next to focus on filtering

* complex nmap filters complete

* nmap printing done

* updated pipfile

* view web-technologies complete

* view searchsploit results complete

* removed filesystem code from amass

* targetlist moved to db only

* targets,amass,masscan all cutover to full database; added view ports

* nmap fully db compliant

* aquatone and webtargets db compliant

* gobuster uses db now

* webanalyze db compliant

* all scans except corscanner are db compliant

* recon tests passing

* web tests passing

* linted files

* added tests for helpers.py and parsers.py

* refactored some redundant code

* added tests to pre-commit

* updated amass tests and pre-commit version

* updated recon.targets tests

* updated nmap tests

* updated masscan tests

* updated config tests

* updated web targets tests

* added gobuster tests

* added aquatone tests

* added subdomain takeover and webanalyze tests; updated test data

* removed homegrown sqlite target in favor of the sqla implementation

* added tests for recon-pipeline.py

* fixed cluge function to set __package__ globally

* updated amass tests

* updated targets tests

* updated nmap tests

* updated masscan tests

* updated aquatone tests

* updated nmap tests to account for no searchsploit

* updated nmap tests to account for no searchsploit

* updated masscan tests

* updated subjack/tkosub tests

* updated web targets tests

* updated webanalyze tests

* added corscanner tests

* linted DBManager a bit

* fixed weird cyclic import issue that only happened during docs build; housekeeping

* added models tests, removed test_install dir

* updated docs a bit; sidenav is wonky

* fixed readthedocs requirements.txt

* fixed issue where view results werent populated directly after scan

* added new tests to pipeline; working on docs

* updated a few overlooked view command items

* updated tests to reflect changes to shell

* incremental push of docs update

* documentation done

* updated exploitdb install

* updated exploitdb install

* updated seclists install

* parseamass updates db in the event of no amass output

* removed corscanner

* added pipenv shell to install instructions per @GreaterGoodest

* added pipenv shell to install instructions per @GreaterGoodest

* added check for chromium-browser during aquatone install; closes #26

* added check for old recon-tools dir; updated Path.resolve calls to Path.expanduser.resolve; fixed very specific import bug due to filesystem location

* added CONTIBUTING.md; updated pre-commit hooks/README

* added .gitattributes for linguist reporting

* updated tests

* fixed a few weird bugs found during test

* updated README

* updated asciinema links in README

* updated README with view command video

* updated other location for url scheme /status

* add ability to specify single target using --target (#31)

* updated a few items in docs and moved tool-dict to tools-dir

* fixed issue where removing tempfile without --verbose caused scan to fail
v0.9.0
2020-04-17 10:29:16 -05:00
epi052
ff801dfc6b added searchsploit to installable tools (#25)
* added searchsploit to installable tools

* formatted with black
v0.8.4
2020-02-22 12:20:13 -06:00
epi052
6613ab69a9 install masscan now adds CAP_NET_RAW linux capability to masscan (#23)
closes #21
v0.8.3
2020-02-20 19:10:59 -06:00
epi052
d1b5470858 Fixed status default value (#19)
On ubuntu, localhost wasn't directing the browswer properly, resulting in a blank URL line.  Default is now 127.0.0.1
v0.8.2
2020-02-09 06:16:49 -06:00
epi052
4277e10414 Add visualization docs (#18)
* added status command; updated install instructions; fixed typo

* #17 - added visualization docs
2020-02-09 06:09:25 -06:00
epi052
1828fa8f35 added status command; updated install instructions; fixed typo (#16) 2020-02-08 06:27:18 -06:00
epi
b92aef58c9 restored Pipfile 2020-02-07 22:58:14 -06:00
epi052
b0534ceb27 Remove OS assumption (#15)
* removed kali specific installs - test 1

* removed kali specific installs - test 2

* removed kali specific installs - test 2

* removed kali specific installs - test 4

* removed kali specific installs - test 5

* removed kali specific installs - test 6

* removed kali specific installs - test 7

* removed kali specific installs - test 8

* removed kali specific installs - test 9

* removed kali specific installs - test 10

* removed kali specific installs - test 11

* removed kali specific installs - test 12

* removed kali specific installs - test 13

* removed kali specific installs - test 14

* all works locally, fixing up tests et al

* trying pipeline again; round 2

* looks good locally; testing pipeline again

* test 18?

* test 19

* 20

* 21
v0.8.1
2020-02-07 22:54:15 -06:00
epi052
25da9574e3 Update README.md 2020-02-07 21:59:08 -06:00
epi052
82e4d3335a Update README.md 2020-02-07 21:58:18 -06:00
epi052
3ea2cc7399 added status command; added --sausage to scan command v0.8.0 2020-02-04 20:55:49 -06:00