Completed store results in a database project (#32)

Co-authored-by: Ryan Good <usafaryangood@gmail.com>

* added initial skeleton; restructured project directories

* removed workers directive from luigi; changed input to tko-subs

* changed masscan command to use config.tool_paths

* linted __init__ files and updated docstring for get_scans

* added per-file-ignores for linting

* recon-pipeline linted

* PoC working for amass results -> db; rudimentary db mgmt commands also

* more linting

* added database management commands to the shell

* db_location passes through to all tasks; masscan results added to db

* removed unused imports from masscan.py

* added ParseNmapOutput class to handle parsing for database storage

* cleaned up repeat code

* searchsploit results stored in db

* lint/format

* gobuster scans now stored in database

* fixed test_recon tests to use db_location

* fixed web tests

* tkosub entries recorded in db

* subjack scan results stored in database

* webanalyze results stored in db

* refactored older commits to use newer helper functions

* refactored older commits to use newer helper functions

* aquatone results stored in database

refactored a few scans to use dbmanager helper functions
refactored db structure wrt headers/screenshots
added 80/443 to web_ports in config.py

* fixed a few queries and re-added webanalyze to FullScan

* view targets/endpoints done

* overhauled nmap parsing

* print all nmap_results good, next to focus on filtering

* complex nmap filters complete

* nmap printing done

* updated pipfile

* view web-technologies complete

* view searchsploit results complete

* removed filesystem code from amass

* targetlist moved to db only

* targets,amass,masscan all cutover to full database; added view ports

* nmap fully db compliant

* aquatone and webtargets db compliant

* gobuster uses db now

* webanalyze db compliant

* all scans except corscanner are db compliant

* recon tests passing

* web tests passing

* linted files

* added tests for helpers.py and parsers.py

* refactored some redundant code

* added tests to pre-commit

* updated amass tests and pre-commit version

* updated recon.targets tests

* updated nmap tests

* updated masscan tests

* updated config tests

* updated web targets tests

* added gobuster tests

* added aquatone tests

* added subdomain takeover and webanalyze tests; updated test data

* removed homegrown sqlite target in favor of the sqla implementation

* added tests for recon-pipeline.py

* fixed cluge function to set __package__ globally

* updated amass tests

* updated targets tests

* updated nmap tests

* updated masscan tests

* updated aquatone tests

* updated nmap tests to account for no searchsploit

* updated nmap tests to account for no searchsploit

* updated masscan tests

* updated subjack/tkosub tests

* updated web targets tests

* updated webanalyze tests

* added corscanner tests

* linted DBManager a bit

* fixed weird cyclic import issue that only happened during docs build; housekeeping

* added models tests, removed test_install dir

* updated docs a bit; sidenav is wonky

* fixed readthedocs requirements.txt

* fixed issue where view results werent populated directly after scan

* added new tests to pipeline; working on docs

* updated a few overlooked view command items

* updated tests to reflect changes to shell

* incremental push of docs update

* documentation done

* updated exploitdb install

* updated exploitdb install

* updated seclists install

* parseamass updates db in the event of no amass output

* removed corscanner

* added pipenv shell to install instructions per @GreaterGoodest

* added pipenv shell to install instructions per @GreaterGoodest

* added check for chromium-browser during aquatone install; closes #26

* added check for old recon-tools dir; updated Path.resolve calls to Path.expanduser.resolve; fixed very specific import bug due to filesystem location

* added CONTIBUTING.md; updated pre-commit hooks/README

* added .gitattributes for linguist reporting

* updated tests

* fixed a few weird bugs found during test

* updated README

* updated asciinema links in README

* updated README with view command video

* updated other location for url scheme /status

* add ability to specify single target using --target (#31)

* updated a few items in docs and moved tool-dict to tools-dir

* fixed issue where removing tempfile without --verbose caused scan to fail

pipeline/recon/helpers.py

@@ -0,0 +1,68 @@
+import sys
+import inspect
+import pkgutil
+import importlib
+import ipaddress
+from pathlib import Path
+from collections import defaultdict
+
+
+def get_scans():
+    """ Iterates over the recon package and its modules to find all of the classes that end in [Ss]can.
+
+    **A contract exists here that says any scans need to end with the word scan in order to be found by this function.**
+
+    Example:
+        ``defaultdict(<class 'list'>, {'AmassScan': ['pipeline.recon.amass'], 'MasscanScan': ['pipeline.recon.masscan'], ... })``
+
+    Returns:
+        dict containing mapping of ``classname -> [modulename, ...]`` for all potential recon-pipeline commands
+    """
+    scans = defaultdict(list)
+
+    file = Path(__file__).expanduser().resolve()
+    web = file.parent / "web"
+    recon = file.parents[1] / "recon"
+
+    lib_paths = [str(web), str(recon)]
+
+    # recursively walk packages; import each module in each package
+    # walk_packages yields ModuleInfo objects for all modules recursively on path
+    # prefix is a string to output on the front of every module name on output.
+    for loader, module_name, is_pkg in pkgutil.walk_packages(path=lib_paths, prefix=f"{__package__}."):
+        try:
+            importlib.import_module(module_name)
+        except ModuleNotFoundError:
+            # skipping things like recon.aquatone, not entirely sure why they're showing up...
+            pass
+
+    # walk all modules, grabbing classes that we've written and add them to the classlist defaultdict
+    # getmembers returns all members of an object in a list of tuples (name, value)
+    for name, obj in inspect.getmembers(sys.modules[__package__]):
+        if inspect.ismodule(obj) and not name.startswith("_"):
+            # we're only interested in modules that don't begin with _ i.e. magic methods __len__ etc...
+
+            for subname, subobj in inspect.getmembers(obj):
+                if inspect.isclass(subobj) and subname.lower().endswith("scan"):
+                    # now we only care about classes that end in [Ss]can
+                    scans[subname].append(f"{__package__}.{name}")
+
+    return scans
+
+
+def is_ip_address(ipaddr):
+    """ Simple helper to determine if given string is an ip address or subnet """
+    try:
+        ipaddress.ip_interface(ipaddr)
+        return True
+    except ValueError:
+        return False
+
+
+def get_ip_address_version(ipaddr):
+    """ Simple helper to determine whether a given ip address is ipv4 or ipv6 """
+    if is_ip_address(ipaddr):
+        if isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv4Address):  # ipv4 addr
+            return "4"
+        elif isinstance(ipaddress.ip_address(ipaddr), ipaddress.IPv6Address):  # ipv6
+            return "6"