Completed store results in a database project (#32)

Co-authored-by: Ryan Good <usafaryangood@gmail.com> * added initial skeleton; restructured project directories * removed workers directive from luigi; changed input to tko-subs * changed masscan command to use config.tool_paths * linted __init__ files and updated docstring for get_scans * added per-file-ignores for linting * recon-pipeline linted * PoC working for amass results -> db; rudimentary db mgmt commands also * more linting * added database management commands to the shell * db_location passes through to all tasks; masscan results added to db * removed unused imports from masscan.py * added ParseNmapOutput class to handle parsing for database storage * cleaned up repeat code * searchsploit results stored in db * lint/format * gobuster scans now stored in database * fixed test_recon tests to use db_location * fixed web tests * tkosub entries recorded in db * subjack scan results stored in database * webanalyze results stored in db * refactored older commits to use newer helper functions * refactored older commits to use newer helper functions * aquatone results stored in database refactored a few scans to use dbmanager helper functions refactored db structure wrt headers/screenshots added 80/443 to web_ports in config.py * fixed a few queries and re-added webanalyze to FullScan * view targets/endpoints done * overhauled nmap parsing * print all nmap_results good, next to focus on filtering * complex nmap filters complete * nmap printing done * updated pipfile * view web-technologies complete * view searchsploit results complete * removed filesystem code from amass * targetlist moved to db only * targets,amass,masscan all cutover to full database; added view ports * nmap fully db compliant * aquatone and webtargets db compliant * gobuster uses db now * webanalyze db compliant * all scans except corscanner are db compliant * recon tests passing * web tests passing * linted files * added tests for helpers.py and parsers.py * refactored some redundant code * added tests to pre-commit * updated amass tests and pre-commit version * updated recon.targets tests * updated nmap tests * updated masscan tests * updated config tests * updated web targets tests * added gobuster tests * added aquatone tests * added subdomain takeover and webanalyze tests; updated test data * removed homegrown sqlite target in favor of the sqla implementation * added tests for recon-pipeline.py * fixed cluge function to set __package__ globally * updated amass tests * updated targets tests * updated nmap tests * updated masscan tests * updated aquatone tests * updated nmap tests to account for no searchsploit * updated nmap tests to account for no searchsploit * updated masscan tests * updated subjack/tkosub tests * updated web targets tests * updated webanalyze tests * added corscanner tests * linted DBManager a bit * fixed weird cyclic import issue that only happened during docs build; housekeeping * added models tests, removed test_install dir * updated docs a bit; sidenav is wonky * fixed readthedocs requirements.txt * fixed issue where view results werent populated directly after scan * added new tests to pipeline; working on docs * updated a few overlooked view command items * updated tests to reflect changes to shell * incremental push of docs update * documentation done * updated exploitdb install * updated exploitdb install * updated seclists install * parseamass updates db in the event of no amass output * removed corscanner * added pipenv shell to install instructions per @GreaterGoodest * added pipenv shell to install instructions per @GreaterGoodest * added check for chromium-browser during aquatone install; closes #26 * added check for old recon-tools dir; updated Path.resolve calls to Path.expanduser.resolve; fixed very specific import bug due to filesystem location * added CONTIBUTING.md; updated pre-commit hooks/README * added .gitattributes for linguist reporting * updated tests * fixed a few weird bugs found during test * updated README * updated asciinema links in README * updated README with view command video * updated other location for url scheme /status * add ability to specify single target using --target (#31) * updated a few items in docs and moved tool-dict to tools-dir * fixed issue where removing tempfile without --verbose caused scan to fail
2025-12-20 15:54:25 +01:00 · 2020-04-17 10:29:16 -05:00
parent ff801dfc6b
commit 6eb3bd8cb0
4682 changed files with 133470 additions and 7368 deletions
--- a/pipeline/models/nmap_model.py
+++ b/pipeline/models/nmap_model.py
@@ -0,0 +1,77 @@
+import textwrap
+
+from sqlalchemy.orm import relationship
+from sqlalchemy import Column, Integer, ForeignKey, String, Boolean
+
+from .base_model import Base
+from .port_model import Port
+from .ip_address_model import IPAddress
+from .nse_model import nse_result_association_table
+
+
+class NmapResult(Base):
+    """ Database model that describes the TARGET.nmap scan results.
+
+        Represents nmap data.
+
+        Relationships:
+            ``target``: many to one -> :class:`pipeline.models.target_model.Target`
+
+            ``ip_address``: one to one -> :class:`pipeline.models.ip_address_model.IPAddress`
+
+            ``port``: one to one -> :class:`pipeline.models.port_model.Port`
+
+            ``nse_results``: one to many -> :class:`pipeline.models.nse_model.NSEResult`
+    """
+
+    def __str__(self):
+        return self.pretty()
+
+    def pretty(self, commandline=False, nse_results=None):
+        pad = "  "
+
+        ip_address = self.ip_address.ipv4_address or self.ip_address.ipv6_address
+
+        msg = f"{ip_address} - {self.service}\n"
+        msg += f"{'=' * (len(ip_address) + len(self.service) + 3)}\n\n"
+        msg += f"{self.port.protocol} port: {self.port.port_number} - {'open' if self.open else 'closed'} - {self.reason}\n"
+        msg += f"product: {self.product} :: {self.product_version}\n"
+        msg += f"nse script(s) output:\n"
+
+        if nse_results is None:
+            # add all nse scripts
+            for nse_result in self.nse_results:
+                msg += f"{pad}{nse_result.script_id}\n"
+                msg += textwrap.indent(nse_result.script_output, pad * 2)
+                msg += "\n"
+        else:
+            # filter used, only return those specified
+            for nse_result in nse_results:
+                if nse_result in self.nse_results:
+                    msg += f"{pad}{nse_result.script_id}\n"
+                    msg += textwrap.indent(nse_result.script_output, pad * 2)
+                    msg += "\n"
+
+        if commandline:
+            msg += f"command used:\n"
+            msg += f"{pad}{self.commandline}\n"
+
+        return msg
+
+    __tablename__ = "nmap_result"
+
+    id = Column(Integer, primary_key=True)
+    open = Column(Boolean)
+    reason = Column(String)
+    service = Column(String)
+    product = Column(String)
+    commandline = Column(String)
+    product_version = Column(String)
+
+    port = relationship(Port)
+    port_id = Column(Integer, ForeignKey("port.id"))
+    ip_address = relationship(IPAddress)
+    ip_address_id = Column(Integer, ForeignKey("ip_address.id"))
+    target_id = Column(Integer, ForeignKey("target.id"))
+    target = relationship("Target", back_populates="nmap_results")
+    nse_results = relationship("NSEResult", secondary=nse_result_association_table, back_populates="nmap_results")