mirror of
https://github.com/aljazceru/recon-pipeline.git
synced 2025-12-24 01:34:26 +01:00
9d5cac6b34
* added tools command with placeholders for un/reinstall along with placeholder tests * added missing docs build dependency * updated documentation to reflect tools vs install * refactored some code for DRY, fixed up prior merge with master * fixed broken tests in test_recon_pipeline_shell * existing tests all passing * added tools list command * added tools list command * added tools reinstall * removed lint * fixed reinstall test * fixed install go test * fixed go install test again
19 lines
1.0 KiB
ReStructuredText
19 lines
1.0 KiB
ReStructuredText
.. _scheduler-ref-label:
|
||
|
||
Using a Scheduler
|
||
=================
|
||
|
||
The backbone of this pipeline is spotify's `luigi <https://github.com/spotify/luigi>`_ batch process management framework. Luigi uses the concept of a
|
||
scheduler in order to manage task execution. Two types of scheduler are available, a **local** scheduler and a
|
||
**central** scheduler. The local scheduler is useful for development and debugging while the central scheduler
|
||
provides the following two benefits:
|
||
|
||
- Make sure two instances of the same task are not running simultaneously
|
||
- Provide :ref:`visualization <visualization-ref-label>` of everything that’s going on
|
||
|
||
While in the ``recon-pipeline`` shell, running ``tools install luigi-service`` will copy the ``luigid.service``
|
||
file provided in the repo to its appropriate systemd location and start/enable the service. The result is that the
|
||
central scheduler is up and running easily.
|
||
|
||
The other option is to add ``--local-scheduler`` to your :ref:`scan_command` command from within the ``recon-pipeline`` shell.
|