# Dynamically Modifying Job Configurations

## Introduction

Subscriptions NG requires the "Python Job" connector for running arbitrary Python code.
A Runner job built using the Python Job connector uses Playwright to create PDF exports
of Zuar Portal dashboards; this is call the Portal Export Job.

The Subscriptions NG job uses data unique to each subscription to dynamically modify the 
behavior of the Portal Export Job.  This document, while focused on Subscriptions NG,
provides the details of how job behavior can be dynamically modified at job runtime.
This is valuable knowledge for anyone building complex Runner deployments.

This document describes the implementation of Subscriptions NG with a lengthy
explanation of how dynamic job parameters and job overrides are used to modify job
behavior based on the contents of subscriptions.

## Portal Export

### `portal_export.py`

We'll begin by looking at the core of the Portal Export implementation, which is in
[portal_export.py](
https://github.com/zuarbase/mitto-plugin-subscriptions-ng/blob/master/subscriptions_ng/export/portal_export.py).
If your instance has `mitto-plugin-subscriptions-ng` installed, this file will be
located at `/var/mitto/plugin/subscriptions_ng/export/portal_export.py`.  This purpose
of the script is to use the Zuar Portal API to create a PDF export of a dashboard.

`portal_export.py` is a standalone Python script that can be run on any system with the
necessary dependencies.  It does not rely on Runner or any Runner code in any way. This
script can be run on any system with Python and a virtualenv that contains the necessary
dependencies.

The script performs the export using a headless Chrome browser
controlled by [Playwright for Python]( https://github.com/microsoft/playwright-python).

The Python dependencies are specified in `subscriptions_ng/export/requirements.txt`.
Additionally, either a Chrome or a FireFox browser must be present; see the next section
for more information.  Depending upon your host, you may need to set an environment
variable to specify where browsers are installed.  On macOS, playwright installs
browsers in `~/Library/Caches/ms-playwright`, in which case you would need to:
`export PLAYWRIGHT_BROWSERS_PATH=~/Library/Caches/ms-playwright`.

The `portal_export.py` script accepts a single mandatory option `--job_config`.  The
format of the `--job-config` value is described [TBD/here](
[here](schemas/subscriptions_ng/export/config_portal_export/index.html).  On the command
line, the value is passed as a JSON string.  Assuming valid credentials, this would
perform an export and save it in a `portal-export.pdf` file in the current directory.

```
python3 portal_export.py --verbose --job_config '{
	"export": {
		"source_url": "https://portal-automation.zuarbase.net",
		"source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCpYweDrUo0",
        "file_path": "portal-export.pdf"
	},
	"subscription": {
		"source": "/p/runner-automation",
        "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
		"json_data": {
			"export_type": "pdf",
			"window_size": "1280x1024"
		}
	}
}'
```

### The Portal Export Job

To allow the `portal_export.py` script to be run as a Runner job, a Portal Export Job (a
Runner Python Job) was created and is installed with the Subscriptions NG connector.
The Portal Export Job is essentially a pre-configured Python Job that runs the
`portal_export.py` script.

When the `mitto-plugin-subscriptions-ng` plugin is installed, it automatically installs
the Chrome and FireFox browsers in of Runner's `webapp` and `scheduler` Docker
containers (see `install_system_dependencies.sh`).  The very first time that the Portal
Export Job runs, the Python Job automatically creates a virtualenv with the dependencies
from `subscriptions_ng/export/requirements.txt`.

Documentation for the Portal Export Job configuration is [here](
schemas/subscriptions_ng/jobs/job_portal_export/index.html).

Example job configurations:

1. For use locally with `mitto-dev-runtime`:

   ```
   {
	   "export": {
		   "source_url": "https://portal-automation.zuarbase.net",
		   "source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCAiYzRDpYweDrUo0",
		   "file_path": "/var/mitto/data/file-exported-from-portal.pdf"
	   },
	   "subscription": {
		   "source": "/p/runner-automation",
		   "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
		   "json_data": {
			   "export_type": "pdf",
			   "window_size": "1280x1024"
		   },
	   },
	   "python_job_config": {
		   "venv_dir": "/var/mitto/data/venvs/portal_export_job/venv",
		   "requirements_file_path": "/var/mitto/data/venvs/portal_export_job/requirements.txt",
		   "requirements_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/requirements.txt",
		   "python_file_path": "/var/mitto/data/py_files/portal_export_job/job_file.py",
		   "python_file_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/portal_export.py",
		   "env_vars": {},
	   }
   }
   ```

1. For use in production:

   ```
   {
	   "export": {
		   "source_url": "https://portal-automation.zuarbase.net",
		   "source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCAiYzRDpYweDrUo0",
		   "file_path": "/var/mitto/data/file-exported-from-portal.pdf"
	   },
	   "subscription": {
		   "source": "/p/runner-automation",
		   "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
		   "json_data": {
			   "export_type": "pdf",
			   "window_size": "1280x1024"
		   },
	   },
	   "python_job_config": {
		   "venv_dir": "/var/mitto/data/venvs/portal_export_job/venv",
		   "requirements_file_path": "/var/mitto/data/venvs/portal_export_job/requirements.txt",
		   "requirements_body": "/var/mitto/plugin/subscriptions_ng/export/requirements.txt",
		   "python_file_path": "/var/mitto/data/py_files/portal_export_job/job_file.py",
		   "python_file_body": "/var/mitto/plugin/subscriptions_ng/export/portal_export.py",
		   "env_vars": {},
	   }
   }
   ```

Note that the `export` and `subscription` sections are identical to that of the previous
section; these are passed by the Portal Export Job to `portal_export.py`.
`python_job_config` is the only new configuration parameters, which are specific to
Runner Python Jobs.

If you have never run the Portal Export job on an instance and then run it using the
above config, it will take the contents of the file at `Source` and place it at
`Destination` (see below), it will build a Python virtualenv, and it will use the
remainder of the job config to run `portal_export.py`, thereby creating a PDF export at
`/var/mitto/data/file-exported-from-portal.pdf`.

| Usage        | Source `*_body`                                                                       | Destination `*file_path` |
|--------------|---------------------------------------------------------------------------------------|--------------------------|
| `m-d-r`      | `/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/requirements.txt` | `/var/mitto/data/venvs/portal_export_job/requirements.txt` |
| `m-d-r`      | `/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/portal_export.py` | `/var/mitto/data/py_files/portal_export_job/job_file.py` |
| `production` | `/var/mitto/plugin/subscriptions_ng/export/requirements.txt`                          | `/var/mitto/data/venvs/portal_export_job/requirements.txt` |
| `production` | `/var/mitto/plugin/subscriptions_ng/export/portal_export.py`                          | `/var/mitto/data/py_files/portal_export_job/job_file.py` |

### Portal Export Job - Static Behavior

The job configurations of the previous section can be used to create a Portal Export job
that does the same thing each time it is run.  This can be useful on its own, but it is
covered here as it is a pre-requisite for dynamic exports that are controlled by
subscriptions.

This document builds on the basics covered in the basic Subscriptions NG
documentation.

To create a Portal Export job:

1. Use the generic job wizard.
1. Modify and use one of the previous job configurations of the previous section.
1. Change the job type to `portal_export`.
1. Click `Run`.
1. When the job completes, the exported PDF file will be in
   `/var/mitto/data/file-exported-from-portal.pdf`.

### Portal Export Job - Dynamic Behavior

The static behavior just described is useful.  But we'd like to generalize it and make
its behavior, dynamic, ultimately based on the contents of subscriptions.  To get
started, let's consider the following use-case:

   As a Runner user, I would like to have a single Portal Export Job that can
   perform exports for different users and dashboards.  I do not want to have to repeatedly
   edit the job configuration to achieve this.

To start, let's look at how Runner starts jobs, first without parameters and then with
parameters.

#### `job_portal_export.py` - Without Parameters

`job_portal_export.py` implements the Runner Portal Export job type. It will appear in
the job type pulldown on Runner.

Let's assume you've created a Portal Export job as described in the [Static Behavior
section](#portal-export-job-static-behavior) and that the job's
configuration resides on disk at
`/var/mitto/conf/example_portal_export_job.json`.

When the job is run on-demand by clicking "Start", it runs `job_portal_export.py`,
which simply passes its job configuration straight to `portal_export.py`, as already
described.

There are multiple ways that the job can be run.

The lowest level is from the cli inside the webapp container:
```
/app/env/bin/python3 /app/mitto/jobs/job_python_export.py /var/mitto/conf/example_portal_export_job.json
```
The above is what is ultimately run when the "Start" button is clicked.

The next lowest level is via the `mitto` cli command:
```
/app/env/bin/mitto job run example_portal_export_job
```
Here, you can identify the job by the name of its configuration file or by its job number.

Finally, the job can also be run via the Runner API (you must have the job number for this):
```
curl -X 'POST' \
  'https://stage-mitto3.zuarbase.net/api/v2/jobs/1/:actions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "action": "START",
  "params": {}
}'
```

All of these methods produce identical results.

#### `job_portal_export.py` - With Parameters

How, then, does Subscriptions perform exports for different users and different
dashboards using a single job configuration?  By passing in a job parameter containing
overrides that are used to dynamically update the job configuration stored on disk.

Let's assume you've created a Portal Export job as described in the [Static Behavior
section](#portal-export-job-static-behavior) and that the job's configuration resides on
disk at `/var/mitto/conf/example_portal_export_job.json`.

Let's also assume that we want to perform an export for a user with these
specifics:

|Field | Value |
|------|-------|
|dashboard| /p/most-excellent-dashboard
|portal user_id| 11111bb9-00dc-46f1-8397-ee0ecf312fa3
|format: format |
|window_size| 1280x1024 |

We'll also assume that that the `export` section of the job configuration is correct and has
valid credentials, etc. 

To get results we want, the `subscription` section of the job configuration
would have to look like this:
```
               "subscription": {
                   "source": "/p/most-excellent-dashboard",
                   "json_data": {
                       "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
                   }
               }
```
But it doesn't look like that....

All Runner jobs support the concept of parameters, which can be passed in via the
`--params` option to modify job behavior.  The value of the option is a JSON
string.  There is a special key named `__config_overrides__` whose value is used to
modify the job configuration.

Let's look at a specific example of how this works.  Let's start with running
`job_portal_export.py` from the cli:
```
/app/env/bin/python3 /app/mitto/jobs/job_python_export.py \
       --params '{
           "__config_overrides__": {
               "subscription": {
                   "source": "/p/most-excellent-dashboard",
                   "json_data": {
                       "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
                   }
               }
           }
       }' \
       /var/mitto/conf/example_portal_export_job.json
```
After reading the job configuration from disk, the value of `__config_overrides__` is
used to modify the job configuration.  In this case, `subscription` is used to modify
the value of `subscription.source` and `subscription.json_data.user_id`.  The resulting
modified job configuration is used to run the job.  We get the export for the user we
wanted.  The modified job configuration is discarded -- the configuration on disk is
unmodified.

Note that we didn't provide `export_type` or `window_size`.  Because of that, the values
for those keys in the configuration on disk are used -- in other words, the original job
configuration is treated as "default values".

The `mitto` command supports this as well:
```
/app/env/bin/mitto job run example_portal_export_job \
       --params '{
           "__config_overrides__": {
               "subscription": {
                   "source": "/p/most-excellent-dashboard",
                   "json_data": {
                       "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
                   }
               }
           }
       }' \
       /var/mitto/conf/example_portal_export_job.json
```

And, of course, we can do this via the API:
```
curl -X 'POST' \
  'https://stage-mitto3.zuarbase.net/api/v2/jobs/1/:actions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
          "action": "START",
          "params": {
              "__config_overrides__": {
                  "subscription": {
                      "source": "/p/runner-automation",
                      "json_data": {
                          "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
                      }
                  }
              }
          }
      }'
```

## The Subscriptions Job

To conclude, let's look at how what's been described so far is used by the
Subscriptions Job to customize subscriptions content.

For each subscription that is to be run:

1. The specifics of the subscription, in particular the `source` (dashboard) and
   `user_id`, are taken from the `subscriptions_ng` table.
1. Those values are used to create `__config_overrides__`.
1. The Subscriptions Job passes those `__config_overrides__` as `--params` to the
   Portal Export job.
1. The Portal Export job performs the export.
1. The Subscriptions Job attaches the PDF file to the outgoing email.

## Generalizing

On its own, it's helpful to know about how Subscriptions NG works.  However, a key goal
of this document is to demonstrate the mechanics of dynamically controlling job
behavior.  `params` and `__config_overrides__` can be used in many creative ways to
create complicated workflows.