Dynamically Modifying Job Configurations¶

Introduction¶

Subscriptions NG requires the “Python Job” connector for running arbitrary Python code. A Runner job built using the Python Job connector uses Playwright to create PDF exports of Zuar Portal dashboards; this is call the Portal Export Job.

The Subscriptions NG job uses data unique to each subscription to dynamically modify the behavior of the Portal Export Job. This document, while focused on Subscriptions NG, provides the details of how job behavior can be dynamically modified at job runtime. This is valuable knowledge for anyone building complex Runner deployments.

This document describes the implementation of Subscriptions NG with a lengthy explanation of how dynamic job parameters and job overrides are used to modify job behavior based on the contents of subscriptions.

Portal Export¶

`portal_export.py`¶

We’ll begin by looking at the core of the Portal Export implementation, which is in portal_export.py. If your instance has mitto-plugin-subscriptions-ng installed, this file will be located at /var/mitto/plugin/subscriptions_ng/export/portal_export.py. This purpose of the script is to use the Zuar Portal API to create a PDF export of a dashboard.

portal_export.py is a standalone Python script that can be run on any system with the necessary dependencies. It does not rely on Runner or any Runner code in any way. This script can be run on any system with Python and a virtualenv that contains the necessary dependencies.

The script performs the export using a headless Chrome browser controlled by Playwright for Python.

The Python dependencies are specified in subscriptions_ng/export/requirements.txt. Additionally, either a Chrome or a FireFox browser must be present; see the next section for more information. Depending upon your host, you may need to set an environment variable to specify where browsers are installed. On macOS, playwright installs browsers in ~/Library/Caches/ms-playwright, in which case you would need to: export PLAYWRIGHT_BROWSERS_PATH=~/Library/Caches/ms-playwright.

The portal_export.py script accepts a single mandatory option --job_config. The format of the --job-config value is described TBD/here. On the command line, the value is passed as a JSON string. Assuming valid credentials, this would perform an export and save it in a portal-export.pdf file in the current directory.

python3 portal_export.py --verbose --job_config '{
    "export": {
        "source_url": "https://portal-automation.zuarbase.net",
        "source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCpYweDrUo0",
        "file_path": "portal-export.pdf"
    },
    "subscription": {
        "source": "/p/runner-automation",
        "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
        "json_data": {
            "export_type": "pdf",
            "window_size": "1280x1024"
        }
    }
}'

The Portal Export Job¶

To allow the portal_export.py script to be run as a Runner job, a Portal Export Job (a Runner Python Job) was created and is installed with the Subscriptions NG connector. The Portal Export Job is essentially a pre-configured Python Job that runs the portal_export.py script.

When the mitto-plugin-subscriptions-ng plugin is installed, it automatically installs the Chrome and FireFox browsers in of Runner’s webapp and scheduler Docker containers (see install_system_dependencies.sh). The very first time that the Portal Export Job runs, the Python Job automatically creates a virtualenv with the dependencies from subscriptions_ng/export/requirements.txt.

Documentation for the Portal Export Job configuration is here.

Example job configurations:

For use locally with mitto-dev-runtime:

{
    "export": {
        "source_url": "https://portal-automation.zuarbase.net",
        "source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCAiYzRDpYweDrUo0",
        "file_path": "/var/mitto/data/file-exported-from-portal.pdf"
    },
    "subscription": {
        "source": "/p/runner-automation",
        "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
        "json_data": {
            "export_type": "pdf",
            "window_size": "1280x1024"
        },
    },
    "python_job_config": {
        "venv_dir": "/var/mitto/data/venvs/portal_export_job/venv",
        "requirements_file_path": "/var/mitto/data/venvs/portal_export_job/requirements.txt",
        "requirements_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/requirements.txt",
        "python_file_path": "/var/mitto/data/py_files/portal_export_job/job_file.py",
        "python_file_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/portal_export.py",
        "env_vars": {},
    }
}

For use in production:

{
    "export": {
        "source_url": "https://portal-automation.zuarbase.net",
        "source_credentials": "TWbMjCSa8_uPEW5CijRabaSwPVHCAiYzRDpYweDrUo0",
        "file_path": "/var/mitto/data/file-exported-from-portal.pdf"
    },
    "subscription": {
        "source": "/p/runner-automation",
        "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
        "json_data": {
            "export_type": "pdf",
            "window_size": "1280x1024"
        },
    },
    "python_job_config": {
        "venv_dir": "/var/mitto/data/venvs/portal_export_job/venv",
        "requirements_file_path": "/var/mitto/data/venvs/portal_export_job/requirements.txt",
        "requirements_body": "/var/mitto/plugin/subscriptions_ng/export/requirements.txt",
        "python_file_path": "/var/mitto/data/py_files/portal_export_job/job_file.py",
        "python_file_body": "/var/mitto/plugin/subscriptions_ng/export/portal_export.py",
        "env_vars": {},
    }
}

Note that the export and subscription sections are identical to that of the previous section; these are passed by the Portal Export Job to portal_export.py. python_job_config is the only new configuration parameters, which are specific to Runner Python Jobs.

If you have never run the Portal Export job on an instance and then run it using the above config, it will take the contents of the file at Source and place it at Destination (see below), it will build a Python virtualenv, and it will use the remainder of the job config to run portal_export.py, thereby creating a PDF export at /var/mitto/data/file-exported-from-portal.pdf.

Usage	Source `*_body`	Destination `*file_path`
`m-d-r`	`/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/requirements.txt`	`/var/mitto/data/venvs/portal_export_job/requirements.txt`
`m-d-r`	`/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/portal_export.py`	`/var/mitto/data/py_files/portal_export_job/job_file.py`
`production`	`/var/mitto/plugin/subscriptions_ng/export/requirements.txt`	`/var/mitto/data/venvs/portal_export_job/requirements.txt`
`production`	`/var/mitto/plugin/subscriptions_ng/export/portal_export.py`	`/var/mitto/data/py_files/portal_export_job/job_file.py`

Portal Export Job - Static Behavior¶

The job configurations of the previous section can be used to create a Portal Export job that does the same thing each time it is run. This can be useful on its own, but it is covered here as it is a pre-requisite for dynamic exports that are controlled by subscriptions.

This document builds on the basics covered in the basic Subscriptions NG documentation.

To create a Portal Export job:

Use the generic job wizard.
Modify and use one of the previous job configurations of the previous section.
Change the job type to portal_export.
Click Run.
When the job completes, the exported PDF file will be in /var/mitto/data/file-exported-from-portal.pdf.

Portal Export Job - Dynamic Behavior¶

The static behavior just described is useful. But we’d like to generalize it and make its behavior, dynamic, ultimately based on the contents of subscriptions. To get started, let’s consider the following use-case:

As a Runner user, I would like to have a single Portal Export Job that can perform exports for different users and dashboards. I do not want to have to repeatedly edit the job configuration to achieve this.

To start, let’s look at how Runner starts jobs, first without parameters and then with parameters.

`job_portal_export.py` - Without Parameters¶

job_portal_export.py implements the Runner Portal Export job type. It will appear in the job type pulldown on Runner.

Let’s assume you’ve created a Portal Export job as described in the Static Behavior section and that the job’s configuration resides on disk at /var/mitto/conf/example_portal_export_job.json.

When the job is run on-demand by clicking “Start”, it runs job_portal_export.py, which simply passes its job configuration straight to portal_export.py, as already described.

There are multiple ways that the job can be run.

The lowest level is from the cli inside the webapp container:

/app/env/bin/python3 /app/mitto/jobs/job_python_export.py /var/mitto/conf/example_portal_export_job.json

The above is what is ultimately run when the “Start” button is clicked.

The next lowest level is via the mitto cli command:

/app/env/bin/mitto job run example_portal_export_job

Here, you can identify the job by the name of its configuration file or by its job number.

Finally, the job can also be run via the Runner API (you must have the job number for this):

curl -X 'POST' \
  'https://stage-mitto3.zuarbase.net/api/v2/jobs/1/:actions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "action": "START",
  "params": {}
}'

All of these methods produce identical results.

`job_portal_export.py` - With Parameters¶

How, then, does Subscriptions perform exports for different users and different dashboards using a single job configuration? By passing in a job parameter containing overrides that are used to dynamically update the job configuration stored on disk.

Let’s assume you’ve created a Portal Export job as described in the Static Behavior section and that the job’s configuration resides on disk at /var/mitto/conf/example_portal_export_job.json.

Let’s also assume that we want to perform an export for a user with these specifics:

We’ll also assume that that the export section of the job configuration is correct and has valid credentials, etc.

To get results we want, the subscription section of the job configuration would have to look like this:

"subscription": {
    "source": "/p/most-excellent-dashboard",
    "json_data": {
        "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
    }
}

But it doesn’t look like that….

All Runner jobs support the concept of parameters, which can be passed in via the --params option to modify job behavior. The value of the option is a JSON string. There is a special key named __config_overrides__ whose value is used to modify the job configuration.

Let’s look at a specific example of how this works. Let’s start with running job_portal_export.py from the cli:

/app/env/bin/python3 /app/mitto/jobs/job_python_export.py \
       --params '{
           "__config_overrides__": {
               "subscription": {
                   "source": "/p/most-excellent-dashboard",
                   "json_data": {
                       "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
                   }
               }
           }
       }' \
       /var/mitto/conf/example_portal_export_job.json

After reading the job configuration from disk, the value of __config_overrides__ is used to modify the job configuration. In this case, subscription is used to modify the value of subscription.source and subscription.json_data.user_id. The resulting modified job configuration is used to run the job. We get the export for the user we wanted. The modified job configuration is discarded – the configuration on disk is unmodified.

Note that we didn’t provide export_type or window_size. Because of that, the values for those keys in the configuration on disk are used – in other words, the original job configuration is treated as “default values”.

The mitto command supports this as well:

/app/env/bin/mitto job run example_portal_export_job \
       --params '{
           "__config_overrides__": {
               "subscription": {
                   "source": "/p/most-excellent-dashboard",
                   "json_data": {
                       "user_id": "11111bb9-00dc-46f1-8397-ee0ecf312fa3",
                   }
               }
           }
       }' \
       /var/mitto/conf/example_portal_export_job.json

And, of course, we can do this via the API:

curl -X 'POST' \
  'https://stage-mitto3.zuarbase.net/api/v2/jobs/1/:actions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
          "action": "START",
          "params": {
              "__config_overrides__": {
                  "subscription": {
                      "source": "/p/runner-automation",
                      "json_data": {
                          "user_id": "e8041bb9-00dc-46f1-8397-ee0ecf312fa3",
                      }
                  }
              }
          }
      }'

The Subscriptions Job¶

To conclude, let’s look at how what’s been described so far is used by the Subscriptions Job to customize subscriptions content.

For each subscription that is to be run:

The specifics of the subscription, in particular the source (dashboard) and user_id, are taken from the subscriptions_ng table.
Those values are used to create __config_overrides__.
The Subscriptions Job passes those __config_overrides__ as --params to the Portal Export job.
The Portal Export job performs the export.
The Subscriptions Job attaches the PDF file to the outgoing email.

Generalizing¶

On its own, it’s helpful to know about how Subscriptions NG works. However, a key goal of this document is to demonstrate the mechanics of dynamically controlling job behavior. params and __config_overrides__ can be used in many creative ways to create complicated workflows.