Introduction¶

“Subscriptions NG” is a Runner connector that creates artifacts (e.g., a PDF document from a Zuar Portal dashboard) and emails the artifact to one or more recipients according to a schedule.

Subscriptions NG requires the “Python Job” connector for running artibtrary Python code. A Runner job built using this Python Job connector uses Playwright to create PDF exports of Zuar Portal dashboards.

An understanding of the Runner Python Job and the Portal Export Job will be helpful in undertanding and configurint Subscriptions NG.

Subscriptions are defined in the subscriptions_ng database table located on Zuar Portal:

create table public.subscriptions_ng
(
    id         uuid                     not null
        primary key,
    user_id    uuid                     not null,
    email      varchar(512)             not null,
    schedule   varchar(8)               not null,
    day_of     integer                  not null,
    send_at    time                     not null,
    tz         varchar(256)             not null,
    source     text                     not null,
    json_data  text                     not null,
    updated_at timestamp with time zone not null,
    created_at timestamp with time zone not null
);

The json_data field contains additional data that can be used to filter modify the subscription.

A subscription_status table exists on Runner. It contains the current status of each subscription in the subscriptions table. The DDL for the table is:

create table public.subscription_status_ng (
  id uuid primary key not null,
  last_checked timestamp with time zone,
  last_run timestamp with time zone
);

ALTER TABLE ONLY public.subscription_status_ng
    ADD CONSTRAINT subscription_status_pkey PRIMARY KEY (id);

A subscriptions job (run via job_subscriptions_ng.py) is scheduled to run regularly (e.g. hourly) on Runner. Each time the job is run, it uses the two tables to determine which, if any, subscriptions should be run. If a subscription is due to run, the specifics of the subscription are obtained from the subscriptions table.

Creation of the file to be exported is implemented by the job_portal_export.py – with the data specified by the subscription record. As job_subscriptions_ng.py processes a subscription, it uses the underlying implementation of job_portal_export.py to create and download the export file to Runner.

Once the export file is downloaded, job_subscriptions.py creates the email message, attaches the export file, and sends the email to the recipient. Finally, the subscription_status_ng table is updated to reflect that the subscription has been processed.

Quickstart¶

Configuring Subscriptions NG isn’t difficult, but it does require a fair amount of effort and many parameters. We’ll simplify this quickstart by breaking it into two distincts parts. The first is to create a working standalone Portal Export job. This will confirm that your export parameters are correct and that you are getting the PDF document you expect. The second is to create a Subscriptions NG job that uses the Portal Export job to create attachments for outgoing subscriptions.

Create a Standalone Portal Export Job¶

Let’s create our standalone Portal Export job.

Portal Export Job configuration reference

Login to Zuar Portal as an admin user.
Use the Zuar Portal OpenAPI documentation at /auth/docs#/default/create_api_key_api_keys_post to create an API Key to be used when impersonating a Zuar Portal user and exporting a dashboard as a PDF document. We’ll assume the API_KEY is TWbMjCSa8_uPEW5Cijslielk8984AiYzRDpYweDrUo0.

Create a new generic job using one of the job configs below.

For use on a production Runner:

{
  export: {
    source_url: https://portal-automation.zuarbase.net
    source_credentials: TWbMjCSa8_uPEW5CijRabaSwPVHCAiYzRDpYweDrUo0
    file_path: file-exported-from-portal.pdf
  }
  subscription: {
    source: /p/runner-automation
    user_id: e8041bb9-00dc-46f1-8397-ee0ecf312fa3
    json_data: {
      export_type: pdf
      window_size: 1280x1024
    }
  }
  python_job_config: {
    venv_dir: /var/mitto/data/venvs/portal_export_job/venv
    requirements_file_path: /var/mitto/data/venvs/portal_export_job/requirements.txt
    requirements_body: /var/mitto/plugin/subscriptions_ng/export/requirements.txt
    python_file_path: /var/mitto/data/py_files/portal_export_job/job_file.py
    python_file_body: /var/mitto/plugin/subscriptions_ng/export/portal_export.py
    env_vars: {
    }
  }
}

For use locally with mitto-dev-runtime:

{
    "export": {
        "source_url": "https://custom-portal-test.zuarbase.net",
        "source_credentials": "TWbMjCSa8_uPEW5Cijslielk8984AiYzRDpYweDrUo0",
        "file_path": "daily-operating-metrics.pdf"
    },
    "subscription": {
        "source": "/p/daily-operating-metrics?GLOBAL_START_DATE:range=Yesterday",
        "user_id": "3ee5e4fb-5b2b-4c52-a3b6-f54ea1d24395",
        "json_data": {
            "export_type": "pdf",
            "window_size": "1280x1024"
        },
    },
    "python_job_config": {
        "venv_dir": "/var/mitto/data/venvs/portal_export_job/venv",
        "requirements_file_path": "/var/mitto/data/venvs/portal_export_job/requirements.txt",
        "requirements_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/requirements.txt",
        "python_file_path": "/var/mitto/data/py_files/portal_export_job/job_file.py",
        "python_file_body": "/app/plugins/mitto-plugin-subscriptions-ng/subscriptions_ng/export/portal_export.py",
        "env_vars": {}
    }
}

Edit the job config:
1. export section
  1. Set source_url to the base URL of the Poral.
  2. Set source_credentials to the Portal’s API_KEY created in step 1.
  3. file_path should be a fully qualified path to the file that the export will be placed in.
2. subscription section
  1. Set source to the path to the dashboard to be exported. This path should begin with /. The path should not include the base URL provided in source_url.
  2. Set user_id to the Portal user_id to impersonate when performing the export.
  3. Set export_type to pdf.
  4. Set window_size to the desired size.
3. Set the job type to job_portal_export.
Run the job.
The export should be in the location you specified in file_path.

Create a Subscription and a Subscriptions NG Job¶

Now that we have a Portal Export job that successfully creates PDF exports from a Zuar Portal, let’s finish up by creating a subscription in Zuar Portal and configuring a Subscriptions NG job in Runner to process subscriptions every hour. The Subscriptions NG job will:

Check the subscriprtions_ng (on Zuar Portal) subscription_status_ng (on Zuar Runner) tables every hour, looking for subscriptions that are due to run at that time.
When it finds a subscription to run, it will:
1. Use the subscription info to modify the Portal Export job created in the last section so that it creates a PDF with the desired content.
2. Send an email to the subscription’s recipients and attach the exported PDF.

Create a Subscription¶

‘subscriptions_ng’ table reference

We’ll assume you have created a subscription in Zuar Portal with the following details:

user_id : 571048a0-5de3-42a0-a910-380890c6de03
email : Steve@greif.com
schedule : DAILY
day_of : 0
send_at : 08:00
tz : America/Chicago
source : /p/daily-operating-metrics?GLOBAL_START_DATE:range=Yesterday
json_data : {"export_type": "pdf", "window_size": "1920x945"}

Create a Subscriptions NG Job¶

Subscriptions NG Job configuration reference

Use Runner’s Generic Job wizard to create a new Subscriptions NG job using the following job configuration:

{
  subscriptions: {
    subscriptions_input: {
      dbo: postgresql://<user>:<pass>@custom-portal-test.zuarbase.net/portal
      schema_name: public
      table_name: subscriptions_ng
    }
    subscription_status_input: {
      dbo: postgresql://<user>:<pass>@db/mitto
      schema_name: public
      table_name: subscription_status_ng
    }
  }
  export: {
    source_url: https://custom-portal-test.zuarbase.net
    source_credentials: TWbMjCSa8_uPEW5Cijslielk8984AiYzRDpYweDrUo0
    file_path: daily-operating-metrics.pdf
    job_type: portal_export
    job_config_path: /var/mitto/data/example_portal_export_job_config.json
  }
  mail: {
    server: in-v3.mailjet.com
    port: 587
    require_tls: true
    credentials: {
      username: 83acd7b55a240a02f79abdffccf5474a
      password: 6cfdae93dabf6835453326cfdaed2b7d
    }
    timeout: 30
    mail_subject: Daily Operating Metrics Subscription
    mail_from: portal-admin-3425@zuar.net
    text: Attached is your Daily Operating Metrics dashboard
    html: <html>Attached is your Daily Operating Metrics dashboard</html>
  }
}

Configure mail for your SMTP server of choice.
Change the job type to subscriptions_ng.
Schedule the job to run hourly.
When the job runs, it will look at each subscription and its status. When a subscription is due to run, the following happens:
1. Values in the subscriptions_ng record are used to create a temporary Portal Export job configuration for the specific export to be created.
2. A Portal Export job is run using the temporary configuration.
3. Values in the subscriptions_ng record are used to create an email, attach the export to the email, and send the email to the recipient(s).