How to use the Notification API¶

This section discusses the usage of the Notification API. It would discuss the pre-requisites, the payload, responses and error code(s) from the error catalogue that the client can expect.

Pre-requisites¶

To use the Notification API, there are a couple of pre-requisites. Invoking the Notification API without these would lead to errors documented in the Notification API error catalogue.

Pre-requisite 1: Set up the recipients¶

Recipients are the groups of users that the notification(s) is(are) intended to be delivered to. These groups can be email distribution lists, Slack channels, FreshService groups, and the like. The Notification API does not currently provide a subscription model where the end users could be subscribed to. This subscription, therefore, needs to be handled by the clients. For example, if the notification(s) is(are) intended to a lab group called "x-lab-group", it is the client's responsibility to set up the distribution list (e.g., xlabgrp@psd.sanger.ac.uk).

Note

For Slack, the distribution list would be a Slack channel¹.

After setting up the distribution lists, you will have to communicate the recipient groups to the C4E. The C4E will provide you with a unique identifier for each recipient you have provided. In the example used in a previous section, the recipient ID given by the C4E was X_LAB_STAKEHOLDERS.

This needs to be done as a pre-requisite as it is the ID populated in the recipients field in the payload.

Pre-requisite 2: Set up the templates¶

Introduction to templates¶

Templates are strings that contain placeholders for dynamically inserting data. These templates are typically rendered into rich-text formats such as Markdown or HTML to display using a Template Processor. The Notification API is currently equipped with Jinja as the template engine, and an internal template processor to combine the template with data.

Check the following Jinja template:

{% extends "base.html" %}
{% block title %}Members{% endblock %}
{% block content %}
  <ul>
  {% for user in users %}
    <li><a href="{{ user.url }}">{{ user.username }}</a></li>
  {% endfor %}
  </ul>
{% endblock %}

If the following arguments are given to the template, the template processor would produce the HTML file listed in the tab group.

Python ArgumentsHTML Output

users = [
  {"username": "alice", "url": "/users/alice"},
  {"username": "bob", "url": "/users/bob"},
  {"username": "charlie", "url": "/users/charlie"}
]

<!DOCTYPE html>
<html>
<head>
  <title>Members</title>
</head>
<body>
  <ul>
    <li><a href="/users/alice">alice</a></li>
    <li><a href="/users/bob">bob</a></li>
    <li><a href="/users/charlie">charlie</a></li>
  </ul>
</body>
</html>

Styles in templates

If the templates are HTML, it is advisable to inline the styles using a tool like MJML or Foundation. For a reference, please find an example MJML template definition implementation for the accessioning use-case discussed in another section.

<!-- This is Accessioning MJML created to generate the Jinja template used to create the accessioning failure notification. -->
<!-- Please feel free to use it as a starting point in setting up your template. -->
<!-- MJML Documentation: https://mjml.io/ -->
<!-- In this template, you have access to a list of every attribute listed inside your `fields` attribute in each notification. -->
<!-- NB: Note the use of <mj-raw>. It is required for MJML transpiler to not to strip-off any Jinja-related syntax. -->

<mjml>
  <mj-head>
      <mj-title>Problems with Accessioning</mj-title>
      <mj-preview>There are validation errors in the samples provided</mj-preview>
      <mj-style inline="inline">
        .email-card {
        box-shadow: 0 6px 20px rgba(0, 0, 0, 0.12);
        }
      </mj-style>
      <mj-attributes>
        <mj-all font-family="Arial, Helvetica, sans-serif" />
        <mj-text font-size="14px" color="#333333" line-height="1.6" align="left" />
      </mj-attributes>
  </mj-head>
  <mj-body background-color="#f4f6f8">
      <mj-wrapper></mj-wrapper>
      <mj-wrapper css-class="email-card" background-color="#ffffff" border="1px solid #e0e0e0" border-radius="8px" padding="0">
        <!-- Header Section -->
        <mj-section background-color="#747067" padding="20px 25px 10px">
            <mj-column>
              <mj-text font-size="20px" font-weight="bold" color="#ffffff" align="left">
                  ⚠️ Problems with Accessioning ⚠️
              </mj-text>
            </mj-column>
        </mj-section>
        <mj-section background-color="#ffffff" padding="20px 25px">
            <mj-column>
              <mj-text>
                  There are validation errors in the samples provided for accessioning. This notification includes details of 
                  <b>
                    <mj-raw> {{ fields | length }} </mj-raw>
                  </b>
                  &nbsp;accessioning samples.
              </mj-text>             
            </mj-column>
        </mj-section>
        <!-- Beginning of content -->
        <!-- If 100 or more samples, display a compact summary -->
        <mj-raw>{% if fields|length >= 10 %}</mj-raw>
        <mj-raw>{% set samples = fields | map(attribute='sample-id') | list | unique | list %}</mj-raw>
        <mj-section background-color="#ffffff" padding="0 25px 12px 25px">
            <mj-column>
              <mj-text>
                  More than 100 samples have errored when accessioning.
              </mj-text>
            </mj-column>
            <mj-column background-color="rgb(251, 233, 173)" border="1px solid #d2c85b" border-radius="6px" padding="0">
              <mj-text font-size="14px" color="#333333" align="left" background-color="#f1f3f5" padding="12px 15px">
                  <mj-raw>{% set unique_attribute_values = fields | map(attribute='attributes') | sum(start=[]) | map(attribute='key') | list | unique | list %}</mj-raw>
                  <mj-raw>{% if unique_attribute_values | length == 0 and fields["manifest-id"] %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                    <mj-raw>{{ fields["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values.
                  <mj-raw>{% elif unique_attribute_values | length > 5 %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                    <mj-raw>{{ fields[0]["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values for fields like 
                  <mj-raw>: {{ unique_attribute_values[:5] | join(", ") }}</mj-raw>
                  .
                  <mj-raw>{% else %}</mj-raw>
                  Some samples in the manifest 
                  <b>
                    <mj-raw>{{ fields[0]["manifest-id"] }}</mj-raw>
                  </b>
                  &nbsp;that have errored, due to incorrect values for fields like 
                  <mj-raw>: {{ unique_attribute_values | join(", ") }}</mj-raw>
                  .
                  <mj-raw>{% endif %}</mj-raw>
              </mj-text>
            </mj-column>
        </mj-section>
        <mj-raw>{% else %}</mj-raw>
        <!-- Display individual fields when fewer than 100 samples -->
        <mj-raw>{% for field in fields %}</mj-raw>
        <mj-section background-color="#ffffff" padding="0 25px 12px 25px">
            <mj-column background-color="#fafafa" border="1px solid #e0e0e0" border-radius="6px" padding="0">
              <mj-text font-size="14px" font-weight="bold" color="#333333" align="left" background-color="#f1f3f5" padding="12px 15px">
                  Sample ID: 
                  <mj-raw>{{ field["sample-id"] }}</mj-raw>
              </mj-text>
              <mj-divider border-color="#e0e0e0" padding="0" />
              <mj-raw>{% for attribute in field.attributes %}</mj-raw>
              <mj-text font-size="13px" color="#555555" padding="6px 15px" align="left">
                  <strong style="color:#000000;">
                    <mj-raw>{{ attribute.key }}</mj-raw>
                    : 
                  </strong>
                  &nbsp;
                  <mj-raw>{{ attribute.value }}</mj-raw>
              </mj-text>
              <mj-raw>{% endfor %}</mj-raw>
            </mj-column>
        </mj-section>
        <mj-raw>{% endfor %}</mj-raw>
        <mj-raw>{% endif %}</mj-raw>
        <!-- End of content -->
        <!-- Footer section -->
        <mj-section background-color="#ffffff" padding="8px 25px 20px 25px">
            <mj-column>
              <mj-text font-size="12px" color="#888888" align="center">
                  This is an automated message. Please do not reply.
              </mj-text>
            </mj-column>
        </mj-section>
      </mj-wrapper>
      <mj-wrapper></mj-wrapper>
  </mj-body>
</mjml>

Using templates for the Notification API¶

Just like the distribution lists, you would have to create the templates and communicate the template to the C4E. The C4E would then provide you with an identifier which you need to include in the payload. In the example used in a previous section, the recipient ID given by the C4E was X_ACCESSIONING_INVALID_INPUTS. Along with the templates, the arguments should be sent in the request payload within the key fields. Invalid arguments are validated and if invalid, the notification is flagged as a failed notification by the Notification API.

Preparing the templates for batched aggregation

Unlike NORMAL and HIGH priority values, preparing the templates batched aggregation requires special attention. The BATCH templates need to make use of the special fields value as mentioned in the section Batch Aggregation.

The Payload¶

The payload for the Notification API is expected to be of a particular schema, which the consumers can find in the API documentation. The idea behind the user guide for explaining the payload is to describe the uses of the attributes of the payload in general.

Let us consider the same payload we referred to in a a previous section.

{
  "channels": [
    {
      "type": "EMAIL",
      "recipient": [
        "X_LAB_STAKEHOLDERS"
      ],
      "subject": "Accessioning Failure: Invalid input data for manifest",
      "content_type": "text",
      "template_id": "X_ACCESSIONING_INVALID_INPUTS",
      "fields": {
        "attributes": [
          {
            "key": "Country",
            "value": "United Kindm"
          },
          {
            "key": "Reference Genome",
            "value": "NONE"
          }
        ],
        "sample_id": "40STDY246690"
      }
    }
  ],
  "priority": "NORMAL",
}

Descriptions of attributes¶

Attribute	Description
`channels`	List of `channel` JSON objects. Each object represents a single channel (e.g., Slack, Email, FreshService).
`type`	Name of the channel. Valid types: `SLACK`, `EMAIL`, `FRESHSERVICE`.
`recipient`	Array of recipient IDs given by the C4E. See Pre-requisite 1 for details.
`subject`	Subject of the notification. For email, it is the email subject (RFC-5422). For FreshService, it is the ticket subject.
`template_id`	Identifier given by the C4E for the pre-populated template. Supported engine: Jinja.
`content_type`	Content type for template rendering. Supported: `text`, `html`.
`fields`	JSON object representing arguments for the template processor. Example: `users` key with a list of user objects.
`priority`	Accepts `NORMAL`, `HIGH`, or `BATCH`. See features section for details.
`aggregator_id`	Mandatory only when `priority` is `BATCH`. Used for notifications to be aggregated and buffered.

Responses and error codes¶

The key attribute returned within the response coming from the Notification API is the notification_id. The calling system would need to store it in a data store if it is required to check the status of the notification using the status check API. An initial status would also be part of the response. DISPATCHED status in the response means that the notification has been submitted to the lower layers of the system, and any failures with validations would result in returning FAILED status for the notification.

Either way, the response would contain a notification_id.

Warning

The Notification ID is deterministically generated based on the notification content, meaning the same content will always produce the same ID. So, if the same notification is received twice, the API would ignore the second notification.

For more information on the response schema and the various error codes, please refer to the API documentation.

Invoking the API¶

The Notification API is an HTTP REST API (endpoint details could be found within the API documentation). Therefore, the clients can use any HTTP client to interact with the Notification API.

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
  "channels": [
    {
      "type": "EMAIL",
      "recipient": [
        "X_LAB_STAKEHOLDERS"
      ],
      "subject": "Accessioning Failure: Invalid input data for manifest",
      "content_type": "text",
      "template_id": "X_ACCESSIONING_INVALID_INPUTS",
      "fields": {
        "attributes": [
          {
            "key": "Country",
            "value": "United Kindm"
          },
          {
            "key": "Reference Genome",
            "value": "NONE"
          }
        ],
        "sample_id": "40STDY246690"
      }
    }
  ],
  "priority": "NORMAL",
}'

The template identified by X_ACCESSIONING_INVALID_INPUTS could be as follows:

<!DOCTYPE html>
<html>
    <body>
        <h1>Accesioning errors for Sample {{ fields.sample_id }}</h1>
        <p>There is a data validation issue in accessioning.</p>
        <ul>
            {% for attribute in fields.attributes %}
                <li>{{ attribute.key }} -> {{attribute.value}}</li>
            {% endfor %}
        </ul>
    </body>
</html>

Status Check¶

The Notification API provides another endpoint /notification/{notification-id} to find the status of a given notification identified by the notification-id. It could be invoked using a GET HTTP call.

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1/{notification-id}' \
--header 'Authorization: Bearer ...'

Batch Aggregation¶

For aggregation priority BATCH, the template needs to account for a list of notifications being aggregated to one single notification. For batched aggregations, you would need to use the fields variable.

The fields variable - in the context of the Jinja template - points to the list of fields in each notification. Take the following three messages sent to be processed as a batch, for example:

First NotificationSecond Notification

{
  "channels": [
    {
      "type": "EMAIL",
      "recipient": [
        "X_LAB_STAKEHOLDERS"
      ],
      "subject": "Accessioning Failure: Invalid input data for manifest",
      "content_type": "text",
      "template_id": "X_ACCESSIONING_INVALID_INPUTS",
      "fields": {
        "attributes": [{
          "key": "Country",
          "value": "United Kindm"
        }],
        "sample-id": "40STDY246690"
      }
    }
  ],
  "priority": "BATCH",
  "aggregator_id": "21784312"
}

{
  "channels": [
    {
      "type": "EMAIL",
      "recipient": [
        "X_LAB_STAKEHOLDERS"
      ],
      "subject": "Accessioning Failure: Invalid input data for manifest",
      "content_type": "text",
      "template_id": "X_ACCESSIONING_INVALID_INPUTS",
      "fields": {
        "attributes": [{
          "key": "Reference Genome",
          "value": "NOT-A-VALID-GENOME"
        }],
        "sample-id": "40STDY246691"
      }
    }
  ],
  "priority": "BATCH",
  "aggregator_id": "21784312"
}

The Jinja2 template can use the fields attribute to iterate through the objects listed in the fields attribute of each notification. Hence, fields attribute is a list, and it is provided into the template as a list only for when priority is equal to BATCH.

Warning

The developers should not confuse between the fields attribute of the payload, and the fields attribute in the template.

For the scenario above, a template like the following could be established:

<!DOCTYPE html>
<html>
<body>
    <h1>Accesioning Errors</h1>
    <p>There are validation errors in the samples provided for accessioning.</p>
    <ul>
        {% for field in fields %}
        <li>Sample ID: {{ field["sample-id"] }}</li>
        <ul>
            {% for attribute in field.attributes %}
              <li>Attribute: {{ attribute.key }}</li>
              <li>Value: {{ attribute.value }}</li>
            {% endfor %}
        </ul>
        {% endfor %}
    </ul>
</body>
</html>

Authentication¶

The client needs to be authenticated by OAuth 2.0 Client Fredentials Flow. The consumers would need to go through the process of obtaining client credentials and proceeding with the credentials to obtain tokens to interact with the API. Please refer to the API documentation.

Because OAuth 2.0 relies on client credentials, the consumers would first have to request them from the C4E in order to invoke the APIs. Since FreshService is still being set up for the Integration Hub, the consumers would need to contact the Integration Hub at integration-hub@sanger.ac.uk and retrieve the credentials from the team. After FreshService is established, this will be moved into a more streamlined process.

Client Libraries

For authentication, we recommend using a OAuth-recommended client library as authentication (and authorisation) deals with sensitive credentials.

Rate Limiting¶

The API has built-in rate limiting. For more information on rate limits, please refer to the API documentation.

Overall Procedure¶

The overall usage of the Notification API can be summarised into the following diagram.

flowchart TD
    %% Preparation Phase
    subgraph Prep[Preparation]
        A([🔍 Identify Notification Need])
        B([👥 Set Up Recipient Groups])
        C([📨 Communicate Recipients to C4E])
        D([🆔 Receive Recipient IDs])
        E([📝 Create Notification Templates])
        F([📨 Communicate Templates to C4E])
        G([🆔 Receive Template IDs])
    end

    %% Payload Construction
    subgraph Payload[Payload Construction]
        H([🛠️ Prepare Notification Payload])
        I([🔗 Include Recipient & Template IDs])
    end

    %% API Interaction
    subgraph API[API Interaction]
        J([🚀 Send Payload to Notification API])
        K([📬 Receive Notification ID])
    end

    %% Post-Processing
    subgraph Post[Post-Processing]
        L{🔄 Check Notification Status?}
        M([🔎 Query Status API])
        N([🏁 End])
    end

    %% Flow
    A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K --> L
    L -- Yes --> M --> N
    L -- No --> N

This is not to be confused with the channel discussed in nomenclature section. This channel refers specifically to Slack Channels. ↩