Example Use Cases¶

The following section aims to provide some use-cases based on a case study scenrio given below. Please note that it is a hypothetical scenario and all the names listed in the description are fictitious and they do not represent entities that currently exists. Although, the processes listed are not fictitious as they represent real-world scenarios.

The use cases are extracted from a hypothetical scenario.

Scenario: ABCLIMS¶

ABCLIMS is a Laboratory Information Management System (LIMS) designed by The LIMS Company® that among many capabilities, provides features such as bulk sample accessioning, and operational features such as deployment notifications. Each of these features are configuration-led, and the former is event-driven. The LIMS allow the administrators to configure webhooks to invoke notifications for each of the capabilities listed above. How the webhooks are configured within ABCLIMS is out of scope of this document.

Accesioning

Accessioning is a feature provided by ABCLIMS that enables indexing of oncoming samples in an external API. In that process, a number (called the "accession number") is generated by the external system for each sample, and recorded within LIMS.

DEFBioLabs, which is a consumer of ABCLIMS, has ABCLIMS installed on-premises, wants to notify different stakeholder groups upon accessioning samples, and operational features such as new deployments of sub-modules of ABCLIMS. The stakeholder groups involved in each use case might be different, and how the stakeholders would get notified is also different depending on the use case and the type of stakeholder group.

Webhooks

Webhooks are APIs exposed to capture events sent by an external system. In the context of ABCLIMS, a webhook refers to a URL exposed by a secondary system that accepts events generated by ABCLIMS. Since ABCLIMS provides configurable webhooks, it does allow configuring the schemas that the system invokes the webhook with.

Types of notifications that ABCLIMS invokes

Failures to accession samples: For failures, ABCLIMS invokes the given webhook with the following content:
- reason: A detailed explanation string on why the accession failed. This includes an error code that maps to an error catalogue.
- invalid-attributes: If failed, which attributes of the sample caused the failure. This is dependent on the reason being invalid attributes.
Nature of accessioning

ABCLIMS performs accessioning in an asynchronous manner. Accessioning samples occur in a per-sample basis, and therefore results in a larger timespan for the whole bulk of samples to be accessioned.
Successful creation of location reports: Because location report generation is a task-queue oriented process (hence, asynchronous), ABCLIMS provides a webhook to notify the stakeholders of the location reports upon successful report generation. ABCLIMS invokes the given webhook with the following content:
- location-report-url: The URL for ABCLIMS' /view-report page.
Successful/Failed deployment notifications of sub-modules: The operational staff (which is a specific role in ABCLIMS) can configure webhooks to notify stakeholders on various deployment events and configuration changes events. ABCLIMS invokes the given webhook with the following content:
- change: A text containing the change made.

DEFBioLabs wants to create a notification system to be notified upon the 1^st and 3^rd types.

DEFBioLabs Design

DEFBioLabs wants to notify the stakeholder groups through a proxy API. This proxy API is a thin API layer developed by DEFBioLabs that transforms payloads coming from ABCLIMS and notifies the stakeholder groups. The idea of the proxy API is to act as a data transformer and data distributor.

However, DEFBioLabs development teams have realised that by adding the stakeholder communication logic into the proxy API - that by design was aimed to be a thin layer - made the API bulkier with business logic than it is required to be. DEFBioLabs development teams have made the point that they would need to maintain different clients for different modes of communication, channel-related heavy configuration management, and infrastructure management for this communication would be a significant piece of work.

Therefore, DEFBioLabs wants to use the Notification API to keep the proxy API layer thin as much as possible.

proxy-api

Note

Please note that the necessity of a proxy API depends entirely on the level of customisation allowed by ABCLIMS. If it is sufficiently customisable, ABCLIMS could invoke the Notification API directly, without any proxy service.

Using the Notification API, DEFBioLabs would be able to perform distribution of notifications conforming to a standard configuration, and would not require them to implement complex handling logic for different communication channels. They would also save operational costs such as infrastructure and dependency management for the client code they would have to write if not for the Notification API.

Use Case 1: Accessioning¶

ABCLIMS performs sample accessioning upon uploading the samples into the system. Samples are contained in labware that contain receptacles holding samples in. Because inserting the samples into the system can be a heavy bulk task, a job-oriented process in ABCLIMS asynchronously performs tasks for accessioning the uploaded samples.

When samples are uploaded into ABCLIMS, the system performs several immediate checks. They can be, among many others:

Data checks to check if the data conforms to the specified types.
Validations on relationships between the data within the sample bulk.
Existence of various sample attributes.

These checks are synchronously performed by ABCLIMS and therefore the user gets prompted (not notified) immediately upon sample insertions.

However, since accessioning is an asynchronous task, ABCLIMS provides webhooks for notifications related to accessioning jobs; they could be accessioning successes, warnings or failures.

So, in this case, DEFBioLabs would need to invoke the Notification API to notify the relevant users. They would use the proxy API as means of transforming the payload. They would not maintain complex client logic for different channels, and they would not have to maintain configuration nor infrastructure to communicate with the channels.

Step 1: Set the pre-requisites up¶

As mentioned in a previous section, DEFBioLabs would need to identify which stakeholders need to be notified for each case. After identifying, they would need to set up distribution lists for each stakeholder group, and communicate the distribution lists to the C4E.

The C4E would then provide with an identifier for each distribution list. These IDs needs to be stored by DEFBioLabs in a store so that the ID could be included in the Notification API payload.

Accordingly, DEFBioLabs identifies the following stakeholder groups for different types of notifications:

Scientific Service Representatives (SSRs): The ID given by the C4E is DEF_BIO_LAB_SSR. This is to notify of any validation issues and study-level configuration mismatches occurred in accessioning the sample.
DEFBioLABS Software Development Group (SDG): The ID given by the C4E is DEF_BIO_LAB_SDG. This is to notify any connectivity-level issues and unhandled exceptions when accessioning the sample. This could also be issues such as rate limits and throttling issues arising from the external API.
DEFBioLABS DevOPS Group: The ID given by the C4E is DEF_BIO_LAB_DOPS: This is to notify any connectivity-level issues.

Stakeholders

The real accessioning use case might be more involved than this simplified version. It's kept intentionally simple in order to get the concepts across. There could be different levels to each stakeholder group - SSRs for certain studies, etc.

Step 2: Configure the recipients and templates in the system¶

As mentioned in a previous section, DEFBioLabs would need to prepare templates that is suitable for each case. After preparing, they would communicate the templates to the C4E.

Accordingly, DEFBioLabs prepares templates for the cases listed below.

Validation errors in sample attributes: A Jinja template is prepared, and communicated to the C4E. The C4E provides the template ID DEF_BIO_LABS_ACCESSIONING_INVALID_INPUTS.

<!DOCTYPE html>
<html>
<body>
    <h1>Accesioning Errors</h1>
    <p>There are validation errors in the samples provided for accessioning.</p>
    <ul>
        {% for field in fields %}
        <li>Sample ID: {{ item["sample-id"] }}</li>
        <ul>
            {% for attribute in field.attributes %}
            <li>Attribute: {{ attribute.key }}</li>
            <li>Value: {{ attribute.value }}</li>
            {% endfor %}
        </ul>
        {% endfor %}
    </ul>
</body>
</html>

Study configuration mismatch: A Jinja template is prepared, and communicated to the C4E. The C4E provides the template ID DEF_BIO_LABS_ACCESSIONING_INVALID_CONFIG.

<!DOCTYPE html>
<html>
    <body>
        <h1>Accesioning Errors</h1>
        <p>There is a configuration mismatch in accessioning.</p>
        <ul>
            <li>Study ID: {{ fields["study-id"] }}</li>
            <li>Study Name: {{ fields["study-name"] }}</li>
        </ul>
    </body>
</html>

Connectivity level issue: A Jinja template is prepared, and communicated to the C4E. The C4E provides the template ID DEF_BIO_LABS_ACCESSIONING_CONNECTIVITY_ERROR.

<!DOCTYPE html>
<html>
    <body>
        <h1>Accesioning Errors</h1>
        <p>There is a connectivity issue accessioning.</p>
        <p><b>Error:</b><pre>{{ fields.error }}</pre>
    </body>
</html>

Step 3: Invoke the API¶

DEFBioLabs has decided to notify the stakeholders on errors coming from the external API. Therefore, when an error occurs, ABCLIMS would invoke the proxy API, and the proxy API business logic then invokes an HTTP client with the parameterised payload.

Invoking based on the error

The arguments given into the payload depends on the error.

For validation errors in sample attributes, the recipients are set to be DEF_BIO_LAB_SSR, and DEF_BIO_LABS_ACCESSIONING_INVALID_INPUTS template is used.
For study configuration mismatch, the recipients are set to be DEF_BIO_LAB_SSR, and DEF_BIO_LABS_ACCESSIONING_INVALID_CONFIG template is used.
For connectivity level issues, the recipients are set to be [DEF_BIO_LAB_SDG, DEF_BIO_LAB_SDG], and DEF_BIO_LABS_ACCESSIONING_CONNECTIVITY_ERROR template is used.

For each of the templates, the arguments are given within the fields attribute.

The proxy API would receive the notification ID as part of the response, and could use the status API to check the status of the notification.

Deciding the Priority

DEFBioLabs decides that the priority used in the notification API would be BATCH, as accessioning in ABCLIMS occur on a per-sample basis. In ABCLIMS, when a bulk of samples is uploaded, accessioning is performed for each sample and a notification is sent for each sample accession. Because this could spam the stakeholders, DEFBioLabs decides that the notifications should be aggregated by the bulk ID (which is the ID of the manifest that each sample is part of and sent along with the webhook by ABCLIMS) using aggregator_id field on the payload. This would therefore result in the Notification API aggregating the notifications for a pre-defined amount of time in an intermediary buffer and send multiple notifications as a bulk.

For invalid data

First InvocationSecond Invocation

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
    "channels": [
        {
            "type": "EMAIL",
            "recipient": [
                "X_LAB_STAKEHOLDERS"
            ],
            "subject": "Accessioning Failure: Invalid input data for manifest",
            "content_type": "text",
            "template_id": "X_ACCESSIONING_INVALID_INPUTS",
            "fields": {
                "attributes": [
                    {
                        "key": "Country",
                        "value": "United Kindm"
                    }
                ],
                "sample-id": "40STDY246690"
            }
        }
    ],
    "priority": "BATCH",
    "aggregator_id": "21784312"
}'

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
    "channels": [
        {
            "type": "EMAIL",
            "recipient": [
                "X_LAB_STAKEHOLDERS"
            ],
            "subject": "Accessioning Failure: Invalid input data for manifest",
            "content_type": "text",
            "template_id": "X_ACCESSIONING_INVALID_INPUTS",
            "fields": {
                "attributes": [
                    {
                        "key": "Reference Genome",
                        "value": "NOT-A-VALID-GENOME"
                    }
                ],
                "sample-id": "40STDY246691"
            }
        }
    ],
    "priority": "BATCH",
    "aggregator_id": "21784312"
}'

For study configuration mismatch

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
    "channels": [
        {
            "type": "EMAIL",
            "recipient": [
                "DEF_BIO_LAB_SSR"
            ],
            "subject": "Accessioning Failure: Study configuration mismatch",
            "content_type": "text",
            "template_id": "DEF_BIO_LABS_ACCESSIONING_INVALID_CONFIG",
            "fields": {
                "study-id": "STDY22231",
                "study-name": "Dengue"
            }
        }
    ],
    "priority": "NORMAL"
}'

For connectivity issues

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
    "channels": [
        {
            "type": "EMAIL",
            "recipient": [
                "DEF_BIO_LAB_DOPS"
            ],
            "subject": "Accessioning Failure: Connectivity Issues",
            "content_type": "text",
            "template_id": "DEF_BIO_LABS_ACCESSIONING_CONNECTIVITY_ERROR",
            "fields": {
                "error": "Error in reaching external API: ABC-API due to throttling.",
            }
        }
    ],
    "priority": "NORMAL"
}'

Here's an overall view of how the accessioning use case utilised the Notification API.

flowchart TD
    A[Samples Uploaded to ABCLIMS] --> B{Immediate Checks}
    B -->|Valid| C[Proceed to Accessioning Job]
    B -->|Invalid| D[Prompt User: Data Error]
    C --> E[Asynchronous Accessioning Process]
    E --> F{Accessioning Result}
    F -->|Success| G[No Notification Needed]
    F -->|Failure| H[Webhook Triggered]
    H --> I[Proxy API Receives Payload]
    I --> J{Error Type?}
    J -->|Validation Error| K[Use SSR Recipient, INVALID_INPUTS Template]
    J -->|Config Mismatch| L[Use SSR Recipient, INVALID_CONFIG Template]
    J -->|Connectivity Issue| M[Use SDG/DOPS Recipients, CONNECTIVITY_ERROR Template]
    K --> N[Invoke Notification API]
    L --> N
    M --> N
    N --> O[Notification Sent to Stakeholders]
    O --> P[Receive Notification ID]
    P --> Q{Check Status?}
    Q -- Yes --> R[Query Status API]
    R --> S
    Q -- No --> S[End]

Use Case 2: Operational Notifications¶

ABCLIMS allows certain stakeholders to be notified when new modules are deployed and/or configured. These stakeholders are primarily operational staff, but they are not limited to operational staff. Some of these notifications can be crucial as some configuration changes could be breaking, and would impact several other processes.

Step 1: Set the pre-requisites up¶

Like the previous example, DEFBioLabs would need to identify which stakeholders need to be notified for each case. After identifying, they would need to set up distribution lists for each stakeholder group, and communicate the distribution lists to the C4E. The C4E would then provide with the necessary IDs.

Following are the groups DEFBioLabs has identified.

Scientific Service Representatives (SSRs): The ID given by the C4E is DEF_BIO_LAB_SSR. This is to notify of any configuration changes for studies and/or projects.
DEFBioLABS Operational Group: DEFBioLABS DevOPS Group: The ID given by the C4E is DEF_BIO_LAB_OPS: This is to notify any sub-module deployments.

Step 2: Configure the recipients and templates in the system¶

As mentioned in a previous section, DEFBioLabs would need to prepare templates that is suitable for each case. After preparing, they would communicate the templates to the C4E.

Accordingly, DEFBioLabs prepares templates for the cases listed below.

Study-level configuration changes: A Jinja template is prepared, and communicated to the C4E. The C4E provides the template ID DEF_BIO_LABS_OPS_STUDTY_CHANGE.
Sub-module deployments: A Jinja template is prepared, and communicated to the C4E. The C4E provides the template ID DEF_BIO_LABS_OPS_MODULE_DEPLOYMENT.

Step 3: Invoke the API¶

Like what is discussed in the previous case, the proxy API would invoke the Notification API with the arguments required for the templates in the fields attribute.

The proxy API would receive the notification ID as part of the response, and could use the status API to check the status of the notification.

Deciding the Priority

DEFBioLabs decides that the priority used in the notification API would be HIGH as there is no need for aggregation, but there is a need for the stakeholders to be notified as soon as possible, with the notification staying in intermediary buffers within the Notification API kept to a minimal.

curl --location 'https://dev.integration-hub.sanger.ac.uk/notifications/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ***' \
--data '{
    "channels": [
    {
        "type": "SLACK",
        "recipient": ["DEF_BIO_LAB_OPS"],
        "subject": "Module re-deployed with critical configuration changed.",
        "content_type": "html",
        "template_id": "DEF_BIO_LABS_OPS_MODULE_DEPLOYMENT",
        "fields": {
            "module-name": "XYZ"
        }
    }
    ],
    "priority": "HIGH"
}
'

Here's an overall view of how the Notification API was used for operational notifications.

flowchart TD
    A[Module Deployed/Configured in ABCLIMS] --> B{Identify Notification Need}
    B --> C[Identify Stakeholder Groups]
    C --> D[Set Up Distribution Lists]
    D --> E[Communicate Lists to C4E]
    E --> F[Receive Recipient IDs]
    F --> G[Prepare Notification Templates]
    G --> H[Communicate Templates to C4E]
    H --> I[Receive Template IDs]
    I --> J[Proxy API Prepares Payload]
    J --> K[Include Recipient & Template IDs]
    K --> L[Invoke Notification API]
    L --> M[Notification Sent to Stakeholders]
    M --> N[Receive Notification ID]
    N --> O{Check Status?}
    O -- Yes --> P[Query Status API]
    P --> Q
    O -- No --> Q[End]