Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This section lists the technical requirements of this Building Block.
In addition to the functional requirements described in this section, GovStack's cross-cutting requirements defined in the architecture specification document and GovStack Security Requirements, apply.
The queuing process varies according to the policy and message type attached to the messaging, if an emergency messaging type applies to it, the messaging gets prioritized, otherwise, the FIFO algorithm (First In, First Out) applies to other message types. It saves all the requests for further processing, furthermore, works in the form of an "AS IS" process analysis without changing anything on the request. The message queueing mechanism will:
save a hash (MHASH) of the original request and a Unique Message ID (UMID);
respond with "Message received" to the sender with generated UMID.
Messaging requests go through a final check to be clean of defects and inconsistencies, to check with external systems as necessary:
check for message duplicates by making an actual request to a database to see if entries with MHASH and UMID exist;
low-level validation of data types and data completeness;
check for inconsistencies;
standard error messages and codes are used as a response to inconsistent requests.
In relation to batch logic, messages are scheduled against the availability of systems, throughput limitations, and rules set by programs.
Regular and repeat messages are scheduled.
Batches may be given prioritization in the queue according to the message settings.
The essential control logic may be included here specific to the individual batch sending and resending.
Additional workflow checks as required, including resending failed transactions.
Handles the initiation of input data processing from all the API messaging calls and makes the API access control verification from other Building Blocks to the Messaging Building Block and vice versa as well as within the Messaging Building Block, it will:
use Identity and access management for authentication;
perform input validation checks to prevent oversized message attacks, SQL injection attacks as well as JSON and XML threats;
require authentication for all API users;
manage access quotas and throttling;
logging of all API calls made;
allow API providers to limit the rate of consumption for all API users;
transform backend error messages into standardized messages so that all error messages look similar (this also eliminates exposing the backend code structure).
This request could come from two sources: external or internal. An external source could be another GovStack Building Block (e.g. the Registration Building Block). Either source must be appropriately authenticated and authorized to initiate the request. The request must contain at a minimum: the contact address (email, phone number, etc.), the message type, the content of the message, and the initiating source’s unique transaction ID.
Find unprocessed requests from a database.
Prepare each request for actual processing, requests may come as single or batch messages and every message needs to be treated as a separate entry.
It prepares unprocessed requests for actual processing.
Each component of the messaging building block should be capable of producing transaction logs. This is important to ensure that the system can be adequately monitored and troubleshooting can be performed efficiently and effectively.
The components should also generate transaction logs for events that capture at least the following information:
transaction date and time;
transaction source;
transaction destination;
supplementary data;
transaction status (success, failed, in progress).
The event logs and the transaction logs should NOT capture any sensitive data such as contact data, or channel authorization credentials.
Fetch and securely archive logs from working nodes is also expected.
This section lists the technical capabilities of the Messaging Building Block. The following components are needed to achieve the technical functionalities of the messaging Building Block.
The Messaging Building Block is part of the GovStack implementation and is prompted to open up a two-way communication channel to deliver and capture messages/data between the GovStack systems and the end users, physical and legal persons.
As a rule, the Messaging Building Block does not own any information but is carrying the minimum amount of (meta) data needed to transfer/publish a message. Messaging Building Block allows other Building Blocks and applications to reach out to customers/citizens in order for them to be able to make a decision or access a service. This Building Block can also be used for broadcasting a disaster message.
Messaging policies are configurable modalities and channels that can be accessed/utilized through the Messaging Building Block in order to reach out to the users or groups. This Building Block is using existing applications/channels, such as e-mail, SMS, and messaging platforms for enabling users to subscribe to an application/channel provided by Messaging Building Block itself for text-based content.
Having published its messaging policies and services at the Information Mediator Building Block, the other Building Blocks and applications can discover and use the services of the Messaging Building Block.
Messaging Building Block allows for asynchronous communication, using either point-to-point communication or a Pub/Sub model (using Information Mediator Building Block) where the user is subscribed to a message room/group and will be receiving all messages intended for the “mother of newborn child” group.
Security: Before allowing any messages to be published, the Messaging Building Block fetches a corresponding ID and a role or a session token available for the user for authentication and access purposes. Incorporates privacy into its design when the purpose of the authentication is not revealed if a service provider sends an authentication request.
This section provides information on the core data structures/data models that are used by this Building Block.
The resource model shows the relationship between data objects that are used by this Building Block.
7.2.1.1 Person
Name
Type
Description
Notes
Surname
String
Family name
First Name
String
First name
Birth Date
Date
DOB
Mobile Number
String
Phone number (mobile) of user
Government ID number
Integer
Government issued ID number
Used when linking to global ID Building Block.
Each model schema MUST have a corresponding JSON Schema (API definition file).
. Should include country code.
This section provides a reference for APIs that should be implemented by this Building Block.
This section provides a reference for APIs that should be implemented by this Building Block. The APIs defined here establish a blueprint for how the Building Block will interact with other Building Blocks. Additional APIs may be implemented by the Building Block, but the listed APIs define a minimal set of functionality that should be provided by any implementation of this Building Block.
The GovStack non-functional requirements document provides additional information on how 'adaptors' may be used to translate an existing API to the patterns described here. This section also provides guidance on how candidate products are tested and how GovStack validates a product's API against the API specifications defined here.
The tests for the Messaging Building Block can be found in this GitHub repository.
This API is designed to send out emails one by one.
Although an email can be sent to multiple recipients by using email
, emailCC
and emailBCC
appropriately, this is still considered as sending a single email.
As a Doctor working in a hospital, I want to notify my patient about a confirmed registration so that the patient would know when to come for a visit.
This API is designed to send out identical emails to multiple (batch) recipients at once.
As a Hospital I want to notify a specified group of my clients about the possibility to register to vacant time slots within a limited period of time so that the clients potentially needing it the most would have a guaranteed chance to get an appointment
This API is designed to accept responses to emails that have been previously sent out by using the Messaging Building Block. The protocol itself is lightweight and expects custom services from anyone using it. The Messaging Building Block is responsible for delivering the message and does not know (or want to know) anything about the content of it, how it will be used, etc.
As a Hospital, I want to get and send responses from and to clients so that I could continue and keep track of email conversations taking place via GovStack
This API is designed to provide information about the status of emails sent to the Messaging Building Block.
As a Doctor working in a hospital I want to know which emails I have sent haven't reached their destination so that I could contact such patients by using different means
Possible status of emails that can be queried are as follows:
scheduled
processing
deferred
bounced
not sent
delivered
There is no keeping track of emails which have been sent successfully, as this would end up creating and maintaining a massive database of emails, their senders and recipients, etc., which would be wrong for so many reasons.
Developed by Rainer Türner (Information System Authority of Estonia), Magnus Hult (Element.io), John Cordeiro and Martin Karner (Estonian ICT Cluster), Bramhanand Jha (India's National e-Governance Division)
This section provides a detailed view of how this Building Block will interact with other Building Blocks to support common use cases.
A workflow provides a detailed view of how this Building Block will interact with other Building Blocks to support common use cases.
This section lists workflows that this Building Block must support. Other workflows may be implemented in addition to those listed.
The main prerequisite for Person-to-Building Block communication is that there is an existing Sender/Source Building Block with the following properties:
the relevant credentials and details about the Person/Citizen to be addressed with;
the content of the message and a reference to a communication channel (contact details) to be used;
additionally, the service discovery service at the Information Mediator Building Block needs to be active. Information Mediator Building Block publishes the list of available services of the Messaging Building Block to the source GovStack Building Block, i.e. Workflow.
A reference token should be carried throughout the communication session in order to save a point of reference for reverse communication back from the Person to the Building Block. In other words, the main prerequisite for Person-to-Building Block communication is the availability of a communication channel and a reference token.
This generic workflow is used to transfer messages between GovStack Building Block and the end user, Person. Data is submitted from a GovStack Building Block front-end application. This workflow shows a connection to GovStack Building Block (such as a government Health System Application) to convey a message that is associated with a real person.
This workflow requires interaction with the Information Mediator Building Block and a source GovStack Building Block, for example, a Health Care Service Building Block or registry Building Block.
The sequence diagram shows the flow of data between Building Blocks for this workflow.
Messaging Building Block accepts a message from GovStack publisher
Publisher Building Block or service conforming to the Messaging BB schema
Reject messages that do not comply with expected schema
Retrieve Person and Contact Channel Data from the incoming Message
Map retreived data with appropriate recipient
Technical mapping is an internal service of the Messaging BB
Publish Message through Communication Channel
Communicate Message Data and User ID to Communication Channel
Reject messages that do not comply with expected schema
Publish Status for the original sender
User and Message IDs with Delivery Status containing date and time
The Message's unique ID is preserved to keep up its status updated
Message sent through the communication channel / service provider
Text message and User ID
Reject messages that do not comply with expected schema
Retrieve Person and Contact Channel Data from the incoming Message
Map retreived data with appropriate recipient
Technical mapping is an internal service of the Messaging BB
Confirm message received
Message Delivery Data Structure following Communication Channel standards with Status
The Message unique ID is collected to keep up other statuses updated
Publish Status for the original sender
User and Message IDs with Delivery Status containing date and time
The Message's unique ID is preserved to keep up its status updated
The following diagram illustrates the internal processes of the Messaging BB to send messages received as external requests and sent via external service providers.
The following diagram illustrates the internal processes of the Messaging BB when providing status report for messages that have been passed on by external partners for further processing by the Messaging BB.
Terminology used within this specification.
Message
The Message is the main entity of communication in this Building Block, it holds all the necessary information to deliver the information to the Publisher or to the Subscriber/client, leaving just a track of delivery Logs with no personal data included. Messages follow a predefined protocol and policy for the chosen or predefined Communication channel. As a rule, the delivery mode of the Message is asynchronous.
Publisher
This is the originator of the Message, holding information about the content of the Message and the (meta) data regarding the Subscriber who will receive the Message. In other words, Messaging Building Block proxies/relays the Message received by a Communication channel to the proper Subscriber/client/person.
Subscriber
In the Publish and Subscribe use case, where the Publisher is relaying the Message into the hands of Messaging Building Block, that will in turn route it to the desired Communication channel, the Subscriber is the one who is listening to the relevant Message Room to receive the Message from the Publisher. The Subscriber has given consent to the Publisher to access or retrieve and store its data for the purpose of carrying out the Messaging event.
Messaging event
Messaging event is the (single) act of asynchronous information transfer between the Publisher and the Subscriber, consisting of the following steps: 1) secure identification and access management of the Publisher and the Subscriber; 2) access to and/or retrieval of relevant Subscriber data and consent for using the data from the Subscriber; 3) selection of relevant data transfer policy and protocol for the Messaging event; 4) Management and processing of Logs and relevant metadata during and after the Messaging event.
Messaging protocol
It is the protocol enabling the Messaging Building Block to communicate and transfer data between appropriate endpoints.
N.B. In its initial versions, GovStack Messaging Building Block is using protocol as a reference point for creating the GovStack Messaging Building Block protocol.
Communication channel
It is the endpoint of the Message delivery, the means used by the Publisher or the Subscriber to communicate. It can be a social media tool, e-mail or SMS service, or a distributed network.
Log
It holds track of Message delivery with the encrypted content to help with the maintenance and troubleshooting of the messaging services.
Routing Policy
Routing policy is the essential, configurable service of the Messaging Building Block, published in the Information Mediator Building Block, and is enabling the Publishers and the Subscribers to pre-define and use the appropriate Routing Policy for the particular Messaging event. In future iterations of this specification, the Routing Policy can be applied to the Messaging Rooms.
Service provider
Service providers provide service(s) to End clients. For example, medical doctors, hospitals, government agencies, etc. In some cases, a Service provider can also be in a role of an End client, being served by another Service provider.
End client
In most cases, End clients are citizens of any given country contacting and/or requesting or being served by a Service provider. For example, a mother/caretaker of a newborn child being serviced by a Health Care Service provider.
This section will highlight important requirements or describe any additional cross-cutting requirements that apply to this Building Block.
The cross-cutting requirements described in this section are an extension of the cross-cutting requirements defined in the . Any implementation MUST adhere to all requirements from .
There are two main reasons for using denormalized databases instead of traditional normalized databases. First, to significantly improve the performance of read queries. This is highly important as databases of the Messaging Building Block will be under a heavy load of requests that in many cases are expected to respond near real-time. Second, to be able to export the content of databases in the format of time-series databases (see below for its reasons). To achieve this, using "UPDATE" and "DELETE" queries in SQL commands SHOULD be disabled. If not, they MUST be avoided by developers and caught by automated tests if still used.
By having this capability, it is possible to delete the content of processed data from production-level server databases right after they have been securely archived. This is important to reduce the impact of potential data breaches by security incidents of potentially N number of application servers hosted by different parties.
If done so, using archived datasets, either fully or partially, allows to recover and/or re-engineer lost databases (or just datasets of interest) automatically and in a matter of seconds if such pipelines have been predefined. Also, by converting the content of relational databases into a format of time-series databases, these purely text-based datasets are easily usable by different kinds of databases, analytics tools, etc., not just by the relational database used at first.
Technical components (with the exception of relational databases) must be stateless in the sense of the .
At least on the level of active-passive nodes based on the principle of "eventually consistent". Pending: initial state for all messages waiting to be queued.
Queued: messages that are in the queue to be sent.
Sent: messages with proper confirmation that was sent to the provider.
Delivered: messages with proper confirmation that was delivered to the end-user.
Errored: messages with an error during delivery.
Failed: messages that are errored and we gave up sending.
If Messaging Building Block protocols are followed, the technical stack used to provide messaging as a service is vendor-neutral, meaning that anyone is free to use technical solutions of their own choice.
Only allowed participants are allowed to provide and use the services of Messaging Building Block.
Messaging Building Block is a dumb messaging proxy that gets all of its relevant input from its clients. This also includes the names and contact information of all relevant participants. Such sensitive information may not exist in its raw and/or re-engineerable form nor in databases or logs.
Every instance of a service must generate audit logs containing information about all of its actions regarding timestamps, related parties, identifiers to operations performed, and HTTP response codes for any given action. Audit logs may not contain the actual content of the message. Audit logs must be in a time series database format. Audit logs must be accessible by a given endpoint by the central service provider only. Audit logs should be periodically removed from their original source after a defined period of time.
All APIs are exposed via secure socket connections (HTTPS).
Manage access quotas and throttling.
Logging of all API calls made.
Allow API providers to limit the rate of consumption for all API users.
Transform backend error messages into standardized messages so that all error messages look similar; this also eliminates exposing the backend code structure.
Contact address (email, phone number, etc.), the message type, the content of the message, and the initiating source’s unique transaction ID.
Asynchronous messaging processing. OpenAPI specifications for calling this function; resource models and data structures, internal and external interfaces.
Stateless architecture.
Message broker tool to enable performant queueing mechanisms such as RabbitMQApache/Kafka, or GRPC for data transfer speed purposes.
Databases with unstructured data should be treated with Elasticsearch/Logstash.
End users should be registered as Message queue clients/subscribers in the Messaging Building Block. Subscription is required to receive a message.
In order to prevent single points of failure, messages must be replicated on at least 2 different service providers. In case of message replication, all unprocessed replicated messages must be kept track of and relevantly updated in order to prevent re-processing them.
Admin of the room must be able to choose the policy profile with the configuration of the message provider, e.g. retrial frequency.
Unsent and unsuccessfully delivered messages must remain in a queue until being successfully delivered or otherwise permanently processed.
Pending (initial state for all messages waiting to be queued); Queued (messages that are in the queue to be sent); Sent (messages with proper confirmation that was sent to the provider); Delivered (messages with proper confirmation that was delivered to the end-user); Errored (messages with an error during delivery); Failed (messages that are errored and we gave up sending).
Messages not delivered in a period of 24 hours (any errored or queued messages more than 24 hours old must be labeled as failed and go out of the queue); Messages retrial (Errored messages must be retried for 24 hours).
Client applications must send authorization tokens in the authorization header of the request to authenticate users and the API Management Gateway will verify whether the token is valid.
Perform input validation checks to prevent oversized message attacks, SQL injection attacks as well as JSON and XML threats.
The following standards are applicable to data structures in the Messaging Building Block:
All services are provided as REST API requests
Only Transport Layer Security (TLS) Server Name Indication (SNI) extension-compatible requests are allowed
REST APIs calls and responses use JSON as the only allowed data format.
Depending on the specific use case, this may mean or .
Using on dates and times
on text input
Key Digital Functionalities describe the core (required) functions that this Building Block must be able to perform.
The Messaging Building Block is a standalone messaging gateway/client/proxy (including all possible modalities: email, SMS, etc.) but can also facilitate connections with existing messaging service providers by enabling safe communication. The communication in this Building Block can be categorized in two different directions as follows:
In scope of v1
informing End clients about their registration;
reminding and requesting End clients' confirmation for appointment booking or other events;
getting confirmation about government services and interactions.
In scope of future releases
delivering information at scale to End clients;
alerting End clients in emergency contexts.
In scope of v1
confirm Client registration for a service;
providing data to the government/service.
In scope of future releases
asking for information about government services.
This section provides context for this Building Block.
The Messaging Building Block provides a standardized, secure communication channel between GovStack service providers and end customers (in most cases citizens). By using GovStack Messaging Building Block, service providers rely on the building block's central functionalities, logging, back upping, security features, etc. without the need to reproduce and maintain them by themselves.
For greenfield developments, it helps developers build communication components on top of different GovStack services, to make use of federated architecture when passing messages, including machine- and human-triggered (both synchronous and asynchronous) ones.
Messaging Building Block is a standalone messaging gateway/client/proxy (including all possible modalities: email, SMS, etc.), but can also facilitate connections with existing messaging service providers, hence making the adoption process less disruptive for end users.
This Building Block can be used for messaging events in many different applications and use cases, such as:
Example A, encrypted: after being enrolled into a Health Care Program for the mothers of a newborn child, a woman/caretaker receives a message from the Health Care Service Provider about a doctor's appointment. The message is routed to an existing communication channel/application, in use and provided by the mother.
Example B, unencrypted: a reminder/notification message from the service provider to the mother about the same appointment. If a doctor decides to hold the consultation remotely, a link is sent to the mother and a video consultation is started (the conference is started and users join the room). The event is triggered by the Scheduler Building Block component and a new message is processed to the client.
Example C: a user-initiated message regarding a doctor consultation, targeted to a Health Care Provider's phone number or a chatbot URL (universal or targeted messaging back to the system).
Messaging Building Block is enabling signing up for and receiving services, such as:
mother and childcare;
social care;
receiving entitlements from the government;
opening bank accounts;
communicating disaster messages or reminders about the voters' rights during elections;
etc.
The use cases are detailed in the section of the following document: ANNEX I - Use Case Tables.
Communication between different government services (government to government) is out of the scope of this Building Block but can be enabled through the Information Mediator Building Block.
Scheduling messages according to some business logic is out of the scope of this Building Block because is done by Scheduler Building Block.
Processing of incoming message content to apply some business logic.
Fully offline and no internet connection scenarios.
Real-time video and audio conferencing.
A historical log of key decisions regarding this Building Block.
A list of topics that may be relevant to future versions of this Building Block.
The version history table describes the major changes to the specifications between published versions.
0.9
Martin Karner, Rainer Türner, John Cordeiro
Technical Reviewers: Tony Shannon, Saša Kovačević, Riham Moawad, Riham Fahmi, Aare Laponin, Manish Srivastava, Palab Saha, Surendra Singh Sucharia, Arvind Gupta, Gayatri. P., Shivank Singh Chauhan, Gavin Lyons
1.0
Martin Karner, Rainer Türner, John Cordeiro, Dr. P. S. Ramkumar, Magnus Hult, Bramhanand Jha
Reviewers: Wesley Brown, Steve Conrad, Valeria Tafoya
Final edits to align content to specification template for GovStack 1.0 release