Communication Platform Proposal

Author: Rodda John
Status: Draft

Overview

Moab will need to send transaction emails and text messages. This MEP outlines the architecture of a communication platform to handle these communications.

Message Templates

A template represents a type of transactional message that is sent by the application.

A template is defined in code (as opposed to e.g. yaml or pkl) as it references models and schemas used elsewhere, as well as has callbacks.

A template exists to centralize and group logic that is related for messages of the same type. Such a centralization also has monitoring and preferences benefits (e.g. all message sending errors of the same type, allowing a user to disable messages of a certain type).

A template governs the two step process necessary to enqueue a message to the message service (described below). The first step is translating the product-aware inputs into JSON for the template engine; the second step is to hit the templating engine to return back the raw data to then enqueue.

The interface between Moab and the template engine should be JSON compatible to allow for an external renderer if necessary. Such a renderer is out of scope here.

For email, the template engine should return HTML, whereas for SMS, raw text is acceptable.

This process (sans message enqueing) is identical to how a PDF renderer would work, as it also takes product aware variables and outputs some result from a templating service.

Template Configuration

A template must allow for specifying the various context pieces that are needed to send a message of this type.

Context is either a raw schema or is an object, which is accompanied by a serialization function.

class Template:
    ...

    context = [
        RawContext(
            name='raw',
            schema=SomeSchema,
        ),
        ObjectContext(
            name='user',
            model=User,
            schema=UserSchema,
            transformer=transform_user,
        ),
    ]

Raw context is used to refer to arbitrary data the sender should pass in.

Object context is used to refer to an object that the sender should pass in, that is then transformed -- using the supplied transformation function -- to the context.

The above example would result in JSON that would look like:

{
    'raw': {
        # Contents of the raw schema
    },
    'user': {
        # Contents of the user schema
    }
}

The template engine can then choose to use this JSON however they want.

The send message callsite for this template may look like

Template.send_message(
    raw=SomeSchema(...),
    user=User.objects.get(id=1),
    ...
)

A template should also have a number of other fields:

Field	Description	Example
`slug`	A unique identifier of the template	`signed-up`
`type`	What sort of communication method this template supports	`EMAIL or SMS`

Templates should ideally be registered using a registry pattern so as to allow static validation. Alternatively, if mypy supports static evaluation of protocols, that is preferable given there is less code required than in the case of registries.

Idempotency

A template shall require an idempotency_function be defined.

This function shall accept the same inputs that the send message call accepts recipients, all objects and context, as well as the timestamp, and must return a single string representing the idempotency key for this specific send_message invocation.

These idempotency keys will be hashed and stored on the MessageQueue objects; but if a second message is enqueud with the same idempotency keys, it will be dropped.

This could be used to do naive rate limiting (e.g. bucket by hour).

Template Engine

Every template will be required to have an engine parameter, which should point to a class that implements a single method designed to render a template.

A template engine can optionally declare a metadata parameter, which is a pydantic schema against which the template_engine_kwargs parameter will be validated against. This is envisioned to allow passing credentials, a path, or a template.

class TemplateEngine:
    def render(self, data: dict) -> str:
        ...

The first template engine will be jinja, and a a sample implementation could look like the following:

env = Environment(
    loader=PackageLoader("app"),
    autoescape=select_autoescape(["html", "xml"]),
)

class JinjaTemplateEngine:
    def render(self, data: dict) -> str:
        env.get_template("template.html").render(**data)

In the case of multiple recipients of a message, the template engine shall be called per recipient. This is to allow template variables like {{ recipient_name }} to function.

By default, recipient_name shall be provided as a template variable. This list may expand in the future.

Recipients

Obviously, a send_message call requires recipient information.

By default, this is passed in as a list of Recipient objects to send_message.

Each Recipient object either has a user or a manual_recipient sub dictionary, which in turn defines an email and or phone number.

A sample implementation of a Recipient base class:

class ManualRecipient(TypedDict):
    email: str
    phone: str
    name: str

class Recipient(TypedDict):
    user: User | None = None
    manual_recipient: ManualRecipient | None = None

Send Message Call

A send message call should create a Message object that stores relevant metadata about the sending of the object.

A template engine should not be hit synchronously in the send_message call. That should instead be queued and polled for. This is to allow for easy scaling if the template service is external.

Message Enqueing

Once a message has been rendered into html or txt, it now needs to be sent.

This will be accomplished by writing a MessageQueue model in PG that indicates a message is queued to be sent.

Field	Description	Examples
`id`	`PK`	`1`
`message_id`	FK to `Message`
`type`	Email or SMS	`EMAIL or SMS`
`business_id`
`recipient`	An email address or phone number
`body`	HTML / TXT stored in a PG `TEXT` field	`rodda@trymoab.com`
`subject`	TXT stored in a PG `TEXT` field	`Hello!`
`attachments`	List of `CustomerUpload` IDs	`[1, 2, 3]`
`status`		`pending`, `sent`, `failed`
`created_at`
`sent_at`
`idempotency_key`	Hashed idempotency key
`remote_id`	The ID of the message in Sendgrid or Twilio

These message queues shall be polled for frequently (every second) and then sent. When handling a message queue, there should be a WITH FOR UPDATE lock acquired on the relevant row in PG.

Thought: why couldn't this simply be all stored on the Message?

Logging

Create a separate MessageLog table that is related to both messages and optionally message queues. This should record all incoming webhooks regarding a message but also store failure information as well as store state transition information as relevant on a message.

Appendices

On Transaction Guarantees

Notice the only thing that happens when sending a message is writing models to a database. This means we get transactional guarantees by default -- namely that if the surrounding transaction is rolled back, the message would not get sent as the INSERT would never be commited.

This ought be preserved.

Scheduled Messages

A message queue could very easily have a scheduled_at field which is then honored in the sending.

Digests

In theory, it should be possible to specify a frequency by which a message type would be digested. This could be intercepted at the message queue level. This is a little complex because the templates may have to be regenerated. Think about this more before implementation.

Message Responses

It will likely be important to handle message responses as well as CTAs within such a platform. This will allow Moab to handle more CRM-lite applications, like text conversations.

These should be handled by specifying a callback function on a message template that is called upon any incoming webhook regarding that message. These webhooks should be sanitized into a single MessageResponse datatype.

This response should include the recipient (match it to a user if possible).

Reliable Asynchronous Tasks

How much do we trust Dramatiq?

There are a two instances of custom polling logic here (Message and MessageQueue); perhaps create a generic action queue concept that is then polled for.

This is functionally creating a PG backed message queue -- which has its downsides, but as opt-in could be good.