In the first part of this blog series we discussed the benefits of using messaging systems for communication among the systems and participants in production workflows, especially those spanning clouds or organizations. In this blog we’ll go deeper into some of the key concepts that apply to these messaging systems and ways in which interoperability can be achieved.
Messages and Messaging Systems
We need to deal with both simple messages, in near real-time, between two participants, like this:
There are many different types of messaging and event distribution systems. What they all have in common is the ability to distribute data payloads from producers to one or more consumers of that data. In a workflow, messaging payloads often consist of notifications – information about something that has happened in a workflow, like a new version having been created, approved, or published – or requests, e.g., “start the next step in the workflow.” At this level, messaging is machine-to-machine communication; the application or service that receives the message may then convert it into something more appropriate for a person, such as an email or a text message.
The Messaging System has three logical components:
- Producers, which create and send messages
- Message Routers, which securely transmit a message from a Producer to Consumers.
- Message Routers support one or more Router Names, which are used to direct messages from producers to appropriate consumers. Note: In this blog, Named Router is short for “A Message Router that supports a particular Router Name.
The Messaging System itself creates Message Routers and gives Producers and Consumers the (implementation dependent) information they need to connect to them. Since there are so many ways to implement these components,1 this blog will only talk about message systems at the level of these components.
The Messaging System has a few major responsibilities. It creates Message Routers and their Router Names, usually as part of a workflow management process, and it hands out information about authorized Producers and Consumers so they can connect to a Named Router.
Message Routers can have multiple components, depending on the underlying implementation. They can also distribute messages in several ways: fan-out, where a single message is delivered to all the consumers connected to Message Router; round-robin, where each message is read by a single one; and filtered, where the message is delivered to consumers by matching some criteria (which may be in the message itself or known to the router based on the Consumer’s stated needs).
Both producers and consumers must have a connection to a Named Router so they can send or receive messages. This is implementation dependent and can be, for example, a URL, a socket connection, or something else. We won’t need that very often in this blog, and use “connection info” as a generic term for it.
Router Names are the primary way Producers and Consumers discover Message Routers and connect to them. Some implementations will support multiple Router Names in a single Message Router, while others have a one-to-one relationship between a router and a name. Some messaging systems call these “topics”, “channels”, or “exchanges”; the conceptual match is good, though there is a great deal of technical variation in how these other terms are applied.
For our examples, we’ll assume that the Messaging System has been asked to create some Named Routers, and that workflow management distributes their connection info to workflow components that need them. Creating these is generally part of workflow setup, and distributing connection info can be part of setup or done on demand when individual components of the workflow get initialized.
Interoperability and Automation
These machine-to-machine messages can then notify individuals or automated processes about things like publication and status; and trigger workflow automation, such as the provisioning of a creative session for an artist or the start of a rendering session.
To transfer messages across different domains, a message Consumer in one domain can read a message from a Named Router, translate or filter or adapt it as needed, and give it to another Named Router inside the second domain.
Although different messaging systems can use many different low-level protocols which have to be translated across domains, interoperability is vastly simplified if there is agreement on the actual content of the messages. Even if Message Routers use different mechanisms, the messages themselves can be standardized and interoperable.
The Contents of Messages
It’s worth going into this interoperability in a little more detail – we can’t make an interoperable message system unless all the participants know what the messages mean. There are two main parts to this:
- A shared way of describing what the message is about. This includes both events and commands . Events say that something has happened and include, for example “this asset has changed” or “this process is complete.” Commands are something that someone wants to happen and include things like “approve this” and “transform this OCF into a proxy.”
- A shared way of describing everything the message has to contain that is needed to understand and act on a command or respond appropriately to an event. For example, an event that indicates that an asset has changed needs a way to identify the asset, and a request to turn an OCF into a proxy needs to say which OCF is involved, and maybe even which service provider is to do the work.
To cover the first point, we are working with industry partners on common names for standard events and commands in the workflow. For the second, much of it can be based on the Ontology for Media Creation (https://mc.movielabs.com), which provides a shared way of describing workflow components, and the use of identifiers in the messages rather than full data payloads (as discussed in our blog on resolvers here: Through the Looking Glass – MovieLabs.)
Examples:
The examples use a streamlined view of the dailies process. We won’t cover all the pieces, but the ones not covered can use messaging in the same way as the parts we look at.
Message Producer and Message Consumer are very generic concepts, and we use them to refer to applications that actually produce or consume messages, such as a dashboard or workflow automation service. Consumers can deal with messages in many ways, for example by acting on the message itself or by sending it on somewhere else.
Often, both producers and consumers are integration points that connect applications and services that are not message-aware to ones that are. For example, in the dailies process, “unload data cards” can be a single application, but it could just as well be a script that waits for something to happen in a watch folder and sends a message when something new appears. Similarly, “review and approve” can be an application that receives and acts on messages directly from “grade color”, or it could be an application that converts the approval request into an email which is them sent to the participant responsible for review and approval; its outbound messages can be sent by an application, through a service fronted by a web form, or by email to a system that converts the email into a message.
Example 1: Messaging in an Approval Workflow
First, we’ll look at the handoff from “grade color” to “review & approve.” Review and approval is an iterative process, and later we’ll expand that in more detail.
Once color grading is done, the result has to be sent to someone to approve. This has two parts: the approver has to be notified that there is something to approve, and then send out the result of the review. Workflow management creates two Named Routers, one called “request color grading” and the other called “review color grading.” The color grading component is a consumer of “grade color” messages from the “request color grading” named router and a producer of “request review” messages which it sends to the “review color grading” named router.
When the Edit stage is done, it sends a message to the “request color grading” router, indicating the Asset, and a note saying something like “reason: initial grading.” The Grade Color task reads from that router and does its work – it can allocate the work to an individual or an internal system that manage pending work (perhaps itself message based). Work happens, and the asset is ready for review. The Grade Color task sends a message to the “review color grading” router. The “Review & Approve” task can send two messages: “approved” and “rejected: reason.” It sends the first one to the Approval Notification message router (see the next example) and the second back to the “request color grading” router, where someone will pick it up.
An important thing to note is that very few messages exist on their own; they are part of some grander context. For this example, the context might be a work order, which is passed with all of the messages.
Example 2: Messaging to Multiple Recipients
In the first example, we saw individual messages going from one producer to one consumer, and how a consumer can receive messages from multiple producers. An equally common case is getting messages to more than one consumer. For that we’ll look at the output of the “Approve Color Grading” task.
The workflow, as diagramed, shows only a single consumer for the approval message. It is much more likely that multiple participants will be interested in the approval. For this example, we’ll add another: a workflow management console application in Edit and Color Management Workflow that is used to track progress, look for late delivery and so on. The message still has to go to the Dailies Screening task, of course.
Review & Approve sends the approval message – an event – to the Approval Notification named router which, for now, has two consumers – the E&CM management console, and the starting point of the Dailies Screening task.
Approval Notification is explicitly a fan-out Message Router – the same message is delivered to multiple consumers. This wasn’t necessary for the other message routers in the previous example, but they could well have been, to support locally centralized monitoring and logging.
So far, so simple, and it’s easy to see how fan-out applies to all sorts of notifications.
Dailies Screening adds its own things to incoming messages’ context and sends it to a Dailies Screening message router. (It can also add an entirely new context.) Since the message contents are standard, there should be very little need to translate the message itself, other than adding or changing the context.
That message is received by two applications: one the local management and tracking console, as before, and the other a “start workflow” task. (It could be set up to send a message directly to the first step of the dailies process, of course.)
“Start Dailies” reads the message and sends a request for work to the “dailies screening: transcode” task, which receives it and starts work.
This may look complicated, but the only new concept it has introduced is fan-out Message Routers. By using standard messaging system components, we are able to:
- Send the same message to multiple recipients
- Translate from one infrastructure to another with an application that receives from one messaging system and sends to a different messaging system.
- Only add locally important information to the message, rather than having to translate the entire thing
There is complexity, of course, in deciding which message routers are needed to support a particular workflow, and even for a single workflow there are many ways to design the system. Workflow designers also have to think about the kinds of context they need in messages, especially when they cross from one domain to another. However, once these systems are established, they can run 24/7 without interruption and as organizations can build “libraries” of messages and messaging systems they can be arranged and rearranged based on the specific needs of each production or task.
Next Steps
As we’ve demonstrated there are considerable opportunities for our industry to automate mundane and repetitive tasks with computer systems. But as our workflows are complex and inherently multi-team and often multi-organizational, we need to take the time upfront to define a flexible communication system to allow those systems to talk to each other. We hope these introductions to Messaging Systems explain why we believe they offer the solution, if correctly designed, to our workflow challenges and provide the basic components required.
We have deliberately not opined on which organizations should build, operate, and integrate with such messaging systems as we believe there’s considerable opportunity for multiple companies to participate and create value by enabling this interoperability. But we do believe that the interoperability and reusability of messaging system integrations can strongly benefit from common practices on the contents of message headers and some aspects of their payloads, especially for recurring patterns like “Review & Approve.” We look forward to working with industry partners on developing these and to seeing them used by workflow management systems and integration platforms to improve automation and reuse.
Expect to see more from MovieLabs as we explore message headers and payload contents, how messaging interfaces with workflow-driven security (see CSAP) and our connected ontologies. In the meantime, feel free to reach out and join this discussion – send us a message and tell us what you think of interoperable message systems for content production.
