[et_pb_section fb_built=”1″ _builder_version=”4.11.4″ _module_preset=”default” custom_margin=”||||false|false” da_disable_devices=”off|off|off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row _builder_version=”4.11.4″ _module_preset=”default” custom_margin=”||||false|false” custom_padding=”0px||||false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.11.4″ _module_preset=”default” pac_dcm_carousel_specific_module_num=”0″ global_colors_info=”{}”][et_pb_text _builder_version=”4.11.4″ _module_preset=”default” background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”||||false|false” custom_padding=”||||false|false” global_colors_info=”{}”]<\/p>\n
Clear communication is critical to the content creation process. And while today\u2019s productions somehow manage to compensate for inefficient communication mechanisms, there is a growing and urgent need to streamline the way we communicate and exchange information as we continue to scale up to meet the increasing demand for content. \u00a0In our blog “Cloud. Work. Flows”<\/a>, we identified some missing components that are required to enable software defined workflows. We highlighted that a more efficient messaging system will be critical to improve communication between participants (which could be people or machines) in a complex workflow system.\u00a0 We\u2019ve addressed communication elements in the Ontology for Media Creation<\/a> which covers some aspects on what<\/em> needs to be communicated. Recently we\u2019ve been turning our attention to the how<\/em>\u00a0to express that communication in the most efficient manner.\u00a0 For example, in the 2030 Vision<\/a> our first principle states that content goes straight to the cloud and does not need to be moved. Once ingestion to the cloud has completed, the first participants in the chain will need to be notified that the content is now in the cloud and ready to be worked on and ideally including a location for that content.\u00a0 A similar workflow notification message is required when a task has been finished and the work is ready for review by another team member.\u00a0 In this post we\u2019ll discuss the benefits of a common approach to communicating these repeating types of workflow messages. In a subsequent post we\u2019ll get into the technicalities of how we think such a system could be built including considerations to enable it to span cloud infrastructures and tools.<\/p>\n We need to deal with both simple messages, in near real-time, between two participants, like this:<\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/movielabs.com\/wp-content\/uploads\/2022\/05\/messaging_system.gif” alt=”Simple Message System” title_text=”Simple Message System” _builder_version=”4.11.4″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][et_pb_text _builder_version=”4.11.4″ _module_preset=”default” background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}”]<\/p>\n Synchronous Real-Time Messages between two known participants.<\/em><\/p>\n And more complex messages, especially as we move to more automated systems, where the participants may not know who will pick up the messages and may not receive replies for hours or even days. Take for an example:<\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/movielabs.com\/wp-content\/uploads\/2022\/05\/render.gif” alt=”Render” title_text=”Render” _builder_version=”4.11.4″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][et_pb_text _builder_version=”4.11.4″ _module_preset=”default” background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”||0px||false|false” custom_padding=”||||false|false” global_colors_info=”{}”]<\/p>\n Asynchronous Messages between one sender and multiple potential recipients.<\/em><\/p>\n In this example the Render Manager (the message sender) doesn\u2019t know which nodes may respond or when they may respond. There are thousands of such nuanced examples in production workflows that we need to consider when thinking about the sorts of messages that could be sent between systems. We need a messaging approach that can accommodate all of these message types, and also the complexity of multi-cloud infrastructure when messages may be flowing between systems that are not all owned\/leased or operated by the same organization and on the same infrastructure.<\/p>\n At MovieLabs we\u2019ve been thinking about approaches to these messaging problems. One approach is using point-to-point API calls between all these disparate systems and while appropriate for many use cases we don\u2019t believe this will scale to whole productions or studios \u2013 there would simply be too many custom integrations to get all the possible components of a workflow to work together[1]<\/span>.<\/a> We see the best way to manage the highly asynchronous delivery of information to multiple (potentially unknown to the sender) destinations is to decouple the mechanics of the communication \u2013 the what<\/em> from the how<\/em>.\u00a0 In software systems this can be managed in a more automated way using Message Queues[2]<\/a><\/span>. A message queue allows a message to be sent blindly (the sender does not need to specifically know who will read it). Specific queues are typically associated with a particular topic, any other participant with an interest in that topic can then subscribe to the queue and receive its messages whenever they\u2019re ready.<\/p>\n Message queues \u2013 or, more broadly, message systems \u2013 are a natural fit for software-defined workflows: their raison d’\u00eatre<\/em> is to provide a communication mechanism where senders and receivers can operate without knowing anything about each other beyond how to communicate (the message queue) and some expectations about the contents of the communication (the messages.) This separation allows applications to run independently and, just as importantly, be developed independently.<\/p>\n As long as the sending and receiving applications can both access the message queue, it doesn\u2019t matter where the applications are running; they can be in the same cloud, in different clouds, on a workstation in a cloud, or even in two organizations. Agreeing on commonality of some aspects of message headers and message contents can enable interoperability especially in that cross-organizational use case. For example, if a message from an editorial department to a VFX house includes a commonly agreed upon place to put shot and sequence identifiers, a workflow management system at the VFX house can route that message to the appropriate recipients internally.<\/p>\n The use of messaging systems rather than point-to-point integrations also makes it easier to gather together operational data for logging, dashboards, and detecting\/acting on exceptions and errors.<\/p>\n As we look to the 10 Principles of the 2030 Vision, we can see that messaging is key to enabling the “publish” function (where access to files is pushed through a workflow, as tasks are created) and enabling participants to \u201csubscribe\u201d to those files, tasks, or changes. Principles 1 and 2 of the 2030 Vision state that assets go to the cloud and do not need to move, which means sharing the location of those files<\/a> becomes a key message that will also need to be shared between systems. By enabling a robust multi-cloud[3]<\/span><\/a> ecosystem with broadly distributed and understandable messages, we hope to unlock the true flexibility of software-defined workflows But we do not need to wait for the entirety of the 2030 Vision ecosystem to be built before we can take advantage of messaging systems \u2013 there are many use cases that can be deployed now to enable a more interoperable and flexible workflow in 2022 and beyond.<\/p>\n In our next post we\u2019ll discuss the software elements of a messaging system, types of messages and the use cases we hope to enable with one.\u00a0 Make sure you stay tuned, so you get the message\u2026[\/et_pb_text][et_pb_divider color=”rgba(86,97,108,0.1)” divider_position=”center” _builder_version=”4.11.4″ _module_preset=”default” custom_margin=”80px||||false|false” custom_margin_tablet=”||||false|false” custom_margin_phone=”30px||||false|false” custom_margin_last_edited=”on|phone” custom_padding=”||||false|false” global_colors_info=”{}” locked=”off”][\/et_pb_divider][et_pb_text _builder_version=”4.11.4″ _module_preset=”default” background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}”]<\/p>\n [1]<\/a> Not to say that API calls don\u2019t have their place in interoperability. We\u2019re very supportive of applications exposing APIs for system-to-system communication with small amounts of data or messages that need a guaranteed quickish response.<\/em><\/p>\n [2]<\/a> Message queues are a familiar element of software engineering, suited to a wide a variety of inter-process communication problems. Operating systems use them internally, and business and process automation software make heavy use of them.<\/em><\/p>\n [3]<\/a> We define \u201ccloud\u201d in the 2030 Vision as private, public and hybrid infrastructures connected to the internet and therefore envision productions that need to span across all of those.\u00a0 See the multi-cloud blog here<\/a>\u00a0for more details.<\/em><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":" Sending messages through workflows – Part 1<\/p>\n","protected":false},"author":11,"featured_media":10582,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"on","_et_pb_old_content":"[et_pb_section fb_built=\"1\" _builder_version=\"4.11.4\" _module_preset=\"default\" da_disable_devices=\"off|off|off\" global_colors_info=\"{}\" da_is_popup=\"off\" da_exit_intent=\"off\" da_has_close=\"on\" da_alt_close=\"off\" da_dark_close=\"off\" da_not_modal=\"on\" da_is_singular=\"off\" da_with_loader=\"off\" da_has_shadow=\"on\"][et_pb_row _builder_version=\"4.11.4\" _module_preset=\"default\" global_colors_info=\"{}\"][et_pb_column type=\"4_4\" _builder_version=\"4.11.4\" _module_preset=\"default\" pac_dcm_carousel_specific_module_num=\"0\" global_colors_info=\"{}\"][et_pb_text _builder_version=\"4.11.4\" _module_preset=\"default\" background_size=\"initial\" background_position=\"top_left\" background_repeat=\"repeat\" global_colors_info=\"{}\"]Clear communication is critical to the content creation process. And while today\u2019s productions somehow manage to compensate for inefficient communication mechanisms, there is a growing and urgent need to streamline the way we communicate and exchange information as we continue to scale up to meet the increasing demand for content. \u00a0In our blog \"Cloud. Work. Flows\"<\/a>, we identified some missing components that are required to enable software defined workflows. We highlighted that a more efficient messaging system will be critical to improve communication between participants (which could be people or machines) in a complex workflow system.\u00a0 We\u2019ve addressed communication elements in the Ontology for Media Creation<\/a> which covers some aspects on what<\/em> needs to be communicated. Recently we\u2019ve been turning our attention to the how<\/em>\u00a0to express that communication in the most efficient manner.\u00a0 For example, in the 2030 Vision<\/a> our first principle states that content goes straight to the cloud and does not need to be moved. Once ingestion to the cloud has completed, the first participants in the chain will need to be notified that the content is now in the cloud and ready to be worked on and ideally including a location for that content.\u00a0 A similar workflow notification message is required when a task has been finished and the work is ready for review by another team member.\u00a0 In this post we\u2019ll discuss the benefits of a common approach to communicating these repeating types of workflow messages. In a subsequent post we\u2019ll get into the technicalities of how we think such a system could be built including considerations to enable it to span cloud infrastructures and tools.\r\n\r\n Synchronous Real-Time Messages between two known participants.<\/em><\/p>\r\n And more complex messages, especially as we move to more automated systems, where the participants may not know who will pick up the messages and may not receive replies for hours or even days. Take for an example:<\/p>[\/et_pb_text][et_pb_image src=\"https:\/\/movielabs.com\/wp-content\/uploads\/2022\/05\/render.gif\" alt=\"Render\" title_text=\"Render\" _builder_version=\"4.11.4\" _module_preset=\"default\" global_colors_info=\"{}\"][\/et_pb_image][et_pb_text _builder_version=\"4.11.4\" _module_preset=\"default\" background_size=\"initial\" background_position=\"top_left\" background_repeat=\"repeat\" global_colors_info=\"{}\"] Asynchronous Messages between one sender and multiple potential recipients.<\/em><\/p>\r\nIn this example the Render Manager (the message sender) doesn\u2019t know which nodes may respond or when they may respond. There are thousands of such nuanced examples in production workflows that we need to consider when thinking about the sorts of messages that could be sent between systems. We need a messaging approach that can accommodate all of these message types, and also the complexity of multi-cloud infrastructure when messages may be flowing between systems that are not all owned\/leased or operated by the same organization and on the same infrastructure.\r\nThe Art of the Message<\/h2>\n
Software Messaging Systems<\/h2>\n
Benefits<\/h2>\n
Next Up…<\/h2>\n
The Art of the Message<\/h2>\r\n\r\nWe need to deal with both simple messages, in near real-time, between two participants, like this:[\/et_pb_text][et_pb_image src=\"https:\/\/movielabs.com\/wp-content\/uploads\/2022\/05\/messaging_system.gif\" alt=\"Simple Message System\" title_text=\"Simple Message System\" _builder_version=\"4.11.4\" _module_preset=\"default\" global_colors_info=\"{}\"][\/et_pb_image][et_pb_text _builder_version=\"4.11.4\" _module_preset=\"default\" background_size=\"initial\" background_position=\"top_left\" background_repeat=\"repeat\" global_colors_info=\"{}\"]
Software Messaging Systems<\/h2>\r\nAt MovieLabs we\u2019ve been thinking about approaches to these messaging problems. One approach is using point-to-point API calls between all these disparate systems and while appropriate for many use cases we don\u2019t believe this will scale to whole productions or studios \u2013 there would simply be too many custom integrations to get all the possible components of a workflow to work together.[1]<\/a> We see the best way to manage the highly asynchronous delivery of information to multiple (potentially unknown to the sender) destinations is to decouple the mechanics of the communication \u2013 the what<\/em> from the how<\/em>.\u00a0 In software systems this can be managed in a more automated way using Message Queues.[2]<\/a> A message queue allows a message to be sent blindly (the sender does not need to specifically know who will read it). Specific queues are typically associated with a particular topic, any other participant with an interest in that topic can then subscribe to the queue and receive its messages whenever they\u2019re ready.\r\n\r\nMessage queues \u2013 or, more broadly, message systems \u2013 are a natural fit for software-defined workflows: their raison d'\u00eatre<\/em> is to provide a communication mechanism where senders and receivers can operate without knowing anything about each other beyond how to communicate (the message queue) and some expectations about the contents of the communication (the messages.) This separation allows applications to run independently and, just as importantly, be developed independently.\r\n\r\nAs long as the sending and receiving applications can both access the message queue, it doesn\u2019t matter where the applications are running; they can be in the same cloud, in different clouds, on a workstation in a cloud, or even in two organizations. Agreeing on commonality of some aspects of message headers and message contents can enable interoperability especially in that cross-organizational use case. For example, if a message from an editorial department to a VFX house includes a commonly agreed upon place to put shot and sequence identifiers, a workflow management system at the VFX house can route that message to the appropriate recipients internally.\r\n\r\nThe use of messaging systems rather than point-to-point integrations also makes it easier to gather together operational data for logging, dashboards, and detecting\/acting on exceptions and errors.\r\n
Benefits<\/h2>\r\nAs we look to the 10 Principles of the 2030 Vision, we can see that messaging is key to enabling the \"publish\" function (where access to files is pushed through a workflow, as tasks are created) and enabling participants to \u201csubscribe\u201d to those files, tasks, or changes. Principles 1 and 2 of the 2030 Vision state that assets go to the cloud and do not need to move, which means sharing the location of those files<\/a> becomes a key message that will also need to be shared between systems. By enabling a robust multi-cloud ecosystem with broadly distributed and understandable messages, we hope to unlock the true flexibility of software-defined workflows But we do not need to wait for the entirety of the 2030 Vision ecosystem to be built before we can take advantage of messaging systems \u2013 there are many use cases that can be deployed now to enable a more interoperable and flexible workflow in 2022 and beyond.\r\n
Next Up...<\/h2>\r\nIn our next post we\u2019ll discuss the software elements of a messaging system, types of messages and the use cases we hope to enable with one.\u00a0 Make sure you stay tuned, so you get the message\u2026[\/et_pb_text][et_pb_divider show_divider=\"off\" _builder_version=\"4.11.4\" _module_preset=\"default\" global_colors_info=\"{}\"][\/et_pb_divider][et_pb_text _builder_version=\"4.11.4\" _module_preset=\"default\" background_size=\"initial\" background_position=\"top_left\" background_repeat=\"repeat\" global_colors_info=\"{}\"][1]<\/a> Not to say that API calls don\u2019t have their place in interoperability. We\u2019re very supportive of applications exposing APIs for system-to-system communication with small amounts of data or messages that need a guaranteed quickish response.\r\n\r\n[2]<\/a> Message queues are a familiar element of software engineering, suited to a wide a variety of inter-process communication problems. Operating systems use them internally, and business and process automation software make heavy use of them.\r\n\r\n[3]<\/a> We define \u201ccloud\u201d in the 2030 Vision as private, public and hybrid infrastructures connected to the internet and therefore envision productions that need to span across all of those.\u00a0 See the multi-cloud blog here [insert link] for more details.[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]","_et_gb_content_width":"","_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[85,84],"tags":[],"ppma_author":[48],"acf":[],"yoast_head":"\n