As recent events have illustrated, the media industry’s need for more sophisticated remote collaboration capabilities in virtually every content creation category has never been greater. With that need comes more urgency regarding the development of interoperable multi-cloud tools across the production chain. And to achieve that goal, many experts believe the only logical path to travel is one involving a significant evolution of remote workflow concepts. To that end, the pursuit of what is increasingly being referred to as “software-defined workflows” is now well underway.
Indeed, a major project in this regard recently picked up steam with the publication of the newest white paper from MovieLabs—an independent industry non-profit consortium that includes five of the major Hollywood film studios. The recent white paper, dubbed “The Evolution of Production Workflows—Empowering Creative Processes with Software-Defined Workflows,” addresses, as the title suggests, what MovieLabs believes is the way that production workflows could be run exclusively using software-based tools in the cloud in coming years. It’s part of a series of recent papers from MovieLabs designed to collectively lay down a path for reimagining the entire production industry over the next 10 years—what MovieLabs is calling its 2030 Vision initiative. Jim Helman, CTO at Movie Labs, explains that the idea of open and interoperative software-defined workflow methodologies lies at the heart of the larger initiative.
“The original 2030 Vision was divided up and has 10 principles for the future of media creation within it,” Helman says. “These include Cloud fundamentals and foundations—the basic idea that we are eventually going to be doing all our work in the Cloud. Assets will go directly to the Cloud and ideally stay in one place, while applications come to them, which is essentially an inversion of the current situation where assets are continually being sent from facility to facility.
“Security and access was a second category, and then the third category was software-defined workflows. That is why we put out this paper on software-defined workflows as a motivational piece to the industry. In order to deploy workflows in the cloud that are less brittle and more easily reconfigured than today’s, you need to build up toward interoperability. The most foundational element of that is to realize that you can’t have common APIs, common interfaces, if you don’t even agree on what things are, what the data model is. So we set out to discuss how we could work out a common set of ontologies for production that would be useful to everyone. We are coming at this from the position of trying to get more flexible and better workflows to support creative work.”
Helman emphasizes that of the 10 principles that need to be applied to design efficient media creation workflows in the Cloud era, the last three directly impact software-defined workflows. The first of those three is that “individual media elements [need to be] referenced as access tracks interrelated using a universal linking system.” The second, number nine, declares that “media workflows are non-destructive dynamically created using common interfaces, underlying data formats, and metadata.” And the final principle states “workflows [should be] designed around realtime iteration and feedbacks.”
All of these points relate to the notion that “you need context for your assets and how to track them and use them,” states Annie Chang, a SMPTE Fellow and VP of Creative Technologies at Universal Pictures, a MovieLabs member studio, who has been promoting the need to work on this issue for many years. “It’s about trying to be smarter and figure out better ways of doing things than [older] asset management processes. The key is to help efficiencies, stop the manual human labor of simply having to find things and replace it with a more machine-to-machine connection. It’s about helping artists and different operators get their jobs done, rather than spending time trying to figure out if they have the right version of an asset to work on. We want all that kind of crazy to go away.”
Chang suggests that if the industry does not aggressively pursue this agenda, then the advantages of Cloud-based production are going to be unnecessarily limited.
“We can all easily go ‘into’ the Cloud,” she says. “But without some common way to track your assets properly through a software-defined workflow as described by the [aforementioned MovieLabs white paper on software-defined workflows], it would be just like going into grandma’s attic—you won’t be able to efficiently find anything.”
Chang says Universal became interested in software-defined workflows after seeing the development of a cloud-native platform from Universal subsidiary DreamWorks Animation. Dubbed the CorPS platform, that system creates workspaces for artists that house related animation assets with context for the work they have to do on a production.
“That system heavily uses concepts from software-defined workflows—I think it’s one of the first in the industry widely using it in this form,” Chang says. “And the key parts of their workflow are what we included in the software-defined workflow white paper. What we discovered is that there are four areas that have to relate together, and if you can achieve that, you can have machines more efficiently talk to each-other and automate certain processes, which is very important for a software-defined workflow.” She explains that those four areas, or pieces to the puzzle, are assets, tasks, participants, and relationships.
“Assets are things like camera files or production sound, in terms of live-action production,” Chang explains. “Tasks mean essentially what do you want to do with each asset? This is where software-defined workflows are different than other ideas that came before, because while they both involve a bit of workflow orchestration, a lot of it is about what is the context for each asset. So a task could be synching picture and sound together or color-correcting certain camera files, or transcoding files for some other purpose.
“Participants are the people who will actually be doing different aspects of the work. They, therefore, need to know about each asset that concerns them and the context or relationship between them. So that brings you to relationships, which are really important. The idea of the relationship is that you want to know what is the context of the asset and the task you need to carry out so that you can understand precisely who needs to access what asset and what they need to do with it. Are you transcoding files for dailies or for editorial? Because those require different types of specs.
“So the point is, if you can identify these four areas in your software-defined workflow, then you have a much better chance of training machines to automate those processes. The idea is to develop a bigger picture view of the entire workflow within the system. Only then can you get applications and systems talking to each other in an efficient way.”
In contextualizing these four puzzle pieces, the MovieLabs White Paper details the need to think of their connections to each—other using the concept of ontologies—essentially the notion of clearly demonstrating the properties and relationships between different elements inside the same domain area. Chang emphasizes that the problem with building different ontologies to define elements inside a domain is that “they can be different depending on the person who may be viewing the relationship.”
“For example, a studio head may only be interested in very high-level components,” she explains. “Those components will have relationships with each other and some dependencies among them. Then, you can go down to a different level, say a visual effects (VFX) producer. That person would be more interested in specifically those elements that will help them get their visual effects shots done. A VFX data wrangler, on the other hand, may only be interested in pictures or scans of the sets and how to bring them into the system properly—that person may really only care about metadata. Again, assets and tasks are related to each other, but only based on the context of what each participant is interested in. So the paper talks about the idea of defining these ontologies ahead of time so that machines can utilize them in a meaningful way as they try to automate various processes. Within the world of feature films, we more or less have the same basic steps, so the thing is, we can create consistency with industry ontologies for specific processes because they are done so similarly. The applications or specifications may be different, but the overall relationship of how things are done is more or less ubiquitous. To build an efficient [software-defined workflow], we need to create and define these ontologies. Each group in the media and entertainment industry basically has their own working style, so ontologies need to be built for each of those, and that is what we are now trying to do.”
Thus, more broadly, as Helman points out and as is stated in the MovieLabs White Paper, a software-defined workflow can be defined as “one that uses a highly configurable set of tools and processes to support creative tasks by connecting them through software mediated collaboration and automation.”
“I like to think of it as trying to take a more holistic view of the systems that support creative processes, and connecting them in a way that different vendors that are creating some of these software tools can have a common way to communicate with each-other,” Helman adds. “Simply defining a bunch of point-to-point interfaces doesn’t get you there. Some patterns—such as review, approve, publish, comment, re-do—occur throughout all stages of content creation. If you can develop a consistent way to automate those processes, building a well-thought-out software-defined workflow to meet the needs of a production becomes much easier. You do not necessarily have to re-do all your integrations when you switch out components in your workflow.”
The overall goal of this effort, therefore, is to eventually build a non-proprietary industry standard for how software-defined media workflows should function for maximum efficiency. Chang, a longtime studio technology executive, emphasizes her belief that such an effort will benefit the entire industry without interfering with classic and healthy competition between studios and other content creators.
“A colleague of mine refers to [this goal] as having a marketplace of services in the future where we have common pipelines, and then we just plug into specific services,” Chang states. “After all, companies compete based on the quality of their services, not how their [pipelines] are designed. In our case, for instance, DreamWorks has already created a very cool system for animation, but obviously, other people will eventually figure it out anyway, so it is only your special secret sauce for so long. So the idea here is to make this concept more open, and create what we call an interchange layer or translation layer or interface layer as MovieLabs calls it—one that is [common] to everyone. If we create this set of ontologies, data mappings, and APIs as industry standards, then we will all know how to connect through them. We won’t have to figure out for each project how to synchronize one vendor’s API with another’s. We could get past all that if we can have standardized ways of translating and exchanging data. After that, what we each do with the service or application—that will be where we compete. All the studios use the same vendors anyway, and the vendors all basically use the same applications and systems. So why not come up with a common language that we can all talk through, and just get on with the work?”
Chang emphasizes that it is still early in the process of figuring out exactly how to standardize what she calls “a nebulous thing called software-defined workflow,” but she adds that she expects to eventually see an industry effort to figure that out. “After all, that’s the kind of thing that SMPTE and others excel at—being able to document things for the industry that are routinely used by the industry so that [such processes and tools] can be interoperable within the industry. So as MovieLabs starts inching further toward this concept of what such an industry interchange layer might look like, it will eventually become clear what makes sense from a standardization standpoint.
Indeed, Helman adds that the effort to get the broader industry on board with the effort beyond MovieLabs core members is well underway.
“This is the vision of the five major MovieLabs studios,” Helman says. “We have positive engagements with most of the major providers of creative tools and the major Cloud services, and we all agree that the current mishmash of proprietary approaches for building workflows doesn’t serve anyone. Right now, studios and their creative vendors have to make large investments in software development to build custom pipelines, and software providers have to implement different features for each of their studio customers, which often does not translate into features for the mass of their customers. We think the value of the key stakeholders will drive consensus in the most important areas, and it’s important that effort be open, and not tied to any single provider.”
As to how long this standardization effort might take, Helman can’t say. But he adds that once there is agreement on the approach, the work can largely be done incrementally, and that elements of that effort are already underway.
“SMPTE has already started some relevant work with its 34CS Technology Committee Microservices Drafting Group working on microservices draft standards to address specific use cases [in partnership with the Open Services Alliance],” he explains. “And the Academy Software Foundation has taken on projects such as OpenCue [an open-source render management system in development] that can provide components of pipelines. MovieLabs is also working on security architectures and open data models to support studio workflows. But in other areas, especially concerning interoperable interfaces to support things like workflow communication and resource management, solutions still need to be incubated and validated with prototypes and proofs of concept before they can be documented and standardized.
“Still, all that said, if you had asked me a year ago, I would have said change management would be a key challenge in getting this done, especially where using Cloud resources means changes to existing working patterns. But COVID forced people to be more flexible in working remotely, and it also has helped show us the gaps. One key will be in getting industry consensus on the most important areas for initial work and standards. We have to focus on achievable pieces, and avoid the temptation to boil the ocean.”