Learning to Love End-to End Cloud Production in Love Hurts
75% time savings in marketing editorial
50-60% reduction in Digital Cinema Package (DCP) production time
325TB of cloud data
Summary
The publication of the MovieLabs® 2030 Vision has inspired studios and production companies to use various cloud-based techniques, tools, and services across parts of the production workflow. Universal Pictures saw an opportunity with the film Love Hurts to build a 2030 Vision-inspired end-to-end cloud-based workflow, featuring Original Camera File (OCF) processing and storage, dailies, VFX plate pulls, trailer conform and finishing, feature conform, post finishing/color grading, and digital cinema mastering, partnering with AWS Company 3, Pixelogic, and other creative vendors.
The Love Hurts production demonstrated the MovieLabs 2030 Vision principles and how centralizing data in a single cloud store dramatically cut unproductive time waiting for file transfers, accelerated the crew’s decision making, expanded opportunities for collaboration, and reduced file duplication, decreasing both waste and the likelihood of data loss, version/sync confusion, and security breakdowns. The new cloud-based¹ workflow enabled the creative teams at the production company, studio, and vendors to collaborate on one set of files in the cloud as a “single source of truth.” The cloud workflow also allowed the teams to continue using familiar tools via virtual workstations interacting with Original Camera Files (OCFs), proxies, and associated metadata, all either uploaded or created natively in the cloud. Importantly, the Love Hurts pilot also identified challenges where further work and collaboration is needed.
Background
Under a long-established workflow for most major movie productions, OCFs are captured on media cards or hard drives and then physically transported to a dailies color vendor to create and manage backups on LTO tapes, conform with on-set audio, provide a dailies color pass, and create dailies available for next day review. The post-production facility often retains and stores OCFs over the length of the production, while proxy versions of the same files follow a different but parallel pipeline for editorial and review/approval of the different edits. Synchronizing the various versions, derivatives, and associated metadata from these two different video pipelines is a complex and time-consuming coordination process across dailies, visual effects, editorial, and other production departments.
The promise of mitigating workflow complexities and inefficiencies that evolved through the film era into the digital era has inspired the industry to implement the principles of the MovieLabs 2030 Vision in a growing array of workflows and tools. Over the years studios have experimented with cloud workflows for some production types (e.g., TV shows) or some production processes (eg. dailies, VFX push/pull, or color grading), Universal decided it was time to take a major step forward by testing end-to-end cloud workflows for a major theatrical release.
Solution
The goal of the Love Hurts project was to understand the real-world implications of a 2030 Vision workflow for a theatrical motion picture. Universal wanted to identify efficiencies, challenges, and key learnings while answering questions about complexity, cost, and scale. To prepare for Love Hurts, in 2023 Universal tested the proposed cloud-based workflow on a 2K trailer (Beast) for both color grading and conform with Company 3 and the Universal marketing team. On Love Hurts, Universal wanted to test a full 4K pipeline for a feature film in the cloud. Four key requirements of the test were: (1) stop the excessive movement of data between departments, vendors, and production personnel; (2) minimize the efforts required to keep versions and metadata in sync; (3) create a more accessible online store of production assets to replace a process that historically leveraged numerous distributed physical components and relatively inaccessible LTO tapes; and (4) understand the costs of doing image workflows in the cloud.
As key partners for the project, the production had chosen Company 3 and Universal chose Pixelogic and AWS. Company 3 specializes in color, finishing, dailies, and had worked previously on cloud-based color grading projects for TV production (including Amazon Studio’s The Rings of Power, Season 1). AWS brought its package of cloud-hosted services, storage, compute, and tools to the project and had a strong interest in testing an end-to-end production beyond Amazon Studios to flesh out issues and limitations, identify and close gaps, and learn more about change management in media workflows. While the project was finishing, the third key partner Pixelogic, leaders in Digital Cinema mastering and versioning, managed the ingest, processing and packaging of the completed DCDM file into compliant Digital Cinema Packages for delivery to theatrical distribution endpoints.
For Love Hurts Universal built cloud workflows for OCF processing and storage, dailies, VFX plate pulls, trailer conform, feature conform, post finishing/color grading2 and digital cinema mastering. Three key workflows remained on-prem. First, feature editorial stayed on-prem because Universal did not want to risk the possibility of the editors not being able to work due to connectivity issues. Second, the creation of visual effects was excluded because Universal did not want to add complexity to the project. Finally, the sound workflow (editing, mixing, dubbing, mastering) was excluded to focus this initial cloud project on the image pipeline. All other elements of the image pipeline were run from cloud-based virtual workstations. Since Universal wanted to preserve familiar post-production pipelines where possible, the cloud workstations ran the same applications used by the teams in on-premises workflows.
After: OCFs are uploaded to a central cloud repository. All departments and vendors work with and sync to one central set of master data that includes the OCFs and proxies.
The key to the Universal solution was to centralize all data storage in NBCUniversal’s tenant of Amazon S3 storage (“Amazon Simple Storage Service (S3)”), which all vendors could access. Universal utilized a vendor (White Ink Digital Post) in the shooting location (Winnipeg, Manitoba, Canada) to upload the Love Hurts OCFs directly to S3 storage. As the data arrived in the cloud, Company 3 Vancouver was able to access the files and start their dailies process. They also downloaded the OCFs in order to make LTO tape backups so that Universal had a physical copy for additional resilience. From that point on vendors and departments relied on and synced with one set of assets maintained in one S3 bucket in the cloud. The VFX vendor received the plate pulls using a Company 3 application that pulled from the NBCUniversal S3 storage, and the editorial media was availed to editorial via S3 storage. Although the editorial team worked on-prem, their Avid NEXIS editorial SAN (Storage Area Network) was synced to the S3 bucket for disaster recovery purposes. All final image assets (editorial, VFX, final theatrical SDR masters) were either uploaded or rendered directly in the central S3 bucket.
Access to the cloud data was subject to strict access management. Even though data was centralized in one bucket, the permissions set for vendors and creative teams limited each group’s access to only the data required to complete their work (using the security principle of “least privilege”). The Universal production and security team also followed principles of the AWS Well-Architected Framework and completed an independent security audit of the environment before production went live.
Architecture
The project architecture was built around a Virtual Private Cloud (VPC) maintained by Universal with a single S3 project bucket. The VPC included a private subnet for trailer marketing with EC2 workstations running AVID Media Composer, Windows, Autodesk Flame workstations running Linux, and Avid NEXIS EBS volumes. Amazon FSx for Lustre was used to create a faster file system on top of the S3 bucket, both for browsing and 4K EXR playback storage during grading. All of the workstations communicated with the S3 bucket using AWS Command Line Interface (CLI) tools and sync. Signiant Media Shuttle portals connected on-prem editorial and VFX users to the bucket. Universal Creative Operations ran offline and online editor workstations connecting to the virtual cloud via NICE DCV. Separately, both Company 3 and Pixelogic maintained distinct AWS VPC environments, linking to the Universal S3 bucket.
All connections in and out of the main S3 bucket were governed by role and bucket access policies. Access controls for direct access to the S3 bucket were managed through either user-based Identity Access Management (IAM) role policies and rotated access keys or S3 bucket policies with cross-account role access for Company 3 and Pixelogic. Thirteen external teams were onboarded and offboarded with access mapped to twelve different policies during the production. All other users accessed the files through Signiant Media Shuttle with access provisioned by the Universal Production Support Group. In total 143 users were onboarded and offboarded during the production.
Below is an example of how Universal organized user-based policies and prefixes within the S3 bucket with AWS access keys and policies rotated every 30 days. In all cases, only Universal employees had “delete” permissions on objects.
OCFs were transported physically from set to White Ink Digital Post, which then AWS MD5 high-speed transfer to upload to the Universal S3 bucket. Once data was uploaded to the main project bucket, the Company 3 Portal application indexed the data and created database records for the application.
For trailer and VFX pulls, a shot list was uploaded to the Company 3 Portal application to pull and render files from the project S3 bucket. For VFX pulls, camera files were read from the S3 bucket and transcoded directly in the bucket, then made available to the VFX vendor through Signiant Media Shuttle. The vendor either returned VFX files to the main S3 bucket for Company 3 to conform or sent VFX editorial files directly to editorial on-prem. For trailers, the media was organized by the Company 3 Portal application and then moved under the marketing sub-path of the S3 bucket for the Creative Operations team. For trailer conform and Flame workflows, scheduled data syncs linked the marketing sub-path of the S3 bucket to ST1 AWS storage attached to EC2 instances running Flame, which pulled data down for marketing playback. The Universal marketing team used Amazon DCV to remote into the EC2 instance for conform and playback.
For digital cinema mastering Pixelogic managed a fully cloud-based workflow in its own AWS environment, sourcing all video masters from the central Universal S3 bucket. The Digital Cinema Distribution Masters (DCDMs) were stored in the Universal S3 bucket and made available to Pixelogic for the creation of all Digital Cinema Packages (DCPs). Using managed services and EC2 workstations, Pixelogic validated the DCDMs in Universal S3 bucket to meet SMPTE specification and used a Digital Cinema mastering software to conduct a visual inspection of the content. The DCDMs were encoded to create the JPEG2000 stream into a Pixelogic S3 bucket. The JPEG2000 stream was registered into Pixelogic’s Asset Manager and validated to meet image compression standards. After transferring the JPEG2000 stream to EFS storage, the DCP was wrapped and packaged before transferring to on-prem storage for QC using the theater playback server. After approval in theater with post execs, DCP’s were transferred to Universal’s S3 bucket from Pixelogic’s on-prem storage.
BENEFITS
By centrally locating the assets in the cloud the production unlocked numerous current benefits, with the potential for more benefits to come as additional processes and systems are added into future versions of this workflow. In Love Hurts multiple departments were able to leverage secure remote storage for their workflows, gaining valuable time by avoiding unnecessary file transfers.
Benefits to the Studio Production Operations Teams:
- The production operations teams saved time that otherwise would have been spent supporting data transfers and managing version control across different vendors and teams. Having a “single source of truth” eliminated or decreased the effort previously spent syncing and managing multiple storage sources and file trees. Instead of copying files and rigorously managing checksums, the support teams simply pointed to the S3 bucket.
- Storage of all the production assets including the camera files on the studio’s S3 storage helped optimize costs and give the studio more control of security and visibility into utilization.
Delivering VFX shot plates to vendors was also simplified That process is typically time-consuming and complex. With the VFX assets maintained in central S3 storage, the process was streamlined. - Centralized storage simplified security for data access by allowing the security team to manage all data in and out from one bucket with consistent role and access policies.
- The centralized store also provided disaster recovery benefits. The team was able to backup the editorial NEXIS to a dedicated S3 bucket throughout production, with all files in S3 backed up in three places as part of the normal AWS cloud setup. Restoring from backup was also streamlined. As an example, about 10 days into the shoot, one on-set production team realized that a local back-up was not working. Instead of having to go back to LTOs to restore the data to local storage, the team realized that the data was already safely stored and available in the S3 bucket.
- Normally, the post-production house holds all the camera files internally during the life of the show. Having the camera files available on the studio’s S3 storage makes it possible to enable other workflows that require camera files. Although the studio did not leverage this advantage during the Love Hurts project, Universal expects to discover and implement new workflows as a result of the camera files being accessible to more than just the post-house.
Benefits to the Studio Marketing Team (Creative Operations):
- Traditionally, the Creative Ops team would follow one of two workflows: (1) submit an order, wait for the files to be transcoded and pushed to Aspera, then manage the download before being able to edit the file, or (2) request the post-house dailies lab to create DNxHD files of the RAW camera data (to a Universal marketing spec) and deliver via hard drives. With the camera files in the cloud, the team was able to skip most of these steps and did not need the extra LTO copy. Instead, they had immediate access to the dailies and even the OCFs. This self-service capability allowed the Creative Ops team to realize approximately 75% time savings compared to the conventional processes (plus savings on shipping hard drives and LTO tapes, as well as the processing necessary to create those materials). The new virtual process gave the team both quicker and cheaper access to a complete set of assets at a time in the production process when marketing ordinarily would have to wait.
- When the trailers were created and conformed, the Creative Ops team also saved time and money delivering the material to the post house for grading. Ordinarily, the team would create a conform of the trailer and package it up to send via Aspera to the finishing house. Instead, they gave Company 3 the ability to pull the conformed trailers directly from the marketing sub-directory in the S3 bucket and start grading. Note that there were issues with the trailer process that will be discussed in the Lessons Learned section.
Benefits to the Post-House:
- Company 3 had direct access to the camera files, which were uploaded near the production and immediately available to the team in Vancouver. The team could start the dailies process as soon as the files were uploaded without waiting for LTO tapes to be written, shipped, and restored in the dailies lab. Camera files also did not have to be restored from LTO tape in the event that access to a backup was required.
- Because the entire production was stored in the cloud, the post-house also was not limited to processing assets with on-prem hardware. Instead, they were able to take advantage of extra compute in the cloud in the event of burst requirements.
- Having a location-independent workflow also enabled location-independent collaboration. Although the primary post-house production teams were in Canada, teams in New York and Los Angeles were able to help from remote locations, allowing Company 3 to add resources quickly when needed.
Benefits to Editorial and VFX:
- There were few direct benefits because the editorial and VFX creative teams were not included in the pilot and continued to work on-prem, either in Vancouver or the VFX vendor. However, as an indirect benefit, the virtual process was essentially invisible to the on-prem teams. The editorial team received their files from the S3 bucket in approximately the same amount of time as from on-prem storage. They were able to maintain the same processes used on other productions.
- All on-prem editorial files were synced to the cloud, using GoodSync to sync the editorial Nexus on-prem to the S3 bucket for disaster recovery. After the show wrapped, the VFX and editorial teams did not have to create hard drives for archive. Instead, they cleaned up their on-prem storage and backed up all materials to the cloud bucket.
Benefits for Digital Cinema Distribution:
- Pixelogic was able to implement a fully cloud-based workflow in its own AWS environment, sourcing all Digital Cinema Distribution Masters (DCDMs) directly from the central Universal S3 bucket. Instead of waiting for final DCDM’s to be delivered and then ingested on-prem, Pixelogic created all DCPs for Love Hurts digital cinema distribution from the DCDMs stored in the S3 bucket, speeding up the DCP production process by an estimated 48 hours. Overall, the time required to transfer content was reduced significantly, resulting in time savings of at least 50-60% over a traditional workflow.
Alignment with MovieLabs 2030 Vision Principles
PRINCIPLE 1
The Love Hurts OCFs were transported on hard drive media to a vendor responsible for uploading to the project bucket. As soon as the files were uploaded to the S3 bucket, vendors and teams in Winnipeg, Vancouver, Los Angeles, and New York were able to begin work immediately without waiting for the delivery and restoration of LTO tapes or hard drives. Even for production workflows excluded from the pilot project, i.e., editorial, creative VFX, and sound, all files created on-prem were synced to the S3 bucket for storage and backup, and final assets were delivered to the S3 bucket for processing and downstream distribution. The assets were copied from AWS US-West to Pixelogic’s US-East because their processing was only deployed in that region.
PRINCIPLE 2
Other than excluded workflows, all media processing for the Love Hurts production happened in the cloud with media assets stored in a centralized S3 bucket. For reading and rendering of dailies and VFX pulls, virtual machines running Colorfront On-Set Dailies (OSD) and Transkoder applications read directly from the S3 bucket for rendering to EXR, ProRes, or DNxHD formats. For higher quality playback, conform, QC, or color, files were synched by prefix subdirectory to faster storage (FSx for Lustre) in AWS, which was able to take advantage of cloud scalability to move the data quickly.
PRINCIPLE 6
Access to shared S3 buckets was managed systematically through a set of user groups each with well-defined authorizations. These were implemented with IAM roles, using user-based role policies within the tenant and bucket policies for cross-tenant access. This also allowed the identity for each user to be managed and authenticated in a single place, namely their home tenant.
Partners
Company 3 played a central role as post-house and facilitator for multiple workflows. When OCF files were uploaded to the Universal S3 bucket, Company 3 immediately began processing dailies, sometimes working remotely from locations in Vancouver, Los Angeles, and New York. It provided cloud-based color grading and post finishing for feature and trailer editorial output. Company 3 also provided a VPC environment (Company 3 Cloud Studio) that facilitated workflows for OCF uploads, debayer, color, dailies, plate pulls, and post finishing. The Company 3 Portal application played a critical role in indexing dailies, VFX files, and OCF files in the Universal S3 bucket, as well as transcoding and rendering for VFX, post and studio trailers.
AWS provided the environment and tools that helped the Universal S3 bucket function effectively as a central store. It also hosted and provided tools for the Company 3 Cloud Studio environment, the Pixelogic digital cinema workflow environment, and the Signiant Media Shuttle portals.
Pixelogic built an innovative fully cloud-based workflow for digital cinema mastering. That workflow relied on the DCDMs in the Universal S3 bucket as the ‘single source of truth’ for the mastering workflow across multiple versions and digital cinema package variations.
Lessons Learned
The primary lesson from the project was that 2030 Vision-inspired workflows can deliver tangible benefits to multiple production and creative teams today on a major studio theatrical release, including production management teams, security teams, post-house teams, trailer marketing, and digital cinema mastering. There were also tangential benefits to the on-prem editorial and VFX creative teams despite those workflows not being included in the pilot.
Universal learned key lessons that should help future productions. Most importantly, the project made clear that the industry—creatives, studios, and vendors—needs more understanding and education about how workflows are evolving in a cloud powered world. There are real potential benefits of handing time back to the creative process, but the operational details require work, coordination and evangilism. In the future, perhaps production workflows can be “all in the cloud,” but getting there will require production workflows to be hybrid and incorporate existing on-prem systems seamlessly with public and private cloud instances. Production support teams need a thorough understanding of the technical requirements and capabilities of the cloud to effectively use it for workflows. Getting cloud workflows to work well, including hybrid cloud/on-prem interfaces, requires new areas of expertise, e.g., around networking and how tools and processes must be adjusted to account for cloud workflows. The industry needs to think through these issues and help make software and hardware more readily adaptable to cloud workflows.
The Trailer team had most of their systems (i.e. graphics) on-prem, so having one trailer working in the cloud was not sustainable. The main problem the team had was once the data was in the cloud, the team did not have a simple and quick way to move and manage the data as needed. This ended up taking more time than it would have if it was on-prem. The decision was made to do the international trailers on-prem instead of in the cloud, due to time constraints. Future shows would need to take into account this hybrid workflow.
There were lessons on change management as well. A key lesson was the need to involve relevant teams early in the process and give stakeholders agency in shaping new workflows for their areas of responsibility. When that process was followed, there were still challenges, but the changes were implemented with a broad level of cooperation across teams. Without sufficient advance planning and collaboration, the pain points became difficult to address and sometimes required abandoning original project goals. A key example was the effort to reduce the need for LTO tapes for physical archiving. Initially, it was hoped that cloud storage could reduce the need for LTO tapes. But the project did not replace existing physical archive processes with a complete set of well-understood new processes. The result was friction among the teams and the need to create LTO tapes to address archive and inter-departmental requirements. However, the project was a good conversation starter allowing the archive team to look at what is needed for cloud to become part of the archive puzzle.
As another example, Universal made every effort to keep the underlying plumbing of the cloud workflows invisible to creative users and filmmakers, but during the finishing on one occasion the creative team encountered an unexpected playback stuttering issue. The issue was resolved, and the finishing process stayed in the cloud. But to give the creative team full confidence in the process and provide reassurance about the quality of the final product, the production team pulled the content down to on-prem for local viewing. The key learning is that even with all the testing in the world, issues can arise in the eleventh hour for any new cloud workflow that may have to be addressed with an on-prem solution.
There were other change management learnings as well. Artists get very comfortable with particular software versions and operating environments, and they can resist change. In addition, cloud versions of applications may not always be the same version that artists are used to or have customized on their on-prem instances which can cause friction. So the team had to plan for the virtual environment to support many different versions of software and different artist environments, some of which introduced inefficiencies to the virtual process, but were nonetheless necessary to support the creatives. Similarly, many creatives and production teams have a long familiarity with the hierarchical structures of POSIX file management. Moving to object storage file management was a new paradigm. Cloud-based production teams need training to highlight the new efficiencies of object storage systems (such as Amazon S3) and, during the transition, may need hybrid environments that allow them to utilize on-prem tools with cloud storage.
For similar reasons, managing the S3 bucket in a clean and cost-effective way also proved a challenge. Standard procedures used in on-prem environments can have unintended consequences in a cloud environment. For example, at one point the team temporarily tripled storage volume almost overnight by turning on versioning without realizing that cloud protocols would generate large numbers of replicated versions across three cloud backups. When the project was complete, the bucket held numerous duplicate files, mistaken files and folders, etc., partly because vendors did not have delete rights so could not clean up their own folders and files. Overall, future projects should include more planning on how to manage files in the bucket efficiently or to use AWS features and services to help with file management. The capability to create a clean archive of production assets in object storage still needs improvement. Industry partners would do well to invest heavily in engineering and digital product teams up front, enabling a more automated and streamlined operations team to rely less on engineering support on a day-to-day basis.
Next Steps
In future productions Universal intends to continue on the journey to implement new workflows inspired by the MovieLabs 2030 Vision. Key next steps will be to test the benefits of virtualizing editorial, working with VFX houses to integrate with the cloud storage, moving audio workflows to the cloud, and piloting cloud-based workflows for home video mastering. The team also will collaborate with the archive team to create new cloud-based archiving workflows. Universal also has continued to expand the set of vendors connecting to their S3 storage so that future productions can take advantage of the full range of resources and vendors available to the industry as the 2030 Vision continues to be adopted across tools and services.
Finally, one key component of the Love Hurts project—cloud-based digital cinema mastering—showed such obvious benefits that it has already been implemented for other productions. Since completing Love Hurts, Universal has worked with two different vendors (Fotokem and Deluxe) on another major title (Megan 2.0) to use Digital Cinema Distribution Masters (DCDMs) directly within a cloud bucket instead of transferring files to vendors to package into Digital Cinema Packages (DCPs). Now, Universal is connecting other vendors such as Eikon, MPS, Qube and Unique into that workflow.
MOVIELABS PERSPECTIVE
When the 2030 Vision was published in 2019, we stated that cloud-based workflows could deliver immediate benefits and the potential for exponential gains as more systems became cloud native. The Universal experience on Love Hurts, bears this out as the team saw significant and immediate benefits in efficiency by cutting out copying, duplication and movement of data but also now can see a roadmap to future gains as more partners and applications can be connected to this cloud-based infrastructure. All partners in this case study expressed appreciation for the learnings, change management, training and opportunity to explore new workflows that this pilot enabled. It is in fact through encountering and overcoming problems that the teams learned the most and will make the next project better.
We always knew it would be a graduated and iterative process to migrate the industry from 100 years of media movement practices but the more creative teams, vendors, application developers and even internal studio departments get exposed to the opportunities, the easier the next production comes to manage and the further down the 2030 path we can get. MovieLabs is very appreciative for the work done on Love Hurts by Universal and its partners and especially about the openness to share what worked and what did not with the industry at large. It is through this shared learning that we can accelerate progress to a more efficient workflow where creatives can focus more on making media instead of moving it.
[2] The cloud-based workflow was used for the theatrical SDR master.
Get the Case Study
Download a free PDF of this case study