Coloring Penelope in the Cloud
Dailies
40% Time Savings
Conform
80% Time Savings
VFX Pulls
98% Time Savings
Summary
Light Iron developed and deployed an all-cloud workflow that was used to complete dailies, VFX pulls, conform, and color on Penelope, a new episodic television project from Duplass Brothers Productions. The workflow covered everything from capture to post-production; camera footage from a live production shoot was ingested into cloud storage daily, while dailies, conform, color, and VFX pull services were performed using cloud workstations acting directly on that ingested media. The workflow also demonstrated enhanced cloud security approaches as access to the original camera media, and any renders or versions created from that media, were provided to clients and other vendors via granular permission polices (as opposed to making copies). Additionally, as all the media was in the cloud, Light Iron was able to service client requests using on-demand, auto-scaling cloud render engines, accelerating the completion of certain processes to hours instead of days. The technologies utilized in the workflow included AWS’s S3, EC2, and FSx, Colorfront’s Express Dailies (ExD) and Transkoder, FilmLight’s Baselight, Light Iron’s Galixy, and Streambox. Now that Light Iron has deployed and proven the success of this workflow, it will become an option for any show with a global creative team.
Background
The legacy methodology for dailies, VFX pulls, conform, and color in the last decade has included lots of less-than-optimal workflows such as multiple terrestrial data movements, project file management, asynchronous status updates and reviews, emails, phone calls and waiting. Dailies typically take 12-16 hours to turn around, usually with a dedicated overnight operator for every show. VFX pulls similarly require a team of people to find, load, and transcode every submitted shot list manually, with a delivery window averaging 24-48 hours per request. Finally, the conform process, which seems simple on its face (replicating a low-resolution timeline with one using the original camera files), often averages a week of work due to the sheer amount of time needed to wrangle hundreds of thousands of frames from archival tape, VFX vendors, titling, and graphics facilities, etc.
The reliance on manual and segmented workflows not only contributes to operational bottlenecks but also increases the likelihood of errors, miscommunications and data loss. Each transition in the production phase requires considerable manual effort, further exacerbating the risk of inaccuracies that necessitate time-consuming corrections and redeliveries. The industry has become numb to this inefficiency simply because it has been the status quo for so long. Technology now exists, however, for better options.
Solution
Partnering with Duplass Brothers Productions, who are passionate about reinventing post-production in the cloud, Light Iron designed a workflow where the original camera files (OCFs) moved to an AWS cloud as fast as possible and stayed there for the duration of the project. All versions, renders, metadata, and other content were created in the same cloud storage location for immediate consumption by the production team, editorial team, and other post-production vendors.
All applications needed for the dailies and finishing phases of the show were colocated in the cloud with the media and metadata, which could be read directly without the need to make intermediate copies. These applications, including FilmLight’s Baselight and Colorfront’s Express Dailies, were remotely driven by editors, colorists, and other operators out of Light Iron offices in Los Angeles and Vancouver, which were fully outfitted with all the necessary monitors, control panels, and test signal devices required for a high-end finishing project.
Even though all the assets were stored in a single cloud location, access permissions to the content repository were allocated at an individual account level. Accounts and roles were governed using the AWS Identity and Access Management (IAM) service, with granular policies specifying access to the project’s content at both the folder and file level. Since access to the content was provided directly to authorized users (both via S3 protocols and through secure transfer services like Signiant Media Shuttle), making copies of media for different vendors and production staff was unnecessary.
Finally, targeting the goal of real-time feedback, Light Iron leveraged its Galixy software service to provide production clients with real-time progress metrics, shot metadata, and on-demand VFX pulls.
Architecture
The pipeline started much the same as any other traditional workflow, with hard drives arriving from set once or twice per day (depending on logistics specific to shooting days) containing all original camera files (OCF), Broadcast Wave Files (BWFs), and relevant metadata files from that day’s shoot. Lower resolution capture proxies were uploaded from set to Frame.io in real-time, but the size of the raw media, and the bandwidth limits of the on-set satellite uplink, necessitated this more traditional path for the full resolution files. The hard drives were connected to a system running the macOS application “Light Iron Agent,” which played a pivotal role in Light Iron’s dailies and on-set operations.
Light Iron Agent performs multiple operations including indexing storage volumes, extracting metadata from various dailies platforms, and distributing hash-verified copies to multiple cloud-hosted storage platforms simultaneously. This is all performed automatically, requiring only a basic initial configuration at the beginning of a project. In the case of Penelope, Light Iron Agent pushed the transfer drive content via a 10Gb encrypted uplink to Light Iron’s AWS S3 bucket (although this workflow could utilize a studio’s or a production’s S3 bucket). That bucket served as the primary storage for all OCF throughout the project, ensuring centralized access and management.
Critically, while the deliverables were written back to the same S3 bucket that contained the OCF source, the OCF were set to read-only upon ingest, using AWS’s IAM policy infrastructure. Using a single bucket and granular user/role policies, as opposed to multiple buckets with broader team-based access policies, cut down the security management overhead significantly. Transferring data between teams required only a simple access policy change so recipients instantly received access to the required media when it was “published” to them at an individual level.
The conform and color environments were also cloud-based, created by leveraging a CloudFormation template designed by FilmLight. CloudFormation is an AWS tool that creates a virtual computing ecosystem using a simple text file “recipe”. The template provided by FilmLight created a fully self-contained Virtual Private Cloud (VPC) with all the required networking, security groups, compute, and storage resources for a single seat Baselight implementation. Light Iron’s DevOps team adjusted this template slightly to spin up the environment in their existing, pre-validated VPC and add a number of conform specific virtual workstations (in addition to the default color workstation). The result was a multi-seat Baselight cluster running in the same region as the show’s
OCF bucket.
Importantly, the CloudFormation template also included a full configuration of an AWS Cloud Digital Interface (CDI) service, already attached to the virtual Baselights. CDI is Amazon’s answer to traditional SDI, allowing the transmission of uncompressed, ultra-low latency video from enabled applications within AWS cloud infrastructure. Baselight uses the CDI service to output its reference monitor signal, which then can be converted into several different formats for streaming to destinations outside of AWS. In this case, Light Iron used Streambox’s Bridge product to convert the CDI signal to Streambox’s proprietary streaming format for delivery to the colorists and editors on site at Light Iron Hollywood (which had an existing Streambox hardware decoder infrastructure). Streambox’s ACT-L5 codec provides near lossless, color accurate video and has been used for final color on major film projects for over a decade.
However, due to the reality of today’s cloud and bandwidth infrastructure, real-time playback directly from the original camera files stored on the main S3 bucket was not possible. It was necessary to use caching, a technique to create temporary local copies of files, to ensure adequate performance for color review. AWS’s FSx product was used as a caching layer in conjunction with the open source “s3fs-fuse” project. The “s3fs-fuse” application enables a direct mount of an S3 OCF bucket and also can set a second mounted file system as a cache. When a file is accessed on that bucket, a copy is delivered to the cache file system. Since s3fs-fuse caches with absolute pathnames that are identical to the pathnames of the source files, all conform and color workstations can access the media at the same time by mounting the FSx file system to the same mountpoint on each system.
BENEFITS
The workflow focused primarily on enhancing efficiency and, ideally, reducing production costs. One of its core advantages was the ability to start aspects of the finishing chain from the very beginning of the production workflow. It also significantly accelerated the availability of dailies for creative and editorial teams, as OCF in the cloud was often available for processing on the same day it was shot. This kind of speed is becoming a crucial advantage in post-production pipelines, given the dynamic and fast-paced nature of contemporary productions. For Penelope, editors could begin an early cut of a scene even before the production crew finished shooting at a given location, which in turn allowed work on visual effect shots to start much earlier than is typical. These incremental improvements, when implemented through the entire pipeline, had a marked effect on the length of shooting schedules and the number of hours spent by stakeholders “waiting around”.
A second benefit was the simplicity of securing a fully cloud-based workflow, versus the traditional facility “perimeter” security approach. From a post facility standpoint, adhering to MPAA controls for network segmentation in an AWS environment was straightforward and easy to implement, and there was no need for a chain of expensive (and easy to misconfigure) firewalls or access control list (ACL)-enforcing Layer 3 routers.
Additionally, having a single source of truth (the OCF bucket) for each part of the process eliminated errors in transfer and copy between various storage volumes (as those steps no longer exist). The time and cost savings from this alone was pronounced for a post-production facility, as additional storage volumes did not need to be purchased, additional I/O hours for manual copies did not need to be logged by an operator, and the need for round-the-clock troubleshooting support was greatly reduced.
All combined, these benefits optimized resource utilization, accelerated project timelines, and reduced production costs.
Alignment with MovieLabs® 2030 Vision Principles
PRINCIPLE 1
This case study demonstrates the benefits of an all-cloud workflow, designed from the beginning to ingest both on-set proxies and OCF straight to the cloud. In this case a single bucket was used with nuanced permissions to control access, making it quick and easy to add new participants to the project and share media with them and their applications.
PRINCIPLE 2
With a single source of truth in the cloud, all of the applications in this workflow could operate on the media where it was stored in the cloud, even simultaneously if required. Light Iron used multiple high performance cloud workstations with remote access to ensure the applications were colocated with the media. Collaborators anywhere in North America successfully used machines based in the AWS US-West region with no latency impact. Although the workflow required caching for the object store to provide sufficient performance for color workflows, the seamless integration on top of S3 minimized any chance of creating rogue copies of data.
PRINCIPLE 10
This workflow demonstrated how overall production time can be cut and workflows made more efficient by using an all-cloud pipeline where applications and participants access the media simultaneously. Processes that used to take hours, days or even weeks now can happen in real time by spinning up compute instances in the cloud with the media instead of waiting for media to be moved to the applications.
PARTNERS
The workflow was primarily a collaboration between Duplass Brothers Productions and Light Iron. Extensive planning was required before, during, and after production to tweak implementation details and workflow components to ensure each phase worked as seamlessly as possible. An earlier version of the workflow was used on the show Biosphere, which piqued interest from both parties in continuing to develop an advanced, cloud-based process.
Duplass Brothers is an independent production company and, as such, is always trying to keep costs low while maintaining quality. They see lots of opportunities for cost savings by utilizing the cloud, and it also allows for more ease of remote access.
The primary application partners were FilmLight and Colorfront, both of whom worked closely with Light Iron to tune a full AWS implementation of their traditional on-premise applications for color and dailies, respectively.
LESSONS LEARNED
Future “color in the cloud” projects will benefit from some enhancements in the underlying technology stack. Primarily, while caching S3 objects with FSx and s3fs-fuse worked to enable real-time playback, it wasn’t a seamless experience. Caching only occurred once the file was first accessed. After creating a fully conformed timeline, the editor had to play it all the way through to ensure that all the media was properly copied before turning the timeline over to the colorist. A number of 3rd party vendors are now addressing this issue by offering fully real-time shared file systems within AWS and other cloud providers. These are expected to offer similar performance and usability to traditional on-premise SAN systems.
Another major improvement would be a vendor interface for interacting with virtual workstations in AWS. Cloud-based production workflows would benefit from an abstraction layer that could serve as a user dashboard for interfacing with the virtual machines, enabling users to connect, start, and reboot machines without needing to learn the ins and outs of the EC2 console. It would be useful to have more options in the future for 1st or 3rd party management of complex high performance virtual workstations, including Hyper-V and Google Compute machines.
Finally, this workflow demanded a considerable amount of expensive FSx storage to accommodate the large, single file RAW formats used on the production. Cloud costs were a major concern for the whole process and required careful management to make sure the cloud resources were used efficiently and effectively. While storage costs continue to decline, Light Iron is providing special emphasis on smart resource scaling in future applications and automated pipelines.
NEXT STEPS
While benefitting from some custom AWS implementations and services, Light Iron expects in the future to explore abstracting cloud workflows away from heavy reliance on one cloud vendor. There are clear potential benefits to picking and choosing services from different cloud vendors based on cost and functionality. With high-speed networking getting less expensive every day, there are fewer roadblocks to spreading an uncompressed finishing workflow across multiple providers. Light Iron believes that taking a multi-cloud approach will allow better service for studios and production companies who have chosen to contract with other providers for their storage or compute needs.
After creative color in the cloud, the next challenge to tackle is performing creative Finishing VFX work in the cloud. Both require low latency and high color accuracy, but VFX work requires even higher-performance systems with significant rendering capabilities and storage speed (i.e., Autodesk Flame).
Additionally, the industry still relies largely on physical LTO tapes for final archives. Frequently, multiple duplicate tapes must be written and verified before camera magazines can be recycled. Having a scalable cloud archive solution would remove a significant burden for a fully cloud-based production. Options exist, but details still need to be considered, with cost and ease of use being primary factors.
MOVIELABS PERSPECTIVE
This case study proves that, technically at least, cloud-based video post-production is now possible. The industry has further work to do on refinements for performance caching and audio post-production workflows, plus figuring out the economics of scaling up from a single production use case to all productions in a multi-vendor ecosystem. But Light Iron has done great work to move this workflow beyond ‘bench testing’ and small proofs of concept at industry demonstrations and tradeshows. As shown in this case study, the workflow can measure up to the rigors of a real project, with real talent, against real budgets and deadlines. The lessons learned here provide a roadmap for future improvements in workflow automation and interoperability, as well as multi-cloud workflows that span infrastructures and tenancies within the same infrastructure.
Get the Case Study
Download a free PDF of this case study