Sony Automates VFX “Pulls” and Pushes the Industry Closer to its Cloud Future
1750 Pulls
Creation of DPX 12-bit VFX files
975 Shots
Over the course of three months
30 TB
Storage for MXF Master files in AWS
Summary
The 2030 Vision is moving to a more efficient media creation ecosystem – where media processing can occur on a single source of truth on media stored in the cloud. The key to this is enabling production teams, studios and their vendors to be able to access those files without moving them to chase the next stage in the process.
Sony Pictures partnered with their sister company Sony Electronics to develop a VFX pull system which moves us toward this cloud future by automating the complex process of coordinating, creating and transferring thousands of VFX shots between multiple production vendors. The solution they created, “Ci Workflow”, a native app on Ci Media Cloud, Sony’s SaaS offering, gave the studio full visibility and control of critical unreleased movie assets and a single source of truth for those files in the cloud. In addition, it saved VFX vendors valuable time and created efficiencies so they could focus on the creative tasks rather than infrastructure and traffic management for file transfers. The solution was deployed on two movies (“The Unholy” and “The Invitation”) and is in production on four more titles.
Background
Major movie productions can have one or several digital visual effects (VFX) vendors providing services on individual frames of video within a production.
The process of sharing those frames with the service provider(s) is called a “VFX Pull” and traditionally includes numerous participants in the production collaborating. In a typical flow (Fig. 1), the VFX vendor’s data team requests the frames they need to access from the organization storing the Original Camera Files (often the DI vendor who will be doing the final color). Meanwhile, the editorial department provides reference video sequences for that edited scene and accompanying Edit Decision Lists (EDLs). All of these elements can continue to change as the production proceeds and revisions are made. These external vendors liaise on the multiple transfers by coordinating via the VFX Supervisor (or their assistant) and initiate many hundreds of file deliveries usually via IP acceleration services (eg. Aspera or Signiant), cloud shares (eg. Dropbox or Box) or hard drives and with supporting files uploaded or even emailed.
Whenever there are many participants collaborating on complex data movement using manual tools, there’s a chance for confusion, error or simply delay. All those people need to coordinate and agree on which files need to be delivered, to which location, which system should move the data, which metadata should accompany it and the work order details (contained in supplementary PDFs and emails including critical project details like timing, deliverables required, formats etc.). These VFX pulls, and the corresponding VFX pushes of finished material for approval, are happening concurrently on different sequences as the production is continuing. With potentially hundreds of pulls to complete per production, there’s a lot of moving parts and a lot that can go wrong.
Figure 1: A traditional VFX pull process
Solution
Sony Pictures wanted to improve the flow of data for VFX pulls and automate much of the manual work required. The COVID pandemic added an extra sense of urgency with workers in various locations adding extra complexity to the old workflows but also a receptivity to try something new.
Working with their Screen Gems affiliate, the Sony Pictures Technology team developed a proof-of-concept VFX automation system code named ‘Tiburon’ leveraging Ci Media Cloud (Ci). The objectives of the project were to:
- Centralize the required original camera file (OCFs) assets in the cloud, acting as a single source of truth, rather than housed at any one vendor location.
- Automate the permissioning of that data to VFX vendors using a single delivery system for all assets (master video, reference video sequences and EDLs) – the VFX “Pull”.
- Reduce paperwork so all work orders are standardized in one system.
- Ingest finished shots from VFX vendors back into the cloud source – the VFX “Push”.
Figure 2: VFX pull process after automation with Ci Workflow
Tiburon has processed and named over 10,000 frames for our show, both in large pull requests for full turnovers, or quick one-offs that need to get turned around fast. Overall, a great experience.”
Partners
The system was successfully deployed on the Screen Gems productions ‘The Unholy’ and ‘The Invitation’ and in partnership with VFX vendor ‘Temprimental’. The files were stored in Amazon Web Service’s S3 object storage and managed by the Ci service. By leveraging Ci’s public APIs the Sony Pictures team were able to expand on the core collaboration and workflow tools provided by Ci and expand the functionality to support VFX Pulls. By using an existing cloud-based tool, the Sony team could take advantage of Ci’s storage, authentication, authorization, media processing and asset management features without needing to custom build them. Based on the success of the proof-of-concept, the solution was productized and added as a new commercially available service within Ci Media Cloud as “Ci Workflow” and announced at NAB 2022.
Architecture
Ci Workflow lets productions directly upload original camera files into Ci while storing them in Sony Pictures’s cloud location. Ci transcodes the files into a high-bitrate mezzanine file (12-bit HQX DNxHR MXF).
For VFX pulls the editorial team creates EDLs and a proxy video file in QuickTime from their AVID system (operating either locally or via Microsoft Azure hosted Avid Media Composer). These are ingested directly into Ci using the Workflow browser app interface.
Screenshot showing web-based file ingest process
The production team can augment the files with searchable metadata that provides extra context about the work to be done.
Ci Workflow automation goes to work and prepares the master files (12-bit HQX DNxHR MXF files) that match the clips referenced in the EDL and if applicable transcodes them to 12-bit DPX files for the VFX vendor. The VFX vendor is automatically sent a Ci MediaBox link to the package of work assets which they can download or copy to their own Ci account. The package contains all the required files for a shot – the master DPX frames, the matching QuickTime proxy and the EDL – all with matching metadata. With files flowing automatically from the production to the VFX vendor, there is no need for assigning traffic departments, producers or other staff to track incoming projects. Productions can save time and money and enable their work flows 24/7, even when vendor facilities are closed.
All progress is tracked in Ci Workflow so everyone can see the status along the way.
Ci was simple, efficient and versatile. Simple EDL workflow for pulls, efficient with automation around the clock and versatile with the ability to send ref QTs and review QTs to both vendors and filmmakers. When needed, tech support was responsive and quick.”
BENEFITS
With Ci Workflow the creative team could focus less on managing media movement and more on creating compelling content. As each user is required to have a Ci account, an audit trail is established, with both users and tasks authenticated and authorized. With an automated workflow sharing cloud media assets, and a common dashboard, there is no lost time with teams wondering where files are in the process – everyone can see the status of orders and deliveries. Studios and content owners retain control of assets and visibility of file movements for enhanced security. Files can be scheduled and published 24/7 without waiting for a producer or traffic department to be on-call to manage data movement and therefore the teams can work to their preferred hours rather than when the media is available. VFX vendors also benefit from lower storage costs as they do not need to hold source OCFs – they remain available on the cloud. Cached copies of the files in use can still be stored locally at the VFX vendor to accelerate performance of the artist’s tools.
Using the Ci interface, any file can be quickly referenced and played back in a standard web browser to provide notes and feedback to the team.
Alignment with MovieLabs 2030 Vision Principles
The 2030 Vision includes 10 principles that imagine a future where all assets, workflows and tasks can be securely run from the cloud using software defined workflows where creatives can focus on creating and not on mundane repetitive tasks such as file copies or asset chasing applications before work can commence. Sony’s Ci Workflow case study demonstrates the benefits of a single source of truth in the cloud, automation of complex but ultimately not differentiating jobs and all tied into a seamless user identity system. This use case demonstrates the following 2030 Vision Principles:
PRINCIPLE 1
Sony’s Ci Media Cloud stores all camera assets and metadata, including the OCFs in one cloud repository for other processes to access and use. By tracking versions and derivatives the cloud acts as a “single source of truth” and doesn’t require sequences to be created, copied, shared and ultimately left with different production vendors.
PRINCIPLE 3
Data movement has the potential to be minimized with assets stored in the Ci cloud and automated notifications for users that assets and tasks are ready for them. By not moving the assets but providing access to them, delivery of files becomes an instant process with a notification arriving that indicates a user or users can now access files in place – no need to wait for a copy to move.
PRINCIPLE 6
Sony recommends every user attached to a project be assigned a Ci account. The studio can then see who is accessing their media, provision media/tasks to specific named users and provision/control groups and teams and users. A consistent user ID and authentication service allows for clear, transparent security and access controls for content owners.
PRINCIPLE 9
By creating an automated software defined workflow that relies on industry standard files (proxy and media files plus EDLs) using published APIs, Sony has demonstrated the benefit of sharing file permissions and directly connecting studios, productions and vendors with a common interfacing system.
NEXT STEPS
Sony is not stopping with the VFX Pull automation system. In future expansions they intend to work with multiple VFX vendors simultaneously, expand to bigger productions with more VFX shots and to include other EDL-based workflows. Also, on the roadmap, are plans to broaden the supported camera formats, add lens metadata, add more source video resolutions and apply color space transformations, when necessary. Sony is also looking to expand to 16-bit EXRs to enable content with extended dynamic range.
As they reviewed the work on the initial Tiburon proof-of-concept and the resulting Ci Workflow, Sony Pictures recognized they would like support from cloud service providers for native object storage support of individual frame file formats (eg. EXR) with the ability to preview a sequence of individual frames as a video file within a browser. If the frame files can be aggregated and handled together (for example all files in a sequence downloaded as a single package) then software systems can be dramatically simplified.
MOVIELABS PERSPECTIVE
Original Camera Files (OCFs) are big, hard to manipulate and required by the VFX process. Unlike other areas of the workflow where proxies are fine (review and approval, dailies, editorial, etc) VFX vendors need the highest quality files with the most bit depth and resolution. So, minimizing OCF movement is one of the first areas we need to look at when optimizing workflows for the 2030 Vision. Sony’s Ci Workflow implementation proves that the VFX turnover process can be automated, file movement minimized and that the time saved immediately pays in more time for creatives, less time waiting and removing the requirements for multiple people to act as ‘air traffic control’ for complex data and metadata movements. With this part of the workflow covered we can now expand to getting those OCFs straight from the camera into the cloud, and expose other OCF based workflows such as color grading and mastering. Currently we’re still seeing caching of these large files at the VFX vendors facilities to improve performance but we’ll unlock the full potential of Principles 1, 2 and 3 when these large files can stay in one cloud with remote workstations/applications, collocated with the media. We look forward to those next steps on the path to 2030.
Get the Case Study
Download a free PDF of this case study