Scaling for the Super Bowl
5m
frames rendered
2 Petabytes
of data created among a globally distributed team
Summary
Gunpowder was founded in 2021 to deliver cloud-first, creative experiences and to help legacy creative visual effects (VFX) studios accelerate their transition to dynamic cloud-based operations and workflows. The goal for Gunpowder is to free production teams to concentrate on delivering their best creative work, by taking care of the cloud infrastructure and management.
House of Parliament, a boutique VFX studio born in the cloud during the pandemic, worked with Gunpowder to support 9 projects simultaneously during the 2024 Super Bowl season. The team more than tripled in size to accommodate the increase in work, involving over 300 artists, 2 PB of data, and thousands of hours of rendering—all over the course of just six weeks.
This case study demonstrates the MovieLabs® 2030 Vision by showing the power of the cloud to scale resources quickly with minimal capital expenditure in response to bursts of demand from major events such as the Super Bowl season.
Background
VFX start-up House of Parliament was launched by industry veterans in early March 2020 with a roster of high-profile projects signed and ready to go. Production was to set to begin on multiple high-profile shoots, several 2021 Super Bowl commercials among them—a week later the global pandemic hit. The Parliament team found itself with no studio, no infrastructure, and very little time to spin up production. Parliament’s co-founder, Phill Crowe, recalls, “The kind of clients that we were used to working with had high expectations, so we had to behave like a large company from day one.”
Their initial plan was to distribute digital assets to the creative team scattered across the globe. The central hub would be a single Flame instance belonging to the lead artist, working from home, who would conform and then transfer digital assets for assigned shots across the Internet to the rest of the team, each of whom also maintained local storage connected to a local Flame instance.
Figure 1: Plan for distribution of creative assets before deployment of the Ignite solution.
- Turnaround delays – Whole days were lost to changes that required redistribution of updated footage among the teams.
- Inconsistent connectivity – Transfer times for large files depended on the remote artists’ Internet bandwidth, which often was constrained or inconsistent, creating delivery bottlenecks.
- Security and versioning – Critical assets had to be stored on each individual’s workstation, so asset security was a concern as well as duplication of assets and version control.
- Inconsistent tool versions – Versions of software tools were inconsistent among artist workstations, increasing collaboration friction.
- Limited IT governance – IT had no control or visibility over the infrastructure.
The model was unsustainable, unscalable, and costly. Significant changes in the fundamental approach were needed to ensure that the technology could support production demands.
Solution
House of Parliament brought in Gunpowder to implement solutions. The effort was guided by the 2030 Vision concept of “bring the artists to the data, and not the data to the artists”. Given the physical constraints, the remote nature of the work, and the global chip shortage, leveraging the cloud really was the only option. Gunpowder created and deployed a virtual studio in the cloud based on the Gunpowder Ignite platform. The first step was to centralize storage of all media in one cloud-based hub under Parliament’s control. The team needed a simple and elegant solution enabling artists to work from anywhere without worrying about latency or continuous file transfers. The core of that solution was WekaFS, a cloud native file system, sitting on top of Google Cloud Storage deployed in the center of the continental USA. With the cloud infrastructure geographically located “in-the-middle”, the platform ensured a smooth experience and low latency for artists based on either the East or West coasts. Gunpowder then created cloud-based workstations for artists to access from the same cloud infrastructure which enabled full memory bandwidth between storage and connected workstations and further minimized latency. Services for license distribution, rendering and asset management were built on top once the basic workstations were implemented. All infrastructure was built with the Terraform infrastructure-as-code tool, which ensured that the team could easily re-create, move or expand infrastructure as needed. Terraform is a provider of infrastructure automation and a fundamental tool for scaling cloud infrastructure; it is agentless and allows teams to change or replicate cloud infrastructure quickly. Led as it was by the MovieLabs 2030 Vision principles, the Gunpowder solution focused on three key service components:
- Asset creation and propagation – Utilizing secure, cloud-only storage, artists at Parliament generated digital VFX assets directly in the cloud using remote workstations. Instead of traditional file-based transfers to the local storage of remote artists in different locations, assets were published in place using WekaFS, ensuring that applications come to the media. Essentially, assets were created, accessed, edited, and stored in place in the cloud.
- Deep libraries & access – Gunpowder architected a custom cloud-native archival solution for Parliament in Google Cloud Storage, facilitating both efficient long-term storage and rapid retrieval capabilities. The archive, structured with metadata tagging, ensured assets were easily searchable, creating a comprehensive digital library with current work, shared assets, and archival assets all available on-demand in the cloud. Rules were put in place to migrate data automatically between tiers based on age.
- Identity & access management – The Ignite platform allowed Parliament to create and maintain granular permissions for the team. Simple, native, role-based access controls ensured artists accessed only relevant project data. Coupled with encryption, multi-factor authentication, and periodic vulnerability assessments, the environment was and continues to be resilient against potential threats. Infrastructure and authentication were designed to meet the industry’s Trusted Partner Network (TPN) requirements from the ground up.
During the six-week Super Bowl season of 2024, Parliament relied on the Ignite platform to triple its team size to meet the demands of clients. Ignite enabled Parliament to hire and onboard hundreds of freelance artists from all over the world to meet the demands of the production schedule. Using traditional workflows and on-premises infrastructure for artists in offices, Parliament would have needed days or weeks to hire, onboard, procure and install equipment to get new artists up and running (assuming equipment was even available in the midst of a global chip shortage). In comparison, using the Gunpowder Ignite platform, it took less than 30 minutes to fully onboard an artist and provide access to the entire asset file tree and a full complement of applications available for the artist to do their job.
ARCHITECTURE
The original infrastructure was deployed using an open-source product called Cloud Orchestrate, which relies on Kubernetes to spin up and shut down infrastructure. The scheduling team for each production was able to create a workstation from scratch in a matter of minutes when onboarding new users.
Infrastructure was split across four main Virtual Private Circuits (VPCs):
- Production – The Production environment for artist workstations & render.
- Shared Services – Active Directory for user management, dynamic storage, etc.
- IO – A file transfer ingress and egress service to bring customer data in and out of the Production and Shared Services environments.
- DMZ (“demilitarized zone” – subnetwork facing the open Internet) – Teradici PCoIP Cloud Access Connectors
Figure 2: Overall architecture of services delivered by the Ignite platform.
Storage
Gunpowder chose WEKA as its partner for managing storage in a cost-effective way. WEKA specializes in a cloud-based storage solution that optimizes quick and efficient data storage using WEKA’s own filesystem software, built on an open-source technology called DPDK (Data Plane Development Kit). DPDK makes data transfer fast and responsive, a necessity for cloud-based collaboration for VFX teams.
When WEKA’s software is installed on cloud-based virtual machines, it manages all file storage tasks. It enables applications on these machines to run as if accessing data from local storage, even when data is actually in the cloud. It also allows most data in the cloud to be stored in more affordable object storage instead (of more expensive options like Solid State Drives or Flash memory) without sacrificing performance.
Using WEKA reduced the overall storage spend by 80% versus more traditional file block storage cloud tiers. In addition, the WEKA storage cluster was able to stream at 7GB/s in its smallest iteration, making it both lower cost and faster than direct object storage (which can introduce issues with long file lists that delay I/O operations). The speed allowed several real-time streams to be played back simultaneously, allowing applications with very high bandwidth requirements (such as Foundry Nuke and color grading) to run using cloud object storage.
Early in the Ignite project, each cloud-based Autodesk Flame machine was attached directly to its own cloud storage volume. One of the major defining moments of the deployment was the migration of that direct attached storage to central servers running on WEKA. Data was no longer copied to locally attached storage on each individual finishing station, but read and written simultaneously over the intra-cloud network. The migration cut down on data duplication and further reduced the storage spend.
Figure 3: Transition from direct attached framestore to a central framestore.
Workstations
The Parliament team originally started with a generic build for all cloud workstations across the company, irrespective of department. They soon found that resources were over-provisioned – some artists simply didn’t need high-end virtual machines all the time. The team implemented a system of “rightsizing” the builds according to the job/task booked on the machine. Initially, the team managed pools of workstations with HP’s PC over IP (PCoIP) virtual workstation management software. Later the team progressed to an in-house tool, built by Gunpowder, that monitors resource usage and suggests the optimal balance of CPU, GPU and memory in real-time. Using Google Cloud to control the virtual machine parameters for memory, CPU and GPU, the Gunpowder Ignite solution can upgrade an artist workstation in minutes in response to automated suggestions from the tool, or shrink it down again to save costs when the task no longer requires the upgraded machine.
Render
The Gunpowder Ignite solution uses the render scheduler to manage a fleet of render nodes configured in a Google Cloud MIG (Managed Instance Group). In its smallest iteration the fleet consists of a single cloud compute instance running continuously behind a kernel-space NFS (network file sharing) daemon cluster (KNFSD) in order to watch the render queue. When a job is submitted by an artist, the MIG scales up to a predetermined maximum number of instances to process the data and run the render as fast as possible. To manage costs, checks and balances ensure a single frame is rendered at half resolution, with additional approval required to submit the entire scene to the farm. Once a render job is complete the render nodes scale back down to the single “listening” node. The metrics used for increasing the render cluster size and node count are a mixture of jobs in the queue, the speed of the current jobs rendering, and the actual CPU/GPU usage on the individual instances.
Figure 4: Illustration of render architecture with caching using the Ignite platform.
Operations Manager User Interface
Gunpowder built a user interface for Ignite that allowed operations managers and schedulers to create cloud instances with a predetermined set of boundaries. The goal was to allow non-technical users to manage resources autonomously without the technical challenges of infrastructure acquisition, configuration and setup. This gave the Parliament team the flexibility and control to perform basic day-to-day operations without needing Gunpowder’s assistance, such as:
- Create/edit/delete non-admin Active Directory users,
- Change passwords,
- Create workstations from templates and assign users,
- Spin up/down workstations and render clusters,
- View pricing prior to creation and monitor ongoing spend to ensure that projects stay within budget.
Figure 5: Example of Ignite UI for operations managers.
Figure 6: Example of artist webpage interface.
BENEFITS
Maximizing creative productivity is at the core of Parliament’s technology investment. Gunpowder turned Ignite into reality by building it on top of a carefully curated integration of cloud technologies, all working in tandem to enable artistic and production talent to collaborate remotely without compromising standards. During the Super Bowl season of 2024 Parliament tripled in size and hired hundreds of freelance artists from all over the world. The Ignite platform enabled the Parliament team to deploy those artists effectively at short notice, allowing them to complete nine high-priority Super Bowl projects over a six-week period. After the commercials were released, the platform also helped offboard talent and right-size the infrastructure back to normal usage. Ensuring the transparency of post-production costs for machines, egress, and render can be challenging, especially when scaling to meet deadlines. Costs can climb quickly and may result in nasty shocks if resources are not actively managed. The Ignite platform includes billing controls that enable managers to ensure, in real-time, that projects remain on budget, while allowing them to spin up infrastructure as and when required. This flexibility and control enabled the Parliament team to meet client requirements during the Super Bowl season while staying within budget.
Alignment with MovieLabs 2030 Vision Principles
The Ignite deployment for House of Parliament illustrates the following principles:
PRINCIPLE 1
All media coming to Parliament from clients is ingested immediately into the cloud in a shared bucket that is available to all artists working on the project. The data doesn’t leave the cloud until the project is finished, video rendered and returned to clients via accelerated file transfer.
PRINCIPLE 2
All artists use virtual machines running in a central cloud instance. Each machine is loaded with the full desktop versions of the applications required by each user. The Ignite platform in the cloud is able to support legacy applications that users access via remote desktop interfaces. By centrally locating the data between the East and West coasts, Gunpowder minimizes the latency for all users and removes the time-consuming and error-prone process of duplicating data for all artists and their workstations.
PRINCIPLE 3
Since the data stays in the cloud, the workflow is driven by in-place publish operations rather than file transfers. VFX assets are created, accessed, edited, and stored directly in the cloud, eliminating the need for traditional file-based transfers to local storage. Instead, when ready to share, the artist hits the publish button in Autodesk Open Clip. Gunpowder, in collaboration with Visional.io, developed a robust timeline workflow where the entire edit is assembled at the filesystem metadata level using symbolic links leaving the assets in place. This approach bypasses the need for data input/output operations when linking materials into the edit, resulting in near-instant updates by which other artists are notified of the newly published assets. Using this workflow, any digital content creation tool such as Flame can publish an image sequence—such as a comp or playblast—into the filesystem, and the timeline is automatically updated, ensuring the most recent versions are always visible.
PRINCIPLE 4
Finished projects and assets (3D and video) are all archived in the cloud permanently and remain available for future reuse, reference and repurposing. Assets from prior productions are moved to an archive tier but remain browsable and searchable for artists to reuse.
PRINCIPLE 10
Pixel streaming and real-time streams are supported to ensure that directors and remote clients can offer live feedback on creative changes, allowing the teams to collaborate from anywhere in the world. The Ignite platform also includes billing controls that enable managers to make real-time decisions to ensure that projects remain on budget. Managers can add render nodes when needed and scale back down automatically when a render job is complete. The platform also allows the team to “rightsize” cloud builds according to the job/task booked on the machine, using a tool that monitors resource usage and suggests the optimal balance of CPU, GPU and memory in real-time. The platform can upgrade an artist workstation in minutes in response to automated suggestions from the tool, or shrink it down again to save costs when the task no longer requires the upgraded machine.
PARTNERS
The Gunpowder solution with House of Parliament requires the integration of multiple software stacks, open-source libraries, and services. The core pieces are:
HP Anywhere (PCoIP) – Remote access software that enables teams to work from any distance. It allows real-time access to the same workstation, desktop, or application from anywhere.
Hashicorp Packer – Tool used to continuously update image templates and ensure compliance across the entire fleet of workstations and render. It is integrated with Terraform to orchestrate the creation of workstations and ensure all users are working in lockstep.
Hashicorp Terraform – Infrastructure-as-code tool that ensures rapid deployments and repeatable creation across multiple regions and zones.
KNFSD – An open-source technology that allows for transparent caching, reducing the amount of data traversing the network to reduce bandwidth consumption and storage requirements.
Cloud Orchestrate – An open-source tool from Google that allows easy deployment and management of fleets of virtual workstations and entire production systems with a high-level API.
Lessons Learned
The Gunpowder team has been working on cloud-based VFX environments since the inception of cloud, initially at the Mill and MPC before the formation of Gunpowder. Key lessons learned through the years include: Adopting a cloud-forward mindset The first step in cloud migration is taking a “cloud-first” mindset that is not based on simply “lifting and shifting” legacy workflows and infrastructure to the cloud. Instead, it is necessary to challenge commonly accepted practices and take fresh approaches to avoid a cloud migration that drives up cost or risk in post-production. Reimagining post in the cloud The novel challenges of the pandemic were a major catalyst for Parliament and Gunpowder reimagining post-production. It revealed new possibilities in cloud-based production. But those opportunities were available well before the pandemic, and cloud technologies only continue to evolve. The cloud presents potential capabilities well beyond compute-intensive rendering and other high-volume processes, and we’re just starting to unlock those efficiencies with a “cloud first” mindset. More and more, infrastructure investment does not mean physical servers Investments in critical infrastructure can be virtual instead of physical, moving costs from CapEx to OpEx. That has implications not just for technology teams but also for finance groups and ultimately for capital structures and how post-production entities are funded and return capital to shareholders. Only pay for what generates revenue Tools in the cloud scale up and down in capacity and compute, but in many usage models they also generate costs only when they generate revenue. This approach enables post and VFX companies to shrink costs and infrastructure when times are lean (for example, when strikes or pandemics hit) and scale up when required to match revenue and demand. Often excess capacity (either in human talent or technology) causes vendors to low-bid a project, perhaps to offset the opportunity cost of excess capacity. With cloud infrastructure the costs are no longer constant, and a virtual workforce of artists can be released to work dynamically for other vendors who need them. Cloud collaboration without lowering the bar While it’s nice to have a creative team in the same room, the downside is needing all that talent and hardware in the same location. Cloud-based projects and collaboration lose the physical presence but gain the enormous advantage of hiring the skilled talent, wherever they are, offering them a job where they don’t have to leave home. VFX studios should focus on developing and expanding collaborative tools so those individuals can work as a team. There are workflow, change management and training challenges in doing so, but with a limited number of trained VFX artists available, the flexibility to hire talent from anywhere is an enormous benefit.
Next Steps
The Ignite solution, in its current form, is a fully capable and comprehensive solution for cloud-based VFX production and post-production. Even so, Gunpowder plans to expand functionality in a number of key areas:
- Embracing generative AI to develop forecasting, analytics and scheduling solutions for assigning work.
- Demystifying cloud technology and virtual workstations by releasing a series of videos and whitepapers to help productions spin up their own infrastructure while getting a handle on costs.
- Continuously removing the technical barrier of entry for productions and allowing faster turnaround times for highly scalable infrastructure.
MovieLabs Perspective
The House of Parliament solution exemplifies many of the benefits of the 2030 Vision by moving entire asset storage and management, as well as creative tasks and workflows, to the cloud. Many of the benefits expected from the original 2030 Vision are now realized in the deployment described in this case study, which demonstrates how the principles of the 2030 Vision can lower time to delivery, enable more dynamic and efficient infrastructure allocation, and make post and VFX workflows more nimble and adaptable. The Gunpowder team has also recognized and responded to the challenges of unpredictable or rapidly accelerating costs in current public cloud models, providing sensible guardrails and tools to empower artists to work quickly and efficiently without spiraling costs. In addition, as House of Parliament continues to deliver projects, teams will see ongoing and escalating benefits from the “always on” archive of legacy projects which can be reused, repurposed or referenced for future projects. In the next exciting iterations for cloud-based VFX and post, we expect clients will become more than just source and destination nodes in a project, but will actively work in the cloud themselves – creating ideas and concepts in the cloud, then developing, shooting and posting those assets in the same virtualized environment.
Get the Case Study
Download a free PDF of this case study