DreamWorks Animation pushes the limits of CG filmmaking with Linux.
All the big film studios primarily use Linux for animation and visual effects. Perhaps no commercial Linux installation is larger than DreamWorks Animation, with more than 1,000 Linux desktops and more than 3,000 server CPUs.
“For Shrek 3, we will consume close to 20 million CPU render hours for the making of the film”, says DreamWorks Animation CTO Ed Leonard. “Each of our films continues to push the edge of what's possible, requiring more and more compute power.” Everyone knows Moore's Law predicts that compute power will double every one and a half years. A little known corollary is that feature cartoon animation CPU render hours will double every three years. In 2001, the original Shrek movie used about 5 million CPU render hours. In 2004, Shrek 2 used more than 10 million CPU render hours. And in 2007, Shrek 3 is using 20 million CPU render hours.
“At any given time, we are working on more than a dozen films”, says Leonard. “Each of those films has its own creative ambition to push the limits of CG filmmaking.” DreamWorks Animation employs about 1,200 people, with about two-thirds in their Glendale studio and the rest in their PDI studio in Redwood City linked by a 2Gb network. (Note that DreamWorks Animation, a publicly traded company led by Jeffrey Katzenberg, isn't Steven Spielberg's DreamWorks live-action that merged with Paramount recently.)
“There were many specific technical advancements on the movie, including advancements in hair, clothing, costuming and crowds as well as bringing the secondary character animation [crowds] to a whole new level of performance”, says Leonard. About 350 people are working on Shrek 3, with about 300 at PDI and 50 in Glendale.
In Shrek 3, Fiona transforms a bevy of classic “rescue me” fairy-tale princesses into action figures to defend the kingdom of Far Far Away from usurper Prince Charming. How to convert Sleeping Beauty's narcolepsy into a weapon or get neat-freak Snow White to dirty her nails fighting bad guys seem like minor challenges compared to the technical obstacles involved.
“DreamWorks Animation R&D provides the tools, libraries and software infrastructure for the creation of world-class CG films”, says Leonard. “We develop and support a suite of application tools for our films, including a proprietary animation system, lighting, rendering and compositing tools, and effects tools for things like fire, water, clothing and crowds, to name just a few.” Leonard estimates they have several millions lines of custom code, mostly written in C (legacy code) and C++ (newer code).
Andrew Pearce leads the DreamWorks Animation R&D group based at PDI. “Long hair may be the biggest technology advance in Shrek 3”, says Pearce. “In all of animation in the past you've seen long hair very little.” “It took months to do the hero-hair flick on Shrek 2”, notes Visual Effects Supervisor Philippe Gluckman. “Hair is everywhere in Shrek 3.” How hair glides across a shoulder looks easy but is very complicated to model. “The way the hair moves had to become much more automated”, says Gluckman. There isn't time for animators to position each hair by hand.
“With clothing we have more interactions, including ripping of the cloth”, says Pearce. “Fast motion is always difficult. In the real world, there's only so fast you can move, but nobody has told our animators that. If you went from 0 to 500 mph in one second, you'd probably leave some clothing behind in the real world.” Animation reality is as much art as physics.
Figure 1. Our print copy of this photo surely doesn't do justice to the astounding detail in the hair of the characters. (Photo credit: all Shrek photos in this article courtesy of DreamWorks Animation LLC.)
It isn't just the challenge of animating some clothing, it's how much clothing. “We have a lot more characters in the same shot”, notes Shrek 3 Co-Director Raman Hui. “Shrek 3 has a huge cast with 48 characters”, says Shrek 3 Director Chris Miller. “We have huge crowd scenes with 40 to 50 characters on a stage and 2,500 in the audience.” “The challenge in crowds is each character needs to look different”, says Pearce.
“If we had to do one setup for each character, that would take too long”, says Character TD Supervisor Lucia Modesto. “We take a generic character and warp that character. We have Man A, Man B and Woman. The big variation you get in a crowd scene is the silhouette of hair and hat. For characters in Shrek, we had one generic man with three variations, now that's 16. Women went from five to 25 variations and 13 hairstyles.” A lot of work is done to make known characters like Shrek look better, but without looking different.
“Global illumination is something lighters everywhere love”, says Pearce. “That bounces the light off all the surfaces.” The challenge is global illumination, often implemented as ray tracing, can be expensive. “It's brilliant for architecture fly-throughs where the lighting is static and baked onto the building”, says Pearce. “Our problem is everything in Shrek 3 is moving.” The DreamWorks software “bakes” (calculates ahead of time) what's not moving and then tries to do the parts that are moving efficiently. “Global illumination is in almost every scene in Shrek 3”, says Pearce. “Scene complexity is what caps us now, such as forests. Ray tracing there is not reasonable.” Advances in Linux computing power and multicore chips and software are pushing render capability higher and higher.
“We use global illumination like a DP does”, says Art Department Production Designer Guillaume Aretos. “We use bounce cards and colored bounce cards. The moviemaking process for us is very close to live action where we use the light that bounces. Shrek 2 was a south of Italy feel, kind of Beverly Hills turned into Italy.” Shrek 3 moves away from the eternal spring of first two movies, with a more Northern European look. “It's very hard to get an overcast effect with bouncing backlight”, says Aretos, “and bright light shining through dark clouds.”
“I'm the head of layout”, says Nick Walker, “which makes me the DP.” Layout is the group that figures out where the virtual actors will stand in virtual sets. “Shrek is seven feet tall and all torso”, says Walker. “Shrek could swallow Fiona's head when moving in for a kiss. Puss n' Boots and Donkey, the two sidekicks, are different sizes. It's difficult to get a two-shot.”
Figure 2. Donkey with Shrimp Skewers on Fire—Final Lit Version of the Scene.
The production moves from story and concept artwork into 3-D modeling and eventually render. DreamWorks Animation uses the popular Linux Maya commercial package for 3-D modeling. Layout positions the characters in the scenes and determines overall lighting. Models are “rigged” with internal skeletons by the Character TDs, then given to the scene animators. Because of the complexity, Shrek 3 animators were assigned in pairs to each of the hundreds of scenes. In the past, it was one animator per scene. Lighting and any special effects are added, such as cloth or flames. Then, the scene is rendered frame by frame on a 3,000+ CPU Linux renderfarm.
Figure 3. Storyboard: a Story Artist's Rendition of the Scene
Each frame is assigned to a different node of the renderfarm by grid software (using Platform LSF, a commercial Linux package), so that many frames can be output simultaneously. The frames are edited into a movie using Avid software (not on Linux). Early in the process, hand-drawn storyboard images are scanned, and a scratch audio track is edited together creating a rough video representation of the movie. As each sequence is completed, it replaces the rough storyboard footage, building the fully rendered movie scene by scene.
Figure 4. Layout: the Layout Artist's Blocking of the Camera Moves and Character Poses
“The fact that our main production pipeline is all on Linux is pretty interesting”, says head of Production Technology Darin Grant. “We've been on Linux for years, but I'm still amazed. Looking back, when using Linux was a radical concept for Digital Domain's renderfarm during Titanic, it wasn't that long till the industry moved toward wholehearted adoption.” The studio Digital Domain, where Grant formerly worked, built the first Linux renderfarm for Titanic (released in 1997). Grant now works at the PDI DreamWorks facility in northern California and commutes to Glendale each week to ensure that his cross-site team is in sync. He also uses VSC, an immersive video teleconferencing system developed by DreamWorks Animation that HP has since taken to market as the Halo room. HAVEN is the DS3 45mbs Halo video exchange network, used extensively to connect between DreamWorks facilities and with HP's desktop division and with AMD.
Figure 5. Animation: at this stage, the animators create the character's performance for the scene.
“The issues with maintaining a large Linux-based pipeline are the same as maintaining a large pipeline on any operating system”, says Grant. “We unified the studio on one standard pipeline a while ago, and now we have all productions at all times using the same pipeline. They stress, push and develop the pipeline in different ways on each production. Linux provides us with many advantages. Solid support for threading, NFS and LAMP toolsets are big pluses for us. Managing developers gets easier each year as the quality of the development tools and IDEs available on Linux improves.”
Figure 6. Lighting: Final Version of the Scene with Lighting and Textures Added to the Frame
“At the desktop, we use HP xw9300 workstations running RHEL 4”, says head of Digital Operations Derek Chan. “The renderfarm uses HP DL145 G2 servers. Our standard for memory is to have 2GB per core. Servers have four cores, so that's 8GB.” Chan says DreamWorks Animation has a good relationship with Red Hat, working closely to ensure that HP workstations work with Red Hat Linux for DreamWorks.
Figure 7. The Huge Server Farm at DreamWorks Animation
“A challenge we overcame on Shrek 3 was the integration of metadata into our pipeline”, says Grant. “In a cross-team effort between R&D and Production Technology, we put in place a system that maintains historical version information, render statistics and other really valuable data in each and every file we produce. That we were able to do this shows one of the key advantages of having a proprietary toolset and file formats.” DreamWorks uses its own TIFF-like file format based on 16-bit binary fixed point, a limited High Dynamic Range (HDR) image format with a color range of 0 to 2.0. Letting images go whiter than white leaves headroom for image adjustments.
DreamWorks Animation technology is organized into three core groups: R&D to create new technology, Production Technology that oversees the production pipeline and Digital Operations that's responsible for the compute, network and storage infrastructure. The production pipeline has hundreds of small tools and applets that form the other glueware, which enables 200 people to work as an orchestrated team. Most legacy pipeline code is written in Perl, and most of the newer code is being written in Python. “We'd love to be all Python”, says Leonard, “but today we still have lots of Perl”.
“Our team has been spearheading the transition from Perl to Python at the facility”, says Grant. “There are three primary reasons for this. The creation of Python bindings to a C++ library is very easy and allows us to utilize core R&D libraries in the rest of the pipeline more quickly. The object-oriented nature of Python is very attractive given our new asset model and should allow us to make changes to that asset model much more easily in the future. And, Python is a first-class citizen in many of the third-party software applications that are used in our industry.”
At the Linux Movies Conference, an all-day conference for motion picture technologists, last held in 2005 [that I chaired], the consensus of studio technologists was that the constraint on renderfarm size was heat. “The wall is still heat and power”, says Chan, “and a little bit of floor space”. DreamWorks Animation is into dual-core and about to go quad-core. “In the next eight months, we'll switch to eight computing cores per desktop”, says Chan. The studio is interested in accelerated computing with GP-GPU. That has the potential to move render processes from overnight to interactive. But, there are daunting technical barriers in that GPU programming is so alien, and there are bandwidth limitations going between CPUs and GPUs on graphics cards. “There are significant performance gains to be had”, says Chan, “especially if we could keep it all on die with an AMD-integrated GPU”.
Shrek 3 consumes 24TB of storage, out of an allocation of 30TB. DreamWorks likes to keep all its movies in near-line storage on big arrays of spinning disks. “People refer to previous movies all the time”, says Chan. “We have the three Shrek movies. Madagascar is getting a sequel.” Although everything is pretty much kept on-line, DreamWorks archives to tape sent off-site for disaster recovery.
Why don't the movie studios contribute some of their millions of lines of Linux code to open source? Many studios have developed proprietary Linux video playback and editing software, an area where open source is deficient. Could they give that to open source? Today's treacherous patent landscape is one obstacle, but beyond that is the cost to maintain it. For example, ILM found it more work to open the OpenEXR image format than expected. The studios are busy making movies.
The film industry does sometimes sponsor outside open-source efforts, such as deep paint support for GIMP in 1999. Unfortunately, 16-bit per channel paint was never released as part of GIMP. It did later see the light of day as CinePaint [an OSS project I lead]. But rather than use CinePaint and have to retrain Photoshop users, DreamWorks Animation, Disney and Pixar provided some funding to CodeWeavers to make Windows Photoshop work on Linux under Wine in 2003.
The film industry may not like free software that cuts too close to its domain. The freeware renderer BMRT, developed by former employees of Pixar, was discontinued as part of an infringement settlement in 2002 between Pixar and NVIDIA (which had acquired a more sophisticated version of the BMRT render technology from the company Exluna to support Cg GPU rendering).
“CG filmmaking is one of the few places where there's a tight bond between technology and the art of filmmaking”, says Leonard. “Technology is enabling artist vision like no other time in the history of our business. We continue to invest heavily into rendering techniques such as global illumination to make lighting better and easier. The original Shrek movie did not use any global illumination. Shrek 2 used it in a very limited way, and Shrek 3 uses it broadly across the film. The result is better lighting to enable better storytelling.”
“Beginning in 2009, we'll be releasing all of our films in native stereoscopic 3-D”, says Leonard. “Our films will be created, from the start, with 3-D stereo in mind. The result will be a whole new level of experience in theaters.” Monsters vs. Aliens (tentative title) and How to Train Your Dragon will be the first 3-D films from the new 3-D pipeline. Since Shrek 3, the studio has built a new system for creating all storyboards digitally from inception in 3-D.
DreamWorks Animation has more Linux geeks on tap than most Linux companies or open-source projects do. If you're interested in working on Linux in the motion picture industry, DreamWorks is advertising job openings for Linux technologists, including Systems Architect, Senior Systems Administrator, Senior Systems Developer, Systems Engineer, Animation Tools Software Engineer, Core Libraries Software Engineer and Software Engineer Manager.
