It is not every day that one of the most successful visual effects and animation studios contacts you for a collaboration on a blockbuster movie. We were really surprised and honored when Sony Pictures Imageworks approached us and asked to integrate our Rebelle watercolor simulation into their VFX pipeline for the production of the Spider-Man: Across the Spider-Verse movie.
"We knew this was something that we definitely wanted to take part in. The movie project was huge and to contribute to it with Rebelle capabilities was a great challenge, but we believed it would work incredibly," says Peter Blaškovič, CEO of Escape Motions.
The task our team of developers faced was to create a special version of Rebelle that Sony Pictures Imageworks could integrate into their pipeline which renders an animation sequence. Because Rebelle is designed for real-time painting, not animation batch rendering, the main challenge was to get a grasp of what the technical pipeline required and translate all the tools, very different in nature, into a productive animation workflow.
From the technical point of view, the main requirement was to set a list of painting commands grouped into animation frames and then at each frame to import, run the watercolor simulation, and export various data layers (RGBA, water, impasto, masks) which dive into the movie rendering pipeline. Because there are post-processing stages in a common movie pipeline, it was necessary to paint and work in 16 bits instead of 8 bits to avoid post-processing artifacts. Moreover, when an artist creates a list of commands to paint and render an animation and then they need to modify it slightly by adding a stroke in the middle of the frame sequence or making some strokes longer, it shouldn't influence other strokes and drippings. This wasn't an issue for our artists who use the software for static paintings. So we had to rework our random generators and command processing carefully. For the VFX pipeline, we had to build a Linux version which is commonly used on screen-less render farms. We created a docker image with Xvfb included which allows Rebelle to run without a physical screen connected to the computer.
Pav Grochola, FX Supervisor on the film and Lead VFX Artist at Sony Pictures Imageworks comments on using this unique approach on the Spot, the villain character: "In terms of the technique, we did something interesting in this film in that we partnered with Escape Motions, which creates this painterly software called Rebelle that has an inbuilt watercolor solver, and you can do cool stuff like wet the canvas. You can put ink on the wet canvas and the ink spreads into the paper. We always tried to simulate the natural organic detail of painting in real life. Right at the beginning, we were working with that tool, and one of our biggest goals was to try to make the movie look as hand-drawn and handmade as possible; that tool to us was the perfect thing for experimenting with natural media. Not only can Rebelle do watercolor but oil paint and charcoal in such a convincing way that would be hard for us to develop from scratch. We merged that software into our software and created this cool 3D and 2D combination where the Spot is made of paint, ink, and moves around leaving ink droplets; he is himself constantly redrawn with natural media." Besides Spot, you can spot (no pun intended) Rebelle in many other scenes as well both for character and environmental FX.
Huge kudos to Sony Pictures Imageworks for their ambitious approach in creating one of the most unique and visually stunning pieces of animated cinematography and for going the distance to make the movie artistically driven.
If you're a fan of Rebelle and animated movies, we highly recommend going to see Spider-man: Across the Spider-verse, now playing in theaters! Not only does it have a great story but the visual effects and animations are mind-blowing. Can you guess what was made with Rebelle? :)
For more information on the creative process of Sony Pictures Imageworks, you can read this article:
https://www.vfxvoice.com/diversity-reigns-supreme-in-spider-man-across-the-spider-verse
EDIT [June 28, 2023]: We plan to release Rebelle's animation solution in the near future. It will enable Rebelle to receive commands from an external custom JSON file which you can group into animation frames and export to be used in larger animations. These features will be integrated into Rebelle 6 Pro and available for free for non-commercial, non-profit use. A commercial license can be purchased upon request - if interested, get in touch with us to set the license up at [email protected].
----
Images courtesy of Columbia Pictures Industries, Sony Pictures Animation, Sony Pictures Imageworks, Marvel