Fluidance | Motion Effect

Finally... it took me a whole year to publish this motion effect. Most of it I had prepared a year ago but I had so much work with Flame Painter (still have), that there was no time left to focus and finish this one. I call it Fluidance - dancing in fluids because it looks great together with dance movements.

This experiment meant to be interactive - with a camera connected to the fluid engine, so you can play online in front of the camera, but unfortunately, it's technically not possible because of Java restrictions and I could use only offline videos as examples.
These examples are not high-quality videos, but they are the best I have found for this purpose.

How does this effect work?

The effect is a combination of three steps:
- video / camera source capture
- motion analysis
- realtime effect


The camera or video source capture is the simplest part, I used a processing library which reads a movie or captures video. (Well, if I had to write a camera capture library, this would not be the easiest part. :] )

The realtime effect is my Fluid Fire Simulation with modifications - similar to Fluid Painter, but instead of painting in fluids with a mouse, the movie (or camera) is the source and also sound analysis is included. There can be used also any other effect, like Flame Painter (I haven't tried this yet), or anything visually interesting.

The most interesting part is the motion analysis. This was the main part of my research. I had to look for more solutions but none of them was suitable for what I needed. When you analyze movie frames with simple picture difference, you get information which pixels in the next frame is changed compared to the previous frame. But you have no info about the motion vectors. Another option is to find blob segments (it's included in some video libraries), analyze them, but this is good for multi-touch finger detection or the whole object movement analysis.
What I needed was the realtime complex picture motion analysis. (huh, what?)
That means, when I have my hand in front of the camera and move one finger, the system detects that only one finger was moved and knows the movement vector of every changed pixel. Or if there are more complicated movements, they can be also detected in an appropriate way.
Something like this you can find in some post-productions plugins (I found a few for Adobe After-Effects), but they are far-far away from realtime.
Voila, and I found a solution! I created an algorithm to analyze a video source in realtime, found the differences between frames and added appropriate local vectors. The movement vectors are then used to move the fluids or any other post-effect. This can be used on any source, it detects the vector movement of persons, cars, even a huge Enterprise spaceship and dancing people inside can be detected! :) It's not bulletproof in every circumstance (nor the system is), but it works fluently and realtime, and you can use all these awesome visuals for live performance.

Here you can find more experiments.

References and video sources:

The video referencies are used from youtube, they are copyrighted to their respective authors.

- A dancing girl from Korea - DanceShower (posted by lpdvv)
- Modern Dance: The angel is smiling - performed by amazing Guo Yaqi (15y.o), Third prize Taoli Cup (posted by littlechristina)
- Dream of Bellydance - performance by fantastic belly dancer Alexandra. (posted by Gagica86)

Realtime effect is programmed realtime in processing.