PuppetShow is a system that allows people use there bodies to control virtual characters. Puppet show uses a webcam to track the movements of a puppet and translates the gestures into bone movements in a game engine. Puppet Show’s goal is remove confusing middleware (keyboards and mice) from the player interface and allow the users to use their body’s natural expressiveness to convey themselves in virtual environments.
PuppetShow was built with Processing and runs within the Unreal Runtime using custom characters. Currently PuppetShow is in the late alpha phases and can track any flat color puppet and relay the actions two different specialized characters: a duck and a panda. Currently the PuppetShow team is integrating the complete PuppetShow animation system into the standard Unreal skeletal system. In the near future PuppetShow will be able to control any Unreal character through the PuppetShow interface.
To download the different alpha packages of PuppetShow visit the downloads page. To see puppet show in action, check out the pictures and video below!
Ok, manifesto might be to strong a term.
PuppetShow was created to liberate Machinima actors and expressive gamers from the shackles of the mouse, keyboard and other abstract middleware. The mouse and keyboard cannot project the human figure’s true expressive range into a virtual environment intuitively. To do so, an interface must be able to understand motions from the whole human body, not just complex motions on a plane (mouse) or keystrokes.
PuppetShow’s goal is to provide a simple, open-source, expressive interface for the Mahinima and expressive games community in the hopes that it will allow for more expressive interactive experiences to be forged. Intuitive and expressive interfaces are the future of human scale computing. PuppetShow is just a concept sketch of that future.
PuppetShow in truth is actually two separate systems that communicate over a TCP link: the puppet tracker and the Unreal puppet animation system. Below is the overall flow diagram of how PuppetShow works.
Puppet tracking is accomplished by using a puppet, a webcam, and the PuppetShow color tracking software. The color tracking software uses a refined blob tracking algorithm to track solid colors on a puppet. Since the tracking is based solely on color almost anything can act as a puppet: origami, gloves, socks, shirts and so forth! The tracking color can be set easily with the dynamic color calibration system. Based on a predetermined scheme, the tracking software then takes the blobs and translates them into usable information which is subsequently sent to the Unreal Runtime running the PuppetShow animation system.
The PuppetShow Mutator takes in values from the color tracker and translates them into bone translations and rotations. These are then applied to the character controlled 3D mesh (in the above diagram: a ducky!) which is subsequently displayed on the screen. An important note: PuppetShow does not interfere with the standard input devices of Unreal. This allows players to move and manipulate their characters as they would usually be able to.
That's it in a nutshell!
Comming soooooooooooooooooon... dot dot
The original character demo video:
And some images of the system being used!
