interactive projections with xbox KINECT

[FuzzyWobble]

i would really like to see someone hack a bunch of xbox kinects ([url]http://www.ladyada.net/learn/diykinect/[/url]), rig up a ton of projectors, and create an offbeat interactive freak space.
i was considering doing this but am way to busy with school and other design project. just though i would throw it out there
maybe someone already is doing this? anyone know?

[AntiM]

Uh, I'm thinking probably not.

[Stickygreen]

Don't worry, I'll make your artistic endeavors reality!

[d6]

interactive video screen @ 3:00 plaza every year since 2000 - come on by.

[C187]

interactive video screen @ 3:00 plaza every year since 2000 - come on by.

Since 2000 you say?
Damn, I really need to get to that part of the city more often.

-----

OP, I would but I really don't want to write any code or make tool kits just so a bunch of hippies can have an easier time face melting.

[lucky420]

love the screen. Didn't see it on playa but did see it at Reno Decomp

[d6]

thanks!
it's subject to the wind, so it may have been in a rejected heap at the time.
in '99 it was over on the 9:00 side in a smaller capacity, we're trying for somewhere in / around 3:00 again this year.
we've hauled it out to random deep playa points as well, which is really fun.


screen will be @ reno yuris night this year.

[nosuchtim]

I'm thinking along these lines as well, providing a space in our camp dedicated to Kinect-based as well as other interactive video things. If you're interested in collaborating or just exchanging ideas, contact me@timthompson.com

...Tim...

[rodiponer]

Hi,
I am doing something along these lines for my KidsVille `hood. Last year we made a walking machine and a morning funnel cake kitchen. We all really enjoyed the kitchen project, but I don't think we have the on-playa energy to do that two years in a row. This project is more setup but should mostly run unattended.

Sorry for the length, this is cut and pasted from an email I sent to the KidsVille list last year, to find collaborators.
------------------------------------------------------------
My goal is: Unsupervised Cooperative Group Silly Play.

Unsupervised: Even very young children can play without parental supervision
Group: Several children can play at once
Cooperative: The children are encouraged to coordinate their play together.
Silly: The focus of the play is silliness, something anyone over two years old can do without thought, and can do it together. This is not a puzzle or a thing where they have to figure it out.

My 4-7 year old children and their 12 year old babysitter love to play with Photo Booth on the Mac. This is a program that records a short video of them making faces and singing or talking in funny voices with various fun-house style filters, then plays it back. This is cooperative group silly play.

I would like to do something a little larger scale for Kidsville 2011. I have a set of four CCTV video cameras and an eight channel analog-digital converter for these cameras (which turn them into >15 FPS IP cameras), extra computers, monitors, a projector, an arcade button interface (which maps durable arcade buttons to keyboard presses), plentiful 24x7 solar electricity on the playa, and a 7x14 trailer that we transport our walking machine in.

My idea is something like a very large Photo Booth with fun-house filters, and a bit more software in there. I don't want a video game, where the focus is the software, but a silly enabler, where the software is more transparent to the children but makes their silly play more fun. The idea is a space where they are encouraged to dance around and be silly, and a little more 'burner' than the simple time delay video cameras some children's museums have.

Maybe something like a projected image that fills the end of the trailer, a camera facing the children that mirrors their movements with a trippy time delay, one or two cameras pointed out of a box where kids can make silly faces that are added to the screen (my daughter, at least, loves making silly faces into a video camera), and maybe one of the night (infrared sensitive cameras) would have a infrared pass filter (that blocks everything except infrared, like the camera on a Wii controller) that picks up the light from little infrared LED sticks or hats that we give to the children. These sticks could then control some kind of swirly pattern (or painting effect that follows the tip of the stick), music, or something. Like a big tangible user interface made out of kids.

Anyways, since I met so many software folks in Kidsville this year, I'd like to see if anyone would like to collaborate on the software side of this.

My idea is that one of us could write the framework, which takes the MPEG4 or motion-JPEG from the IP cameras and displays them on a screen, and that any number of us could write plugin filters that can be run one at a time or stacked and blended by the framework. Each plugin would take an input array of 640x480 pixel values and output an array, so to stack them we'd just feed the output of one into the input of another.

Some of the plugins could be designed to be tied to the 'face' cameras, or the camera of the entire dance floor, or the infrared only camera that see's the white dots from the children's sticks or hats. Or the filters could just be randomly cycled between all of the cameras.

The plugins could be as simple as a time delay, or a blended time delay, or a color cycling effect that takes the last few frames and color cycles them, or whatever. Even a few minimalist plugins like that would be fun for the kids.

With the framework hooked up to the webcam on a laptop, development of a simple fun house effect plugin could take as little as a couple hours. It could be as easy as taking the input pixel array and multiplying it my some matrix transformation where the result is stored in the output pixel array. Add a few variables to tweak the matrix transformation vs. time or vs. inputs from the infrared camera or whatever, and you can have, for example, a trippy effect where the kids movements leave color cycling trails behind them.

The plugins could be written in a language everyone knows, possibly even Kidsville teens, like Python. The framework itself would probably have to be in C or C++, but any language that anyone who wants to help with would be fine. My favorite would be something Linux compatible where each driving computer could be booted off a flash drive and straight into this program, with no interaction or keyboard necessary.

If there is enough interest and time for the people who want to help, we could go further and have the plugins could control the sound or relays within the trailer. The video server boxes (Axis 241Q's) have I/O pins that can be switched from a web request, and these I/O pins could be connected to relays that power a smoke machine, disco ball, bubble machine, or solenoids that bang a pan, or whatever. The input pins could be connected to switches on the floor or walls.

Since the input pixels are coming from IP cameras, multiple computers could be running multiple sets of plugins, and thus each video output (for the back wall, side walls, maybe even ceiling or floor) could be running separately from each other and projecting a different set of silly fun house style filters of the children's play from the same input cameras. This also adds independence and redundancy to the setup.

Anyways, nerds, let me know what you think, and if you would be interested in helping make this, to any degree. And, if anyone has an idea for how to make this in a structure larger than our enclosed trailer (7' wide, 7' tall, 14' long), that would also be cool.

-----------------------------------------------

In the months since this, I've sketched this out and have a 'fun enough' version working-- it's just OpenCV taking MJPEG input from four IP cameras, munging it in some way, and outputting it through OpenGL (to easily add particle and 3D warping effects). I ended up using C++ for everything, I had a performance and OpenCV API bug problem in Python. I've got a few filters that recognize faces and overlay pictures on top of them, so that, the kids heads are turned into animals and flowers, and a few dynamic fun-house mirror type effects that are just projecting the incoming video onto 3D objects that move.

I hope to write the 'more fun' version which will have things like rain that responds to the kids movements, blowing microbubbles that swirl around them, maybe a few old James Bond trippy color cycling delays, and I'd like to add the infrared LED wands that control a reality distortion field (or sparkler particle effect, or particle magnet, or whatever). Anyways, here's the original email, feel free to write to me if you want to collaborate or have ideas.

However, I am currently caught up in an augmented reality idea for a Kim Jong Il Dance Party. I know, it's in very bad taste, but I've been thinking about how fascism will change with ubiquitous augmented reality. How easy the computers we will all wear in the future could be used as tools of monitoring and control. An imam could write a script that tracks how well people follow the daily prayers. And etc. This is a playful interpretation of that future evil-- iPhone software that organizes people into a giant Kim Jong Il Dance Party, sort of like square dancing, but much more complicated because each person can be directed individually (and mistakes corrected). I bet if I put a score on each phone, for how closely an individual follows the computers instructions, then reward them with bacon afterward, I could get very good compliance from a large number of people. Think of a religion or government using the stick, instead.

Sometimes I wonder if Diffie_Hellman, Zimmerman, and the other founders of public key encryption changed the world in a bigger way than we will ever realize. I think they did. They made government monitoring of our communications that much more difficult, at least until the new cloud era where the governments just corrupt the large companies, or the root key servers for SSL. And I wonder if it's possible for someone to make a difference like that with forethought, to try and change the future world and preempt a technological shift with another technology. Like public key cryptography and the key exchange did. I feel that Kim Jong Il Dance Party is somewhere in that exploration.

[nosuchtim]

My idea is that one of us could write the framework, which takes the MPEG4 or motion-JPEG from the IP cameras and displays them on a screen, and that any number of us could write plugin filters that can be run one at a time or stacked and blended by the framework. Each plugin would take an input array of 640x480 pixel values and output an array, so to stack them we'd just feed the output of one into the input of another.

You're describing a fairly standard VJ application (modul8, resolume, etc). VJ apps typically have a lot of filters that you can chain together, and there's a standard called FreeFrame that allows people to write (and sell/share) plugins that work across VJ apps. FreeFrame 1.0 plugins take pixel values in exactly the way you describe. FreeFrame 1.5 plugins are quite different - they work within an OpenGL context.

You could set up presets within the VJ app, and then use MIDI (triggered by the arcade buttons) to switch between them. Not all VJ apps can accept multiple camera inputs.

In the southern california area, you could contact the LA Video Artists user group at http://lava.org to get some input/help on the ideas.

[nosuchtim]

I've made good progress on a Kinect-based musical and graphical instrument, which I call Space Palette. Here it is at Lightning in a Bottle:

http://www.youtube.com/watch?v=PK8kRmMt_s4

I'll have it set up in my camp at Burning Man, doing both music and graphics (at night).

...Tim...

[Token]

After "Dance Dance Immolation" any Kinect hack would be simply mundane.

[nosuchtim]

If anyone's interested in what the Space Palette looked like and how it was enjoyed, see the video clips at http://www.youtube.com/watch?v=7BzbcNnpN7M

[lfchords]

Hi

I'd like to work on an outdoor installation, using the Kinect to control lights and sound. Barring the problem of electricity and power, anyone know if a Kinect will withstand the elements? Assuming it will not, what options are there for encapsulating it to prevent clay/sand/debris from messing with it, while still retaining ability to function? polycarbonite box? will heat then become an issue?

thoughts much appreciated
- Steven

[nosuchtim]

Space Palette (a kinect-based music/visual installation, see http://spacepalette.com ) will be in Illumination Village (2:30 and Esplanade) this year.

...Tim...

[trilobyte]

You guys may want to consider posting a thread on the 2012 event boards.