Touchless Multitouch in Processing

Nikolaus Gradwohl2008-10-10T19:52:00+02:00

update (23.11.2008): i made a processing library out of the codeexample above - guru

Today i read about touchless an opensource sdk microsoft has released. it enables you to track objects using a simple webcam to create apps similar to a multitouch display.

then i spent about 20 minutes coding a processing sketch that does roughly the same :-)

as you can see in the screenshot i cant affort the same cool toys the microsoft coder has in the demo video, so i had to use some post-its instead

below is the complete sourcecode of the app. I didn't export it as an applet this time, because the applet couldn't use the quicktime api. It also only works on mac and windows this time, because processing uses the quicktime api. on linux the gstreamer based replacement from codeanticode can be used.

when the programm starts hold up 2 different colored items (use colors that do not appear in the background) then click on the first color with the left mousebutton and on the second color using the right mousebutton. and then enjoy the "minority report"-experience zooming and rotating the image :-)

colormatcher.png

import processing.video.*;
import java.util.*;

Capture video;
PImage img; 

void setup() {
  size( 640, 480 );
  video = new Capture( this, width, height, 30 );
  img = loadImage( "mangoofdeath.jpg" );
  noStroke();
  smooth();
}

int searchColor1 = color( 128, 255, 0 );
int searchColor2 = color( 255, 0, 0 );

Point e1 = new Point( 0, 0 );
Point e2 = new Point( 640, 480);

boolean s1, s2;


void draw() {
  if ( video.available()) {
     video.read();
     pushMatrix();
     scale(-1,1);
     image( video, -width, 0, width, height );
     popMatrix();

     int idx = 0;

     ArrayList p1 = new ArrayList();
     ArrayList p2 = new ArrayList();
     for ( int y = 0; y < video.height ; y++ ) {
        for ( int x = video.width; x >0; x-- ) {
          if ( match( searchColor1, video.pixels[idx] )) {
            p1.add( new Point( x, y ));
            //fill( 255, 255, 0, 128 );
            //ellipse( x, y, 10, 10 );
          } else if (match( searchColor2, video.pixels[idx] )){
            p2.add( new Point( x, y ));
            //fill( 255, 0, 0, 128 );
            //ellipse( x, y, 10, 10 );
          }
          idx ++;
        }
     }

    noStroke();
    if (p1.size() > 0) e1 = avg( p1 );
    if (p2.size() > 0) e2 = avg( p2 );

    if (s1) {
      fill( 255, 255, 0, 128 );
      ellipse( e1.x, e1.y, 30, 30 );
    }
    if ( s2 ) {  
      fill( 255, 0, 0, 128 );
      ellipse( e2.x, e2.y, 30, 30 );
    }

    if (s1 && s2 ) {
      pushMatrix();
      translate( e1.x, e1.y );
      int dx = e2.x - e1.x;
      int dy = e2.y - e1.y;
      rotate( - atan2( e2.x -  + e1.x, e2.y - e1.y) + atan2( img.width, img.height));
      float zoom = sqrt ( dx * dx + dy * dy ) / 
           sqrt(img.width * img.width + img.height * img.height);
      scale(zoom, zoom ); 


      tint( 255, 200 );

      image( img, 0, 0 );
      strokeWeight( 3 );
      stroke( 255 );
      noFill();
      rect( 0,0,img.width, img.height );
      noTint();
      popMatrix();
    }
  }
}

boolean match( int c1, int c2 ) {
  int limit = 20;
   int sr = c1 >> 16 & 0xFF;
   int sg = c1 >> 8 & 0xFF;
   int sb = c1 & 0xFF;

   int cr = c2 >> 16 & 0xFF;
   int cg = c2 >> 8 & 0xFF;
   int cb = c2 & 0xFF;

   return cr > sr - limit && cr < sr + limit &&
     cg > sg - limit && cg < sg + limit && 
     cb > sb - limit && cb < sb + limit;
}

void mousePressed() {
  if (mouseButton == LEFT) {
      searchColor1 = get( mouseX, mouseY );
      s1 = true;
  }
  if (mouseButton == RIGHT) {
     searchColor2 = get( mouseX, mouseY );
     s2 = true;
  }
}

void keyPressed() {
  s1 = false;
  s2 = false;
}

Point avg( ArrayList l ) {
  if (l.size() == 0) {
    return new Point( 0, 0 );
  }
  int x = 0;
  int y = 0;
  for( Iterator i = l.iterator(); i.hasNext(); ) {
      Point p = (Point)i.next();
      x += p.x;
      y += p.y;
  }
  return new Point( x  / l.size(), y / l.size());
}

public class Point {
  int x;
  int y;

  Point( int x, int y ) {
    this.x = x;
    this.y = y;
  }
}
Tweet This! submit to reddit Digg!   Tags: | 23 comments | no trackbacks

See also:

hexagonal patterns in processing
tree layers in processing
papercut flower
kugelbahn
rotating rainbow

Trackbacks

Comments

Leave a response

  1. Andrea 2008-10-17T13:38:34+02:00

    Great! Very nice and clever application.

    I LOVE PROCESSING :)

  2. Bill Van Loo 2008-10-17T14:33:48+02:00

    Wow, this is fantastic! I can see this as a great jumping-off point for all sorts of things...

    Thanks for sharing not only the idea but also the source code!

  3. n8dgr8 2008-10-17T17:31:05+02:00

    This is awesome! Certainly a great example of the power of Processing! Thanks!

  4. thomas 2008-10-17T19:30:47+02:00

    good show!

  5. Henry 2008-10-18T03:18:17+02:00

    100% Awesome.

  6. James 2008-10-19T22:11:11+02:00

    I couldn't help but noticing the fingers the wrong way around in the picture you put up :)

  7. Bobby 2008-10-20T15:08:30+02:00

    Awesome application. Me being me, I straightaway ported it to As3 for the Flex crowd. This actually should work on OS X, Windows, and Linux without modification.

    Without further ado:

    http://pastebin.com/f5795068b

  8. luclodder 2008-10-21T20:56:20+02:00

    gr8 works like a charm. Wundefull.

  9. Blink 2008-10-21T22:25:01+02:00

    Brilliant!

  10. color_blind 2008-10-26T20:09:40+01:00

    I´m having some issues. When I select the first color with the left mousebutton everything goes fine but then when I choose the second one with the right button it crashes. Any help? Thanks.

  11. color_blind 2008-10-26T20:34:44+01:00

    Problem solved! Briliant! ...my blindness it was not helping me.

  12. Trey 2008-11-04T22:58:56+01:00

    lol ok first off..let the rage go.. I am stuck with coding C#, it's the company's call. Nuff said, no options.

    So please choke back that flame, and tell me, anyone done this very nice code up in C#?

  13. naus3a 2009-05-04T11:11:39+02:00

    it's very easy to do that also without any marker: look at bryan chung's examples or my arkanoid

    :)

  14. kenito 2009-06-09T20:57:31+02:00

    Hi, I'm a french interactive designer... Merci beaucoup pour cette application et le code source : brillant et tellement simple. Congratulation !!!!

  15. ibrahim 2009-07-11T16:31:15+02:00

    perfect...maşallah :D

  16. e238024t 2009-09-05T07:03:48+02:00

    incredible...

  17. aadhavsmail@gmail.com 2010-03-22T10:02:11+01:00

    Can anyone explain how this color tracking working???

  18. Damien 2010-06-12T00:34:57+02:00

    i have the same issue when I clic on the second color,the app crash. do you know how to resolve this ?

  19. Damien 2010-06-12T00:41:21+02:00

    just dowmload or rename a jpg with the name: mangoofdeath.jpg and put it in the same folder than processing and...enjoy!

  20. /dev/null 2011-06-01T08:48:38+02:00

    what about some gloves with led's (rgb) distinct color for every finger and you could frost them to know down the brightness and or dimm them

  21. DekWilde 2011-08-24T16:36:52+02:00

    Great stuff. Very simple

  22. ashshaosh 2011-09-01T21:19:09+02:00

    great

  23. Bubblebloom 2011-12-17T16:39:20+01:00

    Great and thanks ;)

Leave a comment