Virtual Pets

For this week’s assignment I wanted to work with color tracking in Processing. I wanted to alternate objects that a person is supposedly holding in his or her hands and initially I thought of substituting an object of a certain color with a respective picture of another object. However, I didn’t even need to substitute the colored object, I could just display the image in a certain distance from the object instead. I wanted to create the illusion of being able to hold and move around different animals, therefore in my project I have 5 animals each appearing on the screen when there is a certain color present (the colors I used are pink, blue, green, red and yellow). These are the steps of creating my project:

  1. I found pictures of 5 different animals and resized the pictures in Photoshop to approximately 200×200 pixels to make the animals smaller.
  2. I cut thin slips of paper in 5 different colors – one for each animal. I then printed out the RGB values of these colors from the web cam so I could code it for each of the animal. Once the sketch is run, it displays the image from the computer’s web cam. When there is one or several of the five colors present in the range of the web cam, the respective animal shows up on the screen following the object of the color they are assigned to. If this color is not present, then the animal doesn’t show up either. If a person is holding a colorful slip of paper, he or she can then move it around the screen to make the animal follow it and therefore control its motion with his or her hand.
  3. One of the challenges was determining the right threshold value that is the difference between the pixels from the camera and the coded color. In my case this difference has to be very small in order for the animal to show up, otherwise it can get confused and start showing the animal where it is not supposed to show up. However, that also means that if the lighting changes significantly, the RGB value of the color of the slips of paper appearing on the web cam might also change and the animal might not appear.

Here are pictures of the animals that show up on the screen depending of the color:

 

Here is a video of just one animal moving around the screen:

cat video

 

Here is a video of all of the animals that can appear:

all animals

 

Here is the code:

import processing.video.*;
Capture video;
color trackColor;
int locXpuppy, locYpuppy, locXtiger, locYtiger, locXara, locYara, locXbunny, locYbunny, locXcat, locYcat;
PImage puppy;
PImage tiger;
PImage ara;
PImage bunny;
PImage cat;
color doggy=color(255.0, 98.0, 203.0);
color tiger2=color(118.0, 188.0, 91.0);
color ara2=color(116.0, 196.0, 249.0);
color bunny2=color(232.0, 86.0, 103.0);
color cat2=color(255.0, 253.0, 112.0);
boolean drawPuppy=false;
boolean drawTiger=false;
boolean drawAra=false;
boolean drawBunny=false;
boolean drawCat=false;

void setup() {
 size(640, 480);
 video = new Capture(this, 640, 480, 30);
 video.start();
 puppy = loadImage("puppyx.png");
 tiger = loadImage("tigerx.png");
 ara = loadImage("araax.gif");
 bunny = loadImage("bunnyx.png");
 cat = loadImage("catttx.png");
 trackColor=doggy;
}

void draw() {
 if (video.available()) {
 video.read();
 }
 video.loadPixels();
 float dist=20;
 drawPuppy=false;
 drawTiger=false;
 drawAra=false;
 drawBunny=false;
 drawCat=false;
 for (int y=0; y<height; y++) {
 for (int x=0; x<width; x++) {
 int loc = (video.width-x-1)+(y*width);
 color pix=video.pixels[loc];
 float r1=red(pix);
 float g1=green(pix);
 float b1=blue(pix);
 
 float r2=red(trackColor);
 float g2=green(trackColor);
 float b2=blue(trackColor);
 
 float r3=red(tiger2);
 float g3=green(tiger2);
 float b3=blue(tiger2);
 
 float r4=red(ara2);
 float g4=green(ara2);
 float b4=blue(ara2);
 
 float r5=red(bunny2);
 float g5=green(bunny2);
 float b5=blue(bunny2);
 
 float r6=red(cat2);
 float g6=green(cat2);
 float b6=blue(cat2);
 
 
 float diff=dist(r1,g1,b1,r2,g2,b2);
 float diff2=dist(r1,g1,b1,r3,g3,b3);
 float diff3=dist(r1,g1,b1,r4,g4,b4);
 float diff4=dist(r1,g1,b1,r5,g5,b5);
 float diff5=dist(r1,g1,b1,r6,g6,b6);

 
 if (diff<dist){
 drawPuppy=true;
 dist=diff;
 locXpuppy=x;
 locYpuppy=y;
 } 
 
 if (diff2<dist){
 drawTiger=true;
 dist=diff2;
 locXtiger=x;
 locYtiger=y;
 } 
 
 if (diff3<dist){
 drawAra=true;
 dist=diff3;
 locXara=x;
 locYara=y;
 } 
 
 if (diff4<dist){
 drawBunny=true;
 dist=diff4;
 locXbunny=x;
 locYbunny=y;
 } 
 
 if (diff5<dist){
 drawCat=true;
 dist=diff5;
 locXcat=x;
 locYcat=y;
 } 

 }
 }
 video.updatePixels();
 pushMatrix();
 translate(width,0);
 scale(-1,1);
 image(video,0,0);
 popMatrix();
 if(drawPuppy){
 image(puppy, locXpuppy-100, locYpuppy-200);
 }
 if(drawTiger){
 image(tiger, locXtiger-100, locYtiger-200);
 }
 if(drawAra){
 image(ara, locXara-100, locYara-200);
 }
 if(drawBunny){
 image(bunny, locXbunny-100, locYbunny-200);
 }
 if(drawCat){
 image(cat, locXcat-100, locYcat-200);
 }
}

void mousePressed(){
 int loc=(video.width-mouseX-1)+(mouseY*width);
 trackColor=video.pixels[loc];
 println(red(trackColor)+" "+green(trackColor)+" "+blue(trackColor));
}

 

P.S. Because at first the code for creating what I just described didn’t properly work, I started working on a slightly different idea. Even though Aaron helped me fixing the code above (thanks for that!!), I decided to also include the other code. The idea behind it is that there are also 5 animals uploaded to the sketch, however instead of following precoded colors, the color of interest can be adjusted by a mouse press. Once you press the mouse on a, for example, pink object, it will then follow the pink object.  Also, there can only be one animal present at a time, but they can be changed by pressing the key “c”. There is a random function that randomly displays one of the images of the 5 animals.

 

This is the code for the other example:

import processing.video.*;
Capture video;
color trackColor;
int locX, locY;
PImage puppy;
PImage tiger;
PImage ara;
PImage bunny;
PImage cat;
int randomNumber;

void setup() {
 size(640, 480);
 video = new Capture(this, 640, 480, 30);
 video.start();
 puppy = loadImage("puppyx.png");
 tiger = loadImage("tigerx.png");
 ara = loadImage("araax.gif");
 bunny = loadImage("bunnyx.png");
 cat = loadImage("catttx.png");
}

void draw() {
 if (video.available()) {
 video.read();
 }
 video.loadPixels();
 float dist=500;
 for (int y=0; y<height; y++) {
 for (int x=0; x<width; x++) {
 int loc = (video.width-x-1)+(y*width);
 color pix=video.pixels[loc];
 float r1=red(pix);
 float g1=green(pix);
 float b1=blue(pix);
 
 float r2=red(trackColor);
 float g2=green(trackColor);
 float b2=blue(trackColor);
 
 float diff=dist(r1,g1,b1,r2,g2,b2);

 
 if (diff<dist){
 dist=diff;
 locX=x;
 locY=y;
 }
 }
 }
 video.updatePixels();
 pushMatrix();
 translate(width,0);
 scale(-1,1);
 image(video,0,0);
 popMatrix();
 if(randomNumber == 0){
 image(puppy, locX-100, locY-200);
 }
 else if(randomNumber == 1){
 image(tiger, locX-100, locY-200);
 }
 else if(randomNumber == 2){
 image(ara, locX-100, locY-200);
 }
 else if(randomNumber == 3){
 image(bunny, locX-100, locY-200);
 }
 else if(randomNumber == 4){
 image(cat, locX-100, locY-200);
 }
}

void mousePressed(){
 int loc=(video.width-mouseX-1)+(mouseY*width);
 trackColor=video.pixels[loc];
}

void keyPressed(){
 if (key=='c'){
 randomNumber = int(random(0, 5));
 }
}

 

Here is a video of an animal following my selected color by a mouse press in the second version:

animals second version