Totoro and Friend

This has definitely been the assignment I have enjoyed the most this semester. As I wrote on my blogpost earlier, I find computer vision as a form of “magic” (excuse my cheesiness), as it allows for an endless array of possibilities of what could happen on screen. In this same line of thought, for this project I decided to create something that I had always wanted to do.

While brainstorming on the approach I wanted to take for this project, I decided to distract myself by watching a video called “Hayao Miyazaki – The Essence of Humanity” for Communications Lab. As a big fan of Miyazaki’s movies and his iconic and adorable characters, the video inspired me to make a computer vision project that would allow users to interact with the character of Totoro. As perhaps some of my classmates have noted by my computer cover, I am a big fan of Totoro, and making this project would serve as a means of extending this love into the screen. For those who are not familiar with Totoro, here is a glimpse of his adorableness in gifs, which is something I wanted to capture through this project:

Here is a  link to the video of the final project:

https://drive.google.com/a/nyu.edu/file/d/0Bwr6pFsy04OIR3R4azFYVFRYMjQ/view?usp=sharing

And some gifs as a glimpse: 

The overall point of this project is to reunite both Totoros by placing mini Totoro on top of Totoro’s head. The farther away mini-Totoro is from the larger one’s head, the sadder the big Totoro gets. Also, depending on the location of the mini Totoro, the larger one’s eyes follow him as well. However, if mini-totoro is placed on top of Totoro’s body, Totoro won’t be able to see it and will get sad again. This is a rather simple logic, but with the incorporation of animations and Totoro’s adorableness, this project has really made me (and a lot of my friends whom I have shared this code with) happy.

The movement of Totoro’s pupils were done by mapping two ellipses according to the X and Y location of the mini-totoro into Totoro’s eyes.

In order to make the animations, I obtained a transparent .gif file of Totoro’s change from a neutral face to a smile. I separated each of the frames into different png files and adjusted them to a larger image of Totoro.

Possible Improvements and Extension:

As I stated beforehand, this is one of the projects I have enjoyed doing the most. Therefore, I am seriously considering making an extension of this project for my final Intro to IM assignment. Other aspects I could add to these project are the use of different objects with different colors that allow for various interactions with Totoro. Through the projection of Totoro on a wall and the use of a web cam, I could make people fully interact with a real-size image of Totoro. An example of more interactions could be using a different colored glove to rub Totoro on his belly and make him make different growling noises. This is only one of the many ways I could approach this computer vision project, which is why I want to consider an extension of this assignment for my IM final.

Here is the code:

import processing.video.*;

PImage photo;
PImage frame1;
PImage frame2;
PImage frame3;
PImage frame4;
PImage frame5;
PImage frame6;
PImage frame7;
PImage frame8;

Capture video;
color trackColor; 
float locX, locY;
float totoroXleft, totoroYleft;
float totoroXright, totoroYright;


void setup(){
 size(1280,800);
 video = new Capture(this, 1280,800,30);
 video.start();
 photo = loadImage("totoroneutral.png");
 frame1 = loadImage("frame1.png");
 frame2 = loadImage("frame2.png");
 frame3 = loadImage("frame3.png");
 frame4 = loadImage("frame4.png");
 frame5 = loadImage("frame5.png");
 frame6 = loadImage("frame6.png");
 frame7 = loadImage("frame7.png");
 frame8 = loadImage("frame8.png");
}

void draw(){
 if (video.available()) {
 video.read();
 }
 video.loadPixels();
 float dist=500;
 for (int y=0; y<height; y++) {
 for (int x=0; x<width; x++) {
 int loc = (video.width-x-1)+(y*width);
 color pix=video.pixels[loc];
 float r1=red(pix);
 float g1=green(pix);
 float b1=blue(pix);
 
 float r2=red(trackColor);
 float g2=green(trackColor);
 float b2=blue(trackColor);
 float diff=dist(r1,g1,b1,r2,g2,b2);
 
 //wherever the target color is:
 if (diff<dist){
 dist=diff;
 locX=x;
 locY=y;
 }
 
 totoroXleft = map(locX,0,1280,470,542);
 totoroYleft = map(locY,0,800,369,414);
 
 totoroXright = map(locX,0,1280,815,880);
 totoroYright = map(locY,0,800,366,430);
 
 
 
 }
 }
 video.updatePixels();
 pushMatrix();
 translate(width,0);
 scale(-1,1);
 image(video,0,0);
 popMatrix();
 
 
 //changing the animation frames
 if (0 <= locX && locX <= 203 && 650 <= locY && locY <= 1280 || 1104 <= locX && locX <= 1280 && 652 <= locY && locY <= 800){
 image(frame1, 70, -10);
 }
 else if (0 <= locX && locX <= 290 && 561 <= locY && locY <= 649 || 1036 <= locX && locX <= 1280 && 562 <= locY && locY <= 651){
 image(frame2, 70, -10);
 }
 else if (0 <= locX && locX <= 349 && 429 <= locY && locY <= 560|| 964 <= locX && locX <= 1280 && 431 <= locY && locY <= 561){
 image(frame3, 70, -10);
 }
 else if (0 <= locX && locX <= 418 && 324 <= locY && locY <= 428|| 904 <= locX && locX <= 1280 && 328 <= locY && locY <= 430){
 image(frame4, 70, -10);
 }
 else if (0 <= locX && locX <= 475 && 198 <= locY && locY <= 323|| 856 <= locX && locX <= 1280 && 204 <= locY && locY <= 429){
 image(frame5, 70, -10);
 }
 else if (201 <= locX && locX <= 312 && 0 <= locY && locY <= 197|| 1025 <= locX && locX <= 1140 && 0 <= locY && locY <= 203){
 image(frame6, 70, -10);
 }
 else if (312 <= locX && locX <= 579 && 0 <= locY && locY <= 197|| 871 <= locX && locX <= 1024 && 0 <= locY && locY <= 203){
 image(frame7, 70, -10);
 }
 else if(580 <= locX && locX <= 889 && 0<= locY && locY <= 239){
 image(frame8, 70, -10);
 }
 else{
 image(frame1, 70, -10);
 }
 
 //Totoro's eyes 
 fill(0);
 ellipse(totoroXleft,totoroYleft,30,30);
 fill(0);
 ellipse(totoroXright,totoroYright,30,30);
}

//Getting a tracking value
void mousePressed(){
 int loc=(video.width-mouseX-1)+(mouseY*width);
 trackColor=video.pixels[loc];
}