Final Project: “Light is Like Water”, an Interactive Diorama

I must start off by admitting that time, though endless (as my high school Calculus professor used to say, “there’s more time than life”), is often insufficient. That was my experience these past couple of weeks. There is so much I wanted to do for this project that I couldn’t implement not because of technical difficulties, but because of time.

Thus, my biggest takeaway is this: a project that one is excited about could go on forever. I was thrilled to carry out my ideas for this final assignment, because I’m fascinated by the story that inspired it. It was fruitful in the end: I’m proud of what I made. But I could have continued to work on it more, adding more features and fixing others, and refining the “craftsmanship.” I’m glad this is the case, though. It means that this project motivated and inspired me, in a way few projects throughout the semester had.

Inspiration

The story on which I based my work is titled “La luz es como el agua,” or “Light is like water” in English, written by world-famous Colombian author Gabriel García Márquez in 1978. I learned about the text from a friend who read it in high school, and purchased the book where this short story is featured (Doce cuentos peregrinos, or Twelve Pilgrims Stories) last summer.

Throughout the semester, I wanted to work with a track from Pirates of the Caribbean for my final project. However, having used it for one of our weekly assignments, I began to consider other possibilities, and when I remembered García Márquez’s story, it made perfect sense to use it.

This link contains two edited (abridged) versions of the story: one in English, translated by myself, and the other in Spanish. The story was shortened specifically for this project, but the complete text can easily be found online in both Spanish and English.

What I love about García Márquez’s writing is its richness in imagery. His descriptions very easily make the stories come to life for the reader, and thus (it seems to me) there’s a lot to work with if one wishes to depict his narrations.

Two aspects of this story made it particularly adequate for an interactive media project. Firstly, the text deals with electricity: it tells the story of two brothers who cause light (electricity) to “flow” like water, ending on a tragic note. I’ve been interested in working with neopixels ever since our “Stupid Pet Trick” assignment, and thought that they could be used in this project to literally show light around a house, and to make it appear as water.

Secondly, the story allows for interactivity in a fantastic way. The title of the text comes from the narrator’s confession that he once told the two brothers that light is like water.  Thus, the narrator, who tells most of the story in third person, reveals to the reader his role and direct impact on the events that unfold. I wanted the user to be directly implicated in the story’s events as well, having them “cause” said events.

Process, USER TESTING, & Improvements

I was quite lousy regarding the documentation of this project. I took no photographs of the process, the user testing stage, or of its exhibition during the Interactive Media Spring Showcase.

As the following images (taken after the project was exhibited) show, the “main” component of the project is a house I built mostly out of cardboard. The house can be divided into two sections: the top level (or the fifth floor, according to the story) and the bottom level (or the first floor, in my interpretation a basement). The top level contains the setting of the story, plagued with LEDs, and the roof of the building, which has two servo motors hidden inside it. The bottom level is full of wires that connect the top level’s components to power, as well as to Procesing and Arduino through a RedBoard.

Left: View of the complete house. Center: Bottom level (LEDs and wires). Right: Top level (decorative elements, LEDs, and servo motors).

I now include a video of the final project, to serve as the frame of reference for the explanation of the process. This video shows the interaction in English.

I mentioned that the video’s interaction is in English because, as the starting page of the Processing sketch shows, there is also a version of the project in Spanish, which the user can opt for. To me, it was important to include a version of the experience guided by the original story, given that I could never accomplish an accurate imitation of García Marquez’s style in a translation. Because I’m such a big fan of his writing, I wanted his words to be available to Spanish-speaking users.

On a related note, I asked a fellow classmate to help me with the project because I imagined that his voice, specifically, would make the narration of the story much richer than if I had done it myself. Not only is he a great speaker, but he is also Colombian; thus, I thought, it becomes easier to imagine García Márquez himself reciting the text. Perfect casting (thank you, Sebastián!).

In terms of the structure, the bottom level was made by cutting a cardboard box and covering it with black adhesive material (I’m not sure whether to call it paper, plastic…). I cut a small rectangle on one of its sides to let the cables of the RedBoard, neopixel strips, and a small light out, so that I could connect them all to an external power source.

The aforementioned small light was used to illuminate the four wires that the user has to connect. I chose to use a breadboard and breadboard wires for the user interaction for a couple of reasons. In the first place, because the story deals with electric circuits, I wanted the user to have the experience of messing with the house’s actual circuits. I originally left the entire “basement” open, such that the user could easily see not only the four wires they had to manipulate, but also the ones that they didn’t have to use (the ones connected to the RedBoard and a second breadboard). I added the four LEDs that are associated to these wires so that they could act not only as indicators for my own code (of whether or not the wires are connected), but also as indicators for the user.

However, during user testing, my first user expressed that it was confusing to know which wires to connect and disconnect, given that there were so many. To the user, it wasn’t clear what was expected of them. Following his advice, I added a piece of transparent acrylic (hence, it still allows some visibility) to completely separate the four wire-LED pairs and the rest of the circuit.

I also incorporated written instructions in the Processing sketch, right before the narration begins. In them, I tell the user that they must pay attention to the narration (audio) and both the screen and the house (visual). In this way, they know they must be aware of all these components throughout their experience.

The same user also suggested not having written instructions at all, making the computer screen go black after the title clip, with the instructions transmitted through audio. He thought that this would make the experience with the house more immersive, and to separate the narration from the instructions, I could read the latter out loud myself. I recorded these instructions, and was willing to make these changes. But… time. This is definitely an improvement that I would have liked to try, even though I did have one preoccupation: what if the user didn’t understand the instructions right away? Another user who tested the project was slightly confused at the beginning as to where the wires should be connected, but she figured it out after reading the instructions a second time. I think the solution would have been to loop the audio instructions as long as the required task has not been completed.

Another advantage of using a breadboard as the interface is that, by covering up most of the board with tape and leaving one of the positive rails uncovered, I ask the user to connect the wires “along the red line” and don’t have to worry about where exactly they’ll connect them, or in which order, given that all the openings in the rail act the same.

Regarding decorative elements, the bottom level has a large number 47, in reference to the story’s building number, and a set of stairs and floors on one of its sides. I did this because the story mentions that the brothers and their parents live in the fifth floor of their building. Therefore, there’s the bottom level, three floors in between, and the top level, all connected by stairs.

These decorations (as well as the ones in the top level) were very successful during user testing. People appreciate these small details, even if the information of the building is provided until the very end of the narration.

Stairs and floors and stairs and floors and stairs and floors and stairs and floors…

The top level was more complex. I built the box myself, because it needed double walls. The neopixels strips that simulate the water-electricity are glued to the outer walls and the inner walls are covered with translucent paper that allows the user to see the light of the neopixels, diffusing it a bit as well. The same was done with the ceiling.

There are two small openings in the ceiling that go through the cardboard and the paper. I cut a straw to get two small pieces that I glued to these openings. Each of the two servo motors has a wire tied to its arm (which has openings itself, facilitating the process of tying the wire), which moves up and down through the straw when the servo rotates. This mechanism allows the up and down movement of the boat.

The decorative elements inside the top level were printed out on paper and made sturdy by glueing them to cardboard and wooden sticks that go through the cardboard floor. For the four lamps, wires attached to yellow LEDs also go through the cardboard, so that each of the lamps turns on and off in response to the user’s actions. Additionally, there are two other pieces of furniture inside the house: a grand piano and a bar with a wine bottle. These are also referenced in the story.

I decided to make the “flooding” neopixels blue until the very end of the story, when they become yellow and “end” the metaphor of light as water. If one googles this story, the image results mostly show yellow waves and currents, and in the story, the children’s adventures are very explicitly described as occuring in the light, and not water (though water-related terms are constantly used in the text). The advantage of using actual light to depict García Márquez’s water-electricity is that no matter its color, the light is still (actual, physical) light. Thus, in my opinion, the metaphor becomes stronger with lights in blue, like water. I also made them randomly change to different shades of blue with every loop in the Arduino code, to resemble shimmering water.

This is a compilation of Google Images results for “la luz es como el agua”; all of them show the water colored yellow, or alternatively, the light shaped as waves.

The following show the code that was used in this assignment.

Arduino:

#include <Adafruit_NeoPixel.h>
#ifdef __AVR__
 #include <avr/power.h>
#endif

#include <Servo.h>

Servo boatRight;
Servo boatLeft;

const int ledGreen = 10;
const int ledRed = 11;
const int ledBlue = 2;
const int ledYellow = 13;

const int tallLeft= 8;
const int shortLeft = 7;
const int tallRight = 4;
const int shortRight = 3;

const int PIN = 6;

const int NUMPIXELS = 180;

int stage;

int serialComm;

const int colors[] = {47, 86, 233, 45, 100, 245, 47, 141, 255, 51, 171, 249,
52, 204, 255, 82, 219, 255};
const int colorsFire[] = {255, 127, 0, 255, 143, 0, 255, 105, 0, 229, 83, 0};

Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRB + NEO_KHZ800);

void setup() {
 pinMode(ledGreen, INPUT);
 pinMode(ledRed, INPUT);
 pinMode(ledBlue, INPUT);
 pinMode(ledYellow, INPUT);
 pinMode(tallLeft, OUTPUT);
 pinMode(shortLeft, OUTPUT);
 pinMode(tallRight, OUTPUT);
 pinMode(shortRight, OUTPUT);

boatRight.attach(9); 
 boatLeft.attach(5);

setupStuff();

Serial.begin(9600);
 Serial.println("100");
}

void setupStuff(){
 boatRight.write(180); 
 boatLeft.write(0);

pixels.begin();
 pixels.setBrightness(10);
 pixels.clear();
 pixels.show();
 
 stage = -1;

serialComm = 0;
}

void loop() {
 int green = digitalRead(ledGreen);
 int red = digitalRead(ledRed);
 int blue = digitalRead(ledBlue);
 int yellow = digitalRead(ledYellow);
 if(stage == 0){
 if(green == LOW){
 digitalWrite(tallLeft, LOW);
 }
 else if(green == HIGH){
 digitalWrite(tallLeft, HIGH);
 };
 if(red == LOW){
 digitalWrite(shortRight, LOW);
 }
 else if(red == HIGH){
 digitalWrite(shortRight, HIGH);
 };
 if(blue == LOW){
 digitalWrite(tallRight, LOW);
 }
 else if(blue == HIGH){
 digitalWrite(tallRight, HIGH);
 };
 if(yellow == LOW){
 digitalWrite(shortLeft, LOW);
 }
 else if(yellow == HIGH){
 digitalWrite(shortLeft, HIGH);
 };
 if(green == HIGH && red == HIGH && blue == HIGH && yellow == HIGH){
 serialComm = 1;
 boatRight.write(120); 
 boatLeft.write(50); 
 };
 }
 else if(stage == 1){
 if(green == LOW){
 serialComm = 2;
 digitalWrite(tallLeft, LOW);
 for(int i=0; i<9; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=76; i<88; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=115; i<124; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 pixels.show();
 for (int pos = 50; pos <= 120; pos ++) {
 boatLeft.write(pos); 
 boatRight.write(120 - pos);
 delay(15);
 }
 for (int pos = 120; pos <= 50; pos --) {
 boatRight.write(pos - 120);
 boatLeft.write(pos); 
 delay(15);
 }
 };
 }
 else if(stage == 2){
 if(red == LOW){
 serialComm = 3;
 digitalWrite(shortRight, LOW);
 for(int i=0; i<18; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=63; i<88; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=106; i<124; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 pixels.show();
 for (int pos = 50; pos <= 140; pos ++) {
 boatLeft.write(pos); 
 boatRight.write(130 - pos);
 delay(15);
 }
 for (int pos = 140; pos <= 50; pos --) {
 boatRight.write(pos - 130);
 boatLeft.write(pos); 
 delay(15);
 }
 };
 }
else if(stage == 3){
 if(blue == LOW){
 serialComm = 4;
 digitalWrite(tallRight, LOW);
 for(int i=0; i<27; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=49; i<88; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 pixels.show();
 };
 for(int i=97; i<124; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 pixels.show();
 for (int pos = 50; pos <= 160; pos ++) {
 boatLeft.write(pos); 
 boatRight.write(160 - pos);
 delay(15);
 }
 for (int pos = 160; pos <= 50; pos --) {
 boatRight.write(pos - 160);
 boatLeft.write(pos);
 delay(15);
 }
 };
 }
else if(stage == 4){
 if(yellow == LOW){
 serialComm = 5;
 digitalWrite(shortLeft, LOW);
 for(int i=0; i<36; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=38; i<88; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 for(int i=88; i<181; i++){
 int index = int(random(0,6))*3;
 int R = colors[index];
 int G = colors[index + 1];
 int B = colors[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 pixels.show();
 for (int pos = 50; pos <= 180; pos ++) {
 boatLeft.write(pos); 
 boatRight.write(180 - pos);
 delay(15);
 }
 for (int pos = 180; pos <= 50; pos --) {
 boatRight.write(pos - 180);
 boatLeft.write(pos);
 delay(15);
 }
 };
 }
 else if(stage == 5){
 serialComm = 6;
 for(int i=0; i<181; i++){
 int index = int(random(0,4))*3;
 int R = colorsFire[index];
 int G = colorsFire[index + 1];
 int B = colorsFire[index + 2];
 pixels.setPixelColor(i, R, G, B);
 };
 pixels.show();
 boatRight.write(180); 
 boatLeft.write(0);
 }
 else if(stage == 6){
 setupStuff();
 };
 if(Serial.available() > 0){
 
 stage = Serial.read();
 Serial.println(serialComm);
 };
}

Processing:

import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
import ddf.minim.signals.*;
import ddf.minim.spi.*;
import ddf.minim.ugens.*;

Minim minim;
import processing.sound.*;

import processing.serial.*;
Serial myPort;

SoundFile beginSound;
AudioPlayer iESP, iiESP, iiiESP, ivESP, vESP, viESP, viiESP;
AudioPlayer iENG, iiENG, iiiENG, ivENG, vENG, viENG, viiENG;

PFont font1, font2;
boolean startESP, startENG;
int colorChange;
int colorChangerESP, colorChangerESP1, colorChangerESP2, colorChangerESP3;
int colorChangerENG, colorChangerENG1, colorChangerENG2, colorChangerENG3;
String intro1, intro2, intro3, intro4, intro5, intro6, intro7;
int alphaCounter1, alphaCounter2;
boolean alpha;
boolean begin, narration, story, end, restart;
int track;
int bgColor;
String text1, subtext1, text2, text3, text4, text5;
String og, tr, na;
int serialComm;
int stage;

void menu(){
 textAlign(LEFT);
 if(mouseX >= width/10*2.5 && mouseX <= width/10*4.5 
 && mouseY >= height/9*4 && mouseY <= height/9*5){
 if(mousePressed){
 startESP = true;
 }
 else{
 if(colorChangerESP < colorChange){
 colorChangerESP = colorChange;
 }
 if(colorChangerESP1 < colorChange - 30){
 colorChangerESP1 = colorChange - 30;
 }
 if(colorChangerESP2 < colorChange - 60){
 colorChangerESP2 = colorChange - 60;
 }
 if(colorChangerESP3 < colorChange - 90){
 colorChangerESP3 = colorChange - 90;
 }
 if(colorChangerESP3 < 255){
 colorChangerESP+=10;
 colorChangerESP1+=10;
 colorChangerESP2+=10;
 colorChangerESP3+=10;
 }
 };
 }
 else{
 colorChangerESP-=10;
 colorChangerESP1-=10; 
 colorChangerESP2-=10;
 colorChangerESP3-=10;
 };
 if(mouseX >= width/10*5.5 && mouseX <= width/10*7.5 
 && mouseY >= height/9*4 && mouseY <= height/9*5){
 if(mousePressed){
 startENG = true;
 }
 else{
 if(colorChangerENG < colorChange){
 colorChangerENG = colorChange;
 }
 if(colorChangerENG1 < colorChange - 30){
 colorChangerENG1 = colorChange - 30;
 }
 if(colorChangerENG2 < colorChange - 60){
 colorChangerENG2 = colorChange - 60;
 }
 if(colorChangerENG3 < colorChange - 90){
 colorChangerENG3 = colorChange - 90;
 }
 if(colorChangerENG3 < 255){
 colorChangerENG+=10;
 colorChangerENG1+=10; 
 colorChangerENG2+=10;
 colorChangerENG3+=10;
 }
 };
 }
 else{
 colorChangerENG-=10;
 colorChangerENG1-=10; 
 colorChangerENG2-=10;
 colorChangerENG3-=10;
 }
 colorMode(HSB);
 noFill();
 strokeWeight(4);
 stroke(35, 100, colorChangerESP1);
 rect(width/10*2.5, height/9*4, width/10*2, height/9);
 stroke(140, 100, colorChangerENG1);
 rect(width/10*5.5, height/9*4, width/10*2, height/9);
 strokeWeight(3);
 stroke(35, 100, colorChangerESP2);
 rect(width/10*2.5 - 10, height/9*4 - 10, width/10*2 + 20, height/9 + 20);
 stroke(140, 100, colorChangerENG2);
 rect(width/10*5.5 - 10, height/9*4 - 10, width/10*2 + 20, height/9 + 20);
 strokeWeight(2); 
 stroke(35, 100, colorChangerESP3);
 rect(width/10*2.5 - 17.5, height/9*4 - 17.5, width/10*2 + 35, height/9 + 35);
 stroke(140, 100, colorChangerENG3);
 rect(width/10*5.5 - 17.5, height/9*4 - 17.5, width/10*2 + 35, height/9 + 35);
 fill(30, 100, colorChangerESP);
 text("español", width/10*3, height/9*4.65);
 fill(140, 100, colorChangerENG);
 text("english", width/10*6, height/9*4.65);
}

void begin(){
 if(startESP){
 intro1 = "La luz";
 intro2 = " es como el agua:";
 intro3 = "uno abre el grifo,";
 intro4 = " y sale.";
 intro5 = "Esta es una experiencia interactiva audiovisual.";
 intro6 = "La historia será transmitida por audio, las instrucciones aparecerán en la pantalla, y la casa cobrará vida."; 
 intro7 = "Haz click para continuar.";
 bgColor = 140;
 }
 else if(startENG){
 intro1 = "Light";
 intro2 = "is like water:";
 intro3 = "one turns the tap,";
 intro4 = "and out it comes.";
 intro5 = "This is an audiovisual interactive experience.";
 intro6 = "The story will be transmitted by audio, the instructions will appear on screen, and the house will come to life.";
 intro7 = "Click to continue.";
 bgColor = 22;
 };
 if(begin == false && alphaCounter1 >= 20){
 beginSound.play();
 begin = true;
 };
 if(alpha == false){
 alphaCounter1++;
 if(alphaCounter1 > 65){
 alphaCounter2++;
 }
 };
 if(alphaCounter2 >= 240){
 fill(bgColor, 100, 255, alphaCounter2 - 240);
 rect(0, 0, width, height);
 if(mousePressed){
 beginSound.stop();
 narration = true;
 };
 };
 textSize(250);
 fill(0, alphaCounter1);
 text(intro1, width/10*2, height/9*3.5);
 textSize(100);
 fill(0, alphaCounter2);
 text(intro2, width/10*5, height/9*4);
 fill(0, alphaCounter1 - 150);
 text(intro3, width/10*2, height/9*5);
 textSize(170);
 fill(0, alphaCounter2 - 145);
 text(intro4, width/10*3, height/9*6.5);
 textSize(50);
 fill(0, alphaCounter2 - 200);
 text("- Gabriel García Márquez", width/10*6, height/9*8);
 fill(255);
 textFont(font1);
 text(intro5, width/50, height/15);
 text(intro6, width/50, height/15 + 30);
 text(intro7, width/50, height/15 + 60);
}

void story(){
 textAlign(CENTER);
 textFont(font1);
 fill(0);
 if(story == false){
 alphaCounter1 = -10;
 if(track == 1){
 if(startENG == true){
 iENG.play();
 }
 else if(startESP == true){
 iESP.play();
 };
 }
 else if(track == 2){
 if(startENG == true){
 iiENG.play();
 }
 else if(startESP == true){
 iiESP.play();
 };
 }
 else if(track == 3){
 if(startENG == true){
 iiiENG.play();
 }
 else if(startESP == true){
 iiiESP.play();
 };
 }
 else if(track == 4){
 if(startENG == true){
 ivENG.play();
 }
 else if(startESP == true){
 ivESP.play();
 };
 }
 else if(track == 5){
 if(startENG == true){
 vENG.play();
 }
 else if(startESP == true){
 vESP.play();
 };
 }
 else if(track == 6){
 if(startENG == true){
 viENG.play();
 }
 else if(startESP == true){
 viESP.play();
 };
 }
 else if(track == 7){
 if(startENG == true){
 viiENG.play();
 }
 else if(startESP == true){
 viiESP.play();
 };
 };
 story = true;
 }
 else{
 if(track == 1){
 if(startENG == true){
 text1 = "Connect all the cables to the positive rail";
 subtext1 = "(along the red line)";
 }
 else if(startESP == true){
 text1 = "Conecta todos los cables al bus positivo";
 subtext1 = "(a lo largo de la línea roja)";
 };
 if(! iENG.isPlaying() && ! iESP.isPlaying()){
 stage = 0;
 alphaCounter1++;
 fill(0, alphaCounter1);
 text(text1, width/2, height/2);
 text(subtext1, width/2, height/2 + 35);
 if(serialComm == 1){
 story = false;
 track++;
 };
 };
 }
 else if(track == 2){
 if(startENG == true){
 text2 = "Disconnect the green cable";
 }
 else if(startESP == true){
 text2 = "Desconecta el cable verde";
 };
 if(! iiENG.isPlaying() && ! iiESP.isPlaying()){
 stage = 1;
 alphaCounter1++;
 fill(0, alphaCounter1);
 text(text2, width/2, height/2);
 if(serialComm == 2){
 story = false;
 track++;
 };
 };
 }
 else if(track == 3){
 if(startENG == true){
 text3 = "Disconnect the red cable";
 }
 else if(startESP == true){
 text3 = "Desconecta el cable rojo";
 };
 if(! iiiENG.isPlaying() && ! iiiESP.isPlaying()){
 stage = 2;
 alphaCounter1++;
 fill(0, alphaCounter1);
 text(text3, width/2, height/2);
 if(serialComm == 3){
 story = false;
 track++;
 };
 };
 }
 else if(track == 4){
 if(startENG == true){
 text4 = "Disconnect the blue cable";
 }
 else if(startESP == true){
 text4 = "Desconecta el cable azul";
 };
 if(! ivENG.isPlaying() && ! ivESP.isPlaying()){
 stage = 3;
 alphaCounter1++;
 fill(0, alphaCounter1);
 text(text4, width/2, height/2);
 if(serialComm == 4){
 story = false;
 track++;
 };
 };
 }
 else if(track == 5){
 if(startENG == true){
 text5 = "Disconnect the yellow cable";
 }
 else if(startESP == true){
 text5 = "Desconecta el cable amarillo";
 };
 if(! vENG.isPlaying() && ! vESP.isPlaying()){
 stage = 4;
 alphaCounter1++;
 fill(0, alphaCounter1);
 text(text5, width/2, height/2);
 if(serialComm == 5){
 story = false;
 track++;
 };
 };
 }
 else if(track == 6){
 if(! viENG.isPlaying() && ! viESP.isPlaying()){
 stage = 5;
 if(serialComm == 6){
 story = false;
 track++;
 };
 };
 }
 else if(track == 7){
 if(! viiENG.isPlaying() && ! viiESP.isPlaying()){
 stage = 6;
 end = true;
 alphaCounter2 = -10;
 };
 };
 };
}

void credits(){
 fill(bgColor, 100, 255, alphaCounter2);
 alphaCounter2++;
 rect(0, 0, width, height);
 textAlign(CENTER);
 textFont(font2);
 fill(0, alphaCounter2);
 if(startENG == true){
 og = "Original story";
 tr = "Translation and editing";
 na = "Narration";
 }
 else if(startESP == true){
 og = "Cuento original";
 tr = "Edición";
 na = "Narración";
 };
 textSize(70);
 text(og, width/2, height*0.25);
 textSize(50);
 text("Gabriel García Márquez", width/2, height*0.32);
 textSize(70);
 text(tr, width/2, height*0.5);
 textSize(50);
 text("María Laura Mirabelli", width/2, height*0.57);
 textSize(70);
 text(na, width/2, height*0.75);
 textSize(50);
 text("Sebastián Rojas Cabal", width/2, height*0.82);
 if(startESP == true){
 intro5 = "Haz click para finalizar";
 }
 else if(startENG == true){
 intro5 = "Click to end";
 };
 textAlign(LEFT);
 fill(255);
 textFont(font1);
 text(intro5, width/50, height/15);
 if(mousePressed){
 restart = true;
 };
};

void setup(){
 String portName = Serial.list()[2];
 myPort = new Serial (this, portName, 9600);
 myPort.clear();
 myPort.bufferUntil('\n');
 
 fullScreen();
 font1 = createFont("Arvo-Bold.ttf", 30);
 font2 = createFont("Handycheera.otf", 70);
 
 setupStuff();
}

void setupStuff(){
 startESP = startENG = false;
 colorChange = 200;
 colorChangerESP = colorChangerESP1 = colorChangerESP2 = colorChangerESP3 = 0;
 colorChangerENG = colorChangerENG1 = colorChangerENG2 = colorChangerENG3 = 0;
 intro1 = intro2 = intro3 = intro4 = intro5 = intro6 = intro7 = "";
 alphaCounter1 = alphaCounter2 = -10;
 alpha = false;
 begin = narration = story = end = restart = false;
 bgColor = 0;
 track = 1;
 serialComm = 0;
 stage = -1;
 
 beginSound = new SoundFile(this, "beginSound.wav");
 minim = new Minim(this);
 iESP = minim.loadFile("I.wav");
 iiESP = minim.loadFile("II.wav");
 iiiESP = minim.loadFile("III.wav");
 ivESP = minim.loadFile("IV.wav");
 vESP = minim.loadFile("V.wav");
 viESP = minim.loadFile("VI.wav");
 viiESP = minim.loadFile("VII.wav");
 iENG = minim.loadFile("1.wav");
 iiENG = minim.loadFile("2.wav");
 iiiENG = minim.loadFile("3.wav");
 ivENG = minim.loadFile("4.wav");
 vENG = minim.loadFile("5.wav");
 viENG = minim.loadFile("6.wav");
 viiENG = minim.loadFile("7.wav");
}

void draw(){
 background(255);
 textFont(font2);
 if(startESP == false && startENG == false){
 narration = false;
 menu();
 }
 else{
 if(narration == false){
 begin();
 }
 else{
 if(end == false){
 story();
 }
 else{
 if(restart == true){
 setupStuff();
 };
 credits();
 };
 };
 };
}

void serialEvent(Serial myPort){
 String s = myPort.readStringUntil('\n');
 s = trim(s);
 //println(s);
 if(s != null){
 serialComm = int(s);
 };
 myPort.write(stage);
}

Bee Here, Bee Gone

(I know. Cheesy, cheesy title.)

My original idea for this assignment had to do with brightness and ghosts. I wanted to use my laptop’s camera to obtain the video’s brightness and set a brightness threshold. If the room wasn’t dark (above the threshold), text would appear to tell the user they should turn off the lights. Once the brightness was sufficiently low, the background would change (using background substraction) to an image of a ghost. Hopefully, I thought, it would cause jump scares but not heart attacks.

I started coding this but realized that I don’t like how dirty the background subtraction turns out when the video is dark, given that the user’s face is also affected. Thus, I changed the idea to a little ghost that would appear and follow the user.

Somewhere along the way, it occured to me that the ghost should be chasing something specific, and my plan changed completely. I imagined the user taking sweets from a candy bowl and a fly appearing on screen to chase the candy from the user’s hand. I googled “fly gif” and found a nice bee. Classic Google. So I decided to make a bee that chases flower pollen.

I made sure that the following happen in the program:

– The bee follows a green LED covered in pollen. The LED can only be covered in pollen after it’s “dipped” into a flower garden. To do this, I obtained the area in the video that corresponds to being inside the garden; if the LED (which the code tracks by color and brightness) is positioned within this area, an image of pollen (a teeny tiny mountain of pollen) appears over the LED and an animation of a bee flutters above it.

– The bee moves closer to the pollen if the LED remains steady (meaning, if the x-coordinate doesn’t vary too much between video frames). Otherwise, it hovers a bit higher. I think this makes the bee’s movements a bit more realistic.

– If one of the code’s global variables is changed (the boolean named “attraction” is changed to false), then the pollen disappears. This is useful if the video captures a bigger object (or a background) that is green, for it changes the interaction. Instead of having the bee follow the little green light, the user can shoo the bee away while it hovers over the green backdrop.

The following video shows how each version of the program can be used (“Bee Here” followed by “Bee Gone”):

(Music: “Seven Days A Week,” Austin Roberts)

And this is the Processing code. I took Aaron’s color tracking example and made some tweaks:

import processing.video.*;
Capture video;
PImage flower, pollen;
PImage bee0, bee1, bee2, bee3, bee4, bee5, bee6, bee7;
color trackColor;
int locX, locY, prevX;
boolean start;
int counter;
boolean attraction = true; //can be changed to false

void setup() {
 size(640, 480);
 video = new Capture(this, 640, 480, 30);
 flower = loadImage("flower.png");
 pollen = loadImage("pollen.png");
 bee0 = loadImage("bee0.png");
 bee1 = loadImage("bee1.png");
 bee2 = loadImage("bee2.png");
 bee3 = loadImage("bee3.png");
 bee4 = loadImage("bee4.png");
 bee5 = loadImage("bee5.png");
 bee6 = loadImage("bee6.png");
 bee7 = loadImage("bee7.png");
 video.start();
 start = false;
 trackColor = color(0, 255, 0);
 counter = 0;
}

void draw() {
 if (video.available()) {
 video.read();
 }
 video.loadPixels();
 float dist=500;
 for (int y=0; y<height; y++) {
 for (int x=0; x<width; x++) {
 int loc = (video.width-x-1)+(y*width);
 color pix=video.pixels[loc];
 float r1=red(pix);
 float g1=green(pix);
 float b1=blue(pix);
 float r2=red(trackColor);
 float g2=green(trackColor);
 float b2=blue(trackColor);
 float diff=dist(r1,g1,b1,r2,g2,b2);
 
 if (diff<dist){
 dist=diff;
 prevX = locX;
 locX=x;
 locY=y;
 }
 }
 }
 video.updatePixels();
 pushMatrix();
 translate(width,0);
 scale(-1,1);
 image(video,0,0);
 popMatrix();
 fill(trackColor);
 if((locX <= 25 && locY >= 370) || 
 (locX >= 25 && locX <= 135 && locY >= 320)||
 (locX >= 135 && locX <= 195 && locY >= 370) ||
 (locX >= 195 && locX <= 220 && locY >= 400) ||
 (locX >= 220 && locX <= 290 && locY >= 370) ||
 (locX >= 290 && locX <= 320 && locY >= 400) ||
 (locX >= 320 && locX <= 430 && locY >= 370) ||
 (locX >= 430 && locX <= 530 && locY >= 400) ||
 (locX >= 530 && locY >= 320)){
 start = true;
 };
 imageMode(CENTER);
 if(attraction == true){
 if(start == true){
 counter++;
 image(pollen, locX, locY);
 }
 }
 else{
 counter++;
 }
 if(counter > 20){
 int randomX = int(random(-10, 10));
 int randomY;
 if(prevX + 5 >= locX && prevX - 5 <= locX){
 randomY = int(random(30, 50));
 }
 else{
 randomY = int(random(80, 100));
 };
 if(counter%8 == 0){
 image(bee7, locX - randomX, locY - randomY);
 }
 else if(counter%7 == 0){
 image(bee6, locX - randomX, locY - randomY);
 }
 else if(counter%6 == 0){
 image(bee5, locX - randomX, locY - randomY);
 }
 else if(counter%5 == 0){
 image(bee4, locX - randomX, locY - randomY);
 }
 else if(counter%4 == 0){
 image(bee3, locX - randomX, locY - randomY);
 }
 else if(counter%3 == 0){
 image(bee2, locX - randomX, locY - randomY);
 }
 else if(counter%2 == 0){
 image(bee1, locX - randomX, locY - randomY);
 }
 else{
 image(bee0, locX - randomX, locY - randomY);
 }
 };
 imageMode(CORNER);
 scale(0.5);
 image(flower, -100, 600);
}

void mousePressed(){
 int loc = (video.width-mouseX-1)+(mouseY*width);
 color tracked = video.pixels[loc];
 float bright = brightness(tracked);
 if(bright > 200){
 trackColor = tracked;
 };
}

Computer Vision: A Control Freak’s Nightmare (But Also Their Dream)

(Response to “Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers” by Golan Levin)

Levin’s article about computer vision succeeds in describing the field through its many dimensions: he discusses its history, main obstacles, common (and useful) techniques, and popular uses (among other aspects). The text is thus informative, instructive, and a reflection of what computer vision is capable of now and its potential for the future.

Two of Levin’s points struck me the most. None of them were “news” (I was familiar with these facts beforehand), but it was their very obviousness (which, by the way, is a word… I had to check) that made me reflect on them. The first is that we are fascinated by the idea of surveillance. The second is that computer vision is, as Levin puts it, “computationally ‘opaque’… [it] contains no intrinsic semantic or symbolic information.”

I’ll begin with the latter. In developing my own computer vision project for this class, I couldn’t stop thinking about how complicated it is to control how the program works and the users’ experience with it. As Levin correctly explains, computer vision can only analyze pixel by pixel and then interpret the information of these individual units. In this sense, control is particularly challenging. However, because so little can be controlled by the basic algorithms we’re using (in comparison to coding only with text, as Levin states), the “outside” (what’s on screen) often has to be a very controlled environment – light levels, colors, motion, etc. There are numerous conditions that can greatly affect the functioning of the program, which the computer can´t deal with on its own.

In terms of surveillance, I wonder why the camera inspires so many works about this topic. It’s certainly related to our awareness of the fact that our degree of privacy is often an illusion (in this age, technology has evolved to track almost anyone). The camera has the ability to serve as a second pair of eyes, and “see” what we couldn’t see without it. We can act as those who carry out the surveillance, which puts us in a position of power, but we also realize that others can do the same to us, which is worrisome.

From “the Latin Word Putāre, Meaning ‘to Think’”

(Title taken from Merriam-Webster Dictionary’s definition for “compute”)

I shall begin this post in a very cliché fashion. Merriam-Webster Dictionary defines computing as “[determining] especially by mathematical means… [calculating] by means of a computer… [making] a calculation… [using] a computer.”

I grew up using computers. In fact, all throughout elementary school I had to take a class called Computation. However, what’s exciting about computing is not just “to use a computer,” as Merriam-Webster offers, but to be able to understand/manipulate/take advantage of the logic and technologies that allow computation. The satisfaction of building a prototype that works or writing code that runs is the same, at least to me, as the pleasure of solving a mathematical problem. It’s fascinating to create/discover/invent by assembling concepts/tools/pieces that already exist and making them act at your will – some work by the forces of nature (you go, Physics!) and some were made to work by others before you.

It’s a part-by-part logical process, but not necessarily step-by-step, for infinite routes can be taken to achieve the same goal (and the same step can be revisited multiple times, in different order, to fix or improve the computation).

It’s an international, timeless, collaborative endeavour.

I remember a quote from Steve Jobs that was cited on TV once (can’t remember who, when, where, why… but I can remember what), and it stuck with me. Now, I realize that it suits computing really well.

Creativity is just connecting things. When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things.

-Steve Jobs

Catch

When I presented my butterfly animation project in class, Aaron brought up the possibility of using such animations for a game, such that the character(s) moves in a realistic manner. Later that day, Nahil suggested I could use sprite sheets (which our dear W3 Schools defines as “a collection of images” that are meant to be “put into a single image”) the next time I wanted to try an animation, given that they can be easily found as png files with transparent backgrounds online and usually include enough frames to simulate sufficiently smooth movement.

I thought I could follow their advice for this assignment, given that it asked us to use one of our previous Processing projects and implement serial communication with Arduino. I used the butterfly animation code as the skeleton for this new program, but added several new features.

The game imitates “Flutter” in its simple name (#actionverbs, #yay!) and the random character and background selection. The user can either control a zebra, a tiger, or a reindeer.

Zebra sprite sheet

The user has 30 seconds to catch as many yellow circles as they can (there is only one circle on screen at a time; if the user catches it, a new one appears). These circles get smaller as time runs out, thus decreasing the circles’ target radius (the area where the user “touches” the circle) and making the game a bit more difficult.

The animal’s speed can be increased or decreased with the use of a potentiometer and its movement is controlled with four pushbuttons (if no button is pushed, the animal doesn’t move).

Circled in light blue: four pushbuttons that determine direction (left, down, up, right); in pink: potentiometer that determines speed; in yellow: red LED that turns on when the user has 10 seconds or less before the game ends

There are always three numbers on screen: the user’s score, the high score, and the timer. The user’s score keeps track of how many circles the animal has catched. The high score displays the all-time high score. It pulls it from a text file at the start of the game, and if the user surpasses this number, the new high score overrides the previous one in the text file. If this happens, the high score turns yellow and displays the user’s points. Lastly, the timer decreases from 30 to 0, and turns red at the 10-second mark. The LED on the breadboard turns on simultaneously, to warn the user about their limited time if they happen to be watching the board. I included this feature because I looked at the board a lot as I was testing the game. I think this is the problem of using pushbuttons as controllers: it’s very easy to think  you’re pressing a specific one when you’re actually pressing another.

Initially, I wanted to use a joystick to control movement, but the piece I found had surprisingly fragile parts and they came off fairly easily… so that worked for a while, and then everything fell apart (quite literally). I also thought of incorporating a yellow LED that would turn on when the user established a new high score and a piezo buzzer that would accompany each light with a different tune. However, I only had access to my small Sparkfun kit breadboard when I thought of these options, so these many components and their respective wires and resistors wouldn’t fit in it.

In terms of the animation itself, the main thing I want to fix is the position of the animal when it isn’t moving. Right now, the animal stops and faces forward, no matter what it’s previous movement was. I think this can easily be changed by having a variable that saves the movement in each draw loop (up or down or left or right), and if the animal stops, it retains the last movement that was saved, which would determine which images from the sprite sheet to use.

The following video shows three different rounds of the game that I recorded and put together (as well as its title screen):

(Music by YouTube user rakohus)

This is the code from the Processing sketch:

import processing.serial.*;
Serial myPort;

PImage bg;
PImage f1, f2, f3, f4;
PImage l1, l2, l3, l4;
PImage r1, r2, r3, r4;
PImage b1, b2, b3, b4;

PFont font;

Animal user;

int userChoice;
int speed;
int xDirection = 0;
int yDirection = 0;
int counter;
int time;
int points;

boolean object;
boolean ellipse;
boolean start;
int gameOver;

int highScore;

int objectX;
int objectY;
float radius;

void loadImages(int animal){
 if(animal == 1){ //tiger
 bg = loadImage("bg1.png");
 f1 = loadImage("tiger_front1.png");
 f2 = loadImage("tiger_front2.png");
 f3 = loadImage("tiger_front3.png");
 f4 = loadImage("tiger_front4.png");
 l1 = loadImage("tiger_left1.png");
 l2 = loadImage("tiger_left2.png");
 l3 = loadImage("tiger_left3.png");
 l4 = loadImage("tiger_left4.png");
 r1 = loadImage("tiger_right1.png");
 r2 = loadImage("tiger_right2.png");
 r3 = loadImage("tiger_right3.png");
 r4 = loadImage("tiger_right4.png");
 b1 = loadImage("tiger_back1.png");
 b2 = loadImage("tiger_back2.png");
 b3 = loadImage("tiger_back3.png");
 b4 = loadImage("tiger_back4.png");
 }
 else if(animal == 2){ //deer
 bg = loadImage("bg2.png");
 f1 = loadImage("deer_front1.png");
 f2 = loadImage("deer_front2.png");
 f3 = loadImage("deer_front3.png");
 f4 = loadImage("deer_front4.png");
 l1 = loadImage("deer_left1.png");
 l2 = loadImage("deer_left2.png");
 l3 = loadImage("deer_left3.png");
 l4 = loadImage("deer_left4.png");
 r1 = loadImage("deer_right1.png");
 r2 = loadImage("deer_right2.png");
 r3 = loadImage("deer_right3.png");
 r4 = loadImage("deer_right4.png");
 b1 = loadImage("deer_back1.png");
 b2 = loadImage("deer_back2.png");
 b3 = loadImage("deer_back3.png");
 b4 = loadImage("deer_back4.png");
 }
 else if(animal == 3){ //zebra
 bg = loadImage("bg3.png");
 f1 = loadImage("zebra_front1.png");
 f2 = loadImage("zebra_front2.png");
 f3 = loadImage("zebra_front3.png");
 f4 = loadImage("zebra_front4.png");
 l1 = loadImage("zebra_left1.png");
 l2 = loadImage("zebra_left2.png");
 l3 = loadImage("zebra_left3.png");
 l4 = loadImage("zebra_left4.png");
 r1 = loadImage("zebra_right1.png");
 r2 = loadImage("zebra_right2.png");
 r3 = loadImage("zebra_right3.png");
 r4 = loadImage("zebra_right4.png");
 b1 = loadImage("zebra_back1.png");
 b2 = loadImage("zebra_back2.png");
 b3 = loadImage("zebra_back3.png");
 b4 = loadImage("zebra_back4.png");
 };
};

void reset(){
 xDirection = 0;
 yDirection = 0;
 userChoice = int(random(1, 4));
 loadImages(userChoice);
 user = new Animal();
 speed = 15;
 time = 0;
 points = 0;
 counter = 0;
 object = false;
 start = false;
 gameOver = 0;
 ellipse = false;
 radius = 20;
};

void setup(){
 printArray(Serial.list());
 String portName = Serial.list()[2];
 myPort = new Serial (this, portName, 9600);
 myPort.clear();
 myPort.bufferUntil('\n');
 
 int [] txtFile = int(loadStrings("highscore.txt"));
 highScore = txtFile[0];
 
 size(600, 448);
 font = createFont("ARCADECLASSIC.TTF", 50);
 reset();
}

void draw(){
 textFont(font);
 fill(0);
 if(start == false){
 background(255);
 time = 0;
 textAlign(CENTER);
 text("Catch", width/2, height/2 - 60);
 textSize(20);
 text("POT is speed", width/2, height/2 + 10);
 fill(255, 0, 0);
 text("RED is left", width/2, height/2 + 30);
 fill(255, 211, 25);
 text("YELLOW is down", width/2, height/2 + 50);
 fill(34, 139, 34);
 text("GREEN is up", width/2, height/2 + 70);
 fill(0, 0, 255);
 text("BLUE is right", width/2, height/2 + 90);
 fill(0);
 text("Press any button to start", width/2, height/2 + 140);
 textSize(30);
 if(xDirection != 0 || yDirection != 0){
 start = true;
 object = true;
 }
 }
 else{
 imageMode(CORNER);
 background(bg);
 textSize(35);
 time = 30 - round(counter/80);
 if (radius > 10){
 radius = radius - 0.005;
 };
 if(object == true){
 objectX = int(random(40, width - 70));
 objectY = int(random(180, height - 70));
 object = false;
 };
 ellipseMode(CENTER);
 stroke(0);
 fill(255, 255, 0); 
 ellipse(objectX, objectY, radius, radius);
 if(time == -1){
 reset();
 }
 else if(time <= 10){
 gameOver = 1;
 fill(255, 0, 0);
 }
 else{
 fill(0);
 };
 textAlign(RIGHT);
 text(time, width - 40, 40);
 fill(0);
 textAlign(LEFT);
 text("Score "+points, 30, 40);
 textSize(20);
 if(points <= highScore){
 text("High Score "+highScore, 30, 60);
 }
 else{
 fill(255, 255, 0);
 text("High Score "+points, 30, 60);
 String [] high = {points+""};
 saveStrings("highscore.txt", high);
 };
 user.move();
 counter++;
 };
}

class Animal{
 int X;
 int Y;
 int FRAME;
 int HORIZONTAL;
 int VERTICAL;
 
 Animal(){
 X = width/2;
 Y = height - 40;
 FRAME = 1;
 HORIZONTAL = 0;
 VERTICAL = 0;
 }
 
 void move(){
 HORIZONTAL = xDirection;
 VERTICAL = yDirection;
 imageMode(CENTER);
 if(FRAME == 1){
 if(VERTICAL == -1){
 image(f1, X, Y);
 }
 else if(HORIZONTAL == -1){
 image(l1, X, Y);
 }
 else if(HORIZONTAL == 1){
 image(r1, X, Y);
 }
 else if (VERTICAL == 1){
 image(b1, X, Y);
 }
 else if(VERTICAL == 0 && HORIZONTAL == 0){
 image(f1, X, Y);
 };
 }
 else if(FRAME == 2){
 if(VERTICAL == -1){
 image(f2, X, Y);
 }
 else if(HORIZONTAL == -1){
 image(l2, X, Y);
 }
 else if(HORIZONTAL == 1){
 image(r2, X, Y);
 }
 else if (VERTICAL == 1){
 image(b2, X, Y);
 }
 else if(VERTICAL == 0 && HORIZONTAL == 0){
 image(f2, X, Y);
 };
 }
 else if(FRAME == 3){
 if(VERTICAL == -1){
 image(f3, X, Y);
 }
 else if(HORIZONTAL == -1){
 image(l3, X, Y);
 }
 else if(HORIZONTAL == 1){
 image(r3, X, Y);
 }
 else if (VERTICAL == 1){
 image(b3, X, Y);
 }
 else if(VERTICAL == 0 && HORIZONTAL == 0){
 image(f3, X, Y);
 };
 }
 else if(FRAME == 4){
 if(VERTICAL == -1){
 image(f4, X, Y);
 }
 else if(HORIZONTAL == -1){
 image(l4, X, Y);
 }
 else if(HORIZONTAL == 1){
 image(r4, X, Y);
 }
 else if (VERTICAL == 1){
 image(b4, X, Y);
 }
 else if(VERTICAL == 0 && HORIZONTAL == 0){
 image(f4, X, Y);
 };
 };
 
 if(counter%speed == 0){
 if((X < 0 && HORIZONTAL == -1) || (X > width - 70 && HORIZONTAL == 1)){
 X = X;
 }
 else{
 X = X + 10*HORIZONTAL;
 };
 if((Y < 190 && VERTICAL == 1) || (Y > height - 70 && VERTICAL == -1)){
 Y = Y;
 }
 else{
 Y = Y - 10*VERTICAL;
 };
 if(FRAME == 4){
 FRAME = 1;
 }
 else{
 FRAME++;
 };
 };
 
 if(object == false){
 if(X >= objectX - radius 
 && X <= objectX + radius
 && Y >= objectY - radius - 20
 && Y <= objectY + radius + 20){
 points++;
 object = true;
 };
 };
 }
}

void serialEvent(Serial myPort){
 String arduinoInfo = myPort.readStringUntil('\n');
 if(arduinoInfo != null){
 String [] infoString = arduinoInfo.split(",");
 xDirection = int(infoString[0]);
 yDirection = int(infoString[1]);
 String tempSpeed = infoString[2].trim();
 speed = int(tempSpeed);
 };
 myPort.write(gameOver);
}

And this is the Arduino code:

const int ledRed = 2;
const int leftPot = 12;
const int downPot = 11;
const int upPot = 10;
const int rightPot = 9;

void setup() {
 pinMode(ledRed, OUTPUT);
 pinMode(leftPot, INPUT);
 pinMode(downPot, INPUT);
 pinMode(upPot, INPUT);
 pinMode(rightPot, INPUT);
 Serial.begin(9600);
 Serial.println("0,0,1");
}

void loop() {
 int xDirection;
 int yDirection;
 int left = digitalRead(leftPot);
 int down = digitalRead(downPot);
 int up = digitalRead(upPot);
 int right = digitalRead(rightPot);
 int theSpeed = analogRead(A2);
 int mappedSpeed = map(theSpeed, 0, 1023, 15, 5);
 if(Serial.available() > 0){
 int input = Serial.read();
 if(input == 1){
 digitalWrite(ledRed, HIGH);
 }
 else{
 digitalWrite(ledRed, LOW);
 };
 if(down == HIGH){
 yDirection = -1;
 xDirection = 0;
 }
 if(up == HIGH){
 yDirection = 1;
 xDirection = 0;
 }
 if(left == HIGH){
 xDirection = -1;
 yDirection = 0;
 }
 if(right == HIGH){
 xDirection = 1;
 yDirection = 0;
 }
 if(up == LOW && down == LOW && left == LOW && right == LOW){
 xDirection = 0;
 yDirection = 0;
 }
 Serial.print(xDirection);
 Serial.print(",");
 Serial.print(yDirection);
 Serial.print(",");
 Serial.println(mappedSpeed);
 };
}

[Title to Be Determined] – Daniil and María

For this project, we chose to expand on the idea of the butterfly animation that María worked on last week. With it, we decided to create a visualization for the world’s birth and death rates.

The Data

Initially, we thought it would be fascinating to use real-time data and found a website that simulates births and deaths around the globe by the second. However, without an API, we didn’t know how to extract the data.

Thus, we chose to use information from a database and came across The World Factbook (TWF), published by the CIA. This publication, available online and already updated for 2017, includes lists of the world’s countries by descending birth and death rates (per 1,000 persons).

Python code to organize TWF’s data and generate a text file (which can be used by Processing):

def convertTxt(txtFile):
 txtData = open(txtFile,'r')
 txtText = txtData.read()
 txtArray = []
 txtArray2 = txtText.splitlines()
 for i in range(0, len(txtArray2)):
 tempArray = txtArray2[i].split(" ")
 ranking = tempArray.pop(0)
 for j in range(0, len(tempArray)):
 tempArray[j].strip()
 if tempArray[j] != "" and tempArray[j] != " ":
 stringArray = tempArray[j].split(", ")
 firstWordArray = stringArray[0].split()
 firstWord = " ".join(firstWordArray)
 first = stringArray.pop(0)
 stringArray.insert(0, firstWord)
 if len(stringArray) == 2:
 stringArray.reverse()
 string = " ".join(stringArray)
 else:
 string = ", ".join(stringArray)
 txtArray.append(string)
 return txtArray

def combineTxt(txtBirth, txtDeath):
 birthArray = convertTxt(txtBirth)
 deathArray = convertTxt(txtDeath)
 completeArray = []
 for i in range(0, len(birthArray)):
 if i%2 == 0:
 for j in range(0, len(deathArray)):
 if deathArray[j] == birthArray[i]:
 completeArray.append(birthArray[i])
 completeArray.append(birthArray[i+1])
 completeArray.append(deathArray[j+1])
 return completeArray

def writeTxt(txtBirth, txtDeath, txtNew):
 array = combineTxt(txtBirth, txtDeath)
 string = "_".join(array)
 newFile = open(txtNew,"w")
 newFile.write(string) 
 newFile.close()

writeTxt("rawdata_2054.txt", "rawdata_2066.txt", "birthDeath.txt")

Text file that the previous code generated (such that it shows country_birthRate_deathRate):

Niger_44.80_12.10_Mali_44.40_12.60_Uganda_43.40_10.40_Zambia_41.80_12.40_Burundi_41.70_9.00_Burkina Faso_41.60_11.50_Malawi_41.30_8.10_Somalia_40.00_13.30_Angola_38.60_11.30_Mozambique_38.30_11.90_Afghanistan_38.30_13.70_Nigeria_37.30_12.70_Ethiopia_36.90_7.90_Sierra Leone_36.70_10.60_South Sudan_36.20_8.00_Chad_36.10_14.00_Tanzania_36.00_7.80_Cameroon_35.80_9.80_Benin_35.50_8.00_Guinea_35.40_9.20_Republic of the Congo_35.10_9.70_Central African Republic_34.70_13.50_Gabon_34.30_13.10_Democratic Republic of the Congo_34.20_9.90_Senegal_34.00_8.30_Liberia_33.90_9.50_Timor-Leste_33.80_6.00_Togo_33.70_7.10_Sao Tome and Principe_33.30_7.00_Rwanda_33.30_8.80_Guinea-Bissau_32.90_14.10_Equatorial Guinea_32.80_8.00_Gaza Strip_32.30_3.20_Madagascar_32.10_6.70_Zimbabwe_31.90_9.90_Iraq_30.90_3.80_Mauritania_30.90_8.10_Ghana_30.80_7.10_Egypt_30.30_4.70_Eritrea_30.10_7.30_The Gambia_30.10_7.10_Western Sahara_29.80_8.20_Yemen_29.20_6.10_Sudan_28.50_7.50_Cote d'Ivoire_28.20_9.50_Namibia_27.90_8.10_Comoros_26.90_7.40_West Bank_26.70_3.50_Jordan_25.50_3.80_Solomon Islands_25.30_3.80_Kenya_25.10_6.80_Lesotho_25.10_14.90_Marshall Islands_25.00_4.20_Vanuatu_24.50_4.10_Guatemala_24.50_4.70_Nauru_24.40_5.90_Oman_24.30_3.30_Swaziland_24.30_13.40_Belize_24.30_6.00_Papua New Guinea_24.00_6.50_Philippines_24.00_6.10_Laos_23.90_7.50_Tajikistan_23.80_6.10_Tuvalu_23.80_8.60_Djibouti_23.60_7.60_Cambodia_23.40_7.60_Haiti_23.30_7.70_Algeria_23.00_4.30_American Samoa_22.90_4.80_Honduras_22.80_5.20_Tonga_22.60_4.90_Kyrgyzstan_22.60_6.60_Bolivia_22.40_6.50_Pakistan_22.30_6.40_Syria_21.70_4.00_Kiribati_21.30_7.10_Botswana_20.70_13.30_Samoa_20.60_5.30_South Africa_20.50_9.60_Federated States of Micronesia_20.30_4.20_Cabo Verde_20.20_6.10_Nepal_19.90_5.70_Kuwait_19.60_2.20_Mongolia_19.60_6.30_Malaysia_19.40_5.10_India_19.30_7.30_Turkmenistan_19.30_6.10_Venezuela_19.20_5.20_Bangladesh_19.00_5.30_Fiji_19.00_6.10_Kazakhstan_18.70_8.20_Dominican Republic_18.60_4.60_Mexico_18.50_5.30_Saudi Arabia_18.40_3.30_Israel_18.30_5.20_Burma_18.20_7.90_Ecuador_18.20_5.10_Panama_18.10_4.90_Jamaica_18.00_6.70_Morocco_18.00_4.80_Peru_18.00_6.00_Nicaragua_17.90_5.10_Iran_17.80_5.90_Libya_17.80_3.60_Bhutan_17.50_6.60_Brunei_17.20_3.60_Northern Mariana Islands_17.20_3.80_Argentina_17.00_7.50_Uzbekistan_16.90_5.30_Guam_16.70_5.20_Paraguay_16.50_4.70_Tunisia_16.40_6.00_Indonesia_16.40_6.40_Colombia_16.30_5.40_El Salvador_16.30_5.70_Azerbaijan_16.20_7.10_Turkey_16.00_5.90_Suriname_16.00_6.10_Maldives_16.00_3.90_Antigua and Barbuda_15.80_5.70_Grenada_15.80_8.10_Costa Rica_15.70_4.60_Turks and Caicos Islands_15.70_3.20_Vietnam_15.70_5.90_Guyana_15.50_7.40_Sri Lanka_15.50_6.20_The Bahamas_15.40_7.10_United Arab Emirates_15.30_2.00_New Caledonia_15.20_5.60_Dominica_15.20_7.90_French Polynesia_15.00_5.10_North Korea_14.60_9.30_Ireland_14.50_6.50_Lebanon_14.40_4.90_Greenland_14.40_8.60_Brazil_14.30_6.60_Cook Islands_14.10_8.30_Gibraltar_14.10_8.40_Faroe Islands_14.00_8.70_Seychelles_13.90_6.90_Curacao_13.80_8.30_Iceland_13.80_6.30_Chile_13.70_6.10_Bahrain_13.50_2.70_Saint Lucia_13.50_7.60_Saint Vincent and the Grenadines_13.40_7.30_Saint Kitts and Nevis_13.30_7.10_New Zealand_13.30_7.40_Wallis and Futuna_13.30_5.20_Armenia_13.30_9.40_Mauritius_13.10_7.00_Trinidad and Tobago_13.10_8.70_Albania_13.10_6.70_Sint Maarten_13.10_5.00_Uruguay_13.00_9.40_Anguilla_12.70_4.60_Aruba_12.50_8.30_Georgia_12.50_10.90_United States_12.50_8.20_China_12.40_7.70_France_12.30_9.30_Norway_12.20_8.10_Australia_12.10_7.20_Jersey_12.10_7.70_United Kingdom_12.10_9.40_Cayman Islands_12.10_5.70_Sweden_12.00_9.40_Barbados_11.80_8.50_Moldova_11.80_12.60_Macedonia_11.50_9.10_Belgium_11.40_9.70_Luxembourg_11.40_7.30_Cyprus_11.40_6.70_Russia_11.30_13.60_Bermuda_11.30_8.40_Palau_11.20_8.00_Thailand_11.10_7.90_British Virgin Islands_11.00_5.10_Isle of Man_11.00_10.10_Montserrat_11.00_6.30_Netherlands_10.90_8.80_Falkland Islands (Islas Malvinas)_10.90_4.90_Cuba_10.80_8.60_Puerto Rico_10.80_8.80_Finland_10.70_9.90_Ukraine_10.50_14.40_Belarus_10.50_13.30_Switzerland_10.50_8.20_Liechtenstein_10.40_7.30_Denmark_10.40_10.30_Estonia_10.30_12.50_Canada_10.30_8.50_Montenegro_10.20_9.60_Virgin Islands_10.20_8.90_Malta_10.10_9.20_Lithuania_10.00_14.50_Latvia_9.90_14.40_Slovakia_9.80_9.80_Guernsey_9.80_8.90_Saint Helena, Ascension, and Tristan da Cunha_9.70_7.70_Qatar_9.70_1.50_Poland_9.60_10.30_Czechia_9.50_10.40_Austria_9.50_9.50_Spain_9.40_9.10_Hong Kong_9.10_7.20_Hungary_9.10_12.80_Portugal_9.10_11.10_Romania_9.00_11.90_Croatia_9.00_12.10_Serbia_9.00_13.60_Bulgaria_8.80_14.50_Bosnia and Herzegovina_8.80_9.90_Macau_8.80_4.40_Italy_8.70_10.30_San Marino_8.60_8.60_Germany_8.50_11.60_Greece_8.50_11.20_Taiwan_8.40_7.30_South Korea_8.40_5.80_Singapore_8.40_3.50_Slovenia_8.30_11.50_Andorra_7.80_7.10_Japan_7.80_9.60_Saint Pierre and Miquelon_7.20_9.80_Monaco_6.60_9.60

The Visualization

Having obtained the data we needed, we adapted the butterfly code from María’s previous assignment such that it responds to each country’s birth and death rates.

We wanted to allow the user to select a country from a drop-down menu. In return, the sketch generates as many butterflies as the birth rate minus the deat rate in the chosen country (thus, a representation of its population´s growth rate) during 2016. Each butterfly flutters for some time and then “dies” (begins to fall and eventually disappears). If the total number of butterflies that are “alive” at any time is less than the country’s growth rate, then more butterflies – as many as needed to maintain the population – are generated (“born”) to repopulate the space. Once the country is changed on the menu,  the butterfly births and deaths adjust to the new population.

Processing sketch (this is actually a mash-up of three sketches – the butterfly class, the drop-down menu and the text file “reader”, and the sketch that runs the code):

import controlP5.*;

PFont font;

String data[];

ControlP5 cp5;
ControlFont cf1;
DropdownList d1;
int country;

ArrayList<Butterfly> bfs = new ArrayList <Butterfly>();
PImage bf1, bf2, bf3, bf4, bf5, bf6, bf7, bf8, bf9;
int widthLimit;
int heightLimit;
int counter;
int index;
int rate;
boolean start;

void setup(){
 size(900, 900);
 font = createFont("BebasNeue Regular.ttf", 30);
 loadImages();
 start = false;
 
 String[] txtFile = loadStrings("birthDeath.txt");
 data = split(txtFile[0], '_');
 
 cp5 = new ControlP5(this);
 d1 = cp5.addDropdownList("Country")
 .setPosition(100, 110)
 .setSize(200, 200); 
 customize(d1);
 
 counter = 0;
 index = -1;
 widthLimit = 250;
 heightLimit = 250;
 for (int i = 0; i < rate; i++){
 Butterfly b = new Butterfly();
 bfs.add(b);
 };
}

int getRate(int country){
 int indexCountry = country*3; //countries from 0 to 225
 int rate = Math.round(float(data[indexCountry + 1])) - Math.round(float(data[indexCountry + 2]));
 if(rate < 0){
 rate = 0;
 };
 return rate;
}

void customize(DropdownList ddl) {
 ddl.setBackgroundColor(color(190));
 ddl.setItemHeight(20);
 ddl.setBarHeight(15);
 ddl.getCaptionLabel().set("Countries");
 for (int i = 0;i < data.length/3; i++) {
 ddl.addItem(data[i*3], i);
 }
 ddl.setColorBackground(color(60));
 ddl.setColorActive(color(255, 128));
}

void controlEvent(ControlEvent theEvent) {
 if (theEvent.isController()) {
 index = int(theEvent.getController().getValue()); 
 rate = getRate(index);
 index = index*3;
 println(rate);
 }
}

void loadImages(){
 bf1 = loadImage("bf1.png");
 bf2 = loadImage("bf2.png");
 bf3 = loadImage("bf3.png");
 bf4 = loadImage("bf4.png");
 bf5 = loadImage("bf5.png");
 bf6 = loadImage("bf6.png");
 bf7 = loadImage("bf7.png");
 bf8 = loadImage("bf8.png");
 bf9 = loadImage("bf9.png");
}

class Butterfly{
 int X;
 int Y;
 int FRAME;
 int DIRECTION;
 int HORIZONTAL;
 int VERTICAL;
 int SPEED;
 int CENTERX;
 int CENTERY;
 int centerCounter;
 int centerCounterLimit;
 float alpha;
 
 Butterfly(){
 FRAME = int(random(1, 10));
 SPEED = int(random(10, 20));
 DIRECTION = int(random(0, 2));
 X = DIRECTION*width;
 Y = int(random(heightLimit, height-heightLimit));
 CENTERX = int(random(widthLimit, width-widthLimit));
 CENTERY = int(random(heightLimit, height-heightLimit));
 HORIZONTAL = (CENTERX - X)/SPEED;
 VERTICAL = (CENTERY - Y)/SPEED;
 alpha = 255;
 centerCounter = 0;
 centerCounterLimit = int(random(100, 300));
 }
 
 float fly(){
 tint(255, alpha);
 pushMatrix();
 scale(0.75);
 scale((float)Math.pow(-1, DIRECTION), 1);
 switch(FRAME){
 case 1: image(bf1, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf1.width)), Y);
 break;
 case 2: image(bf2, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf2.width)), Y);
 break;
 case 3: image(bf3, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf3.width)), Y);
 break;
 case 4: image(bf4, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf4.width)), Y);
 break;
 case 5: image(bf5, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf5.width)), Y);
 break;
 case 6: image(bf6, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf6.width)), Y);
 break;
 case 7: image(bf7, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf7.width)), Y);
 break;
 case 8: image(bf8, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf8.width)), Y);
 break;
 case 9: image(bf9, ((float)Math.pow(-1, DIRECTION))*(X + (DIRECTION*bf9.width)), Y);
 break;
 };
 popMatrix();
 if((DIRECTION == 0 && X >= CENTERX) || (DIRECTION == 1 && X <= CENTERX)){
 HORIZONTAL = 0;
 VERTICAL = 0;
 centerCounter++;
 };
 if(centerCounter > centerCounterLimit){
 HORIZONTAL = 0;
 VERTICAL = (height - CENTERY)/SPEED;
 alpha -= 1;
 };
 if(counter%7 == 0){
 if(FRAME == 9){
 FRAME = 1;
 }
 else{
 FRAME++;
 };
 };
 if(counter%SPEED == 0){
 X += HORIZONTAL;
 Y += VERTICAL;
 };
 return alpha;
 }
 }

void draw(){
 background(255);
 textFont(font);
 fill(0, 120);
 if(start == false){
 d1.hide();
 textAlign(CENTER);
 text("This is a data visualization of the world's 2016 birth and death rates for each country.", width/2, height/30*14);
 text("The data was taken from the CIA's 'The World Factbook'.", width/2, height/30*15);
 fill(0);
 text("Click to continue.", width/2, height/30*18);
 if(mousePressed){
 start = true;
 }
 }
 else{
 d1.show();
 fill(0, 120);
 textAlign(LEFT);
 if(index == -1){
 text("Choose a country ", 100, 100);
 }
 else{
 text("Country: "+data[index], 100, 100);
 textAlign(RIGHT);
 text("Birth rate: "+data[index+1], width - 100, height - 133);
 text("Death rate: "+data[index+2], width - 100, height - 100);
 textAlign(LEFT);
 };
 
 counter++;
 if(bfs.size() < rate){
 for(int i = 0; i < rate - bfs.size(); i++){
 Butterfly a = new Butterfly();
 bfs.add(a);
 };
 };
 for (int i = 0; i < bfs.size(); i++){
 Butterfly b = bfs.get(i);
 float opacity = b.fly();
 if(opacity < 0){
 bfs.remove(b);
 };
 }; 
 };
}

There was nothing particularly challenging about this project, however, it still took us time to figure out how to connect the text with the butterfly reproduction (generation) rate AND the menu at the same time.

Video of running sketch (a few countries were chosen to show the differences in the amount of butterflies):

On the Cost of Digitization

(Response to “The Digitization of Just About Everything” by Erik Brynjolfsson and Andrew McAfee)

One of the main points in Brynjolfsson and McAfee’s chapter is the acknowledgment of the “unique economic properties of digital information: such information is non-rival, and it has close to zero marginal cost of reproduction.”

The chapter proceeds to accurately explain how digitization reduces the cost of information. However, it is easy to forget, even ignore, that the processes behind digitization (and the management and storage of digital information) has its own set of unique costs.  As the authors put it, “… digitization yields truly big data”… and big data can’t be handled cost-free.

Last semester, I read an article titled “The Environmental Toll of a Netflix Binge,” which intends to describe the often-irresponsible energy consumption of massive data centers. The article indicates, for instance, that “in its 2013 sustainability report, Facebook stated its data centers used 986 million kilowatt-hours of electricity—around the same amount consumed by Burkina Faso in 2012.”

Brynjolfsson and McAfee also wrote that “digitization can also help us better understand the past. As of March 2012 Google had scanned more than twenty million books published over several centuries.” Again, I’ve read an article that denounces the cost of digitizing books at a massive scale, represented by the human labour behind the task (their working contracts, conditions, and lack of recognition – the case of the mechanical turkers). The article (“The Ladies Vanish”) says:

Of course books don’t digitize themselves. Human hands have to individually scan the books, to open the covers and flip the pages. But when Google promotes its project—a database of “millions of books from libraries and publishers worldwide”—they put the technology, the search function and the expansive virtual library in the forefront. The laborers are erased from the narrative, even as we experience their work firsthand when we look at Google Books.

Though I agree with the arguments that this chapter presents, I believe there is a lot about digital information that we, as users, are not aware of and should be. As users (and now, in this class, as makers), we share the responsibility of what happens “behind the screen” (in the back-end, if you will), far away in data centers and digitization facilities. Thus, it’s incorrect to believe that digitization, and digital environments themselves, have no costs at all.

Flutter

While googling illustrations for my first idea for this assignment (I’ve been changing my mind a lot with these projects lately), I found an image of a butterfly fluttering, composed of nine frames.

“Monarch Butterflies in Motion” (istockphoto)

Given that we have been focusing on visualization in motion, I thought it would be interesting (and conventiently straightforward) to create an animation of the butterfly in movement (Processing sketch here). The frames in the image I found allow said movement to be smooth and realistic. By creating a butterfly class, the animation can contain as many butterflies as desired with varying characteristics. For these objects in particular, motion and position offer the possibilities for variation, as the code shows.

PImage bf1, bf2, bf3, bf4, bf5, bf6, bf7, bf8, bf9;
PImage bg;
Butterfly bfA, bfB, bfC, bfD, bfE, bfF, bfG, bfH, bfI, bfJ, bfK, bfL, bfM, bfN;
int counter;
boolean start;

void setup(){
 size(1000, 1000);
 bf1 = loadImage("bf1.png");
 bf2 = loadImage("bf2.png");
 bf3 = loadImage("bf3.png");
 bf4 = loadImage("bf4.png");
 bf5 = loadImage("bf5.png");
 bf6 = loadImage("bf6.png");
 bf7 = loadImage("bf7.png");
 bf8 = loadImage("bf8.png");
 bf9 = loadImage("bf9.png");
 //frames of butterfly animation (9 in total)
 bfA = new Butterfly(0, 450, 1, 10, 10);
 bfB = new Butterfly(200, 350, 2, 10, 2);
 bfC = new Butterfly(300, 300, 3, 5, 5);
 bfD = new Butterfly(400, 500, 1, 15, 10);
 bfE = new Butterfly(50, 400, 3, 10, 10);
 bfF = new Butterfly(300, 600, 2, 10, 5);
 bfG = new Butterfly(200, 700, 4, 5, 10);
 bfH = new Butterfly(100, 800, 3, 15, 15);
 bfI = new Butterfly(10, 900, 1, 10, 10);
 bfJ = new Butterfly(50, 750, 2, 10, 10);
 bfK = new Butterfly(400, 1000, 2, 10, 10);
 bfL = new Butterfly(100, 650, 3, 10, 10);
 bfM = new Butterfly(10, 850, 2, 5, 10);
 bfN = new Butterfly(10, 700, 4, 10, 5);
 //butterflies (objects); can be increased or decreased, unlike animation frames
 counter = 0;
 //counter increases with each draw() loop
 start = false;
 //controls start of animation
 
 int i = int(random(6));
 switch(i){
 case 0:
 bg = loadImage("white.jpg");
 break;
 case 1:
 bg = loadImage("meadow.jpg");
 break;
 case 2:
 bg = loadImage("beach.jpg");
 break;
 case 3:
 bg = loadImage("house.jpg");
 break;
 case 4: 
 bg = loadImage("city.jpg");
 break;
 case 5: 
 bg = loadImage("water.jpg");
 break;
 }
 //random number and switch determine which background will be showed alongside the butterflies (6 options)
}

void draw(){
 if(start == false){
 background(255);
 fill(255,140,0);
 textSize(50);
 text("Flutter", 420, 480);
 //displays "title screen"
 if(mousePressed){
 start = true;
 }
 //animation starts until mouse is pressed
 }
 else{
 background(bg);
 bfA.fly();
 bfB.fly();
 bfC.fly();
 bfD.fly();
 bfE.fly();
 bfF.fly();
 bfG.fly();
 bfH.fly();
 bfI.fly();
 bfJ.fly();
 bfK.fly();
 bfL.fly();
 bfM.fly();
 bfN.fly();
 counter++;
 };
}

class Butterfly{
 int X;
 int Y;
 int FRAME;
 int SPEED;
 int FORWARDS;
 int UPWARDS;
 
 Butterfly(int x, int y, int speed, int forwards, int upwards){
 X = x;
 Y = y;
 FRAME = int(random(1, 10));
 SPEED = speed;
 FORWARDS = forwards;
 UPWARDS = upwards;
 //original position of butterfly, speed (related to counter), and forward/upward movement are uniquely assigned for each object 
 //FRAME is randomly assigned and determines which is the starting frame for the animation of each butterfly
 }
 
 void fly(){
 if(FRAME == 1){
 image(bf1, X, Y);
 }
 else if(FRAME == 2){
 image(bf2, X, Y);
 }
 else if(FRAME == 3){
 image(bf3, X, Y);
 }
 else if(FRAME == 4){
 image(bf4, X, Y);
 }
 else if(FRAME == 5){
 image(bf5, X, Y);
 }
 else if(FRAME == 6){
 image(bf6, X, Y);
 }
 else if(FRAME == 7){
 image(bf7, X, Y);
 }
 else if(FRAME == 8){
 image(bf8, X, Y);
 }
 else if(FRAME == 9){
 image(bf9, X, Y);
 };
 //image to display per loop is determined by FRAME
 if(counter%SPEED == 0){
 X = X + FORWARDS;
 Y = Y - UPWARDS;
 //SPEED determines after how many draw() loops the position of the butterfly changes; FORWARDS and UPWARDS determine the horizontal and vertical components of the displacement
 if(FRAME == 9){
 FRAME = 1;
 }
 //this "if" statement allows the cycle of animation frames to restart after the ninth one is displayed
 else{
 FRAME++;
 };
 //"else" statement allows progression of animation frames
 };
 }
}

I separated the frames in the original image and made their white background transparent (not perfectly, as the white traces in the animation give away). I also decided to allow the background to change randomly, to play with realism (hopefully to augment it, in most cases). Most background are realistic. The white background is the basic, plain, void one. The under-the-sea background is just for the laughs or the intense confusion, whichever compels the user first.

 

On Why New Media is New, But Also Kinda Old, and Everything In Between

(Response to The Language of New Media by Lev Manovich)

The sections of Manovich’s book that we were assigned attempt to define new media in technological, cultural, social, and historical ways. His chapters pack a lot of information about the concept, not only for the sake of definition but also to lay out what he thinks are the characteristics of new media, to validate or debunk commonly-held beliefs about it, to provide examples of new media by developers, artists, etc. and to trace the history of new media within the context of media in general.

It is this last aspect of the book that struck me the most. It often feels that Manovich wants to deal with too many faces of new media at once, a notion that I possibly have because I don’t fully understand all that he explains, but a common thread throughout the sections is his desire to tie new media back to its antecedents, and through this show how the new technologies and interactions that it encompasses did not come out of the blue. Manovich does this from the start, by bringing Charles Babbage’s Analytical Machine and J.M. Jacquard’s loom into the conversation, and continues to do so until the end, when he mentions Leon Battista Alberti’s On Painting among others.

I find that the “timeline” given by Manovich makes the development of new media more interesting, especially given that the book was published in 2001, and as a 2017 reader I can fill it in with new information. Understanding the many dimensions of new media, where it takes its influences from, and how it adapts them in different ways, makes the concept seem much richer, complex, and valuable than just a vague term to describe “what computers do.”

Cubism, the Pyramids, and a Rainbow

For this week’s assignment, I was torn between two illustrations from Computer Graphics and Art that I found interesting. One was more complex than the other, or so it seemed to me, and thus I began trying out the more difficult one. Halfway through (realizing that option one would take a while), I decided to try my second option. I was eager to see it coded because, frankly, to me it is the more aesthetically pleasing one.

I copied Random Squares by Bill Kolomyjec, which appeared in Computer Graphics and Art in August, 1977.

The Processing sketch I made can be found online.

The generation of the squares is mostly based on randomness, as the code shows, in order to stay true to Kolomyjec’s title. I also added other features (stroke color and weight) that are not present in the publication’s image, which the user can control.

int columns = 7;
int rows = 5;
int gridSquare = 100;
//sides of each square on the grid
//thus, 7x5 squares on the grid
int squareSide = 20;
//sides of the "originating", smaller square within each square on the grid
boolean mouseP = false;
boolean keyP = false;
int hue = 0;
//used to set the stroke() in HSB color mode
int weight = 1;
//used to set the strokeWeight()
int loop = 0;

void setup(){
 size(700, 500);
 //canvas size corresponds to columns, rows, and gridSquare
 background(255);
 fill(0);
 textSize(22);
 text("Click here", 300, 250);
 //instructs the user to click on the canvas
}

void draw(){
 if(mousePressed && mouseP == false){
 background(255);
 textSize(18);
 text("Press UP or DOWN to change color.", 200, 210);
 text("Press LEFT or RIGHT to change stroke weight.", 150, 250);
 text("Press ANY OTHER KEY for black.", 200, 290);
 //lets user know about the features of the program
 fill(255);
 mouseP = true;
 }
 
 if(keyPressed && keyP == false && mouseP == true){
 background(255);
 if(keyCode == UP){
 colorMode(HSB);
 stroke(hue, 240, 240);
 if(hue != 230){
 hue = hue + 10;
 };
 }
 //UP key increases hue (230 is the maximum)
 else if(keyCode == DOWN){
 colorMode(HSB);
 stroke(hue, 240, 240);
 if(hue != 0){
 hue = hue - 10;
 };
 }
 //DOWN key decreases hue (0 is the minimum)
 else if(keyCode == LEFT){
 if(weight != 1){
 weight--;
 };
 strokeWeight(weight);
 }
 //LEFT key decreases stroke weight (1 is the minimum)
 else if(keyCode == RIGHT){
 if(weight != 10){
 weight++;
 };
 strokeWeight(weight);
 }
 //RIGHT key increases stroke weight (10 is the maximum)
 else{
 stroke(0);
 };
 //any other key sets stroke to black
 for(int i = 0; i < (columns*gridSquare); i = i + gridSquare){
 line(i, 0, i, rows*gridSquare);
 };
 //creates grid's rows
 for(int i = 0; i < (rows*gridSquare); i = i + gridSquare){
 line(0, i, columns*gridSquare, i);
 };
 //creates grid's columns
 line((columns*100) - 1, 0, (columns*100) - 1, (rows*100) - 1);
 line(0, (rows*100) - 1, (columns*100) - 1, (rows*100) - 1);
 
 for(int i = 0; i < (columns*gridSquare); i = i + gridSquare){
 for(int j = 0; j < (rows*gridSquare); j = j + gridSquare){
 int xDisplacement = int(random(10, gridSquare - squareSide - 10));
 int yDisplacement = int(random(10, gridSquare - squareSide - 10));
 rect(i + xDisplacement, j + yDisplacement, squareSide, squareSide);
 //creates "originating" square
 int divisions;
 if(xDisplacement < 40 || yDisplacement < 40 || xDisplacement > 60 || yDisplacement > 60){
 if(xDisplacement < 20 || yDisplacement < 20 || xDisplacement > 80 || yDisplacement > 80){
 divisions = int(random(2, 4));
 }
 else{
 divisions = int(random(4, 6));
 };
 }
 else{
 divisions = int(random(6, 10));
 };
 //randomly selects how many rectangles there will be between the "originating" square and the grid square
 int upperDivisions = yDisplacement/divisions;
 int lowerDivisions = (gridSquare - (yDisplacement + squareSide))/divisions;
 int leftDivisions = xDisplacement/divisions;
 int rightDivisions = (gridSquare - (xDisplacement + squareSide))/divisions;
 for(int k = 1; k < divisions; k++){
 line(i + (leftDivisions*k), j + (upperDivisions*k), (i + gridSquare) - (rightDivisions*k), j + (upperDivisions*k));
 line(i + (leftDivisions*k), j + (upperDivisions*k), i + (leftDivisions*k), (j + gridSquare) - (lowerDivisions*k));
 line((i + gridSquare) - (rightDivisions*k), j + (upperDivisions*k), (i + gridSquare) - (rightDivisions*k), (j + gridSquare) - (lowerDivisions*k));
 line(i + (leftDivisions*k), (j + gridSquare) - (lowerDivisions*k), (i + gridSquare) - (rightDivisions*k), (j + gridSquare) - (lowerDivisions*k));
 };
 // creates said rectangles
 };
 };
 keyP = true;
 }
 else if(!keyPressed && keyP == true){
 keyP = false;
 };
}

There is something visually satisfying about the illusion of depth in a 2D space that the squares accomplish. The effect makes the “denser” squares on the grid look like pyramids that point either inwards, towards the screen, or outwards, towards the user. I added the change of hue because I’m a great fan of bright colors and their variation along the visible light spectrum (which I also used in my previous assignment). Finally, I enabled changing the stroke weight because the effect the image completely, and in my opinion, makes it more interesting. It somewhat reminded me of Cubist paintings…

Maybe? Just a little bit?