Week 10: April 1: Working with Data + Visualizations

Itinerary

  • Critique Organisms
  • Agnes Varda
  • Examples of data and art
  • Review importing data + p5 /Formatting data files
  • Using CSV Files
  • In Class Data Collection
  • Studio for Data Visualization
  • Data Sets!

Visualization + Data

Simple Data

As we’ve seen throughout the semester so far, the for loop helps to repeat actions in our programming. Primarily we’ve used it to move and repeat graphics and create multiple objects. In this instance of working with data, we are going to use it to iterate through a set of information. In this first case, the data is an array. An array can contain any set of data – objects, integers, strings, colors. This example contains numbers.

To read the value of an array, we can use the for loop to cycle through the data.

let num = [10,20,30,40];
for(let i=0; i<num.length; i++){
      print(num[i]);
}

Here we use the value of i to change the index point of the array, then print the result to the console.

The simplest data 

Formatting Data + Importing into Web Editor

You will want to save your data as a .csv file to bring into the web editor or brackets. Before you bring in the data, you’ll need to tidy it up. Remove any columns of information in Excel or Google Drive that are unnecessary. We will want to make this file as small as possible so that it loads quickly in our webpage. Make sure at the top of the each of your rows, you have a simple way to identify each column. Eg: Miles, Mood, Smell. Don’t write “The amt of MiLeS Rannnnnn”. This will likely cause you to make an error in trying to re-write accessing this in p5.js.

At the top of the web editor, select Sketch, then Add File, and then drag and drop your CSV file into the pane. Close out the pane when it’s done loading.

Simple CSV Data Reading

But what happens when we we have a two-dimensional array, a matrix of information. A CSV, or comma separated value, file – a spreadsheet, has data going in both rows and columns. Fortunately, p5.js has some functions to help us read through the data more easily. There is some work on our end to design the spreadsheet more efficiently, however.

let table = loadTable(“data.csv”, “csv”,”header”);

let numRows = table.getRowCount();

let miles = table.getColumn(“INSERT COLUMN NAME HERE”);

First, we have to store our csv file somewhere. We use loadTable() to store our data. Writing csv afterwards says the file type is a csv, and using the attribute header lets this function know that we are going to use the very first row of information to give names to our columns of data. You’ll see this in the next line. Pretty nifty.

.getRowCount(); allows us to do the same thing as .length for an array. It uses the value of the tables rows to get the total number of rows. We will use this value to iterate over the csv file.

table is the name of the variable where we stored our data in loadTable. We use dot notation to attach the function of .getColumn, which gets all of the data for a particular column and stores it in an array. Inside the parenthesis we use the header of the column as a string to access the data. For example, I used “Miles”. Case matters here, and you have to be consistent otherwise the function won’t access the data.

for(let i=0; i<numRows; i++){
     print(miles[i]);
}

The stored value of the column is set to an array that we can iterate over using a for loop. We can use this method to use the visualize the data immediately.

Simple CSV to Screen

map() Review

As we’ve seen previously in the semester, we can use the map() function to re-contextualize information by changing its range.

let newValue = map(oldValue, oldMinimumRange, oldMaximumRange, newMinRange, newMaximumRange);

Previously, we’ve used it to remap color values in particular. For example, if we wanted to change the color of the background of our canvas when our mouse moves across it, we would rescale the value of mouseX. We would specifically rescale the width of the canvas to be equal to the values of an RGB value – which is on the scale from 0-255.

In practice this would look like:

newBgColor = map(mouseX, 0, width, 0, 255);
background(newBgColor);

When working with data – we might want to rescale the value that we’ve been working with to work most clearly within the canvas. In the example of our running data set, the amount of miles ran goes between 2 and 7. If I only showed a difference between 2 and 7 pixels on the canvas, then the visual difference to our eyes would be minuscule. So I would want to rescale this value for the canvas. In the next examples, we’ll see how to get the minimum and maximum values for our original values without having to manually look at our code. In this case, with only 17 values, its easy to go look at the data. In situ:

 //iterate over the number of rows
for (let i = 0; i < numRows; i++) {

  //change the length of the rectangle based on the amount of miles ran and the range of miles
  let mappedMiles = map(miles[i], 2, 7, 10,50);

  //use the value of i to move the rectangle down
  let mover = 20 + (i * 20)

  //draw the rectangle
  rect(50, mover, mappedMiles, 10);
}

The difference here, original without remapped value and then with. Big Visual Difference!

     

Conditionals with Strings

When working with data, we may see text data in addition to numerical data. In order to process and change the data based on text, or strings, then we will have to use conditional statements. For example, in the example running data there is a column for smells.

trash
river
grass
hoagie
gravel

I want to change the color of the shapes drawn based on each of these points of data. Using an
if/else if/else statement will allow me to do this.

We will use two equal signs together == inside the parenthesis for the conditional part of the if statement to see if the string text matches what is inside our column of data.

if ("string" == data) {
  //do something
}

Below, using the smell column we are changing the variable value of col, which will store our color. If the string in the conditional statement matches the data, the color of the fill will change in the rectangle that’s drawn to the screen because we use fill(col) right before we draw the rectangle. The whole code:

//create a variable to store the color
let col;

//create a variable to store the amount of rows in the CSV
let numRows = table.getRowCount();

//get the columns titled Miles and Smell
let miles = table.getColumn('Miles');
let smell = table.getColumn('Smell');
//iterate over the number of rows
for (let i = 0; i < numRows; i++) {

  if ("trash" == smell[i]) {
   //saddle
   col = color('#8B4513');
  } else if ("river" == smell[i]) {
   //blue
   col = color(0, 50, 200);
  } else if ("grass" == smell[i]) {
   //green
   col = color(25, 255, 100);
  } else if ("hoagie" == smell[i]) {
   //yellowish
   col = color('#CCCC00');
  } else if ("gravel" == smell[i]) {
   //light grey
   col = color('#D3D3D3');
  }

  //use the value of i to move the rectangle down
  let mover = 20 + (i * 20)

  //change the length of the rectangle based on 
  //the amount of miles ran and the range of miles
  let mappedMiles = map(miles[i], minMiles, maxMiles, 20, width - 80);

  //draw the rectangle
  fill(col);
  rect(50, mover, mappedMiles, 10);
}

Data code example here

Storing Max and Min Values

Often times when working with data we’ll need to figure out a range of values for a given data point – a range being the minimum and maximum values. This comes up especially when considering latitude and longitude, where if we are creating a map that might change size based on the windowWidth and windowHeight, or only within a specific dimension, we want our resulting data set to fit proportionally inside that form.

While we could manually search through the data to find these values, we can do this more efficiently using p5’s built in functions. The functions min and max will search through an array to find the minimum and maximum values respectively. Since we are storing the information from each column in an array, we can call min or max on the variable the column is stored in, and save that as a specific variable to use later in our visualization. For example:

//store the column of miles as a variable
let miles = table.getColumn('Miles');

//store the min value of miles in a new variable
let minMiles = min(miles);

//store the max value of miles in a new variable
let maxMiles = max(miles);

//debug and print to the console
print(minMiles,maxMiles);

We can then use these new variables to get a more developed image. Previously, our image looked like this:

While this works, the visual difference between each element is pretty small. By using the min and max values and map, we can get a more developed visual representation of the data, such as the image below.

To do this, inside the for loop where we access each element of the array of data, we remap the original value to a new value.  Full example in situ here.

//use a for loop to cycle through the data
for (let i = 0; i < numRows; i++) {
  
  // in a new variable, map the value of each mile from its min value and its max value to a
  // new range, in this case, a min of 20 to just less than the width of the canvas.
  let mappedMiles = map(miles[i], minMiles, maxMiles, 20, width-50);
  
  //use that new variable to change the width of each rectangle
  rect(20, 20 + (i * 20), mappedMiles, 10);
}

Description of Latitude and Longitude

Coordinates are measured in degrees, minutes, and seconds (DMS). For example, Shanghai’s degree coordinates are 31.2304° N, 121.4737° E . When working in the canvas space of p5.js and graphics oriented images, we want the decimal notation of the coordinates to be able to convert these values to x and y polar coordinates. Google Maps will do this for us if we put the DMS coordinates into the search bar. So the decimal notation for Shanghai is 31.230400, 121.473700. Most spreadsheets you are working with will have the decimal notation for coordinates embedded into it.

Longitude goes from -180 degrees to 180 degrees
Latitude goes from -90 degrees to 90 degrees

Data to Screen with Lat and Long

SAMPLE DATA – Food Poisoning

Food Poisoning Data to Screen

In order to display latitude and longitude in a consistent manner, we have to do a remapping of values to the canvas.

let lngMap = map(lng, LONGITUDE MIN VALUE, LONGITUDE MAX VALUE, 0, width);
let latMap = map(lat, LATITUDE MIN VALUE, LATITUDE MAX VALUE, height, 0);
point(lngMap,latMap);

Note here a few things. We need the min and max values for the longitude and the latitude. We can find those using min and max functions, and storing them in a variable using the method we used above.

IMPORTANT, do not use long as a variable name in p5. long is a datatype, and the program will get confused if you use this as your variable name, so use lng instead

//store the column of latitude and longitude as a variable
let lat= table.getColumn('lat');
let lng = table.getColumn('lng');

//store the min and min value of latitude in new variables
let minlat= min(lat);
let maxlat = max(lat);

//store the min  and min value of longitude in new variables
let minlng= min(lng);
let maxlng = max(lng);

//debug and print to the console
print(minlat, maxlat);
print(minlng, maxlng);

Or, we can use the bounding box website to get a general idea of where our the limits of our values might be located. Secondly, notice that in the second mapped value, the values of zero and height are reversed. This is because the canvas goes from zero at the top to height at the bottom corner. In order to get this to draw right side up, we have to switch these values around.

//run a loop through the rows of the table
for (let i = 0; i < numRows; i++) {

  //remap the latitude and longitude to fit on the screen using our previously stored values
  let lngMap = map(lng[i], minlng, maxlng, 0, width); 
  let latMap = map(lat[i], minlat, maxlat, height, 0); 

  //draw to the screen
  //fill with white and partial opacity
  fill(255,125);
  ellipse(lngMap,latMap,2,2);
}

USA Visualization Simplest

Homework

  • Post six images to Are.na about visualizations
  • Continue working on your visualization – come in next week with it working on screen
  • Continue to update Portfolio page