Parsons School of Design, The New School
School of Art, Media and Technology
Master of Science, Data Visualization
Wednesdays, 12:10pm - 2:50pm
online course | Zoom info in Canvas
Faculty: Aaron Hill | Office Hours: Fridays 3:00pm - 4:00pm: no appointment needed; Zoom info in Canvas
Slack: I encourage use of the #data-structures-20 channel in the ms-dataviz Slack for collaboration and work on assignments, including the 12 weekly and 3 final assignments. All questions/issues about the assignments should be posted here, building a shared knowledge base in the process. Questions/issues raised in office hours should also be posed here prior to the office hour meeting and the answer/resolution will be posted by the person who originally posed the question.
Course description: Curating raw data presents issues of technology, speed, and efficiency, as well as broader ethical considerations of what it means to represent and make conclusions about groups and individuals from their data. This course covers the database, semi-structured data, and unstructured data. Students will gain familiarity with underlying data structures; techniques and tools, including acquisition, augmentation, and restructuring; data storage and aggregation; access to parallel and distributed computing; high-volume data, disparate sources, and performance; and streaming data, real time, and dynamic queries.
Prerequisites: Basic knowledge of JavaScript, HTML/CSS, Linux command line, Git, and GitHub.
| Duration | Description |
|---|---|
| Week 1 | Introductions and overview of semester |
| Weeks 2 - 4 | Fundamental data structures; modeling structured data |
| Week 5 | Modeling semi-structured data |
| Week 6 | The Query |
| Weeks 7 - 9 | Physical sensors and IoT; data cleaning |
| Week 10-11 | Interface design; serverless computing |
| Weeks 12 - 14 | Reliability, scalability, maintainability, sustainability; Labs |
| Week 15 | Final presentations (video) and reflections (Canvas) |
Introduction to data structures.
Weekly Assignment 1, due Tuesday 9/8 at 6:00pm:
You will "get" some HTML files, save them, and study their structure.
The web as a database. "Raw data."
- Gitelman, Introduction and Color Plates
- Hills, Part One (pp. 17-42)
- Dourish, Chapter 1
- Express/Node Introduction (read carefully the "Node" introduction and quickly skim the "Express" introduction)
Weekly Assignment 2, due Tuesday 9/15 at 6:00pm:
You will parse one of the HTML files you saved last week and log essential data to the console.
Fundamental data structures. Data augmentation and integration. GitHub for open source collaboration.
- Gitelman, Chapter 5
- Hills, Chapters 10, 11, and 12
- Javascript data types and structures
- Working with objects (JavaScript)
Weekly Assignment 3, due Tuesday 9/22 at 6:00pm:
Using data you parsed last week, you will interface with TAMU GeoServices to augment the data you collected.
- TAMU GeoServices
- TAMU GeoServices Geocoding APIS
- Node Async module
- Tutorial: creating and using environment variables in Linux.
Data models I: SQL database.
- Gitelman, Chapter 2
- Hills, Chapters 13, 14, 15, and 16
- Dourish, Chapter 4
Weekly Assignment 4, due Tuesday 9/29 at 6:00pm:
You will design a schema, create a new managed PostgreSQL database, and write SQL code to populate your database with your AA data.
Final Assignment 1 distributed: Map of AA meetings in Manhattan.
Data models I: NoSQL database.
- Gitelman, Chapter 6
- Hills, Chapter 17
- Dourish, Chapter 5
Weekly Assignment 5, due Tuesday 10/6 at 9:00am:
You will design a (non)schema, create a new managed DynamoDB database, and write code to populate your database with your "Process Blog" data.
Final Assignment 2 distributed: "Process Blog," a source of semi-structured, qualitative data.
The Query. Indexes, joins, query optimization, and data restructuring. The E-R model. Data provision.
- Hills, Chapter 5
- Database as Symbolic Form, Lev Manovich
- Gitelman, Data Flakes: An Afterword to "Raw Data" Is an Oxymoron
Weekly Assignment 6, due Tuesday 10/13 at 6:00pm:
You will write queries for both the SQL and NoSQL databases you have created, filtering, aggregating, and/or restructuring the data in the process.
Data cleaning. Logging issues.
Data cleaning workshop.
Weekly Assignment 7, due Tuesday 10/20 at 6:00pm:
Finish parsing and cleaning the rest of the AA data in all ten Manhattan "zones", and update/replace the data in your Postgres table(s).
- Gitelman, Chapter 4
- For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insights, The New York Times, August 17, 2014
Note: IEEE VIS conference is October 25-30
Sensors. Data gathering and surveillance.
Weekly Assignment 8, due Tuesday 11/10 at 6:00pm:
Switching gears from previous assignments, you will use an IoT development board and sensors to collect and log data to your local computer. You will also sketch your first iteration of the design of an interactive visualization of the data you will collect from the sensor.
- Gitelman, Chapter 7
[slides]
IoT.
Note: IEEE VIS conference is October 25-30
Weekly Assignment 9, due Tuesday 11/10 at 6:00pm:
You will begin writing sensor data to a database.
Final Assignment 3 distributed: Surveilling your life.
DUE Wednesday 11/4 at 12:00pm: IEEE VIS conference presentation summary
- Gitelman, Chapter 3
- Klepman, Chapter 2
- Dourish, Chapter 6
From database to visual representation. Interface design.
Weekly Assignment 10, due Tuesday 11/10 at 6:00pm:
You will submit the final designs for the interfaces for each of your three data sources.
Debate: SQL vs. NoSQL [1/2 | 3/4]
- Gitelman, Chapter 1
- New York Times: Slump in Air Travel Hindered Weather Forecasting, Study Shows
Serverless computing.
Weekly Assignment 11, due Tuesday 12/1 at 6:00pm:
You will migrate your work to a serverless computing environment.
Discussion: Reliability. Lab.
Weekly Assignment 12, due Tuesday 12/1 at 6:00pm:
You will create endpoints to connect your data to the design of your front-end interface.
Coding demo: making a map in mapbox
- Klepmann, Chapter 1. Read from the beginning through the "Reliability Section."
- How the internet works
Discussion: Scalability. When do you need a database(s)? When do you not? Lab.
- Klepmann, Chapter 1. Read the "Scalability Section."
- MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat
- immens paper
Discussion: Maintainability. Lab.
- Klepmann, Chapter 1. Read the "Maintainability Section" and through the end of the chapter.
- Gitelman, Chapter 8
- What is code?
Final Assignments 1, 2, and 3 due Tuesday 12/15 at 6:00pm:
In the final class meeting, you will present your work in groups, by theme. Group and theme assignments:
- Reliability: Dallas, Isabel, Justin, Tiffany, and Zhibang
- Scalability: Amina, Baihan, James, and Morgane
- Maintainability: Jessie, Lee, Ripley, and Sherry
Terminating and cleaning up your AWS resources and credentials. [checklist]
By the successful completion of this course, students will be able to demonstrate:
- Proficiency with various techniques to acquire, process, restructure, and analyze raw data.
- Ability to program and work within structured and unstructured databases to deliver and exchange data.
- Knowledge of methods for web development, hosting, and processing of data for visualizations, often from very large data sources.
- An understanding of computer science principles that support computational efficiency and speed.
- An understanding of the limitations and ethical considerations when using data to represent individuals and groups.
| Description | % |
|---|---|
| Engagement and process blog | 5% |
| IEEE VIS presentation summary | 2% |
| Weekly Assignments (12 total) | 48% |
| Final assignment 1 | 15% |
| Final assignment 2 | 15% |
| Final assignment 3 | 15% |
Weekly assignments are worth 4 points each, unless otherwise indicated. They are due 6:00pm the Tuesday after they are assigned, unless otherwise indicated. The quality of your documentation is a part of the grade for every assignment, including all the weekly and final assignments. All documentation must be kept in your (well organized!) data-structures repository, using markdown.
If an assignment is less than 48 hours late, the maximum possible grade is 75%.
If an assignment is 48-96 hours late, the maximum possible grade is 50%.
If an assignment is 96-144 hours late, the maximum possible grade is 25%.
If an assignment is more than 144 hours late, no credit will be given.
The Stuff of Bits : An Essay on the Materialities of Information, Paul Dourish. This book is available electronically through course reserves.
"Raw Data" Is an Oxymoron, Lisa Gitelman, 2013. This book is available electronically through course reserves.
NoSQL and SQL Data Modeling: Bringing Together Data, Semantics, and Software, Ted Hills, 2016. This book is not available through course reserves and should be purchased (~$27 on Amazon).
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017. This book is available electronically through course reserves.
Database Aesthetics : Art in the Age of Information Overflow, Victoria Vesna. This book is available electronically through course reserves.
Fridays, 3:00pm - 4:00pm
Above all other responsibilities this semester, our primary purpose is to take care of ourselves, which is particularly urgent in the middle of a global pandemic concurrent with social and political tumult. Normally, I am strict about deadlines and adherence to the specifications of an assignment, but these are not normal times. I am going to give everyone the benefit of the doubt and assume we’re all doing the best we can (myself included). My primary expectation of you this semester is that you take care of yourself, first and foremost. And that means you don’t need to ask me for what you need; just take it. Should you have to miss a deadline or take time away, you don’t need to justify it to me, you certainly don’t have to tell me why. I ask that you let me know if you have to step away and miss classes, assignments, or other obligations (so that I don’t worry about you), but you do not have to qualify it by disclosing why you need to do so. That said, I hope you feel comfortable coming to me with anything you want to discuss and I’ll do my best to help or find someone who can. This time is stressful enough as it is; I don’t want your work in this class to add to it. If you need to depart from the syllabus in any way, I will work with you to figure out a way to fulfill the requirements of this course in a way and at a time that will work for you. Please put your physical and mental health at the very top of your list of priorities right now. Everything else can be sorted out later.
In this course we will rely exclusively on open source software and we will work with the software by writing code: JavaScript, SQL, and some basic Linux.
We will also rely on cloud-based services to provide infrastructure, platforms, and computational power:
- Amazon Web Services (or similar) for computing infrastructure, platforms, and computational power.
- AWS Cloud 9 for a working environment to use for prototyping and experimentation.
- A modern web browser. Preferred: Google Chrome. Acceptable: Firefox. Not acceptable: Internet Explorer, Safari.
- Bring your laptop to every class.
- A (free) GitHub account
- A (free) Amazon Web Services (AWS) account
- A (free) AWS Educate account
- A microcontroller and sensors (as assigned; you will be given a list of the things you'll need to purchase, ~$50)
- It is also helpful to have a text editor installed locally. Sublime and Atom are popular choices. Atom integrates well with GitHub.
- If you decide to work locally (instead of in Cloud9), you may need to install and maintain additional software such as Git (installed with Xcode in OSX), Node.js, npm (which is installed with Node.js), and other software.
The university provides many resources to help students achieve academic and artistic excellence. These resources include:
- The University (and associated) Libraries: http://library.newschool.edu
- The University Learning Center: http://www.newschool.edu/learning-center
- University Disabilities Service: http://www.newschool.edu/student-disability-services/
In keeping with the university's policy of providing equal access for students with disabilities, any student with a disability who needs academic accommodations is welcome to meet with me privately. All conversations will be kept confidential. Students requesting any accommodations will also need to contact Student Disability Service (SDS). SDS will conduct an intake and, if appropriate, the Director will provide an academic accommodation notification letter for you to bring to me. At that point, I will review the letter with you and discuss these accommodations in relation to this course.
The Making Center is a constellation of shops, labs, and open workspaces that are situated across the New School to help students express their ideas in a variety of materials and methods. We have resources to help support woodworking, metalworking, ceramics and pottery work, photography and film, textiles, printmaking, 3D printing, manual and CNC machining, and more. A staff of technicians and student workers provide expertise and maintain the different shops and labs. Safety is a primary concern, so each area has policies for access, training, and etiquette that students and faculty should be familiar with. Many areas require specific orientations or trainings before access is granted. Detailed information about the resources available, as well as schedules, trainings, and policies can be found at http://resources.parsons.edu.
A Work of exceptional quality
A- Work of high quality
B+ Very good work
B Good work; satisfies course requirements
Satisfactory completion of a course is considered to be a grade of B or higher.
B- Below-average work
C+ Less than adequate work
C Well below average work
C- Poor work; lowest possible passing grade
F Failure
GM Grade missing for an individual
Grades of D are not used in graduate level courses.
Grade of W:
The grade of W may be issued by the Office of the Registrar to a student who officially withdraws from a course within the applicable deadline. There is no academic penalty, but the grade will appear on the student transcript.
Grade of Z:
This grade is to be assigned to students who have never attended or stopped attending classes. Exceptions can be made if the student has completed enough work to warrant a grade (including a failing grade), and arrangements have been made with the instructor(s) and the Dean’s Office prior to grade submission. The Z grade does not calculate into the student’s GPA.
Grades of Incomplete:
The grade of I, or temporary incomplete, may be granted to a student under unusual and extenuating circumstances, such as when the student's academic life is interrupted by a medical or personal emergency. This mark is not given automatically but only upon the student's request and at the discretion of the instructor. A Request for Incomplete form must be completed and signed by student and instructor. The time allowed for completion of the work and removal of the I mark will be set by the instructor with the following limitations:
Work must be completed no later than one year following the end of the class. Grades of I not revised in the prescribed time will be recorded as a final grade of N by the Registrar’s Office.
A comprehensive overview of policy may be found under Policies: A to Z. Students are also encouraged to consult the Academic Catalog for Parsons.
Use of Canvas may be an important resource for this class. Students should check it for announcements before coming to class each week.
The use of electronic devices (phones, tablets, laptops, cameras, etc.) is permitted when the device is being used in relation to the course's work. All other uses are prohibited in the classroom and devices should be turned off before class starts.
Students are responsible for all assignments, even if they are absent. Late assignments, failure to complete the assignments for class discussion and/or critique, and lack of preparedness for in-class discussions, presentations and/or critiques will jeopardize your successful completion of this course.
Class participation is an essential part of class and includes: keeping up with reading, assignments, projects, contributing meaningfully to class discussions, active participation in group work, and coming to class regularly and on time.
Parsons' attendance guidelines were developed to encourage students’ success in all aspects of their academic programs. Full participation is essential to the successful completion of coursework and enhances the quality of the educational experience for all, particularly in courses where group work is integral; thus, Parsons promotes high levels of attendance. Students are expected to attend classes regularly and promptly and in compliance with the standards stated in this course syllabus.
While attendance is just one aspect of active participation, absence from a significant portion of class time may prevent the successful attainment of course objectives. A significant portion of class time is generally defined as the equivalent of three weeks, or 20%, of class time. Lateness or early departure from class may be recorded as one full absence. Students may be asked to withdraw from a course if habitual absenteeism or tardiness has a negative impact on the class environment.
I will assess each student’s performance against all of the assessment criteria in determining your final grade.
Compromising your academic integrity may lead to serious consequences, including (but not limited to) one or more of the following: failure of the assignment, failure of the course, academic warning, disciplinary probation, suspension from the university, or dismissal from the university.
Students are responsible for understanding the University’s policy on academic honesty and integrity and must make use of proper citations of sources for writing papers, creating, presenting, and performing their work, taking examinations, and doing research. It is the responsibility of students to learn the procedures specific to their discipline for correctly and appropriately differentiating their own work from that of others. The full text of the policy, including adjudication procedures, is found on the university website under Policies: A to Z. Resources regarding what plagiarism is and how to avoid it can be found on the Learning Center’s website.
The New School views "academic honesty and integrity" as the duty of every member of an academic community to claim authorship for his or her own work and only for that work, and to recognize the contributions of others accurately and completely. This obligation is fundamental to the integrity of intellectual debate, and creative and academic pursuits. Academic honesty and integrity includes accurate use of quotations, as well as appropriate and explicit citation of sources in instances of paraphrasing and describing ideas, or reporting on research findings or any aspect of the work of others (including that of faculty members and other students). Academic dishonesty results from infractions of this "accurate use." The standards of academic honesty and integrity, and citation of sources, apply to all forms of academic work, including submissions of drafts of final papers or projects. All members of the University community are expected to conduct themselves in accord with the standards of academic honesty and integrity. Please see the complete policy in the Parsons Catalog.
The New School (the "university") seeks to encourage creativity and invention among its faculty members and students. In doing so, the University affirms its traditional commitment to the personal ownership by its faculty members and students of Intellectual Property Rights in works they create. The complete policy governing Intellectual Property Rights may be seen on the university website, on the Provost's page.