TMI-Web is a social science research tool for managing, analyzing, coding, and visualizing qualitative survey data on identities. It presents identity and experience in a network graph, encouraging tactile exploration of intersectional identities and facets of privilege and marginalization.
TMI-Web has three primary components: the web application, the survey response database, and the persona database.
Survey data is exported from your survey platform as a comma-separated-value (CSV) file, then imported into TMI-Web using the browser interface. TMI-Web stores the survey response data in a relational database, and creates personas in a graph database as the survey data is analyzed.
(To make sure that the web application is always responsive, certain long-running operations are carried out in the background, requiring a key-value store database that's used for a background job queue.)
Instructions for installing and running TMI-Web locally are provided below.
Depending on the operating system you use, there may be different ways for you to access your local command line. Understanding how to navigate the file system and run programs from the command line is essential, so you're encouraged to find tutorials if you need help getting started.
In your terminal, navigate to the folder you use to store your other code projects. Or, you can navigate to any convenient location in your home directory.
Make a local copy of this repository by typing:
git clone git@github.com:identity-research-lab/tmi-web.git
Navigate to the tmi-web directory with:
cd tmi-web/
The rest of the instructions below assume that you are in the root tmi-web directory.
TMI-Web requires Ruby version 3.2.2 or higher. To check what version of Ruby your computer has installed, type:
ruby -v
Upgrade your Ruby version as needed using the package manager of your choice. If you need help, refer to the Ruby installation guide at https://www.ruby-lang.org/en/documentation/installation/
The first database to set up is the relational database, PostgreSQL. (This is the database that survey responses are stored in.)
You can install and run PostgreSQL using the instructions at https://www.postgresql.org/download/
The next database to set up is the graph database, Neo4j. (This is where interactive personas and identity and experience codes are stored.)
You can install Neo4j from https://neo4j.com/product/neo4j-graph-database/
Once installed, open the Neo4j native application and use the interface to create an empty database called tmi_graph. You will need to keep the Neo4j application open and the tmi_graph database started while running the Rails application locally.
The final database to install is Redis. (Redis is used by TMI-Web as a queue for background jobs.) You can find instructions at https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/
TMI-Web is built on the Ruby on Rails web application framework. It also uses a number of specialized Ruby libraries called gems. To install everything that the web application needs, make sure that you're in the tmi-web directory that you cloned from Github. Then type:
bundle
If you get any errors, you may need to do some searching on the web to fix your installation problems. Read the error message carefully to see which Ruby gem is having trouble being installed, and search for that along with the name and version of your operating system.
Take heart! If everything goes smoothly, you're all set. And if something went wrong, diagnosing the installation will be the hardest part, and it's all smooth going from there.
From the root tmi-web directory, run the following commands:
rake db:drop
rake db:create
rake db:migrate
If you get an error, check that postgresql is running on your system.
From the root directory of tmi-web, type:
rake neo4j:migrate
If you get an error, make sure that Neo4j is running and that the tmi_graph database is started.
Before you start the application for the first time, there are some global environment variables that you need to set. Rails will look for them in a file named .env in the root tmi-web directory.
IMPORTANT! This file will contain your sensitive API keys, so you want to keep it secure. Do not commit the .env file to a source code repository, even a private one.
To create your local .env file, make a copy of .env.example. (This filename starts with a "." so it will be hidden by default. Use ls -a in your terminal to list all files in the directory, including invisible ones.) To make a copy of the example file, type:
cp .env.example .env
Now, when you open the .env file in your text editor, you'll see a list of key/value pairs that need filling in. These are configuration options including database credentials, customizations, and the API keys that TMI-Web needs to communicate with third-party services. Follow the instructions provided in the file to register for the appropriate API keys.
rspec --format documentation spec/
To always run rspec with the documentation flag:
echo "--format documentation" > .rspec
From the root directory of tmi-web, launch Sidekiq by typing:
bundle exec sidekiq
You will see the log files for Sidekiq scroll into view. Sidekiq will continue running in the background until you terminate it with control-c.
To start TMI-web, type
bin/rails server
You should see the output logs from the application begin to scroll into view. The application will continue running in that terminal shell until you interrupt the process (with control-c) or otherwise terminate the application. Shutting down the application can be safely done at any time.
Now that the application has been started, visit http://localhost:3000 in your web browser.
In your browser, use the Upload navigation link. Click on the file selector, and navigate to the "data" directory inside the TMI-Web project directory. Select the file named sample_data.csv and click the Upload and Merge button. This creates 100 survey response records with random text.
Since TMI-Web is designed to merge sets of survey responses, note that you will need to completely reset your database before importing real survey responses. Refer to Prepare the PostgreSQL database above for instructions.
If you forgot to open the Neo4j application and start the tmi_graph database, the Rails application will hang as it tries to connect, and time out after 60 seconds. You'll always need to keep Neo4j running in the background while you're using TMI-Web in your local environment.
You may get an error that has to do with database migrations. This means that your local database is missing updates and changes to the database structure since the last time you ran the application.
To update the Postgresql database, from the root directory of tmi-web type:
rake db:migrate
To update the Neo4j database, type:
rake neo4j:migrate
If you're new to Ruby, but know how to program in another language, there's a great guide to applying what you already know to Ruby at https://www.ruby-lang.org/en/documentation/ruby-from-other-languages/.
The code in TMI-Web is thoroughly commented, and the test suite is not only functional but acts as further documentation. The code has been written to be as simple and readable as possible, sometimes at the expense of elegance or performance.
You can explore code documentation in your browser by typing:
open doc/index.html
Launch the local interactive Rails console:
bin/rails console
At the prompt, type:
Sidekiq.redis(&:flushdb)
To exit the console:
exit
Navigate in your terminal to the db directory and run these commands:
heroku pg:backups:capture --app tmi-web
heroku pg:backups:download --app tmi-web
Navigate in your terminal to root directory of the project and type
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U postgres -d tmi_web_development db/latest.dump
heroku pg:backups:restore --app tmi-web
rake neo4j:migrate
rake export:sample_data
or
rake 'export:sample_data'
rdoc -op doc
There are four types of nodes in the graph database. They all center on the Persona. Each pair of nodes is connected by a uniquely labeled edge.
A Persona is a representation or data avatar of someone who completed the survey.
A Category is a label applied to a group of related Codes within a provided context. For example, a category may refer to a subset of the codes related to "age".
Categories are machine-derived. As such, they are influenced by biases in external training data. Careful human discernment of categories is required to identify and address these biases.
A Code is a label applied to a group of related responses within a provided context. For example, a Code like "self-reflects" may be be applied to one or more responses to the Age Experience question.
Since codes are contextual, they are not unique. If "self-reflects" is coded for both Gender Experience and Age Experience, there will be two distinct Codes, each with the appropriate context.
An Identity is a word or phrase used by a survey participant to self-describe. Identities have associated contexts.
Keywords are the nouns extracted from a 'corpus' consisting of the exact text of certain freeform response fields. The extraction is performed using AI assistance, so results are non-determinate and must be assessed for bias by the researchers.