Getting up and running with the basic Stroop task
Perhaps the best way to learn about psiTurk is to go through the steps of configuring and running an experiment. This tutorial will take you through the steps required to run the basic Stroop experiment that ships default with psiTurk. This project can be a great starting place for developing your own experiment.
This guide assumes that you have already successfully installed psiTurk.
Currently, psiTurk shell is only available on the dev branch. To install from dev, issue the following commands in your terminal:
$ git clone -b dev git@github.com:NYUCCL/psiTurk.git
$ cd psiTurk
$ sudo python setup.py install*Note, if you've installed psiTurk in the past you'll first need to run pip uninstall psiturk.
The Stroop effect is the finding that people show interference from reading while naming the font color of words. The task is used to suggest that reading has become a highly "automatic" cognitive skill. You can read more about the Stroop task here. This guide won't comment much on the psychology of it, rather focusing on the technical aspect of running such an experiment online that consists of a sequence of trials and which records response time and key presses.
The first step is to obtain the archive of code and resources specific to the Stroop demo. In the near future, lots of additional experiments will be shared on the psiTurk experiment exchange. However, the Stroop demo comes bundled within the psiturk command line tool.
First use the psiturk-setup-example command to place fresh copies of the files into a new folder:
$ psiturk-setup-exampleafterward you should have a new folder in the current directory names "psiturk-example" with the following listing of files:
$ cd psiturk-example
$ ls -la
total 16
drwxr-xr-x 7 gureckis staff 238 Oct 18 22:05 .
drwx------ 47 gureckis staff 1598 Oct 18 22:05 ..
-rw-r--r-- 1 gureckis staff 748 Oct 18 22:05 config.txt
drwxr-xr-x 7 gureckis staff 238 Oct 18 22:05 static
drwxr-xr-x 14 gureckis staff 476 Oct 18 22:05 templatesOne of the files generated by psiturk-setup-example is the config.txt file, which contains a variety of experiment and server parameters. These values can be changed by altering the file in any text editor.
The default config.txt file is already mostly configured to help you test the Stoop demo. Three options you might want to adjust to begin with are:
- Enter your AWS info in the "AWS Access" section. It's possible to skip this step for now, but you will only be able to test the experiment locally.
- In the "Server" section ensure that the port listed is one that is available on your computer (answer is usually yes unless you have particular firewall software running).
- In the "Server" section ensure that the host is either localhost (if just testing/debugging) or set to your current IP address (if planning to test live on the AMT site).
All user commands to psiTurk, such as creating a HIT, launching the experiment server, or approving workers, are issued through the psiTurk shell. To open the shell, run psiturk in the current folder. You should see something like this (though probably colorized on your display):
$ psiturk
psiTurk version 1.0.3dev
Type "help" for more information.
[psiTurk server:off mode:sdbx #HITs:0]$ The psiTurk shell prompt displays several useful pieces of information: whether the experiment server is on, whether you are in sandbox or live mode, and how many hits are online in your current mode (more on all of these below). While in the psiTurk shell, all commands entered will be executed by psiTurk. To exit the shell, type quit.
Starting the experiment server is as simple as typing server on:
[psiTurk server:off mode:sdbx #HITs:0]$ server on
Experiment server launching...
Now serving on http://localhost:22362
[psiTurk server:on mode:sdbx #HITs:0]$ Note that the command prompt has changed from showing server:off to server:on (and also changed form red to green on colorized terminals). You can start or stop the server at any time using the server on and server off commands. Typically you want to have the server running when you are testing locally, testing on the AMT "sandbox", or running your actual experiment. If the server stops when running your actual experiment, Internet users will no longer be able to participate in your experiment even if you still have HITs posted on AMT's website. Thus, you should think of the experiment server as meaning you experiment is current "live."
Frequently you would like to test your experiment in your browser locally without involving Amazon's servers at all. To do so, ensure that the experiment server is running (the prompt should show server:on). Then enter the command debug. A new browser tab will open with the first screen of the experiment. The URL string for this will look something like this:
http://localhost:22362/ad?assignmentId=debug1382500678820&hitId=debug1382500678820&workerId=debug1382500678820#
The http://localhost:22362/ part is set in the configuration options under "Server Parameters" in the fields "host" and "port". The default value, http://localhost:22362/ is a special term that refer to your own computer. As mentioned above, if you wanted to run this experiment publically you would want to change the host option to the public hostname or ip address of your computer. For example, if your laptop or desktop computer had a static ip address of research.uni.edu, you'd want to change the host to research.uni.edu.
The remaining part of the URL created random (i.e., fake) identifiers which stand-in for the values that Amazon provides identifying the user, hit, etc... Since by default psiTurk does not allow individuals to take the same experiment more than once (it checks for you to see if the worker has already completed the task or read too far into the instructions) these random values are helpful during debugging.
The first page that you see in the experiment looks something like this:
This is the page the AMT worker would see when they first accept the hit. When you click the link, a full screen window will open up which will run the experiment. You can test it now if you like just to get a sense of things. If you want to stop midway through that is no problem. Just close that browser window. Running debug again will open a new browser window and let you repeat the process.
Now that you've tested the experiment locally, you may want to see how it would appear on mturk before running it live with paid workers. Amazon offers this ability through the worker sandbox, a simulated environment that allows developers to test their HITs.
To create a hit in the worker sandbox, first check that the server is on and that you are in sandbox mode; the psiTurk prompt should say on next to server and sdbx next to mode. If you are in live mode, enter the command mode to switch to sandbox mode.
To create a hit, enter the command create_hit, and then answer the prompts to set up the HIT. Your choices for the prompt answers are arbitrary for now, since the HIT won't be completed by real workers:
[psiTurk server:on mode:sdbx #HITs:0]$ create_hit
number of participants? 10
reward per HIT? 1.00
duration of hit (in hours)? 4
*****************************
Creating sandbox HIT
HITid: 28K4SMEZZ2MI004SE8G9138T8GVL4Z
Max workers: 10
Reward: $1.00
Duration: 4 hours
Fee: $1.00
________________________
Total: $11.00
Ad for this HIT now hosted at: https://psiturk.org/ad/65?assignmentId=debugNJ7917&hitId=debugUD88IS
[psiTurk server:on mode:sdbx #HITs:1]$ You can also run create_hit non-interactively by providing arguments when you run the command, for example create_hit 10 1.00 4
You should now see the number "1" next to "#HITs:" in the psiTurk prompt, denoting that you have one active HIT in the worker sandbox. If you type the command list_active_hits, you should see a description of your HIT.
Note that the "ad" for the hit is hosted on https://psiturk.org/. This reflects a very useful feature of psiTurk since ad will not display properly for all browsers if not served using a certified SSL certificate. Our Ad Server lets you host a secure ad on a different computer than the local server, but one that has a valid SSL certificate.
To test your HIT, go to the worker sandbox and search for your HIT by entering the name of your requester account in the search bar. You should see something like this:
Click "view a HIT in this group" to open a hit. You should see an ad for your HIT appear on the screen. Click "accept HIT", then click the link in the HIT ad to open the experiment in a full-screen window.