united Project

United For Iran prisoner database web site.

Dependencies

• OpBeat:
• Google Analytics:

Monitoring / Administration

• Django Admin
• OpBeat

Development

Code Location

ssh://git@github.com/smallmedia/united.git

Development Prerequisites

• Install the latest VirtualBox 5.0.x.
• Install Vagrant
• sudo pip install 'ansible<2.1'

Initial Database Preparation

To get setup you'll need to have a wordpress dump from United from Iran - serve this database using mysql and check that you're able to connect correctly. Django uses a standard postgres database.

To migrate, run the commands below.

There's a print out but essentially the first command will create all the datatypes as well as create judges, courts and the relationship between judges and courts. The second command will actually create the prisoners and their relationship to the data. It also handles the mapping from the old data to the new data as well as creating a report which flags certain profiles for further attention.

python manage.py core_types

python manage.py data_migration

Be aware that data_migration pulls all the images from their live server which can slow their site down somewhat.

Acquire Database Backup

• SSH into the production machine (details below).
• sudo su - ipa
• Backup the PostgreSQL database: PGPASSWORD="$DB_PASSWORD" pg_dump --host=$DB_HOST -U $DB_USER $DB_NAME > ~/db.sql -p 5433
• Copy it to your local machine

Development Setup

• Acquire a copy of the most recent database and media directory backups.
• Create and provision the new Vagrant box: vagrant up
• Open a spare terminal and cd back to this project directory.
• SSH into the Vagrant box: vagrant ssh
• Drop the stub database generated by provisioning: PGPASSWORD="$DB_PASSWORD" dropdb -U $DB_USER $DB_NAME
• Create a new empty database: PGPASSWORD="$DB_PASSWORD" createdb -U $DB_USER $DB_NAME
• Restore the real DB dump: PGPASSWORD="$DB_PASSWORD" psql -U $DB_USER $DB_NAME -f db.sql
• Create a user account: python manage.py createsuperuser
• Start the Django development webserver: python manage.py runserver 0:8000
• Visit http://192.168.33.10:8000/admin/ and login with your new user.
• Verify you can now visit http://192.168.33.10:8000/ and the site is presented.
• Run gulp dev for both frontend and dashboard watching and concat.
• Run gulp for both frontend and dashboard concat and minify js and css.
• Edit files.
• Commit!

Running Unit Tests

• DJANGO_SETTINGS_MODULE=ipa.test_settings ./manage.py test

Staging

Servers

• 52.213.110.153

Application UNIX Account

• united

Login with your personal UNIX account then sudo to the application account.

Directories

• /webapps/united

Deploying A New Version

• cd deploy
• ansible-playbook staging.yml -t deploy

Production

DNS Domains

• https://ipa.united4iran.org (DNS not hosted by us)

SSL Certificates

• Lets Encrypt

Servers

• sm-app-j.positive-dedicated.net (requires VPN for SSH)

Support Contacts

• support@positive-internet.com

Deploying A New Version

• cd deploy
• ansible-playbook production.yml -t deploy

Restarting The Site

• sudo supervisorctl restart united

Running Django Management Commands

• Login to sm-app-f.positive-dedicated.net using your personal UNIX account.
• sudo su - united
• cd united
• python manage.py <command>

Cron

Update Prisoners CSV File

30 6,18 * * * . /webapps/united/bin/activate && . /webapps/united/bin/postactivate && cd /webapps/united/united && python manage.py generate_prisoners_csv --settings=united.settings.production >/dev/null

API

1. There is a single common api.viewsets.ModelViewSet which overrides the default rest_framework ModelViewSet to implement automatic search, filtering, and access control.

2. Items are invisible to anonymous users if their model class has the core_types.models.PublishableMixin, and its is_published field is False.

3. Items cannot be marked published unless the requesting user is logged in, and a member of the Publishing Managers group (settings.APP_SUPERUSER_GROUP_NAME). This group is automatically created by core_types/migrations/0005_create_publisher_managers_group.py group.

4. ViewSet list operations can be filtered using the "q" query string parameter. Strings passed here are matched against the list of fields from the ViewSet's search_fields attribute. That means the frontend only needs to know about the "q"` parameter, and allows us to tweak how searching works on the backend without having to change the frontend later.

5. All ViewSets support "?limit=N" and "?offset=N" parameters. The default limit is 25.

6. All ViewSets support ordering by any supported attribute, e.g. ?ordering=-name\_en. See http://www.django-rest-framework.org/api-guide/filtering/#orderingfilter

7. All ViewSets support an optional "?verbose=1" parameter which causes the output to include created_by and updated_by user fields. These may be necessary later on, but they're turned off by default to save on SQL/response time, since they're present on every object in the database.

Prisoner Filters

The /api/prisoners/prisoner/ collection additionally supports the following filter GET parametrs:

Query	Description
?gender=M	Gender
?is_detained=0, ?is_detained=1	Detention status
?religion=Shiaa, ?religion=شیعه	Religion (English or Farsi input)
?ethnicity=Fars, ?ethnicity=فارس	Ethnicity (English or Farsi input)
?activity_persecuted_for=Student Activist	Activity persecuted for EN
?activity_persecuted_for=فعال دانشجوئی	Activity persecuted for FA
?has_comments=0, ?has_comments=1	Has comments?
?arrest_year_min=0, ?arrest_year_max=9999	Arrested within year range

File Upload

For judges/pictures/NN/ and prisoners/pictures/NN/, these endpoints expect a regular HTML form containing a file attachment. The file variable can be named anything.

CSV Export

The /api/prisoners/prisoner/ endpoint additionally supports a CSV output mode. In this mode, pagination is disabled (i.e. the full data set is returned), though filters are still applied. To export a CSV, use:

/api/prisoners/prisoner/.csv

The request takes around 30 seconds, due to the overhead of slurping down basically the entire DB.