Skip to content

Course utilities

Jump to bottom Edit New page
François Briatte edited this page Feb 21, 2019 · 16 revisions

Note: this page is my attempt to document as precisely as possible the codebase of the SRQM course. Developing the course utilities documented below has taught me a fair share of Stata programming, and has allowed me (and others) to run the course rather smoothly over the years, using many different versions of Stata on a vast array of student and university machines. You should probably not be reading this page, unless you are interested in adapting the course to your own needs. Please do not report the insane amount of technical details below to my employer, parents or psychiatrist.

Contents

The SRQM course relies on the profile.do file located at the root of the SRQM folder, which itself relies on a set of ado-files located in the setup/ folder.

The following 'core' utilities are used to set up Stata for the course:

  • srqm: check the integrity of the course utilities
  • srqm_data: restore and/or prepare the course datasets
  • srqm_grab: add files to the course material by copying them from a remote source
  • srqm_link: link to the SRQM folder from the Stata application folder
  • srqm_pkgs: install Stata packages used in the course
  • srqm_scan: check the integrity of the course material (data and code)

Some additional utilities are used for testing the course material:

The [setup/][setup] folder also contains some utilities that are not prefixed with srqm_*:

  • debug.do: a do-file to help students report issues with their setup
  • require.do: a do-file to check whether specific Stata packages are installed
  • utils.ado: various short utilities, some of which are used by other course utilities
  • sbar and stab: 'simple' utilities targeted at students

Last, the setup/dev/ folder contains even more work-in-progress code, most of which is completely undocumented and will probably never leave that folder.

Notes:

  • All commands above are intended to be run from the SRQM folder, within the context of the SRQM course. Some of the utilities not prefixed with srqm_* might also be useful outside of the course, even though they have never been released separately from it.

  • In Stata, use which to get descriptions of each utility, including common usage and optional arguments. Alternatively, type srqm, v or srqm, verbose to get descriptions for all utilities located in the [setup][setup] folder.

profile.do

Note: this section is about the profile.do file found in the SRQM root folder. The profile.do file found in the [setup/][setup] is a different file: see srqm_link.

In the first week of class, students are instructed to type run profile after setting the working directory to the SRQM folder. This executes the profile.do file located in the SRQM folder, which does the following:

  1. Set the global macros used by the course
  2. Open a log file at srqm.log (see $SRQM_LOGFILE in the course macros)
  3. Set some Stata system parameters: memory (on Stata 11-), maxvars, more, etc.
  4. Run srqm_scan to check the integrity of the [setup/][setup], code/ and data/ course folders
  5. Run srqm_link to check or create a link to the SRQM folder from the Stata application folder
  6. Run srqm_pkgs to check that all course-required packages are installed
  7. Print a 'hello' message to welcome students to the course

Notes:

  • The code in profile.do is executed on every startup, which ensures that the course material stays complete and functional throughout the semester. If something goes missing at any point, the code will try to restore the files; see srqm_scan in particular.

  • The code in profile.do and in srqm_scan will generate 'fatal' errors when the course material is incomplete and could not be restored one way or another. This happens from time to time when students try to install the course on a saturated hard drive, which results in only part of the course material being copied.

Course macros

The following (global) macros/variables are set in profile.do.

Course folders

Course contents

  • $SRQM_DATASETS: names of the datasets stored in the data/ folder and used by the course do-files
  • $SRQM_PACKAGES: names of the Stata packages used by the course do-files

Course utilities

  • $SRQM_ADOFILES: paths to all course ado-files
  • $SRQM_DOFILES: paths to all course do-files

Course log files

  • $SRQM_LOGFILE: relative path to the course log file (opens at startup)
  • $SRQM_LOGNAME: name of the course log file

Those macros are not currently used outside of profile.do.

SRQM course utilities

srqm

Signs the architecture of the SRQM folder, a.k.a the 'Teaching Pack', by checking that all course utilities are present in the folder, and by printing information related to the course macros.

Notes:

  • [TODO]: if utilities are absent, restore them via srqm_grab?

srqm_data

Checks that the course datasets exist in the data/ folder.

If a dataset is missing, srqm_data tries to restore it by looking for a zipped backup copy in that same folder. If no backup is found, it instead looks (in the same folder) for a 'data preparation' do-file named after the dataset, and runs it if it exists.

The 'data preparation' do-files in the data/ folder either download the raw data directly from the source, or do so from a personal access point – using srqm_grab – when that is not possible. The do-files also download selected data codebooks and perform some minimal data preparation tasks, which vary from one dataset to another.

After running a 'data preparation' do-file, srqm_data will label and annotate the resulting dataset, remove any empty variables from it, and save it to the data/ folder, along with a zipped backup.

See the course datasets page for further details.

Compatibility:

  • All datasets are currently saved in Stata 12+ (.dta-format 115) format.
  • Some datasets might contain more than 2,047 or 2,048 variables, which might be an issue with older versions and/or restricted "flavours" of Stata.
  • The command requires an Internet connexion.

Notes:

srqm_demo

[TODO] old text, rewrite

Runs the course do-files, separately or sequentially.

  • Use the s() option to select the sessions to run, as in s(1/12) to run the entire course.
  • Use the test option to perform a clean run of the course that tests the require package calls
  • Use the wipe option to clean all workfiles at the end of the demo
  • You can log that command by passing a using argument to it.

srqm_grab

Downloads files from an online access point to the course folders. Sometimes used in class to distribute updated versions of the code, programs and slides for the course.

Compatibility:

  • The default access point is set to allow HTTP connexions only, which means that this utility should work with older versions of Stata (12-).
  • I was not able to support HTTPS connexions due to my server using an AES encryption scheme that is not supported by Stata 13.

Notes:

  • Formerly called srqm_copy.
  • [TOFIX] Type srqm_get demo.do for an example.

srqm_link

[TODO] old text, rewrite, and check Stata 12 option (see first note)

Checks whether Stata is being told to automatically run from the SRQM folder by checking the folder path to the global macro $SRQM_WD. If this is not the case, the macro is written into a profile.do file located in the Stata application folder.

The profile.do file saved to the Stata application folder acts as a symbolic link that also runs the profile.do file from the SRQM folder, which modifies some Stata settings and performs further integrity checks on the course setup with srqm_scan and srqm_pkgs.

Compatibility:

  • This command was written back in the days when Stata 11 (and possibly Stata 12) did not have an option to start in the last session's current working directory.

  • On Windows Vista and above, the command requires running Stata as administrator, or it will fail due to a system restriction in recent versions of Windows.

srqm_pkgs

Checks whether the additional Stata packages used in the course do-files have been installed and installs them if necessary.

Compatibility:

  • The command requires an Internet connexion.
  • The command relies on the pkgs utility located in utils.ado.

Notes:

  • The course packages include the burd graph scheme, which was developed for the course.
  • The command tries to run as quickly as possible by skipping packages that are already installed.
  • When necessary, the command will try to bypass admin restrictions by cycling through several install locations.

srqm_scan

[TODO] old text, rewrite

Checks whether the data/ and code/ folders exist in the SRQM folder, and restores the course datasets from their ZIP archives if they have been deleted.

Folder-by-folder details:

  • code: check all course do-files are present
    • [TOFIX] if not, tries to download them via srqm_grab
    • [TODO] use GitHub as last-resort access point?
  • data: check all course datasets are present
    • if not, unzip the backup copy (routine: srqm_scan.ado)
    • if no backup copy, try to assemble from source (routine: srqm_data + do-files in data/ folder)

srqm_wipe

A very short wrapper around wipe that cleans up the SRQM root and code/ folders by erasing temporary files produced by the course do-files (log files and plaintext tables).

Notes:

  • The command relies on the wipe utility located in utils.ado.

'Simple' utilities

The following utilities were written to help students with doing very basic things that can end up being painful to code in Stata.

sbar

Produces categorical plots that use coloured gradients to draw the bars.

Notes:

stab

Produces summary statistics tables in plaintext format, for maximum compatibility with external editors like Google Documents.

Notes:

Other utilities

debug.do

A do-file that produces a log file named debug.log in the SRQM folder, containing useful information to understand the machine on which it was executed.

This do-file is intended to help students report issues with their setup. It is very rarely used in practice, as most issues with running the course material boil down to either working directory issues or to insufficient hard drive space, which produces trickier issues like missing content and copy/write errors.

require.do

A very simple wrapper around srqm_pkgs that is used in the course do-files.

Towards the top of many course do-files, one will often find a line that reads like:

run setup/require fre spineplot

This line will first run srqm_scan to check the integrity of the course material, and will then (quietly) check that the fre and spineplot Stata packages are installed.

utils.ado

Mostly undocumented short commands:

  • find: a quicker lookfor command
  • load: a quicker use command
  • pkgs: determines whether a folder can be used to install Stata packages
  • science: a Viennese Easter egg
  • updates: updates all installed packages
  • wipe: clean up temporary files
Add a custom footer
Add a custom sidebar
Clone this wiki locally