-
Notifications
You must be signed in to change notification settings - Fork 18
Course utilities
Note: this page is my attempt to document as precisely as possible the codebase of the SRQM course. Developing the course utilities documented below has taught me a fair share of Stata programming, and has allowed me (and others) to run the course rather smoothly over the years, using many different versions of Stata on a vast array of student and university machines. You should probably not be reading this page, unless you are interested in adapting the course to your own needs. Please do not report the insane amount of technical details below to my employer, parents or psychiatrist.
The SRQM course relies on the profile.do
file located at the root of the SRQM
folder, which itself relies on a set of ado-files located in the setup/
folder.
The following 'core' utilities are used to set up Stata for the course:
-
srqm
: check the integrity of the course utilities -
srqm_data
: restore and/or prepare the course datasets -
srqm_grab
: add files to the course material by copying them from a remote source -
srqm_link
: link to theSRQM
folder from the Stata application folder -
srqm_pkgs
: install Stata packages used in the course -
srqm_scan
: check the integrity of the course material (data and code)
Some additional utilities are used for testing the course material:
The [setup/
][setup] folder also contains some utilities that are not prefixed with srqm_*
:
-
debug.do
: a do-file to help students report issues with their setup -
require.do
: a do-file to check whether specific Stata packages are installed -
utils.ado
: various short utilities, some of which are used by other course utilities -
sbar
andstab
: 'simple' utilities targeted at students
Last, the setup/dev/
folder contains even more work-in-progress code, most of which is completely undocumented and will probably never leave that folder.
Notes:
-
All commands above are intended to be run from the
SRQM
folder, within the context of the SRQM course. Some of the utilities not prefixed withsrqm_*
might also be useful outside of the course, even though they have never been released separately from it. -
In Stata, use
which
to get descriptions of each utility, including common usage and optional arguments. Alternatively, typesrqm, v
orsrqm, verbose
to get descriptions for all utilities located in the [setup
][setup] folder.
Note: this section is about the profile.do
file found in the SRQM
root folder. The profile.do
file found in the [setup/
][setup] is a different file: see srqm_link
.
In the first week of class, students are instructed to type run profile
after setting the working directory to the SRQM
folder. This executes the profile.do
file located in the SRQM
folder, which does the following:
- Set the global macros used by the course
- Open a log file at
srqm.log
(see$SRQM_LOGFILE
in the course macros) - Set some Stata system parameters:
memory
(on Stata 11-),maxvars
,more
, etc. - Run
srqm_scan
to check the integrity of the [setup/
][setup],code/
anddata/
course folders - Run
srqm_link
to check or create a link to theSRQM
folder from the Stata application folder - Run
srqm_pkgs
to check that all course-required packages are installed - Print a 'hello' message to welcome students to the course
Notes:
-
The code in
profile.do
is executed on every startup, which ensures that the course material stays complete and functional throughout the semester. If something goes missing at any point, the code will try to restore the files; seesrqm_scan
in particular. -
The code in
profile.do
and insrqm_scan
will generate 'fatal' errors when the course material is incomplete and could not be restored one way or another. This happens from time to time when students try to install the course on a saturated hard drive, which results in only part of the course material being copied.
The following (global) macros/variables are set in profile.do
.
-
$SRQM_WD
: path to theSRQM
folder -
$SRQM_FOLDERS
: names of thecode/
,data/
and [setup/
][setup] folders
-
$SRQM_DATASETS
: names of the datasets stored in thedata/
folder and used by the course do-files -
$SRQM_PACKAGES
: names of the Stata packages used by the course do-files
-
$SRQM_ADOFILES
: paths to all course ado-files- used by
srqm
- used by
-
$SRQM_DOFILES
: paths to all course do-files- used by
srqm
- used by
-
$SRQM_LOGFILE
: relative path to the course log file (opens at startup) -
$SRQM_LOGNAME
: name of the course log file
Those macros are not currently used outside of profile.do
.
Signs the architecture of the SRQM
folder, a.k.a the 'Teaching Pack', by checking that all course utilities are present in the folder, and by printing information related to the course macros.
Notes:
- [TODO]: if utilities are absent, restore them via
srqm_grab
?
Checks that the course datasets exist in the data/
folder.
If a dataset is missing, srqm_data
tries to restore it by looking for a zipped backup copy in that same folder. If no backup is found, it instead looks (in the same folder) for a 'data preparation' do-file named after the dataset, and runs it if it exists.
The 'data preparation' do-files in the data/
folder either download the raw data directly from the source, or do so from a personal access point – using srqm_grab
– when that is not possible. The do-files also download selected data codebooks and perform some minimal data preparation tasks, which vary from one dataset to another.
After running a 'data preparation' do-file, srqm_data
will label and annotate the resulting dataset, remove any empty variables from it, and save it to the data/
folder, along with a zipped backup.
See the course datasets page for further details.
Compatibility:
- All datasets are currently saved in Stata 12+ (
.dta-format 115
) format. - Some datasets might contain more than 2,047 or 2,048 variables, which might be an issue with older versions and/or restricted "flavours" of Stata.
- The command requires an Internet connexion.
Notes:
- [TODO] Also download the dataset
_readme.txt
files if they are missing. - See the following pages on variable restrictions in Small Stata and Stata IC:
- https://www.stata.com/help.cgi?maxvar (Stata 15, max 2,048 for IC)
- https://www.stata.com/help14.cgi?maxvar (max 2,047 for IC)
- https://www.stata.com/help13.cgi?maxvar (max 2,047 for IC, 99 for Small)
- https://www.stata.com/help12.cgi?maxvar (max 2,048 for IC)
- [TODO] Save the course datasets in
- Stata 10/11 (
.dta-format 114
) format, for better backward compatibility? - Stata 8/9 (
.dta-format 113
) format, for maximum backward compatibility?
- Stata 10/11 (
- See the following pages for information on the Stata
.dta-format
:- https://www.loc.gov/preservation/digital/formats/fdd/fdd000471.shtml
- http://www.radyakin.org/transfer/saveto9/cs/index.htm
- https://www.stata.com/help.cgi?dtaversion (Stata 15)
- https://www.stata.com/help.cgi?saveold (Stata 15; can save in Stata 11-14) format)
- https://www.stata.com/help14.cgi?dtaversion
- https://www.stata.com/help14.cgi?saveold (can save in Stata 11-13 format)
- https://www.stata.com/help13.cgi?dtaversion
- https://www.stata.com/help13.cgi?saveold – saves in Stata 12 format
- https://www.stata.com/help12.cgi?saveold – saves in Stata "9/10" (?) format
- https://www.stata.com/help11.cgi?saveold – saves in Stata 8/9 format
[TODO] old text, rewrite
Runs the course do-files, separately or sequentially.
- Use the
s()
option to select the sessions to run, as ins(1/12)
to run the entire course. - Use the
test
option to perform a clean run of the course that tests therequire
package calls - Use the
wipe
option to clean all workfiles at the end of the demo - You can
log
that command by passing ausing
argument to it.
Downloads files from an online access point to the course folders. Sometimes used in class to distribute updated versions of the code, programs and slides for the course.
Compatibility:
- The default access point is set to allow HTTP connexions only, which means that this utility should work with older versions of Stata (12-).
- I was not able to support HTTPS connexions due to my server using an AES encryption scheme that is not supported by Stata 13.
Notes:
- Formerly called
srqm_copy
. - [TOFIX] Type
srqm_get demo.do
for an example.
[TODO] old text, rewrite, and check Stata 12 option (see first note)
Checks whether Stata is being told to automatically run from the SRQM
folder by checking the folder path to the global macro $SRQM_WD
. If this is not the case, the macro is written into a profile.do
file located in the Stata application folder.
The profile.do
file saved to the Stata application folder acts as a symbolic link that also runs the profile.do
file from the SRQM folder, which modifies some Stata settings and performs further integrity checks on the course setup with srqm_scan
and srqm_pkgs
.
Compatibility:
-
This command was written back in the days when Stata 11 (and possibly Stata 12) did not have an option to start in the last session's current working directory.
-
On Windows Vista and above, the command requires running Stata as administrator, or it will fail due to a system restriction in recent versions of Windows.
Checks whether the additional Stata packages used in the course do-files have been installed and installs them if necessary.
Compatibility:
- The command requires an Internet connexion.
- The command relies on the
pkgs
utility located inutils.ado
.
Notes:
- The course packages include the burd graph scheme, which was developed for the course.
- The command tries to run as quickly as possible by skipping packages that are already installed.
- When necessary, the command will try to bypass admin restrictions by cycling through several install locations.
[TODO] old text, rewrite
Checks whether the data/
and code/
folders exist in the SRQM
folder, and restores the course datasets from their ZIP
archives if they have been deleted.
Folder-by-folder details:
-
code
: check all course do-files are present- [TOFIX] if not, tries to download them via
srqm_grab
- [TODO] use GitHub as last-resort access point?
- [TOFIX] if not, tries to download them via
-
data
: check all course datasets are present- if not, unzip the backup copy (routine:
srqm_scan.ado
) - if no backup copy, try to assemble from source (routine:
srqm_data
+ do-files indata/
folder)
- if not, unzip the backup copy (routine:
A very short wrapper around wipe
that cleans up the SRQM
root and code/
folders by erasing temporary files produced by the course do-files (log files and plaintext tables).
Notes:
- The command relies on the
wipe
utility located inutils.ado
.
The following utilities were written to help students with doing very basic things that can end up being painful to code in Stata.
Produces categorical plots that use coloured gradients to draw the bars.
Notes:
- Experimental, not currently used in the course material.
- Relies on (and mostly wraps around) Nick Cox's
catplot
command. - Type
sbar_demo
for an example, or see this blog post from 2013.
Produces summary statistics tables in plaintext format, for maximum compatibility with external editors like Google Documents.
Notes:
- Used in the course do-files for Weeks 5, 9, 10, 11 and 12.
- Relies on Ben Jann's
estout
command. - Type
stab_demo
for an example, or see this blog post from 2013. - A much older version of this utility used to also produce correlation matrices.
A do-file that produces a log file named debug.log
in the SRQM
folder, containing useful information to understand the machine on which it was executed.
This do-file is intended to help students report issues with their setup. It is very rarely used in practice, as most issues with running the course material boil down to either working directory issues or to insufficient hard drive space, which produces trickier issues like missing content and copy/write errors.
A very simple wrapper around srqm_pkgs
that is used in the course do-files.
Towards the top of many course do-files, one will often find a line that reads like:
run setup/require fre spineplot
This line will first run srqm_scan
to check the integrity of the course material, and will then (quietly) check that the fre
and spineplot
Stata packages are installed.
Mostly undocumented short commands:
-
find
: a quickerlookfor
command -
load
: a quickeruse
command -
pkgs
: determines whether a folder can be used to install Stata packages -
science
: a Viennese Easter egg -
updates
: updates all installed packages -
wipe
: clean up temporary files