Merge pull request #170 from askap-vast/notebook-v2.0

v2.0

BUILDLIGHTCURVES.md

@@ -1,26 +1,21 @@
 # build\_lightcurves.py
 
-This script allows you to quickly build lightcurves of sources you have extracted using `find_sources.py`.
+This script allows you to quickly build lightcurves of sources you have extracted using `find_sources.py`. While in v2.0 `find_sources.py` can now output lightcurve plots directly, this script can be helpful to run the lightcurve plotting again on a complete `find_sources.py` output.
 
-The script will output a csv file containing:
-* The start and end datetimes of the observation
-* Peak (default) or integrated flux density and associated uncertainty.
-* The image noise in the local region
-* A flag stating whether the measurement is a detection or an upper limit.
-
-By default the script will also plot the lightcurve, although this can be disabled using the `--no-plotting` flag. Also peak fluxes are used by default, integrated fluxes can be used by using the `--use-int-flux` flag.
+Peak fluxes are used by default, integrated fluxes can be used by using the `--use-int-flux` flag.
 
 # Running the script
-Prior to running this script you should query the survey data using `find_sources.py` with the `--vast-pilot` flag set to `all`. Then run `build_lightcurves.py FOLDER` where `FOLDER` is the output folder of the previous query.
+Prior to running this script you should query the survey data using `find_sources.py` with the `--vast-pilot` flag set to your desired epochs. Then run `build_lightcurves.py FOLDER` where `FOLDER` is the output folder of the previous query.
 
 ## Usage
 
-Most options should be self explanatory. The lightcurve plots and csv files are saved in the same directory as the input
+Most options should be self explanatory. The lightcurve plots are saved in the same directory as the input.
 
 ```
-usage: build_lightcurves.py [-h] [--use-int-flux] [--no-plotting] [--quiet] [--debug] [--min-points MIN_POINTS]
-                            [--min-detections MIN_DETECTIONS] [--mjd] [--grid] [--yaxis-start {auto,0}]
-                            [--nice NICE]
+usage: build_lightcurves.py [-h] [--use-int-flux] [--quiet] [--debug] [--min-points MIN_POINTS]
+                            [--min-detections MIN_DETECTIONS] [--mjd] [--grid]
+                            [--yaxis-start {auto,0}] [--use-forced-for-limits] [--use-forced-for-all]
+                            [--hide-legend] [--nice NICE]
                             folder
 
 positional arguments:
@@ -29,7 +24,6 @@ positional arguments:
 optional arguments:
   -h, --help            show this help message and exit
   --use-int-flux        Use the integrated flux, rather than peak flux (default: False)
-  --no-plotting         Write lightcurves to file without plotting (default: False)
   --quiet               Turn off non-essential terminal output. (default: False)
   --debug               Turn on debug output. (default: False)
   --min-points MIN_POINTS
@@ -39,7 +33,12 @@ optional arguments:
   --mjd                 Plot lightcurve in MJD rather than datetime. (default: False)
   --grid                Turn on the 'grid' in the lightcurve plot. (default: False)
   --yaxis-start {auto,0}
-                        Define where the y axis on the lightcurve plot starts from. 'auto' will let matplotlib
-                        decide the best range and '0' will start from 0. (default: 0)
+                        Define where the y axis on the lightcurve plot starts from. 'auto' will let
+                        matplotlib decide the best range and '0' will start from 0. (default: 0)
+  --use-forced-for-limits
+                        Use the forced fits values instead of upper limits. (default: False)
+  --use-forced-for-all  Use the forced fits for all datapoints. (default: False)
+  --hide-legend         Don't show the legend on the final plot. (default: False)
   --nice NICE           Set nice level. (default: 5)
+
 ```

MANIFEST.in

@@ -1 +1,2 @@
-include vasttools/data/*
+recursive-include vasttools/data/ *
+recursive-include notebook-examples/ *

PILOTFIELDSINFO.md

@@ -41,23 +41,25 @@ See the section below for details on the other options available when running.
 Most options should be self explanatory. The lightcurve plots and csv files are saved in the same directory as the input
 
 ```
-usage: pilot_fields_info.py [-h] [--psf] [--common-psf] [--all-psf] [--save] [--quiet] [--debug]
-                            [--nice NICE]
+usage: pilot_fields_info.py [-h] [--psf] [--largest-psf] [--common-psf] [--all-psf] [--save] [--quiet]
+                            [--debug] [--nice NICE]
                             fields [fields ...]
 
 positional arguments:
-  fields        Fields to query (or csv file containing fields).
+  fields         Fields to query (or csv file containing fields).
 
 optional arguments:
-  -h, --help    show this help message and exit
-  --psf         Include the largest PSF of the 36 beams that make up the field. (default: False)
-  --common-psf  Include the common PSF of the 36 beams that make up the field. (default: False)
-  --all-psf     Include all the PSF information for the field. (default: False)
-  --save        Save the resulting information. Files will be saved to the current working directory
-                in the form of 'VAST_XXXX+/-XXA_field_info.csv'." (default: False)
-  --quiet       Turn off non-essential terminal output. (default: False)
-  --debug       Turn on debug output. (default: False)
-  --nice NICE   Set nice level. (default: 5)
+  -h, --help     show this help message and exit
+  --psf          Include the used PSF of the 36 beams that make up the field. Usually set from beam
+                 00. (default: False)
+  --largest-psf  Include the largest PSF of the 36 beams that make up the field. (default: False)
+  --common-psf   Include the common PSF of the 36 beams that make up the field. (default: False)
+  --all-psf      Include all the PSF information for the field. (default: False)
+  --save         Save the resulting information. Files will be saved to the current working directory
+                 in the form of 'VAST_XXXX+/-XXA_field_info.csv'. (default: False)
+  --quiet        Turn off non-essential terminal output. (default: False)
+  --debug        Turn on debug output. (default: False)
+  --nice NICE    Set nice level. (default: 5)
 ```
 
 ## Example

README.md

@@ -21,23 +21,24 @@ There is a requirements.txt included in the repository that you can use to insta
 pip install -r requirements.txt
 ````
 
+## Notebook Usage
+
+As of version v2.0 the module has been made 'notebook friendly' and can be used interactively. See the `notebook-examples` directory in this repository for examples on how to use the module in a notebook environment. Note that for large queries it is better to use the `find_sources.py` script as pre version v2.0.
+
+## System Variables
+
+To save specifying the data directories in every call to VAST tools there are two system variables you can set that will be read by the module:
+
+* `VAST_DATA_DIR`: The path to the VAST Pilot data, i.e. the path which contains the `EPOCHXX` folders.
+* `PIPELINE_WORKING_DIR`: The path to the VAST Pipeline directory containing the pipeline runs. 
+
 ## Current Scripts
 As part of the installation the following scripts are made available in your path:
 
-* **get\_vast\_pilot\_dbx.py** - A script to allow simpler downloading of the VAST Pilot survey from Dropbox.
-    - See [VASTDROPBOX.md](VASTDROPBOX.md) for full instructions.
 * **pilot\_fields\_info.py** - A script to get basic information of VAST Pilot survey fields.
     - See [PILOTFIELDSINFO.md](PILOTFIELDSINFO.md) for full instructions.
 * **find\_sources.py** - A tool to swiftly search VAST Pilot data at chosen coordinates (also supports RACS if available).
-    - See [FINDSOURCES.md](FINDSOURCES.md) for full instructions.
-* **build\_lightcurves.py** - A script to allow easy creation of source lightcurves.
+    - See [FINDSOURCES.md](FINDSOURCES.md) for full instructions and options.
+* **build\_lightcurves.py** - A script to allow easy creation of source lightcurves on output of `find_sources.py`.
+    - As of v2.0 `find_sources.py` can also output lightcurves. This script can be useful if you'd like to regenerate them using different settings.
     - See [BUILDLIGHTCURVES.md](BUILDLIGHTCURVES.md) for full instructions.
-
-## Workflow Example
-
-This is an example of a basic workflow to query three sources of interest. The steps are:
-
-1. Determine which VAST Pilot fields contain the sources of interest. Do this using the `--find-fields ` mode in `find_sources.py` (see [FINDSOURCES.md](FINDSOURCES.md)).
-2. Download the required fields by passing the above output to `get_vast_pilot_dbx.py` (see [VASTDROPBOX.md](VASTDROPBOX.md)).
-3. Run `find_sources.py` for a second time, making sure to point to the newly downloaded data above, to gather information and cut outs of the sources and their crossmatches (see [FINDSOURCES.md](FINDSOURCES.md)).
-4. Run `build_lightcurves.py` on the output directory to create lightcurves of the sources (see [BUILDLIGHTCURVES.md](BUILDLIGHTCURVES.md)).