-
Notifications
You must be signed in to change notification settings - Fork 4
Syncing smoke test
Tom Aldcroft edited this page Aug 6, 2019
·
7 revisions
This shows steps to demonstrate the installation and basic functionality of maintaining a copy of the cheta archive using the new sync infrastructure.
The steps are written for Mac/linux, and probably need some translation into Window-ese.
This was done using 0043f31.
In your usual platform-dependent fashion, install the master branch of
the eng_archive repo to a local dev installation of Ska3.
NOTE: this branch is currently installed on ska3-test on HEAD linux (/proj/sot/ska3/test).
This includes only a few MSIDs from the acis4eng content type.
(ska3-test) neptune$ cd tmp # or wherever
(ska3-test) neptune$ mkdir test_sync
(ska3-test) neptune$ cd test_sync
<HOST> below can be either `chimchim` or `kadi` or `ccosmos`.
(ska3-test) neptune$ rsync -av <HOST>:/proj/sot/ska/data/cheta/test_sync/ ./
aldcroft@kadi.cfa.harvard.edu's password:
receiving file list ... done
./
data/
data/acis4eng/
data/acis4eng/1PIN1AT.h5
data/acis4eng/1WRAT.h5
data/acis4eng/1WRBT.h5
data/acis4eng/TIME.h5
data/acis4eng/archfiles.db3
data/acis4eng/colnames.pickle
data/acis4eng/5min/
data/acis4eng/5min/1PIN1AT.h5
data/acis4eng/5min/1WRAT.h5
data/acis4eng/5min/1WRBT.h5
data/acis4eng/daily/
data/acis4eng/daily/1PIN1AT.h5
data/acis4eng/daily/1WRAT.h5
data/acis4eng/daily/1WRBT.h5
sent 314 bytes received 68878288 bytes 3935920.11 bytes/sec
total size is 68868941 speedup is 1.00
Normally cheta / Ska.engarchive looks in $SKA/data/eng_archive for data,
but we set an environment variable to point it elsewhere.
(ska3-test) neptune$ pwd
/Users/aldcroft/tmp/test_sync
(ska3-test) neptune$ export ENG_ARCHIVE=$PWD
Also check that cheta and Ska.engarchive both import OK and have the
right version.
(ska3-test) neptune$ ipython --matplotlib
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.
Using matplotlib backend: MacOSX
In [1]: import Ska.engarchive
In [2]: Ska.engarchive.__version__
Out[2]: '4.45'
In [3]: import cheta
In [4]: cheta.__version__
Out[4]: '4.45'
In [5]: from Ska.engarchive import fetch
fetch: using ENG_ARCHIVE=/Users/aldcroft/tmp/test_sync for archive path
In [6]: fetch.get_time_range('1wrat', format='fits')
Out[6]: ('1999-12-31T22:41:37.260', '2019-07-20T00:18:20.222')
In [7]: dat = fetch.Msid('1wrat', '2019-07-18')
In [8]: dat.plot()
In [9]: dat = fetch.Msid('1wrat', '2019-07-18', stat='5min')
In [10]: dat = fetch.Msid('1wrat', '2019-07-18', stat='daily')
You need to be on the OCC or SI VPN to see icxc. The code can also do the update using a
sync repository on local disk, but in practice this is mostly just for testing.
For stats data (5min, daily), for the first update it digs back a little in time to make sure it has overlap. Later once it knows (and remembers) which sync files it has applied then it only downloads what is necessary.
The --date-stop argument is just for testing, in practice one would almost always
not supply this, in which case it applies all available sync data to bring the local
archive up to date.
(ska3-test) neptune$ python -m cheta.update_client_archive --date-stop 2019-07-22 --log-level=20
fetch: using ENG_ARCHIVE=/Users/aldcroft/tmp/test_sync for archive path
2019-08-03 13:07:59,216
2019-08-03 13:07:59,216 Processing full data for acis4eng
2019-08-03 13:07:59,390 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:07:59,924 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-20T0018z/full.pkl.gz
2019-08-03 13:08:00,163 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-21T1339z/full.pkl.gz
2019-08-03 13:08:00,232
2019-08-03 13:08:00,232 Processing 5min data for acis4eng
2019-08-03 13:08:00,310 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:08:00,433 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-15T1209z/5min.pkl.gz
2019-08-03 13:08:00,598 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-17T0147z/5min.pkl.gz
2019-08-03 13:08:00,784 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-18T1323z/5min.pkl.gz
2019-08-03 13:08:00,917 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-20T0018z/5min.pkl.gz
2019-08-03 13:08:01,046 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-21T1339z/5min.pkl.gz
2019-08-03 13:08:01,085
2019-08-03 13:08:01,085 Processing daily data for acis4eng
2019-08-03 13:08:01,176 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:08:01,325 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-15T1209z/daily.pkl.gz
2019-08-03 13:08:01,463 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-17T0147z/daily.pkl.gz
2019-08-03 13:08:01,638 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-18T1323z/daily.pkl.gz
2019-08-03 13:08:01,813 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-20T0018z/daily.pkl.gz
2019-08-03 13:08:02,007 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-21T1339z/daily.pkl.gz
(ska3-test) neptune$ ipython --matplotlib
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.
Using matplotlib backend: MacOSX
In [1]: from cheta import fetch
fetch: using ENG_ARCHIVE=/Users/aldcroft/tmp/test_sync for archive path
In [2]: fetch.get_time_range('1wrat', format='fits')
Out[2]: ('1999-12-31T22:41:37.260', '2019-07-23T01:01:49.837')
In [3]: dat = fetch.Msid('1wrat', '2019-07-18')
In [4]: dat.plot()
This time it only downloads the required data. (The fact that it reads the same index file 3 times will be fixed).
(ska3-test) neptune$ python -m cheta.update_client_archive --date-stop 2019-07-24 --log-level=20
fetch: using ENG_ARCHIVE=/Users/aldcroft/tmp/test_sync for archive path
2019-08-03 13:10:37,412
2019-08-03 13:10:37,412 Processing full data for acis4eng
2019-08-03 13:10:37,516 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:10:37,977 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-23T0102z/full.pkl.gz
2019-08-03 13:10:38,046
2019-08-03 13:10:38,046 Processing 5min data for acis4eng
2019-08-03 13:10:38,134 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:10:38,240 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-23T0102z/5min.pkl.gz
2019-08-03 13:10:38,284
2019-08-03 13:10:38,284 Processing daily data for acis4eng
2019-08-03 13:10:38,370 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:10:38,470 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-23T0102z/daily.pkl.gz
This is the normal calling pattern. Note the use of the command line script
cheta_update_client_archive that is installed in the same path with python or ipython.
(ska3-test) neptune$ cheta_update_client_archive --log-level=20
2019-08-03 13:11:27,211
2019-08-03 13:11:27,211 Processing full data for acis4eng
2019-08-03 13:11:27,327 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:11:27,741 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-24T1328z/full.pkl.gz
2019-08-03 13:11:27,993 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-26T0139z/full.pkl.gz
2019-08-03 13:11:28,205 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-27T1347z/full.pkl.gz
2019-08-03 13:11:28,430 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-29T0125z/full.pkl.gz
2019-08-03 13:11:28,688 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-30T1330z/full.pkl.gz
2019-08-03 13:11:28,911 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-01T0026z/full.pkl.gz
2019-08-03 13:11:29,110 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-02T0348z/full.pkl.gz
2019-08-03 13:11:29,165
2019-08-03 13:11:29,165 Processing 5min data for acis4eng
2019-08-03 13:11:29,244 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:11:29,340 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-24T1328z/5min.pkl.gz
2019-08-03 13:11:29,475 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-26T0139z/5min.pkl.gz
2019-08-03 13:11:29,607 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-27T1347z/5min.pkl.gz
2019-08-03 13:11:29,743 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-29T0125z/5min.pkl.gz
2019-08-03 13:11:29,890 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-30T1330z/5min.pkl.gz
2019-08-03 13:11:30,101 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-01T0026z/5min.pkl.gz
2019-08-03 13:11:30,260 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-02T0348z/5min.pkl.gz
2019-08-03 13:11:30,301
2019-08-03 13:11:30,301 Processing daily data for acis4eng
2019-08-03 13:11:30,391 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/index.ecsv
2019-08-03 13:11:30,478 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-24T1328z/daily.pkl.gz
2019-08-03 13:11:30,715 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-26T0139z/daily.pkl.gz
2019-08-03 13:11:30,918 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-27T1347z/daily.pkl.gz
2019-08-03 13:11:31,147 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-29T0125z/daily.pkl.gz
2019-08-03 13:11:31,329 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-07-30T1330z/daily.pkl.gz
2019-08-03 13:11:31,513 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-01T0026z/daily.pkl.gz
2019-08-03 13:11:31,684 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis4eng/2019-08-02T0348z/daily.pkl.gz