-
Notifications
You must be signed in to change notification settings - Fork 0
/
dataproducts.tex
190 lines (132 loc) · 11.1 KB
/
dataproducts.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
\section{Data Products} \label{sec:dataproducts}
Rubin Observatory’s LSST Science Pipelines (\S~\ref{sec:softproducts}) will produce the \emph{science-ready data products}.
These data products have been carefully designed to enable the vast majority of LSST science without the need to access the raw pixels, nor for users to reprocess the data.
There will however be some science cases where pixel access or a reprocessing of the data is warranted are, such as estimating and subtracting a different background (LSB science), reprocessing a small fraction of images to develop the systematics budget for weak lensing studies (Dark Energy science), or injecting fake objects into images and reprocessing them to develop models for artifact rejection.
In all such cases involving image reprocessing, we anticipate that users will start from images that have been corrected for instrumental effects and photometrically and astrometrically calibrated.
The Data Products Definition Document, \citep{LSE-163} was used to describe the data products produced by the LLSST and guide the development of the Data Management System.
In this section we provide a high-level overview of the LSST science-ready data products.
A detailed description of the LSST data products and their scientific performance on the early LSST commissioning data is given in \citep{PSTN-024}.
% Types of data product
% -- the different types
% Categories of data product
% -- the different categories are designed to address different goals, and are delivered on a range of cadences.
%Describe the science that they will enable - i.e why are we creating these data products?
%How are they created, how are they distributed and or served
\subsection{Types of Data Product} \label{sec:dp-types}
LSST produces several types of data products.
\paragraph {\tt Images}~
processed visit images (PVI) are images that have been corrected for instrumental effects and photometrically and astrometrically calibrated.
raw single visit images, calibrated processed visit images (PVI), coadd images, cutouts (postage stamps)
Rubin images are rich data products, which, in addition to storing the image pixel data also contain the PSF model, WCS and mask plane, ... what else
Include a description of cutout images and how they will be accessed
What is the maximum size of a cutout, how many at a time?
Image data products also includes calibration frames (darks, flats, biases, fringe, etc.)
coadds -- We reiterate that not all coadds will be kept and served to the public
template coadds\
RGB color images derived from coadds
All calibration frames (darks, flats, biases, fringe, etc.) will be preserved and made available.
Provide the full list of calibration images and the data products that come out of cp\_pipe.
\paragraph {\tt Spectra}~
AuxTel data ...
All auxiliary telescope data, both raw (images with spectra) and processed (calibrated spectra,
derived atmosphere models), will be preserved and made available for download.
\paragraph {\tt Catalogs}~
DR includes Object, Source, DIASource, DIAObject,
Object
`
Source
ForcedSource
ShearObject
\paragraph {\tt Alerts}~
A composite data product that includes image cutouts (postage stamps) and extracts of catalog data.
Alerts packets are distributed via the alert distribution system (\S~ref), one alert for each object that has changed in brightness or position on the sky.
% 5Sigma alerts
% Sub-threshold alerts
In addition to the alerts detected on DIASources above the nominal detection threshold of 5$\sigma$, we also measure and store a small sample of DIASources detected the nominal 5$\sigma$ threshold.
There are several drivers for these \emph{sub-threshold alerts}, for example to enable monitoring of difference image analysis quality or to assess the danger posed by a potentially hazardous asteroid.
A set of criteria, described in \citep{dmtn-228} was defined based on key science cases.
\paragraph {\tt Calibration Data Products} ~
\paragraph {\tt Survey Property Maps}~
Several types of survey property maps will be generated and served to users.
The properties are typically the mean or total values determined from the images input to generate the deep coadd.
The types of maps will include the total exposure time; the point-source 5-sigma AB magnitude limit; the weighted mean of the PSF moments; the weighted mean of the sky background and sky noise; and the average effect of differential chromatic refraction (DCR) in the
right ascension and declination directions, and in the PSF moments.
Property maps based on statistics measured on deep coadds might also be generated.
\subsection{Categories of Data Product} \label{sec:dp-categories}
LSST defines three main categories of data products to be served by Rubin.
The different categories are designed to enable different types of science.
Each category of data product may comprise any or all of the data product types described in \S~\ref{sec:dp-categories}.
% For each category, describe 1) the science that they enable, 2) how they are produced, 3)what the data products in each category are and 4) how and on what latency they are served. }
% Some general comment somewhere about the various metadata products that are also produced during nightly processing and made available to users.
\subsubsection{Prompt data products} \label{sec:dp-prompt}
Prompt data products are designed to enable time domain science, the rapid discovery, characterization and follow up of objects that have been observed to change in position or brightness on the sky.
{\it Add in a list of science cases that will be enabled on the various time scales}
These data products are fully processed single visit images, difference images, and the catalogs produced by difference image analysis (DIA) (sec ref to software products).
DIA outputs consist of, the sources detected in difference images (DIASources), the astrophysical objects that the sources are associated to (DIAObjects),
characterizations of hitherto identified Solar System objects (SSObject), and discoveries of new Solar System objects.
Prompt data products are the result of nightly processing.
Prompt data products are all based on difference imaging, and as such require transient-free templates to exist for each pointing and filter. The production of templates
Prompt data products are release on a continual and ongoing basis.
Two latencies, 60s for alerts and 24hrs for the catalogs. Data on likely optical transients, will be released publicly with a latency of at most 60s.
They are generated continuously every observing night, including both alerts to objects that have changed brightness or position,
which are released with 60-second latency,
and other catalog and image data products that are released with 24-hour latency.
Prompt image data products include:
\paragraph {Image data products} PVIs,
\paragraph {Catalog data products} DIASource, DIAObject catalogs,
\paragraph {Alerts}
\subsubsection{Data Release data products} \label{sec:dp-release}
A Data Release (DR) is specific, fixed {\tt snapshots} of the data at a given time.
Data Releases are made periodically and that can be used and
unambiguously referenced in published analyses.
The catalogs that form the data release will include an extensive list of quantities measured on sources detected in images and
enable a variety of science analyses without the need for users to access or reprocess the images
These data products will be made available as part of an LSST Data Release (\S~???) as the result of coherent
processings of the entire science data set to date.
These will include calibrated images, measurements of positions, fluxes, and shapes, variability information such as orbital
parameters for moving objects, and an appropriate compact description of light curves.
The Data Release data products will include a uniform reprocessing of the difference imaging-based Prompt data products.
\input{dptable}
\subsection{Other categories of data products}
\label{sec:dpother}
\subsubsection{User Generated data products} \label{sec:dp-user}
User Generated data products data products will originate entirely from the community, including project teams.
These will be created and stored using suitable Application Programming Interfaces (APIs)
that will be provided by the LSST Data Management System.
The system will allow the science teams to use the full power of the Rubin database systems and
Science Platform for the storage, access, and analysis of their results.
It will provide for users and groups to maintain access control over the data products they create,
enabling them to have limited distribution or to be shared with the entire LSST community.
The Rubin Science Platform (\S~???) will allow for the creation of User Generated data
products and will enable science cases that greatly benefit from co-location of
user processing and/or data within the LSST Archive Center.
% SRD
%data products are generated continuously every observing night, including alerts
%to objects that have changed brightness or position.
%
%data products will be made available as annual Data Releases and will include
%images and measurements of positions, fluxes, and shapes, as well as variability information such as orbital parameters for moving objects and an appropriate compact
%description of light curves.
%data products will be created by the community, including project teams, using
%suitable Applications Programming Interfaces (APIs) that will be provided by the LSST
%Data Management System. The Data Management System will also provide at least 10%
%of its total capability for user-dedicated processing and user-dedicated storage. The key
%aspect of these capabilities is that they will reside ?next to" the LSST data, avoiding the
%latency associated with downloads. They will also allow the science teams to use the
%database infrastructure to store their results.
The first two, {\tt Prompt} and {\tt Data Release} data products are produced and delivered by the DM system described in this paper.
The third, {\tt User Generated} data products are produced by the Rubin Science Community using the {\tt Prompt} and {\tt Data Release} together possibly with data from other surveys.
The data product categories are outlined in \citet{LPM-231}
In operations Data Production will use the software outlined in \secref{sec:softproducts} to produce the various data products.
Show mapping from data product type to category. i.e prompt contains images, catalogs, but not he same ones as DR/
UG catalogs can be federated with DR/PP catalogs.
These data product categories are defined in the SRD \citep{LPM-17} and have been a driver for DM
(add more detail about why)
\subsection{Special programs data products}
Say something about data products from Special Programs.
The special programs data products will be processed and stored as for all other data products.
Maybe doesn't need to be a subsection
\subsection{Custom data products}
During processing, many intermediate data products are created. If is not feasible nor efficient to store them all.
The DM system provides services to generate data products.
Describe the generation of custom data products, in particular to generate flavours of coadds.