V3 #17
Merged
V3 #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cutouts are stored as an AVRO packet, using ZTF alert schema 3.3
with all keys other than candid and cutouts removed. This means
that science, reference, and difference cutouts are retrieved
together for each alert. The underlying assumption is that cutout
queries will be rare.
Adds:
- a new endpoint, PUT /alert/{candid} for inserting new alerts
- roles for tokens. Only writer tokens can insert new alerts
- s3 backend for cutout storage
Removes:
- with_cutouts arguments to alert query endpoints.
Instead of storing individual cutouts in s3, store entire chunks of alerts. These are only moderately larger than just the cutouts themselves. The alert table tracks the URI of the blob in which the alert was stored, along with the byte range of the target block, which can be retrieved with a range request.
This is ~linearly faster than the previous implementation in the number of pixels queried, as postgres does not seem to elide multiple calls to healpix_ang2pix_nest, even though it is marked IMMUTABLE. Generated columns require postgres >= 12
Inserting alerts in bulk in multiple concurrent transactions causes deadlocks like:
```
2022-02-16 12:11:30.643 CET [2834] STATEMENT: INSERT INTO upper_limit (jd, fid, pid, diffmaglim, programid, rbversion, nid, rcid, field) VALUES (2459622.7773727, 2, 1868277372915, 19.48270034790039, 1, 't17_f5_c3', 1868, 29, 515) ON CONFLICT DO NOTHING RETURNING upper_limit.upper_limit_id
2022-02-16 12:12:19.164 CET [2832] ERROR: deadlock detected
2022-02-16 12:12:19.164 CET [2832] DETAIL: Process 2832 waits for ShareLock on transaction 53604; blocked by process 2835.
Process 2835 waits for ShareLock on transaction 53602; blocked by process 2832.
Process 2832: INSERT INTO upper_limit (jd, fid, pid, diffmaglim, programid, rbversion, nid, rcid, field) VALUES (2459622.7773727, 2, 1868277370115, 19.436500549316406, 1, 't17_f5_c3', 1868, 1, 515) ON CONFLICT DO NOTHING
Process 2835: INSERT INTO upper_limit (jd, fid, pid, diffmaglim, programid, rbversion, nid, rcid, field) VALUES (2459624.668044, 2, 1870168043615, 19.528600692749023, 1, 't17_f5_c3', 1870, 36, 258) ON CONFLICT DO NOTHING
2022-02-16 12:12:19.164 CET [2832] HINT: See server log for query details.
2022-02-16 12:12:19.164 CET [2832] CONTEXT: while inserting index tuple (377163,4) in relation "upper_limit"
```
Cutouts may be explicitly retrieved for a single candidate, but do not make sense in bulk queries.
Non-ZTF users should see only programid 1. To accomplish this, all bulk alert query endpoints now have an optional programid parameter, and tokens have a partnership field, set to true if the owner's identity is in the partnership group. Non-partnership tokens must set programid to 1, or omit it, in which case it defaults to 1. Values other than 1 raise 401 Unauthorized. Partnership queries pass the value of programid through. As a side-effect, fixes #5.
In addition to jd/spatial constraints, requests may specify a filter
to apply to the candidate in each alert, in a format similar to a
MongoDB match expression. For example:
``` {"fid": 1, "drb": {"$gt": 0.95}} ```
selects only alerts where fid==1 and drb>=0.95. The operators $gt,
$gte, $lt, $lte, $eq, $ne, $in, and $nin are supported.
Allow for long-running queries by creating the group outside a transaction and running the actual query in a background task.
This is _much_ faster than dropping and recreating the primary key (2 minutes vs 1 hour for ~200k updates)
get_alert did not require auth, so drop it here too
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.