Support S3-compatible blob storage#1071

Merged

alxndrsn merged 760 commits intogetodk:masterfrom alxndrsn:s3-blob-storage-wip

Sep 13, 2024

+3,294-230

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 21, 2024

Add TODOs
alxndrsn
committed
Add TODO
alxndrsn
committed
Add TODOs
alxndrsn
committed
Add TODO
alxndrsn
committed
Add TODO
alxndrsn
committed
add more TODO
alxndrsn
committed

Commits on Jun 24, 2024

stream content into blobs
alxndrsn
committed
s3-mock: use Map
alxndrsn
committed
fix streamBriefcaseCsvs()
alxndrsn
committed
try reverting more changes
alxndrsn
committed
reduce changeset
alxndrsn
committed
remove debug
alxndrsn
committed
streamClientAudits() - minimise changes
alxndrsn
committed
Drop unnecessary props
alxndrsn
committed
reduce changes
alxndrsn
committed
fix copyright year
alxndrsn
committed
fix?
alxndrsn
committed

Commits on Jun 26, 2024

Move Blobs._ensureWithStatus() into test file
alxndrsn
committed

Commits on Jun 27, 2024

Makefile: split fake-s3-server into 2 versions
alxndrsn
committed
config: move to default.external.s3blobStore
alxndrsn
committed
Remove resolved FIXME
alxndrsn
committed
Test encrypted submission fetching from s3
alxndrsn
committed
remove resolved TODO
alxndrsn
committed
Update TODOs
alxndrsn
committed
Resolve FIXME
alxndrsn
committed
blobs
alxndrsn
committed
Remove call to non-existent fn
alxndrsn
committed
tone down TODO
alxndrsn
committed
re-org and cull TODOs
alxndrsn
committed
Add test case; resolve TODO
alxndrsn
committed
Add form test
alxndrsn
committed
Update TODOs
alxndrsn
committed
lint
alxndrsn
committed
.only
alxndrsn
committed
fix location check
alxndrsn
committed
delete done TODO
alxndrsn
committed
Add test for changed attachemnt
alxndrsn
committed
Add TODOs
alxndrsn
committed
More TODOs from github comments
alxndrsn
committed
more TODO structure
alxndrsn
committed
re-order TODOs
alxndrsn
committed
async
alxndrsn
committed
remove TODO
alxndrsn
committed
e2e: make cli() async
alxndrsn
committed
e2e: move TODOs to failing tests
alxndrsn
committed

Commits on Jul 1, 2024

s3/run-tests: clean up better
alxndrsn
committed
Check if server already running & kill safer
alxndrsn
committed
Clean up fully
alxndrsn
committed
Add new test case
alxndrsn
committed
Another test
alxndrsn
committed
make final test work
alxndrsn
committed

Commits on Jul 3, 2024

Prepare for multiple tests
alxndrsn
committed
Add 4 to every 4-attachment
alxndrsn
committed
twp tests
alxndrsn
committed
cleanup after each test
alxndrsn
committed

Commits on Jul 25, 2024

TODO: add more info to crontab config
alxndrsn
committed

Commits on Aug 5, 2024

lint
alxndrsn
committed
lint
alxndrsn
committed
Make failure easier to read
alxndrsn
committed
re-add debug
alxndrsn
committed
prevent simultaneous update of same blob
alxndrsn
committed
expectFailure()
alxndrsn
committed
await & log sleeping
alxndrsn
committed
Log pending uploads before beginning uploads
alxndrsn
committed
laxer
alxndrsn
committed
Initial uploaded
alxndrsn
committed
Correct count
alxndrsn
committed
e2e: expect upload to be stuck in-progress
alxndrsn
committed
pipefail
alxndrsn
committed
+e
alxndrsn
committed
Make things more flexible
alxndrsn
committed
form-4: remove unnecessary attachments
alxndrsn
committed
Fix first test?
alxndrsn
committed
Remove row-locks
alxndrsn
committed
Revert "Remove row-locks"
alxndrsn
committed
Revert "Fix first test?"
alxndrsn
committed
fix uploaded expectation
alxndrsn
committed
Revert "Revert "Fix first test?""
alxndrsn
committed
Revert "Revert "Remove row-locks""
alxndrsn
committed
Revert "Revert "Revert "Fix first test?"""
alxndrsn
committed
Revert "Revert "Revert "Revert "Fix first test?""""
alxndrsn
committed
remove pointless attachments from form 2
alxndrsn
committed
Revert "Revert "Revert "Remove row-locks"""
alxndrsn
committed
simplify wait loop
alxndrsn
committed
remove debug
alxndrsn
committed
Merge branch 'master' into s3-blob-storage-wip
alxndrsn
committed
TODO: more notes
alxndrsn
committed
manual transaction management
alxndrsn
committed

Commits on Aug 6, 2024

Revert "manual transaction management"
alxndrsn
committed
resolve TODO in lib/task/s3
alxndrsn
committed
remove isTesting param
alxndrsn
committed

Commits on Aug 15, 2024

Add TODOs
alxndrsn
committed
Fix test cases; add new ideas
alxndrsn
committed
possible approach to transaction-wrap for upload
alxndrsn
committed
lint
alxndrsn
committed

Commits on Aug 16, 2024

Revert uploadBlobIfAvailable()
alxndrsn
committed
Extract countByStatus();
alxndrsn
committed
Fix the test?
alxndrsn
committed
Extract untilUploadInProgress() fn
alxndrsn
committed
Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)Add test #5 (pending)
alxndrsn
committed
Rename expectRejectionFrom()
alxndrsn
committed
countAllByStatus(): convert to numbers
alxndrsn
committed
numbers not strings
alxndrsn
committed
Add final test killing minio
alxndrsn
committed
add sleep
alxndrsn
committed
assertBlobStatuses()
alxndrsn
committed
only
alxndrsn
committed
add setInProgressToPending
alxndrsn
committed
fix s3SetInProgressToPending()
alxndrsn
committed
Add TODO
alxndrsn
committed
track when minio terminated
alxndrsn
committed
not just only
alxndrsn
committed
resolve TODOs
alxndrsn
committed
try more docker-centric approach
alxndrsn
committed
new approach?
alxndrsn
committed

Commits on Aug 17, 2024

fake in_progress count
alxndrsn
committed
enable SIGTERM test
alxndrsn
committed
update expectations
alxndrsn
committed
Expect success
alxndrsn
committed
docker debug
alxndrsn
committed
update SIGTERM expectations
alxndrsn
committed
remove debug
alxndrsn
committed
nicer debug
alxndrsn
committed
docker debug 1 & 2
alxndrsn
committed
awk
alxndrsn
committed
add TODO
alxndrsn
committed
match
alxndrsn
committed
Add TODO
alxndrsn
committed
better match
alxndrsn
committed
fix eregex
alxndrsn
committed
Safer regex
alxndrsn
committed
Add FIXME
alxndrsn
committed
lint
alxndrsn
committed
remove in_progress explicit status
alxndrsn
committed
remove more refs to setInProgressToPending()
alxndrsn
committed
Fix error bubbling?
alxndrsn
committed
Fix integration test
alxndrsn
committed
lint
alxndrsn
committed
remove bad command
alxndrsn
committed
no big
alxndrsn
committed
fix test
alxndrsn
committed
more simpler assertions
alxndrsn
committed
restreict args
alxndrsn
committed
remove no-op
alxndrsn
committed
Avoid timeout
alxndrsn
committed
smaller timeout
alxndrsn
committed

Commits on Aug 20, 2024

make regex moire reliable
alxndrsn
committed
DEBUG: add timings to uploadFormWithAttachments()
alxndrsn
committed
Add comment and another error type
alxndrsn
committed
Resolve FIXME
alxndrsn
committed
Revert "DEBUG: add timings to uploadFormWithAttachments()"
alxndrsn
committed
Try decrease TIMEOUT
alxndrsn
committed
decrease TIMEOUT even more
alxndrsn
committed
increase timeout
alxndrsn
committed
big number notation
alxndrsn
committed
TIMEOUT: 60s
alxndrsn
committed
timeout: 120s
alxndrsn
committed
Add TODOs, remove brackets
alxndrsn
committed
cleanup
alxndrsn
committed
lint
alxndrsn
committed
lint2
alxndrsn
committed
e2e: add SIGINT test
alxndrsn
committed
Remove resolved TODO
alxndrsn
committed

Commits on Aug 21, 2024

increase timeout; sigint=2
alxndrsn
committed
add TODO
alxndrsn
committed

Commits on Aug 22, 2024

Make 3.xml unique
alxndrsn
committed
wip
alxndrsn
committed
remove .only
alxndrsn
committed
remove logging
alxndrsn
committed
remove logging
alxndrsn
committed
lint
alxndrsn
committed
Re-intro corect output
alxndrsn
committed
remove logging
alxndrsn
committed
tone down logging
alxndrsn
committed
tidy up
alxndrsn
committed
remove logging; resolve TODO
alxndrsn
committed
restructure TODOs
alxndrsn
committed
TODO -> FIXME
alxndrsn
committed
resolve TODO
alxndrsn
committed
lower timeout
alxndrsn
committed
multiple bigfiles for test7
alxndrsn
committed
double tiny
alxndrsn
committed
fix filenames
alxndrsn
committed
remove TODO
alxndrsn
committed
resolve TODO
alxndrsn
committed

Commits on Aug 23, 2024

await before sleep()
alxndrsn
committed
3: fix big bin name
alxndrsn
committed
7: fix wait code
alxndrsn
committed
lint
alxndrsn
committed
recursive
alxndrsn
committed
no sleep
alxndrsn
committed
2: fix form ID
alxndrsn
committed
Add FIXMEs
alxndrsn
committed
Enable encryption
alxndrsn
committed
temporary fix - not handling differing etags yet
alxndrsn
committed
new assertion
alxndrsn
committed
catch unexpected success
alxndrsn
committed
special pending count
alxndrsn
committed

Commits on Aug 24, 2024

Add missing awaits
alxndrsn
committed
postgres: trim log line lengths
alxndrsn
committed
Count unlocked rows without locking them
alxndrsn
committed
fix pending count
alxndrsn
committed
Add another expected error
alxndrsn
committed
lint
alxndrsn
committed
lint - don't await in promise
alxndrsn
committed

Commits on Aug 26, 2024

Update etag handling
alxndrsn
committed
Update RFC quote
alxndrsn
committed
Add new ETag test
alxndrsn
committed
Fix tests for new etag handling
alxndrsn
committed
FIXME -> TODO
alxndrsn
committed
remove etag handling from blob upload
alxndrsn
committed
add TODO item
alxndrsn
committed
add link to blocking issue
alxndrsn
committed
remove TODO
alxndrsn
committed
Add test TODO
alxndrsn
committed
simplify purge code
alxndrsn
committed

Commits on Aug 27, 2024

change object name to include blob id
alxndrsn
committed
lint
alxndrsn
committed
Add check for null/undefined id
alxndrsn
committed
atomic purge?
alxndrsn
committed
notes
alxndrsn
committed
purge optimisation
alxndrsn
committed
Fix s3 mock
alxndrsn
committed
remove console.log
alxndrsn
committed
Include sha in object URL
alxndrsn
committed
return sha!
alxndrsn
committed
blobIds
alxndrsn
committed
lint
alxndrsn
committed
Fix blob streaming
alxndrsn
committed
blobId
alxndrsn
committed
Merge branch 'master' into s3-blob-storage-wip
alxndrsn
committed

Commits on Aug 29, 2024

reintro worker queue
alxndrsn
committed
external/s3: move require()s to top
alxndrsn
committed
migration: drop extension on unmigrate
alxndrsn
committed
Update comment
alxndrsn
committed
re-order vars to make comparison of fns easier
alxndrsn
committed
revert whitespace change
alxndrsn
committed
clarify test explanation
alxndrsn
committed

Commits on Aug 30, 2024

comment
alxndrsn
committed
Add option: objectPrefix
alxndrsn
committed

Commits on Sep 3, 2024

Update TODOs
alxndrsn
committed
Update TODOs
alxndrsn
committed

Commits on Sep 11, 2024

try include stack in Error
alxndrsn
committed
remove TODO file
alxndrsn
committed
Makefile: revert changes
alxndrsn
committed
Merge branch 'master' into s3-blob-storage-wip
alxndrsn
committed

Commits on Sep 13, 2024

Remove unnecessary linter exception
alxndrsn
committed
Reswet package-lock
alxndrsn
committed
update migration name
alxndrsn
committed
remove down migration
alxndrsn
committed