Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

efs fixes #32

Merged
merged 1 commit into from
Jun 15, 2023
Merged

efs fixes #32

merged 1 commit into from
Jun 15, 2023

Conversation

geertvandeweyer
Copy link
Collaborator

  • support paths with over 127 characters :

  • fix delocalization of efs-based globs : path as incorrect, integrity/size check failed due to symlinks

geertvandeweyer
geertvandeweyer commented Jun 13, 2023
View reviewed changes
...nds/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchAsyncBackendJobExecutionActor.scala
@@ -417,8 +417,10 @@ class AwsBatchAsyncBackendJobExecutionActor(override val standardParams: Standar
val output = if (configuration.efsMntPoint.isDefined &&
configuration.efsMntPoint.getOrElse("").equals(disk.toString.split(" ")(1)) &&
! runtimeAttributes.efsDelocalize) {
AwsBatchFileOutput(makeSafeAwsBatchReferenceName(womFile.value), makeSafeAwsBatchReferenceName(womFile.value), relpath, disk)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makeSafeAwsBatchReferenceName() converts the path to md5 hash when over 127 characters.

geertvandeweyer
geertvandeweyer commented Jun 13, 2023
View reviewed changes
supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchJob.scala
@@ -276,7 +277,7 @@ final case class AwsBatchJob(jobDescriptor: BackendJobDescriptor, // WDL/CWL
| s3_content_length=$$($awsCmd s3api head-object --bucket "$$bucket" --key "$$key" --query 'ContentLength') ||
| { echo "Attempt to get head of object failed for $$s3_path." && return 1 ; }
| # local
| local_content_length=$$(LC_ALL=C ls -dn -- "$$local_path" | awk '{print $$5; exit}' ) ||
| local_content_length=$$(LC_ALL=C ls -dnL -- "$$local_path" | awk '{print $$5; exit}' ) ||
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-L makes sure that the symlink target is evaluated, not the link itself

geertvandeweyer
geertvandeweyer commented Jun 13, 2023
View reviewed changes
supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchJob.scala
case output: AwsBatchFileOutput if output.name.endsWith(".list") && output.name.contains("glob-") =>
Log.debug("Globbing : check for EFS settings.")
case output: AwsBatchFileOutput if output.s3key.endsWith(".list") && output.s3key.contains("glob-") =>
Log.debug("Globbing : check for EFS settings.")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

output.name is the SafeName, which can be md5-hash. use the full s3key instead.

@henriqueribeiro henriqueribeiro merged commit 473321f into henriqueribeiro:develop_aws Jun 15, 2023
henriqueribeiro added a commit that referenced this pull request Sep 18, 2023
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri @geertvandeweyer
Develop AWS (#35)
8395859 
* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: geertvandeweyer <geertvandeweyer@gmail.com>
henriqueribeiro added a commit that referenced this pull request Jan 8, 2024
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri @geertvandeweyer @xquek
Develop AWS (#42)
83ca277 
* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* Options to publish status only (#36)

* add options to publish status only

* updated readme.md

---------

Co-authored-by: quekx <quekx@gene.com>

* Fix aws unit tests (#39)

* checkpoint

* fix ecr and batch tests

* fix AwsBatchJobSpec.scala

---------

Co-authored-by: quekx <quekx@gene.com>

* return bucket directly instead of listing and checking it (#38)

Co-authored-by: quekx <quekx@gene.com>

* Add evaluteOnExit for aws batch retry (#40)

Co-authored-by: quekx <quekx@gene.com>

* Improved tagging support (#37)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* support for tagging instances and volumes used in jobs, some support for spaces in file paths

* corrected workflow id in tagging

* redirect exit code 137 to retry-with-more-memory routine

---------

Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* add gpu count (#41)

* add gpu count

* fix typo

---------

Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: geertvandeweyer <geertvandeweyer@gmail.com>
Co-authored-by: xquek <46759870+xquek@users.noreply.github.com>
Co-authored-by: quekx <quekx@gene.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@geertvandeweyer @henriqueribeiro