Generate security evidence by documenting security testcases #11306
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nickchapman-da
force-pushed
the
nick-evidencing-daml-security
branch
from
d2c5667 to
950d71d
Compare
CHANGELOG_BEGIN CHANGELOG_END
nickchapman-da
force-pushed
the
nick-evidencing-daml-security
branch
from
950d71d to
68e95a6
Compare
nickchapman-da
changed the title
nickchapman-da
requested review from
cocreature and
remyhaemmerle-da
as code owners
cocreature
reviewed
Oct 21, 2021
daml-lf/engine/src/test/scala/com/digitalasset/daml/lf/engine/AuthPropagationSpec.scala
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,17 @@ | |||
| # Security tests, by category | |||
There was a problem hiding this comment.
Doesn’t have to be in this PR but I think it would make sense to have a machine-readable format for this whether that’s JSON or CSV or something else. That should make it easier to eventually integrate it in the docs or produce a spreadsheet.
There was a problem hiding this comment.
@stefanobaghino-da fyi, this is still in very early stages but just so you get an idea of what we have in mind.
| # Security tests, by category | ||
|
|
||
| ## Authorization: | ||
| - Engine level tests for _authorization_ check.: [AuthPropagationSpec.scala](daml-lf/engine/src/test/scala/com/digitalasset/daml/lf/engine/AuthPropagationSpec.scala#L39) |
There was a problem hiding this comment.
What do you think of checking the current commit when generating the file and then using a link to that instead of current main? The latter is bound to get invalid whereas the former might point to an old file but at least still to the right location.
There was a problem hiding this comment.
This seems a good idea.
There was a problem hiding this comment.
Actually, I don't think we should link to the current commit, as this will make the generated file unstable.
| @@ -17,6 +17,7 @@ import org.scalatest.matchers.should.Matchers | |||
|
|
|||
| import org.scalatest.Inside | |||
|
|
|||
| // SECURITY_TEST: Authorization: Unit test _authorization_ computations in: `CheckAuthorization`. | |||
There was a problem hiding this comment.
Doesn’t have to be in this PR but I think it would be nice to be more specific, e.g., "positive testcase for authorization of create" "negative testcase for authorization of create", …. We don’t have to be this specific for all tests but authorization is so central that I think it deserves that level of specificity.
There was a problem hiding this comment.
Sure.
The text can be as general or as specific as we like. The few example I added so far were really just to test the framework.
security/EvidenceSecurity.hs
Outdated
| redirected to the name of the generated file: | ||
|
|
||
| ``` | ||
| git grep --line-number SECURITY_TEST\: | bazel run security:evidence-security > security-evidence.md |
There was a problem hiding this comment.
I think what we could do for keeping it up2date is to add a check to CI that runs this and checks that the result is identical. If it isn’t fail CI and print the command users have to run to update it. Could be as part of ./fmt.sh if it’s fast or a separate CI job if it isn’t.
There was a problem hiding this comment.
This seems a good approach.
There was a problem hiding this comment.
It is quick to update, so I will look at adding to ./fmt.sh
cocreature
reviewed
Oct 25, 2021
azure-pipelines bot
pushed a commit
that referenced
this pull request
Oct 27, 2021
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @akshayshirahatti-da is in charge of this release. Commit log: ``` e474b2d [JSON-API] Websockets fix for matchedQueries (#11361) 4a34b68 KV: port V2 errors to self-service errors framework [KVL-1143] (#11326) 811a6d3 Fixed AuthorizationInterceptorSpec again (#11418) c8006b8 ScenarioRunner: enrich incomplete transactions (#11384) d9c7031 ACS testing - payload support [DPP-661] (#11308) d87d3d4 deal with deadlocks while fetching contracts in json-api Oracle (#11391) 8212c0b Make submission ID optional [KVL-1107] (#11011) 3587eb8 Use Timestamp instead of Instant (#11356) ea5f09e sandbox: Deprecate the `--eager-package-loading` flag. (#11404) 9f882f2 remove search index on json fields that harm insert and pruning performance (#11041) 70b90f4 optimize max event_sequential_id query for oracle (#11297) b1fed31 Fix missing script results (#11395) 03db0aa Auto run/check security evidence generation in ./fmt.sh (#11407) c928f0e [Short] Typo (#11400) ba6c2be Add missing TransactionId to com.daml.error.ErrorResource (#11396) a2a1571 Generate security evidence by documenting security testcases (#11306) 8d17882 Allocate parties sequentially in script export tests (#11389) 1309c2f DPP-587 Use Timestamp instead of Instant (#11183) 82f9873 Rotate release rotation (#11394) b14077a Fix AuthorizationInterceptorSpec flake (#11387) 7090f2d update NOTICES file (#11367) ad42dfa Update gRPC to the latest (1.41.0) and Protobuf (#11380) 54c400a Update wording in Deploying to a generic Daml ledger (#11327) 4461ed1 Fix log output (#11374) 613aac3 Add support for non-star-kinded type synonyms in data-dependencies (#11293) f89ecc6 interfaces: add an experimental `toTypeRep` builtin. (#11378) 5654d5c fix es ingest for missing files (#11375) 03cfd12 Configurable assertions in Ledger API test tool by feature descriptors (#11328) 96b7b58 [DPP-648][Self-service error codes] Adopt ApiPartyManagementService (#11338) 9e94ae0 LF: move repl exception-auth test from dev to stable (#11369) 5365d68 LF: Remove PartialTransaction out from ScenarioRunner/IdeLedgerClient (#11368) 79037c8 [DPP-646][Self-service error codes] Adopt ApiPackageManagementService (#11314) 0ee59f5 Command submission in the ledger-api-bench-tool. (#11296) 8d5cab5 LF: Simplify seeds generation in scenario runnner (#11353) 9e5b788 Speedup daml repl integration tests (#11335) 3bc0db3 fix contract_tpid_fkey-related race condition (#11330) ab8a863 [docs] Add Daml Driver for VMBC to the commercial integrations section (#11360) c95db72 Fix Bazel cache download retry (#11238) e8d0ccb [DPP-611][Self-service error codes] Adapt ApiCommandService (#11325) a89079b [DPP-647][Self-service error codes] Adopt ApiParticipantPruningService (#11324) cc8ec28 [Self-service error codes] Adapt GrpcHealthService (#11354) c60c94b [DPP-645][Self-service error codes] Adapt ApiConfigManagementService (#11312) e6da1f7 Add step in ghc-lib guide for getting submodules to work (#11351) f3057ea Increase timeout for non-repudation tests on Postgres (#11340) 176f470 interface: adding interfaces to the TS codegen (#11280) 355352f DPP-650 Remove the mutating schema (#11211) 443b64d [DPP-621][Self-service error codes] Adopt error codes in ApiVersionService (#11302) ed9dbed interfaces: Add fixed choice collision check in typechecker (Haskell) (#11337) c37ecd1 [Short] Pass correct loggingContext to withValidatedPackageId (#11307) 0d305cf [Short] Move field before logging statement (ApiTimeService) (#11313) 73c94b5 Increase timeout for non-repudation test (#11281) 88c607b [Self-service error codes] Adapt ApiTransactionService [DPP-613] (#11094) 07ad3e0 Suport multi-party readAs in triggers (#11299) 76eb165 Interface fixed choices: ghc parser (#11275) da27a1e [DPP-619][Self-service error codes] Adopt error codes in ApiVersionService (#11303) 5f5af30 [DPP-628][Self-service error codes] Adapt error codes in ApiTimeService (#11295) f9e67ad [Self-service error codes] Adapt error responses in ledger-api-auth [DPP-617] (#11223) 7282965 Fix component status for triggers (#11311) 17776f3 Factor out npm install step of create-daml-app tests (#11294) 3a8b685 [Short] Fix docs for Dispatcher#startingAt (#11304) f315a90 release 1.18.0-snapshot.20211019.8113.0.8ff347d8 (#11300) 8a3abce [DPP-618][Self-service error codes] Adapt error codes in ApiPackageService (#11284) 50ea92f Use ApiTypes.Party instead of String in the trigger runner (#11298) 2267429 [DPP-656] Assert on self-service error code details in ErrorFactoriesSpec (#11289) c06faf2 LF: remove imperative environment from Speedy compiler (#11285) d3dad75 [DPP-592] Generate docs for self-service error codes. (#11129) ``` Changelog: ``` [JSON-API] fixes a bug related to the matchedQueries value returned for websocket multiqueries, this only happens for patterns where the multiqueries contain a mixture of queries with and without offsets. - [Integration Kit] - ledger-api-bench-tool can generate test contracts with configurable payload size. - [Integration Kit] - Added multi-template support for command submission in the ledger-api-bench-tool - [Sandbox] The ``--eager-package-loading`` flag has been deprecated. It hasn't actually done anything for many releases; packages are always loaded eagerly. This does not affect Daml on SQL, which does support lazy package loading. - [Daml Studio] Fix a bug where script results in Daml Studio sometimes do not show up. - [Integration Kit] - The ledger-api-bench-tool is now capable of generating test contracts for testing purposes. - [JSON API] Fixed a rare error that manifested as ‘violates foreign key constraint "contract_tpid_fkey" Detail: Key (tpid)=(...) is not present in table’ when attempting to run queries and goes away on JSON API restart. See `issue #11330 <https://github.com/digital-asset/daml/pull/11330>`__. - [Participant] All participants now use the new append-only schema. Existing databases will automatically upgrade to the new schema the first time a participant/ledger is started. - [Daml Triggers] Triggers now support readAs parties. They can be specified via `--ledger-readas a,b,c`. As part of this change ``testRule`` gained an extra argument to specify the `readAs` parties. If you previously used ``` testRule trigger party acsBuilder commandsInFlight s ``` you now need to use ``` testRule trigger party [] acsBuilder commandsInFlight s ``` ``` CHANGELOG_BEGIN CHANGELOG_END
akshayshirahatti-da
pushed a commit
that referenced
this pull request
Oct 27, 2021
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @akshayshirahatti-da is in charge of this release. Commit log: ``` e474b2d [JSON-API] Websockets fix for matchedQueries (#11361) 4a34b68 KV: port V2 errors to self-service errors framework [KVL-1143] (#11326) 811a6d3 Fixed AuthorizationInterceptorSpec again (#11418) c8006b8 ScenarioRunner: enrich incomplete transactions (#11384) d9c7031 ACS testing - payload support [DPP-661] (#11308) d87d3d4 deal with deadlocks while fetching contracts in json-api Oracle (#11391) 8212c0b Make submission ID optional [KVL-1107] (#11011) 3587eb8 Use Timestamp instead of Instant (#11356) ea5f09e sandbox: Deprecate the `--eager-package-loading` flag. (#11404) 9f882f2 remove search index on json fields that harm insert and pruning performance (#11041) 70b90f4 optimize max event_sequential_id query for oracle (#11297) b1fed31 Fix missing script results (#11395) 03db0aa Auto run/check security evidence generation in ./fmt.sh (#11407) c928f0e [Short] Typo (#11400) ba6c2be Add missing TransactionId to com.daml.error.ErrorResource (#11396) a2a1571 Generate security evidence by documenting security testcases (#11306) 8d17882 Allocate parties sequentially in script export tests (#11389) 1309c2f DPP-587 Use Timestamp instead of Instant (#11183) 82f9873 Rotate release rotation (#11394) b14077a Fix AuthorizationInterceptorSpec flake (#11387) 7090f2d update NOTICES file (#11367) ad42dfa Update gRPC to the latest (1.41.0) and Protobuf (#11380) 54c400a Update wording in Deploying to a generic Daml ledger (#11327) 4461ed1 Fix log output (#11374) 613aac3 Add support for non-star-kinded type synonyms in data-dependencies (#11293) f89ecc6 interfaces: add an experimental `toTypeRep` builtin. (#11378) 5654d5c fix es ingest for missing files (#11375) 03cfd12 Configurable assertions in Ledger API test tool by feature descriptors (#11328) 96b7b58 [DPP-648][Self-service error codes] Adopt ApiPartyManagementService (#11338) 9e94ae0 LF: move repl exception-auth test from dev to stable (#11369) 5365d68 LF: Remove PartialTransaction out from ScenarioRunner/IdeLedgerClient (#11368) 79037c8 [DPP-646][Self-service error codes] Adopt ApiPackageManagementService (#11314) 0ee59f5 Command submission in the ledger-api-bench-tool. (#11296) 8d5cab5 LF: Simplify seeds generation in scenario runnner (#11353) 9e5b788 Speedup daml repl integration tests (#11335) 3bc0db3 fix contract_tpid_fkey-related race condition (#11330) ab8a863 [docs] Add Daml Driver for VMBC to the commercial integrations section (#11360) c95db72 Fix Bazel cache download retry (#11238) e8d0ccb [DPP-611][Self-service error codes] Adapt ApiCommandService (#11325) a89079b [DPP-647][Self-service error codes] Adopt ApiParticipantPruningService (#11324) cc8ec28 [Self-service error codes] Adapt GrpcHealthService (#11354) c60c94b [DPP-645][Self-service error codes] Adapt ApiConfigManagementService (#11312) e6da1f7 Add step in ghc-lib guide for getting submodules to work (#11351) f3057ea Increase timeout for non-repudation tests on Postgres (#11340) 176f470 interface: adding interfaces to the TS codegen (#11280) 355352f DPP-650 Remove the mutating schema (#11211) 443b64d [DPP-621][Self-service error codes] Adopt error codes in ApiVersionService (#11302) ed9dbed interfaces: Add fixed choice collision check in typechecker (Haskell) (#11337) c37ecd1 [Short] Pass correct loggingContext to withValidatedPackageId (#11307) 0d305cf [Short] Move field before logging statement (ApiTimeService) (#11313) 73c94b5 Increase timeout for non-repudation test (#11281) 88c607b [Self-service error codes] Adapt ApiTransactionService [DPP-613] (#11094) 07ad3e0 Suport multi-party readAs in triggers (#11299) 76eb165 Interface fixed choices: ghc parser (#11275) da27a1e [DPP-619][Self-service error codes] Adopt error codes in ApiVersionService (#11303) 5f5af30 [DPP-628][Self-service error codes] Adapt error codes in ApiTimeService (#11295) f9e67ad [Self-service error codes] Adapt error responses in ledger-api-auth [DPP-617] (#11223) 7282965 Fix component status for triggers (#11311) 17776f3 Factor out npm install step of create-daml-app tests (#11294) 3a8b685 [Short] Fix docs for Dispatcher#startingAt (#11304) f315a90 release 1.18.0-snapshot.20211019.8113.0.8ff347d8 (#11300) 8a3abce [DPP-618][Self-service error codes] Adapt error codes in ApiPackageService (#11284) 50ea92f Use ApiTypes.Party instead of String in the trigger runner (#11298) 2267429 [DPP-656] Assert on self-service error code details in ErrorFactoriesSpec (#11289) c06faf2 LF: remove imperative environment from Speedy compiler (#11285) d3dad75 [DPP-592] Generate docs for self-service error codes. (#11129) ``` Changelog: ``` [JSON-API] fixes a bug related to the matchedQueries value returned for websocket multiqueries, this only happens for patterns where the multiqueries contain a mixture of queries with and without offsets. - [Integration Kit] - ledger-api-bench-tool can generate test contracts with configurable payload size. - [Integration Kit] - Added multi-template support for command submission in the ledger-api-bench-tool - [Sandbox] The ``--eager-package-loading`` flag has been deprecated. It hasn't actually done anything for many releases; packages are always loaded eagerly. This does not affect Daml on SQL, which does support lazy package loading. - [Daml Studio] Fix a bug where script results in Daml Studio sometimes do not show up. - [Integration Kit] - The ledger-api-bench-tool is now capable of generating test contracts for testing purposes. - [JSON API] Fixed a rare error that manifested as ‘violates foreign key constraint "contract_tpid_fkey" Detail: Key (tpid)=(...) is not present in table’ when attempting to run queries and goes away on JSON API restart. See `issue #11330 <https://github.com/digital-asset/daml/pull/11330>`__. - [Participant] All participants now use the new append-only schema. Existing databases will automatically upgrade to the new schema the first time a participant/ledger is started. - [Daml Triggers] Triggers now support readAs parties. They can be specified via `--ledger-readas a,b,c`. As part of this change ``testRule`` gained an extra argument to specify the `readAs` parties. If you previously used ``` testRule trigger party acsBuilder commandsInFlight s ``` you now need to use ``` testRule trigger party [] acsBuilder commandsInFlight s ``` ``` CHANGELOG_BEGIN CHANGELOG_END Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here is a first cut at a simple generation framework. Currently there are four categories. We can extend as we like. I've added markers to 6 testcases. We can add more in subsequent PRs, and work out what nice descriptive text we want in each case.
The markdown output file is generated by:
The generated file is committed to the repo, for ease of viewing.
I'm not sure yet of the best way to:
(1) run the generation auto-magically by the bazel build
(2) integrate the generated file into existing internal documentation
We can sort this out in subsequent PRs.