update variables and executor loading #708

fokion · 2023-08-04T09:16:14Z

modify the code to avoid having open readers as it was a potential memory leak, I pass the path instead and use the same ReadFile that closes its reader by default
load the executors ( dont interpolate them with values as we expect to have them interpolated when we execute a test )
update the variables to remove any cloning and avoid any other duplication
reporting of testsuites as they get executed to the file format and only the html will be done in the end ( that can be improved further but it is a start)
added a flag ( you need to set it to ON ) for adding a timestamp with hours.minutes.seconds.milliseconds in the dump.json files and that is useful if you have an executor that is reused multiple times and you want to see all the states and not only the last one.
skip is updated ( due to the variables change mostly )

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion · 2023-08-04T09:18:04Z

Linking #701

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

… not to overwrite the dump of a reused executor Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

… based on the documentation Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion · 2023-08-10T15:10:47Z

Any thoughts @yesnault , @fsamin ?

ovh-cds · 2023-09-07T09:26:14Z

CDS Report build-venom-a#87.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2023-09-07T09:37:17Z

CDS Report build-venom-a#87.1 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2023-09-07T10:14:24Z

CDS Report build-venom-a#88.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

yesnault · 2023-09-07T12:55:43Z

@fokion Thank you for this proposal. There is so many new code in this Pull-Request, new env var, new code on assertions, configData update. It's pretty hard to review that and will be impossible to rollback the commit if there some issue with something.

Can you re-open previous PR (or new prs) dedicated on each fix / feature please? We had discussions on some of them, as #685

About flag, is there any perf data before and after adding the flag VENOM_NO_JSON_EXPANSION ?

The Integration Tests are failing with:

Testcase "", step #0-0: Assertion "result.systemout ShouldEqual \"bar test\"" failed. expected: bar test  got: {{.cat-json.foo}} {{.cat-json.bar}} (exec.yml:48)

fokion · 2023-09-07T14:23:08Z

is there any perf data before and after adding the flag VENOM_NO_JSON_EXPANSION ?

Yes when you have big json payloads that you only need a one or two fields , it crawls after a while while the memory goes really high.

I will have a look at that test , but in general it is stable enough as I have been using this fork for a while

yesnault · 2023-09-07T14:29:41Z

cmd/venom/run/cmd.go

@@ -158,6 +170,7 @@ type ConfigFileData struct {
 	StopOnFailure  *bool     `json:"stop_on_failure,omitempty" yaml:"stop_on_failure,omitempty"`
 	HtmlReport     *bool     `json:"html_report,omitempty" yaml:"html_report,omitempty"`
 	Variables      *[]string `json:"variables,omitempty" yaml:"variables,omitempty"`
+	Secrets        *[]string `json:"secrets,omitempty" yaml:"secrets,omitempty"`


backport on #715

yesnault · 2023-09-07T14:29:49Z

cmd/venom/run/cmd.go

@@ -195,6 +208,11 @@ func initFromReaderConfigFile(reader io.Reader) error {
 			variables = mergeVariables(varFromFile, variables)
 		}
 	}
+	if configFileData.Secrets != nil {


backport on #715

fokion · 2023-09-18T11:12:02Z

@fokion Thank you for this proposal. There is so many new code in this Pull-Request, new env var, new code on assertions, configData update. It's pretty hard to review that and will be impossible to rollback the commit if there some issue with something.

Can you re-open previous PR (or new prs) dedicated on each fix / feature please? We had discussions on some of them, as #685

About flag, is there any perf data before and after adding the flag VENOM_NO_JSON_EXPANSION ?

The Integration Tests are failing with:
Testcase "", step #0-0: Assertion "result.systemout ShouldEqual \"bar test\"" failed. expected: bar test  got: {{.cat-json.foo}} {{.cat-json.bar}} (exec.yml:48)

I could not replicate the test error , will try to run it as a unit test case.

fokion · 2023-09-27T22:31:35Z

started splitting it into parts

add flag VENOM_LOGS_WITH_TIMESTAMP #722 ( flag)
WIP --- Update how we are loading executors #723 ( executor loading )

yesnault · 2023-09-28T08:56:43Z

see splitted pr above.

fokion · 2023-10-23T08:18:53Z

#732

dont use the readers and use the os.ReadFile to read from a path instead

3a87a5c

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion force-pushed the feat/update-variables branch from e8dc6e2 to a8d42f0 Compare August 4, 2023 09:16

fokion mentioned this pull request Aug 4, 2023

Cache properly the executors #705

Closed

This was referenced Aug 4, 2023

Update the skip as it was not working correctly #685

Closed

reporting optimisation, generate the reports as we execute the tests #704

Closed

fokion added 2 commits August 4, 2023 14:54

adding output on a single test suite

6b87cc9

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

return a clean test suite after execution

d2dd81c

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion force-pushed the feat/update-variables branch from 6bfca73 to c99c55f Compare August 4, 2023 13:54

fokion added 6 commits August 5, 2023 13:33

update test suite with a clean version

1f1ab70

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

add flag VENOM_LOGS_WITH_TIMESTAMP in order to have the timestamp and…

48f5874

… not to overwrite the dump of a reused executor Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

update how we read executors

d0e05d8

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

prevent doing multiple ranges if we have range not enabled

560ec5f

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fail if we cannot process variable assignments

b79d322

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

on demand json expansion ( useful on big payloads)

bbf99b0

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion force-pushed the feat/update-variables branch 2 times, most recently from 91e08f4 to 824f207 Compare August 7, 2023 10:45

fokion added 2 commits August 9, 2023 20:37

process array and maps from json blobs as well as actual types

ed92ea5

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

more tests and adding local vars

b3b2473

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

fokion force-pushed the feat/update-variables branch from 824f207 to b3b2473 Compare August 9, 2023 19:38

fokion added 7 commits August 10, 2023 18:09

fix cmd test

e6acc24

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

adding unit test for run cmd that starts the tests/assertions

44e6128

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

have a json locally generated instead of calling an api

37d2e8e

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

remove warnings in cmd

5e63f9b

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

rename file to be the same pattern with run and update

7bd0fe6

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

adding test in the root

3300464

Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

add test for ShouldNotExist and fixing the logical issue that existed…

5447f53

… based on the documentation Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>

Merge branch 'master' into feat/update-variables

95d0769

yesnault reviewed Sep 7, 2023

View reviewed changes

yesnault closed this Sep 28, 2023

This was referenced Oct 23, 2023

Lazy expansion of json payloads that we get from user executors #735

Closed

Adding test in the root package in order to test the run command #733

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update variables and executor loading #708

update variables and executor loading #708

fokion commented Aug 4, 2023

fokion commented Aug 4, 2023

fokion commented Aug 10, 2023

ovh-cds commented Sep 7, 2023

ovh-cds commented Sep 7, 2023

ovh-cds commented Sep 7, 2023

yesnault commented Sep 7, 2023

fokion commented Sep 7, 2023

yesnault Sep 7, 2023

yesnault Sep 7, 2023

fokion commented Sep 18, 2023

fokion commented Sep 27, 2023 •

edited

Loading

yesnault commented Sep 28, 2023

fokion commented Oct 23, 2023

update variables and executor loading #708

update variables and executor loading #708

Conversation

fokion commented Aug 4, 2023

fokion commented Aug 4, 2023

fokion commented Aug 10, 2023

ovh-cds commented Sep 7, 2023

ovh-cds commented Sep 7, 2023

ovh-cds commented Sep 7, 2023

yesnault commented Sep 7, 2023

fokion commented Sep 7, 2023

yesnault Sep 7, 2023

Choose a reason for hiding this comment

yesnault Sep 7, 2023

Choose a reason for hiding this comment

fokion commented Sep 18, 2023

fokion commented Sep 27, 2023 • edited Loading

yesnault commented Sep 28, 2023

fokion commented Oct 23, 2023

fokion commented Sep 27, 2023 •

edited

Loading