Add config file support to integration tests

[pracucci]

What this PR does:
While reviewing the PR #1878 it popped up that the PR may change the query-frontend config requirements (discussion still on-going). This made me realise that we don't have any integration test running with the config instead of CLI flags.

This PR is a first step to ease writing tests where Cortex gets started with a config file (CLI flags are still used for some overrides).

In particular:

• Added query-frontend support to local tsdb-blocks-storage-s3 dev env
• Introduced CortexService to have cleaner integration tests
• Added to writeFileToSharedDir() the ability to create the path of directories
• Enhanced query-frontend tests to run with config file too

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

LGTM.

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

[pracucci]

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

Me too. What're your thoughts about the refactoring? What would refactor?

[tomwilkie]

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

I used Table testing pattern but with a generated table here: https://github.com/cortexproject/cortex/blob/master/pkg/distributor/distributor_test.go#L284

Not sure its super useful in this case, but it does allow you to ensure you hit all the combinations of different parameters.

[pracucci]

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

I used Table testing pattern but with a generated table here: https://github.com/cortexproject/cortex/blob/master/pkg/distributor/distributor_test.go#L284

Not sure its super useful in this case, but it does allow you to ensure you hit all the combinations of different parameters.

Not sure it would help much (will end up writing the same logic but in a single test). I've an idea to introduce fixtures support to the e2e framework, in order to be able to write a single test and then run it against different set of fixtures (different configs). I will give it a try when I've some spare time.

[pstibrany]

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

Me too. What're your thoughts about the refactoring? What would refactor?

Oh, I've never replied to this. Something like what Tom linked to or what querier_test.go in Cortex does to test various combinations of encodings and chunk merging algorithms.

[pracucci]

I'm little worried how many combinations we will be testing in query_frontend_test.go. Perhaps it would be worth doing some refactoring there, but it can also lead to big test obfuscation.

Me too. What're your thoughts about the refactoring? What would refactor?

Oh, I've never replied to this. Something like what Tom linked to or what querier_test.go in Cortex does to test various combinations of encodings and chunk merging algorithms.

Yes, but we need custom setup for each test (reason why there's the setup function) so I don't see a real benefit (to me it's looks like testing style: single function, like table testing, or multiple functions). However, if we introduce a generic fixture support (moving the custom setup logic to re-usable fixtures), we may actually simplify these kinds of tests and make fixtures reusable across tests, but to me it's a work decoupled from this PR.

[pstibrany]

However, if we introduce a generic fixture support (moving the custom setup logic to re-usable fixtures), we may actually simplify these kinds of tests and make fixtures reusable across tests,

👍

but to me it's a work decoupled from this PR.

👍