Consolidate jobs #279
Merged
Consolidate jobs #279
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ics. Provision single topic by default.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: zhilingc The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
zhilingc
requested review from
davidheryanto and
woop
and removed request for
tims and
pradithya
|
/retest |
…ics. Provision single topic by default.
* Update prow config and test script so it works for Feast 0.3 * Update CI test script for golang to use correct path * Set pipefail option so test exit code is correct For those commands that pipe the test output * Update group id and version grpc-spring-boot-starter dependency - Newer version is hosted on Maven central which is more maintained than other repo * Add JVM heap settings for Maven surefire Otherwise it fails to run some tests due to insufficient memory * Fix potential setup/teardown error when running ImportJobTest - Set project to empty (DataflowOptions in Beam require project to be not null) - Ignore error when shutting down Kafka server during tear down. This should not affect test result. * Update remote URI to download cached Maven packages - This tar archive contains packages used by Feast 0.3 rather than previous version of Feast * Fix path to logs artifacts * Update maven enforcer rule for Maven and JDK versions - Allow more ranges of versions as long as they are not breaking changes * Use batch mode when initializing Maven for cleaner build log * Skip maven enforcer when running test for specific project * Fix incorrect order for java-sdk test script * Install database and kafka dependencies for Feast end to end test * Update Python SDK to not set source in FeatureSet when using default source Update in progress script for integration test: TODO.sh * Fix Python SDK type mapping - float64 should map to ValueType.DOUBLE instead of ValueType.FLOAT - Ensure Python SDK installation use protobuf v3.10 from pypi. Earlier version of protobuf do not support accessing Enum value in Proto object directly. - Ensure version is passed as INT in e2e test. Set default allow_dirty to true so it's easier to debug error when running e2e test locally. * Refactor prow scripts such that every test is defined in a separate script - Since there are only 6 tests, there is not much benefit of combining tests into a single script Having each test defined on diff script file makes debugging easier * Remove reference to actual project id * Log current stage in end to end test script - Update test case DirectRunnerJobManagerTest.shouldStartDirectJobAndRegisterPipelineResult to set project option to empty string for DirectRunner - Update /usr/sbin/policy-rc.d used in Maven Docker image, so Redis installation will succeed * Fix missing option to specify miniconda download target - Also use cached maven packages when running e2e test * Fix incorrect path to installing Python SDK * Add small wait after ingestion, before validating the saved features, in feast e2e test * Increase wait time before checking, after applying feature set * Make log cleaner
…ast into zl/0.3-dev-consolidate-jobs
|
/retest |
|
/lgtm |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to address the issue of running many jobs and introducing a lot of unnecessary overhead, particularly since we don't have a jobs API.
Changes made: