Add docker-compose file with Jupyer notebook #328

khorshuheng · 2019-11-26T12:51:59Z

This addresses #272.

This PR added a new Jupyter notebook docker image preinstalled with Feast Python SDK. A quick start notebook has also been included for the corresponding docker compose service.

By default, the docker compose file will not rebuild the image. Instead, it will pull images based on the environmental variable defined under infra/docker-compose/.env. Users can either use the default setting, in which the official Feast images are used, or substitute one or more images with their own local image.

Additional settings, such as google credential key, Springboot configurations can be configured via the .env file as well.

Developers who wish to rebuild everything from scratch may run docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d.

Pending tasks before this can be merged:

Need to manually build and upload the jupyter docker image to official repository

feast-ci-bot · 2019-11-26T12:52:15Z

Hi @khorshuheng. Thanks for your PR.

I'm waiting for a gojek member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

woop · 2019-11-26T16:24:48Z

This is awesome, thanks @khorshuheng.

Did you run any tests to confirm that this was working?

khorshuheng · 2019-11-28T05:36:19Z

This is awesome, thanks @khorshuheng.

Did you run any tests to confirm that this was working?

The run output is included in the quick start jupyter notebook: https://github.com/gojek/feast/blob/b92602c2af2e5599e20e2a5f7bc72e657974b8e4/infra/docker-compose/jupyter/notebooks/feast-quickstart.ipynb

I don't think i can write any unit test / integration test for this, but i can potentially include more scenarios in the quick start notebook to test different cases. What other scenarios do you think will be useful for the notebook example?

We can also add additional command in the Makefile for starting / stopping the docker compose services, though i am not sure if that is needed for now.

woop · 2019-11-28T06:24:40Z

This is awesome, thanks @khorshuheng.
Did you run any tests to confirm that this was working?

The run output is included in the quick start jupyter notebook: https://github.com/gojek/feast/blob/b92602c2af2e5599e20e2a5f7bc72e657974b8e4/infra/docker-compose/jupyter/notebooks/feast-quickstart.ipynb

I don't think i can write any unit test / integration test for this, but i can potentially include more scenarios in the quick start notebook to test different cases. What other scenarios do you think will be useful for the notebook example?

We can also add additional command in the Makefile for starting / stopping the docker compose services, though i am not sure if that is needed for now.

Hi @khorshuheng. Yea I dont think its necessary to automate the process. We just need to have a basic script where we can apply, ingest, and retrieve successfully. I just want a way to make sure that the docker compose file and configuration works.

woop · 2019-11-29T04:47:44Z

/ok-to-test

woop

Seriously happy about this :)

I think it would be great to add a few lines in a README.md here just to explain how somebody can get up and running (most likely an end user, not a developer)

woop · 2019-12-03T14:56:01Z

infra/docker/test-runner/conftest.py

+def pytest_addoption(parser):
+    parser.addoption("--core_url", action="store", default="core:6565")
+    parser.addoption("--serving_url", action="store", default="serving:6566")
+    parser.addoption("--allow_dirty", action="store", default="True")


Is there a reason why you are setting allow-dirty to be true?

My intention is to run the e2e test as follows:
docker-compose -f docker-compose.yml -f docker-compose.override.yml -f docker-compose.e2e.yml run e2e

That way, the e2e test run is entirely self contained. However, the above command will keep the services running even after the e2e test run finished. So for convenience i simply set allow dirty equals to true, otherwise we will need to restart the other services everytime. But i do agree that i should have probably set it to false, for consistency and accuracy of test.

That being say, this e2e test setup will be invalidated once we added test for batch retrieval, because my docker-compose setup has not included batch serving yet, only online serving.

The only reason I added that flag was for safety, just in case somebody accidentally runs the test while connected to a production cluster. If you think that situation won't occur, then I guess its fine to make it true.

woop · 2019-12-03T14:56:34Z

infra/docker/test-runner/Dockerfile

+ADD infra/docker/test-runner/conftest.py ./e2e/
+ADD infra/docker/test-runner/run-e2e-test.sh ./
+RUN chmod +x run-e2e-test.sh
+CMD ["./run-e2e-test.sh"]


Is this just running the normal e2e tests?

Yes. This is also a way to test the docker compose setup.

woop · 2019-12-03T14:57:07Z

infra/docker-compose/.env.sample

@@ -0,0 +1,16 @@
+COMPOSE_PROJECT_NAME=feast
+
+FEAST_VERSION=0.3.0


Should we perhaps default to latest and just ensure that we always push the latest tagged docker image?

woop · 2019-12-03T14:57:31Z

infra/docker-compose/.env.sample

+FEAST_SERVING_STORE_CONFIG_PATH=/etc/feast/feast-serving/store.yaml
+FEAST_SERVING_GCP_SERVICE_ACCOUNT_KEY=placeholder
+
+FEAST_JUPYTER_IMAGE=gcr.io/kf-feast/feast-jupyter


Not finding this. Did you end up pushing it, or should I help with that?

I haven't pushed it yet. The current dockerfile always build from master branch, which is fine, but for production we should probably install via pip instead. For which versions should the Feast jupyter images be available? All versions after and including 0.3? (0.3.0, 0.3.2)

Yes, ideally all versions.

khorshuheng · 2019-12-04T01:38:36Z

Seriously happy about this :)

I think it would be great to add a few lines in a README.md here just to explain how somebody can get up and running (most likely an end user, not a developer)

Will add reference link to the Gitbook in the README.md

khorshuheng · 2019-12-06T02:47:27Z

Add capability for batch serving using Big Query. Other changes:

Remove docker-compose.dev. It is probably easier to achieve the same functionality by modifying the .env variable to use a local image instead of the official repository.
Remove docker-compose.e2e, as this will be broken once we add batch testing to our e2e test. It is also not user friendly enough. Will fix this using alternative approach in a separate PR.
New notebook to demonstrate batch feature retrieval.

woop · 2019-12-08T07:50:42Z

/test test-golang-sdk

woop · 2019-12-08T08:04:57Z

/test test-golang-sdk

woop · 2019-12-08T08:05:13Z

/approve
/lgtm

feast-ci-bot · 2019-12-08T08:05:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khorshuheng, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [woop]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…vn dependencies download

woop · 2019-12-08T12:47:28Z

/lgtm

khorshuheng requested review from davidheryanto, pradithya, thirteen37, tims, woop and zhilingc as code owners November 26, 2019 12:52

feast-ci-bot added the needs-ok-to-test label Nov 26, 2019

feast-ci-bot added the size/XL label Nov 26, 2019

khorshuheng force-pushed the docker-compose-setup branch from 77dbbbe to b92602c Compare November 27, 2019 09:13

This was referenced Nov 28, 2019

GetFeastCoreVersion failed with code "StatusCode.UNIMPLEMENTED" #318

Closed

Add docker-compose for Feast #312

Closed

feast-ci-bot added ok-to-test and removed needs-ok-to-test labels Nov 29, 2019

khorshuheng force-pushed the docker-compose-setup branch from e18d4d7 to 23584c0 Compare December 3, 2019 01:14

woop requested changes Dec 3, 2019

View reviewed changes

khorshuheng force-pushed the docker-compose-setup branch from 23584c0 to c2389c3 Compare December 4, 2019 12:26

feast-ci-bot added size/XXL and removed size/XL labels Dec 6, 2019

feast-ci-bot assigned woop Dec 8, 2019

feast-ci-bot added the lgtm label Dec 8, 2019

feast-ci-bot added the approved label Dec 8, 2019

khorshuheng and others added 5 commits December 8, 2019 20:08

Add docker-compose file with Jupyer notebook

77bae5c

Add development version of dockerfile for Feast developer, to avoid m…

e153713

…vn dependencies download

Add end to end test runner

58bc6ce

Add documentation

e6b5a00

Add batch serving functionality to docker compose setup

c613115

khorshuheng force-pushed the docker-compose-setup branch from 7411765 to c613115 Compare December 8, 2019 12:13

feast-ci-bot removed the lgtm label Dec 8, 2019

feast-ci-bot added the lgtm label Dec 8, 2019

feast-ci-bot merged commit 4d1b4f9 into feast-dev:master Dec 8, 2019

woop mentioned this pull request Dec 17, 2019

Docker compose for Feast #272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docker-compose file with Jupyer notebook #328

Add docker-compose file with Jupyer notebook #328

khorshuheng commented Nov 26, 2019 •

edited

Loading

feast-ci-bot commented Nov 26, 2019

woop commented Nov 26, 2019

khorshuheng commented Nov 28, 2019 •

edited

Loading

woop commented Nov 28, 2019

woop commented Nov 29, 2019

woop left a comment •

edited

Loading

woop Dec 3, 2019

khorshuheng Dec 4, 2019

woop Dec 4, 2019

woop Dec 3, 2019

khorshuheng Dec 4, 2019

woop Dec 3, 2019

khorshuheng Dec 4, 2019

woop Dec 3, 2019

khorshuheng Dec 4, 2019

woop Dec 4, 2019

khorshuheng commented Dec 4, 2019

khorshuheng commented Dec 6, 2019

woop commented Dec 8, 2019

woop commented Dec 8, 2019

woop commented Dec 8, 2019

feast-ci-bot commented Dec 8, 2019

woop commented Dec 8, 2019

		@@ -0,0 +1,16 @@
		COMPOSE_PROJECT_NAME=feast

		FEAST_VERSION=0.3.0

Add docker-compose file with Jupyer notebook #328

Add docker-compose file with Jupyer notebook #328

Conversation

khorshuheng commented Nov 26, 2019 • edited Loading

feast-ci-bot commented Nov 26, 2019

woop commented Nov 26, 2019

khorshuheng commented Nov 28, 2019 • edited Loading

woop commented Nov 28, 2019

woop commented Nov 29, 2019

woop left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khorshuheng commented Dec 4, 2019

khorshuheng commented Dec 6, 2019

woop commented Dec 8, 2019

woop commented Dec 8, 2019

woop commented Dec 8, 2019

feast-ci-bot commented Dec 8, 2019

woop commented Dec 8, 2019

khorshuheng commented Nov 26, 2019 •

edited

Loading

khorshuheng commented Nov 28, 2019 •

edited

Loading

woop left a comment •

edited

Loading