Unable to retrieve feature from Redis

[sbryfcz]

Team,
Thanks for the great progress on this project. Looks very promising. I am trying to evaluate its current capabilities but ran into an issue running through the quickstart. I am trying to set up a minimum GCP environment to play and experiment. I followed the quickstart material and everything seems to be provisioned and running well. I was able to load your example data and the GCP consoles all look green and good. However, when I try to access the data via the python serving api, I receive an error about "Unable to retrieve feature from Redis". Any insights into what might cause this and/or how I could go about diagnosing and debugging?

python3 - <<EOF
> from feast.sdk.client import Client
> from feast.sdk.resources.feature_set import FeatureSet
> 
> feast_client = Client(core_url="${FEAST_CORE_URI}", serving_url="${FEAST_SERVING_URI}")
> feature_set = FeatureSet(entity="entity_1", features=["entity_1.feature_1", "entity_1.feature_2"])
> 
> print(feast_client.get_serving_data(feature_set, entity_keys=["0","1"]))
> EOF

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "/Users/sam/anaconda3/lib/python3.7/site-packages/feast/sdk/client.py", line 269, in get_serving_data
    feature_set, self._serving_service_stub.QueryFeatures(request), start, end
  File "/Users/sam/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/sam/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "Unable to retrieve feature from Redis"
        debug_error_string = "{"created":"@1568054518.425475000","description":"Error received from peer ipv4:XXX.XXX.XXX.XXX:6565","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Unable to retrieve feature from Redis","grpc_status":13}"

[woop]

Hi @sbryfcz, can you give us a bit more details on how you have set up your environment? Did you follow the quickstart exactly? Are you seeing any other error messages on the logs (for example Serving)?

[sbryfcz]

Sure thing. I believe I have setup my environment following the Quickstart exactly.
I did use follow the following direction though:

• I removed the --no-address flag when setting up the REDIS store. I still set FEAST_SERVING_REDIS_HOST to the INTERNAL_IP address
• I removed core.service.annotations, core.service.loadBalancerSourceRanges, serving.service.annotations, serving.service.loadBalancerSourceRanges from feast-helm-values.yaml. See below for my current configuration.

I will note that the dataflow seems to succeed and shows data flowing into the redis store. Not sure if that helps with debugging.

feast-helm-values.yaml

global:
  postgresql:
    existingSecret: ${FEAST_HELM_RELEASE_NAME}-postgresql

core:
  projectId: ${GCP_PROJECT}
  jobs:
    runner: DataflowRunner
    options: '{"project": "${GCP_PROJECT}","region": "${GCP_REGION}","subnetwork": "regions/${GCP_REGION}/subnetworks/${GCP_SUBNETWORK}", "maxNumWorkers": "${GCP_DATAFLOW_MAX_NUM_WORKERS}", "autoscalingAlgorithm": "THROUGHPUT_BASED"}'
  service:
    type: LoadBalancer

serving:
  service:
    type: LoadBalancer

dataflow:
  projectID: ${GCP_PROJECT}
  location: ${GCP_REGION}

store:
  serving:
    options: '{"host": "${FEAST_SERVING_REDIS_HOST}", "port": 6379}'
  warehouse:
    options: '{"project": "${GCP_PROJECT}", "dataset": "${FEAST_WAREHOUSE_BIGQUERY_DATASET}"}'

serviceAccount:
  name: ${FEAST_HELM_RELEASE_NAME}-service-account
  key: service-account.json

[pradithya]

Hi @sbryfcz , that error is usually because serving service is unable to reach redis.

Can you check whether ${FEAST_SERVING_REDIS_HOST} is reachable from the service?

  serving:
    options: '{"host": "${FEAST_SERVING_REDIS_HOST}", "port": 6379}'

[NicholaiStaalung]

@sbryfcz did you solve it?

Im having the same issue. Did the config exactly like mentioned above or at least i followed the same steps.

@pradithya i did a curl https://${FEAST_SERVING_REDIS_HOST}:6379 from inside the gke-feast-default-pool and it just hangs.

[woop]

@sbryfcz did you solve it?

Im having the same issue. Did the config exactly like mentioned above or at least i followed the same steps.

@pradithya i did a curl https://${FEAST_SERVING_REDIS_HOST}:6379 from inside the gke-feast-default-pool and it just hangs.

Thanks for the interest @NicholaiStaalung!

I would recommend moving to Feast 0.3. You can find more information on installing it on this PR: #289

We've added quite a bit of functionality since Feast 0.1. We will be moving that over to the master branch shortly.

[NicholaiStaalung]

@sbryfcz did you solve it?
Im having the same issue. Did the config exactly like mentioned above or at least i followed the same steps.
@pradithya i did a curl https://${FEAST_SERVING_REDIS_HOST}:6379 from inside the gke-feast-default-pool and it just hangs.

Thanks for the interest @NicholaiStaalung!

I would recommend moving to Feast 0.3. You can find more information on installing it on this PR: #289

We've added quite a bit of functionality since Feast 0.1. We will be moving that over to the master branch shortly.

@woop thanks but i keep running into errors. Are you updating the quickstart guide? I cant find any in 0.3-dev.

Thanks for building Feast. Much needed!

[woop]

@sbryfcz did you solve it?
Im having the same issue. Did the config exactly like mentioned above or at least i followed the same steps.
@pradithya i did a curl https://${FEAST_SERVING_REDIS_HOST}:6379 from inside the gke-feast-default-pool and it just hangs.

Thanks for the interest @NicholaiStaalung!
I would recommend moving to Feast 0.3. You can find more information on installing it on this PR: #289
We've added quite a bit of functionality since Feast 0.1. We will be moving that over to the master branch shortly.

@woop thanks but i keep running into errors. Are you updating the quickstart guide? I cant find any in 0.3-dev.

Thanks for building Feast. Much needed!

Hi @NicholaiStaalung!

We're very close to shipping this release, but it's not officially out. Right now it can be found on the 0.3-dev branch. You can see an example of the Python SDK if you look at one of our end to end tests: https://github.com/gojek/feast/blob/0.3-dev/tests/e2e/test_e2e.py#L59

If you are ambitious you can also look at the contributing.md file on that 0.3-dev branch to see how to get the system up and running locally.

Otherwise I would love to get feedback once we release!

[NicholaiStaalung]

I think you can close this one. I had no problem retrieving features from the online serving with 0.3.x docker-compose installation

[woop]

Thanks @NicholaiStaalung