complex graph only expose endpoint for last metioned container and does not pass forward the output of the parent model

[maorlipchuk]

Hi all,

I was trying to make my own complex graph similar to what is described here:
https://docs.seldon.io/projects/seldon-core/en/latest/graph/inference-graph.html?highlight=graph#example-of-graph-with-pre-processor-and-post-processor

I used my own graph structure almost similar to the pre-processor example mentioned above:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: seldon-model
  namespace: seldon-system
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - name: steptwo
          image: docker056588154/mymodel:1.0.2.27
        - name: stepone
          image: docker056588154/translator:1.0.0.1
    graph:
      name: stepone
      endpoint:
        type: REST
      type: MODEL
      children:
      - name: steptwo
        endpoint:
          type: REST
        type: MODEL
        children: []
    name: example
    replicas: 1

Once I apply the following template, I can see that a new pod is being created containning 3 containers, stepone, steptwo and a sidecar
but I only see one endpoint that is being discovered which belong to the last model wrapper metioned in containers:

➜ k get endpoints
NAME                           ENDPOINTS                               AGE
seldon-model-example           xxx.xxx.xxx.xxx:8000,xxx.xxx.xxx.xxx:8000   22s
seldon-model-example-stepone   xxx.xxx.xxx.xxx:9001                      47s
seldon-model-example-steptwo   <none>                                  47s

further more when I use port-forward and pass a JSON payload it seems that the message does not get forward to steptwo:

kubectl port-forward -n seldon-system svc/seldon-model-example-stepone 9001:9001

I get only response from stepone model and seems that the message does not get forward to steptwo:

➜ curl -g http://localhost:9001/predict --data-urlencode 'json={"data": {"names": ["message"], "ndarray": ["Wie läuft dein Tag"]}}'
{"data":{"names":[],"ndarray":["How is your day"]},"meta":{}}

The logs of stepTwo are empty

[maorlipchuk]

Here is an opposite example when I switch the order of the containers stepone with steptwo in the template:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: seldon-model
  namespace: seldon-system
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - name: stepone
          image: docker056588154/translator:1.0.0.1
        - name: steptwo
          image: docker056588154/mymodel:1.0.2.27
    graph:
      name: stepone
      endpoint:
        type: REST
      type: MODEL
      children:
      - name: steptwo
        endpoint:
          type: REST
        type: MODEL
        children: []
    name: example
    replicas: 1

➜ k get endpoints
NAME                           ENDPOINTS                               AGE
seldon-model-example           xxx.xxx.xxx.xxx:8000,xxx.xxx.xxx.xxx:8000   54s
seldon-model-example-stepone   <none>                                  79s
seldon-model-example-steptwo   xxx.xxx.xxx.xxx:9001                      79s
seldon-webhook-service         10.110.66.163:443                       49d

kubectl port-forward -n seldon-system svc/seldon-model-example-steptwo 9001:9001

➜ curl -g http://localhost:9001/predict --data-urlencode 'json={"data": {"names": ["message"], "ndarray": ["Wie läuft dein Tag"]}}'
{"data":{"names":[],"ndarray":[7]},"meta":{}}

{"data":{"names":[],"ndarray":[7]},"meta":{}} is the dummy output of steptwo (What I expected to get when I passed this message in the first place to stepone)

[maorlipchuk]

@cliveseldon

[ukclivecox]

Seldon treats each component as an isolated container which means the developer does not need to know how each container is used in different graphs. Seldon adds a Service Orchestrator (a golang sidecar called the "executor") to manage the graph defined. So to get the graph functionality you need to call the top level endpoint. If you call any of the container endpoints you will just get the functionality exposed in that container only.

Hope that makes sense.

[maorlipchuk]

Hi Clive,
Thanks for the explanation, that does make more sense now. My mistake

I tried what you suggested and it worked

➜ kubectl port-forward -n seldon-system svc/seldon-mymodel-classifier-seldon-combiner 8000:8000
Forwarding from 127.0.0.1:8000 -> 8000
apiVersion: machinelearning.seldon.io/v1alpha2
Forwarding from [::1]:8000 -> 8000

and call a curl:
curl -g http://localhost:8000/api/v1.0/predictions --data-urlencode 'json={"data": {"names": ["message"], "ndarray": ["Wie läuft dein Tag"]}}''

Response:
{"data":{"names":[],"ndarray":[7]},"meta":{}}

[SwathiAunooru-wavelabs]

hi maorlipchuk I am trying something similar to yours, did you make any changes in the graph, and are using a different port number since initially you used 9001 and later you used 8000.

I have given a graph like this, The response I'm getting is only from one container and not getting passed to the second container.

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: graph-metadata-input
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: preprocess-rul-transform:latest
          name: node-input-transformer
        - image: postprocess-rul-transform:latest          
          name: node

    graph:
      name: node-input-transformer
      type: TRANSFORMER
      children:
      - name: node
        type: TRANSFORMER
        children: []
    name: example
    replicas: 1

Can you please help me with this
Thank you.