Some errors during deploy and invoke functions

[pzg250]

Describe the bug
Deployed vhive on 2 ubuntu20.04(1 master, 1 worker), cannot deploy and invoke example functions

To Reproduce
On master

1. source /etc/profile && pushd ./examples/deployer && go build && popd && ./examples/deployer/deployer
2. pushd ./examples/invoker && go build && popd && ./examples/invoker/invoker

Expected behavior
A clear and concise description of what you expected to happen.

Logs
pods status

ubuntu@ip-172-31-21-207:~/vhive$ kubectl get pods --all-namespaces
NAMESPACE          NAME                                       READY   STATUS      RESTARTS   AGE
istio-system       cluster-local-gateway-fffb9f589-bv2nc      1/1     Running     0          6m32s
istio-system       istio-ingressgateway-778db64bb6-rsknq      1/1     Running     0          6m32s
istio-system       istiod-85bf857c79-cjrkq                    1/1     Running     0          6m40s
knative-eventing   eventing-controller-6b5b744bfd-bkrqs       1/1     Running     0          6m12s
knative-eventing   eventing-webhook-75cdd7c68-dwpzw           1/1     Running     0          6m12s
knative-eventing   imc-controller-565df566f8-45vnv            1/1     Running     0          6m9s
knative-eventing   imc-dispatcher-5bf6c7d945-dwbv5            1/1     Running     0          6m9s
knative-eventing   mt-broker-controller-575d4c9f77-7rmdc      0/1     Pending     0          6m6s
knative-eventing   mt-broker-filter-746ddf5785-trjzl          0/1     Pending     0          6m7s
knative-eventing   mt-broker-ingress-7bff548b5b-b6j8s         0/1     Pending     0          6m7s
knative-serving    activator-5cc89f4c4d-x4ppb                 1/1     Running     0          6m18s
knative-serving    autoscaler-6fb596f4bb-2h57s                1/1     Running     0          6m18s
knative-serving    controller-6b5874c54-sh5t2                 1/1     Running     0          6m18s
knative-serving    default-domain-pn85m                       0/1     Completed   0          6m15s
knative-serving    domain-mapping-5b6c878f85-prj26            1/1     Running     0          6m18s
knative-serving    domainmapping-webhook-59f98dc77b-6njg7     1/1     Running     0          6m18s
knative-serving    net-istio-controller-777b6b4d89-jwvbj      1/1     Running     0          6m14s
knative-serving    net-istio-webhook-78665d59fd-xbdxd         1/1     Running     0          6m14s
knative-serving    webhook-79f8449d8f-ws65v                   1/1     Running     0          6m18s
kube-system        calico-kube-controllers-567c56ff98-qw68x   1/1     Running     0          7m35s
kube-system        calico-node-6sjwr                          1/1     Running     0          7m35s
kube-system        calico-node-jczfv                          1/1     Running     0          7m35s
kube-system        coredns-565d847f94-5rlsb                   1/1     Running     0          8m19s
kube-system        coredns-565d847f94-xqhfh                   1/1     Running     0          8m18s
kube-system        etcd-ip-172-31-21-207                      1/1     Running     0          8m31s
kube-system        kube-apiserver-ip-172-31-21-207            1/1     Running     0          8m31s
kube-system        kube-controller-manager-ip-172-31-21-207   1/1     Running     0          8m31s
kube-system        kube-proxy-6xlcf                           1/1     Running     0          8m19s
kube-system        kube-proxy-t564b                           1/1     Running     0          7m47s
kube-system        kube-scheduler-ip-172-31-21-207            1/1     Running     0          8m31s
metallb-system     controller-844979dcdc-x2v49                1/1     Running     0          7m31s
metallb-system     speaker-7jrfj                              1/1     Running     0          7m6s
metallb-system     speaker-8d4wp                              1/1     Running     0          7m20s
registry           docker-registry-pod-n7z68                  1/1     Running     0          6m15s
registry           registry-etc-hosts-update-g4nh6            1/1     Running     0          6m15s

Deploy

ubuntu@ip-172-31-21-207:~/vhive$ source /etc/profile && pushd ./examples/deployer && go build && popd && ./examples/deployer/deployer
~/vhive/examples/deployer ~/vhive
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading golang.org/x/sys v0.0.0-20191026070338-33540a1f6037
~/vhive
WARN[0002] Failed to deploy function pyaes-0, configs/knative_workloads/pyaes.yaml: exit status 1
Warning: Kubernetes default value is insecure, Knative may default this to secure in a future release: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation, spec.template.spec.containers[0].securityContext.capabilities, spec.template.spec.containers[0].securityContext.runAsNonRoot, spec.template.spec.containers[0].securityContext.seccompProfile
Creating service 'pyaes-0' in namespace 'default':

  0.162s The Route is still working to reflect the latest desired specification.
  0.223s Configuration "pyaes-0" is waiting for a Revision to become ready.
  0.728s Revision "pyaes-0-00001" failed with message: 0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..
  0.845s Configuration "pyaes-0" does not have any ready Revision.
**Error: RevisionFailed: Revision "pyaes-0-00001" failed with message: 0/2 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling..**
Run 'kn --help' for usage

INFO[0002] Deployed function pyaes-0
INFO[0005] Deployed function pyaes-1
INFO[0005] Deployed function helloworld-0
INFO[0005] Deployment finished

Invoke

ubuntu@ip-172-31-21-207:~/vhive$ pushd ./examples/invoker && go build && popd && ./examples/invoker/invoker
~/vhive/examples/invoker ~/vhive
go: downloading github.com/containerd/containerd v1.5.2
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/google/uuid v1.2.0
go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.20.0
go: downloading go.opentelemetry.io/contrib v0.20.0
go: downloading google.golang.org/grpc v1.47.0
go: downloading google.golang.org/protobuf v1.28.0
go: downloading golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
go: downloading go.opentelemetry.io/otel v0.20.0
go: downloading go.opentelemetry.io/otel/exporters/trace/zipkin v0.20.0
go: downloading go.opentelemetry.io/otel/sdk v0.20.0
go: downloading go.opentelemetry.io/otel/trace v0.20.0
go: downloading golang.org/x/net v0.0.0-20220722155237-a158d28d115b
go: downloading google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21
go: downloading github.com/openzipkin/zipkin-go v0.2.5
go: downloading go.opentelemetry.io/otel/metric v0.20.0
go: downloading golang.org/x/text v0.3.7
~/vhive
INFO[2023-05-31T05:51:18.849692611Z] Reading the endpoints from the file: endpoints.json
WARN[2023-05-31T05:51:18.890550474Z] Failed to invoke helloworld-0.default.192.168.1.240.sslip.io:80, err=rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/plain; charset=utf-8"
WARN[2023-05-31T05:51:20.111533843Z] Failed to invoke pyaes-0.default.192.168.1.240.sslip.io:80, err=rpc error: code = Unimplemented desc =
WARN[2023-05-31T05:51:21.113206730Z] Failed to invoke pyaes-1.default.192.168.1.240.sslip.io:80, err=rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/plain; charset=utf-8"
WARN[2023-05-31T05:51:21.864744566Z] Failed to invoke helloworld-0.default.192.168.1.240.sslip.io:80, err=rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/plain; charset=utf-8"
WARN[2023-05-31T05:51:22.853863164Z] Failed to invoke pyaes-0.default.192.168.1.240.sslip.io:80, err=rpc error: code = Unimplemented desc =
INFO[2023-05-31T05:51:23.850308374Z] Issued / completed requests: 5, 0
INFO[2023-05-31T05:51:23.850345393Z] Real / target RPS: 0.00 / 1
INFO[2023-05-31T05:51:23.850360212Z] Experiment finished!
INFO[2023-05-31T05:51:23.850372454Z] The measured latencies are saved in rps0.00_lat.csv

Notes
Currently, we support only Ubuntu 18 (x86) bare-metal hosts, however
we encourage the users to reports Issues that appear in different settings.
We will try to help and potentially include these scenarios into our CI
if given enough interest from the community.

[ustiugov]

what is your question? the error messages indicate the scheduler as the problem.

[pzg250]

Hi @ustiugov , my question is why I cannot deploy demo pyaes functions.

the error messages indicate the scheduler as the problem

oh, sorry, does that mean the cores of the cpu is not enough for running pods ? (system issue)

[ustiugov]

you can check the scheduler's logs. it's definitely one of the problems.