Condition ConfigurationsReady is True but LCR != LRR

[navidshaikh]

In what area(s)?

/area API

What version of Knative?

nightly build - Nov 18
from: https://storage.googleapis.com/knative-nightly/serving/latest/serving.yaml

Expected Behavior

LCR = latestCreatedRevision
LRR = latestReadyRevision

After service update, when service's all 3 conditions reporting True, expected LCR = LRR.

Actual Behavior

After service update, when service reported status as True, LCR != LRR. After a few seconds, service's status then report LCR = LRR.

Expected ConfigurationsReady condition to set correct revision name in LRR before it reported True.

Steps to Reproduce the Problem

1. Setup
   • minikube: 1.5.2
   • kube: v1.14.0
   • knative serving: nightly
   • Istio lean : 1.2.7
2. Save the following test script to create a service, generate 3 revisions and get the status of conditions, LRR, LCR.

echo "Create a new service 'echo'"
cat <<-EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: svc
spec:
  template:
    metadata:
      name: svc-v1
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
EOF

echo "Update 'echo' to generate 2nd revision"
cat <<-EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: svc
spec:
  template:
    metadata:
      name: svc-v2
    spec:
      containers:
      - env:
        - name: TARGET
          value: v2
        image: gcr.io/knative-samples/helloworld-go
  traffic:
  - latestRevision: true
    percent: 10
  - revisionName: svc-v1
    percent: 90
EOF

echo "Update 'echo' to generate 3rd revision"
cat <<-EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: svc
spec:
  template:
    metadata:
      name: svc-v3
    spec:
      containers:
      - env:
        - name: TARGET
          value: v3
        image: gcr.io/knative-samples/helloworld-go
  traffic:
  - latestRevision: true
    percent: 10
  - revisionName: svc-v1
    percent: 45
  - revisionName: svc-v2
    percent: 45
EOF

for i in {1..100}; do
  kubectl get ksvc svc -o=jsonpath="{range .status.conditions[*]}{.type}{':'}{.status}{' '}{end}{' '}{.status.}{'LCR: '}{.status.latestCreatedRevisionName}{' '}{'LRR: '}{.status.latestReadyRevisionName}"
  echo ""
  kubectl get ksvc svc | grep True
  if [[ $? != 0 ]]; then
    continue
  fi
  echo "reconciled.."
  break
done

sleep 5
echo "After sleep 5"
kubectl get ksvc svc -o=jsonpath="{range .status.conditions[*]}{.type}{':'}{.status}{' '}{end}{' '}{.status.}{'LCR: '}{.status.latestCreatedRevisionName}{' '}{'LRR: '}{.status.latestReadyRevisionName}"
echo ""

kubectl delete ksvc svc

3. Run the test script

  ✗ bash lrr.sh 
  1 Create a new service 'echo'
  2 service.serving.knative.dev/svc created
  3 Update 'echo' to generate 2nd revision
  4 service.serving.knative.dev/svc configured
  5 Update 'echo' to generate 3rd revision
  6 service.serving.knative.dev/svc configured
  7 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v2 LRR: 
  8 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
  9 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 10 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 11 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 12 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 13 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 14 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 15 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 16 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 17 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 18 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 19 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 20 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 21 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 22 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 23 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 24 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 25 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 26 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 27 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 28 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 29 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 30 ConfigurationsReady:Unknown Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: 
 31 ConfigurationsReady:True Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: svc-v2
 32 ConfigurationsReady:True Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: svc-v2
 33 ConfigurationsReady:True Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: svc-v2
 34 ConfigurationsReady:True Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: svc-v2
 35 ConfigurationsReady:True Ready:Unknown RoutesReady:Unknown  LCR: svc-v3 LRR: svc-v2
 36 ConfigurationsReady:True Ready:True RoutesReady:True  LCR: svc-v3 LRR: svc-v2
 37 svc    http://svc.default.example.com   svc-v3          svc-v2        True                                                                                                                
 38 reconciled..
 39 After sleep 5
 40 ConfigurationsReady:True Ready:True RoutesReady:True  LCR: svc-v3 LRR: svc-v3
 41 service.serving.knative.dev "svc" deleted   

Line no: 36 service reports all the conditions as True, while LCR != LRR.
After a few seconds we see at line no: 40, LCR = LRR.

Question: Should service's condition ConfigurationsReady report True only after LRR is set to LCR in status?

[navidshaikh]

This is seen like 2 out of 5 times when the mentioned test script is ran. In client we're getting e2e tests failure against serving nightly job specifically in traffic splitting e2e tests where multiple service create/update operations are performed (knative/client#500). CI jobs against serving v0.10.0 or prior run fine.

[taragu]

/assign