synse server configuration merging failures on k8s when service clusterIP is not None

[edaniszewski]

Description

I noticed this a little while ago, just getting to opening the issue now.

When I was playing around with k8s configs for synse server, we had the service's clusterIP field as None. I commented this out when trying some things (e.g. so it used the default value for clusterIP) and it caused things to blow up.

vapor@vec1-c1-austin:~/tmp$ kubectl get pods -o wide
NAME                            READY     STATUS             RESTARTS   AGE       IP           NODE
rs485-plugin-7d56f57949-lm7s6   1/1       Running            0          56s       10.193.1.3   vec3-c1-austin
synse-75964b48cf-2bsfm          1/2       Error              3          56s       10.193.1.5   vec5-c1-austin
synse-75964b48cf-95b8x          1/2       Error              3          56s       10.193.1.2   vec2-c1-austin
synse-75964b48cf-9z4xr          1/2       Error              3          56s       10.193.1.1   vec1-c1-austin
synse-75964b48cf-mlk52          1/2       CrashLoopBackOff   3          56s       10.193.1.4   vec4-c1-austin
synse-75964b48cf-pqfwz          1/2       Error              3          56s       10.193.1.3   vec3-c1-austin
synse-75964b48cf-w4grd          1/2       Error              3          56s       10.193.1.6   vec6-c1-austin
vapor@vec1-c1-austin:~/tmp$ kubectl logs synse-75964b48cf-2bsfm
Error from server (BadRequest): a container name must be specified for pod synse-75964b48cf-2bsfm, choose one of: [synse-server i2c-plugin]
vapor@vec1-c1-austin:~/tmp$ kubectl logs synse-75964b48cf-2bsfm synse-server
Traceback (most recent call last):
  File "/synse/runserver.py", line 10, in <module>
    config.load()
  File "/synse/synse/config.py", line 104, in load
    parse_env_vars()
  File "/synse/synse/config.py", line 179, in parse_env_vars
    set(key, v)
  File "/synse/synse/config.py", line 281, in set
    scope = scope[k]
TypeError: string indices must be integers

My guess is that because the ENV parsing is looking for environment variables with a particular prefix, when the clusterIP is not None, k8s injects some ENV variables in there with the same SYNSE_ prefix. Something about those injected variables are making the configuration here pretty unhappy.

This isn't critical right now, but it is definitely a problem that needs to get solved. The first step is to find a way to reproduce this issue in a simple manner locally (e.g. not having to set things up on k8s..)

[edaniszewski]

i managed to recreate this locally with the following config

version: "3"
services:
  synse-server:
    container_name: synse-server
    image: vaporio/synse-server:2.0.0
    ports:
      - 5000:5000
    environment:
      - SYNSE_SYNSE_PORT=tcp://10.108.105.178:5001
      - SYNSE_SYNSE_PORT_5001_TCP=tcp://10.108.105.178:5001

And I believe I understand why this bug is happening so I'll tackle it next.