LED Statuses

[geerlingguy]

For the Kubernetes cluster, it might be more interesting to set LEDs based on the Kubernetes node status (e.g. what you get with kubectl get nodes).

• Green = Ready
• Red = NotReady (or any other status for now?)

[geerlingguy]

So I think, in order to get this working with the PoE HATs in place, I'm going to go with a pre-fabbed solution, the BlinkStick Nano. It has a python library which should be (hopefully) easy enough to wire up: https://github.com/arvydas/blinkstick-python

One pattern that could work is to have a DeamonSet running a pod on each node which just sets the LED status. The pod could have a cluster role that gets node status or something, maybe? See some docs here: https://kubernetes.io/docs/tasks/debug-application-cluster/monitor-node-health/

It would be really nice if there were some pre-hardwired way to just run a command on the node itself and say "hey, am I joined and Ready in a Kubernetes cluster right now?"

[geerlingguy]

Just playing around with the BlinkStick Nano I received in the mail today on my Mac:

# Install libusb (dependency of `blinkstick` Python library).
$ brew install libusb

# Install `blinkstick` Python library.
$ pip install blinkstick

# Make the Blinkstick make one red pulse (top LED).
$ blinkstick --pulse red

# Turn the Blinkstick red (top LED).
$ blinkstick --set-color red

# Turn the Blinkstick green (top LED).
$ blinkstick --set-color green

# Turn the Blinkstick off
$ blinkstick off

# Set mode to 3 so both top and bottom LEDs light.
$ blinkstick --set-mode 3
$ blinkstick --set-color red

Worked great, this seems pretty spiffy!

Note that if I set the mode to 1, it seems to never let me change it back to 2 or 3. Not sure what mode 1 is used for. Upstream issue: arvydas/blinkstick-python#59

[geerlingguy]

So for the simplistic purposes of this issue, I would want:

• Ansible installs the blinkstick pip package
• Have a script run at reboot that runs:

  blinkstick --set-mode 3 && blinkstick --set-color red
  

• Have a cron script that runs every minute and runs:

  # TODO: Something to see if node is Ready or NotReady, setting:
  blinkstick --set-color red # for NotReady
  blinkstick --set-color green # for Ready
  

[geerlingguy]

Additionally, I would like to have a few Ansible playbooks I can run which show how orchestration works—e.g. turn all green, turn all blue. Then do it serially, then in sets of two...

[geerlingguy]

Tasks remaining:

• Break out GPIO and Blinkstick setup into separate roles (so either can be used)
• Set up the Blinkstick role (basics)
• Build Python script which checks status of Kubernetes node every 5s or 10s and turns LED green if change to Ready, red if change to NotReady
• Build a few scripts that allow easy remote control of LEDs (for use with demoing Ansible ad-hoc/playbook orchestration).

[geerlingguy]

Created issue to ask about Kubernetes node status monitoring: How can I check a Kubernetes node's status from the node itself?

[geerlingguy]

curl http://localhost:10248/healthz

So maybe a Python script / systemctl service that runs and checks every 10s if that URL returns ok. If any other condition (e.g. no response, failure to connect, or some other message), turn it red. If ok, and blinkstick is currently green, leave it green.

Also the @reboot cron job to power up the red LED seems to not currently be working :/

[geerlingguy]

Follow-up issue #143 for the other scripts. For now I have a totally-awesome dramble-node-monitor service that can be started and stopped cleanly with systemctl. And when you start it, it does green for Ready, red for NotReady... then when you stop it it turns off the LED. Very nice!

[geerlingguy]

[geerlingguy]

https://twitter.com/geerlingguy/status/1098070027645812736