Single daemonset vs one per deployment

[AverageMarcus]

Look at implementing an option to pick between:

• having a single (or as few as possible - see Respect node affinity #3) daemonset with all images pulled by it
• having a dedicated daemonset created per deployment or possibly per image.

The former (single DS) is "cleaner" as there's less noise when looking up resources but a single image failing to pull will mark the whole pod as having a problem.

The latter could potentially end up with many daemonsets and therefore many pods running on every node. This could lead to pod exhaustion (or in the case of AWS EKS, IP exhaustion).

---

From what I've been able to find out (but not confirm) there is no limit on the number of containers a pod can contain. The limit comes from the number of containers the node can suport as a whole. There is a limit to the number of pods though (usually 100 by default but that's not always the case - e.g. AWS EKS).
Tests using Kind have been able to succesfully create pods with at least 100 300 containers.

[AverageMarcus]

Thinking about the numbers:

Single daemonset

Given the scenario where we have:
50 deployments in our cluster, each requiring 3 replicas
5 nodes in our cluster

Total deployment pods = 150
Assuming even spread over the nodes that leave 30 pods per node (plus any other pods from daemonsets, stsatefulsets, etc)

With a single DaemonSet for our pre-fetch we then use ✨ 31 ✨ pods per node

Daemonset per deployment

Given the same scenario as above but with a DaemonSet per deployment we'd end up with ✨ 80 ✨ pods per node

---

Implications

With the scenario above the latter option could end up having significant cost implications (due to using up more limited "resource")

For example, if our scenario was deployed onto AWS EKS with all 5 nodes using t3.large instances the first option would likely be fine as each node suports a max of 36 IPs and therefore supports 35 pods per node.

But our second option would lead to pod exhaustion, preventing new pods from being scheduled. We can't solve this by increasing the number of nodes in our cluster as the number of deployments we're looking to prefetch (50) is already above the max pod limit of each node (35). To use this option we'd need to change instance type to one that supports more IPs, the cheapest that supports enough IPs is a m2.xlarge. Making this change will increase the cost of nodes by more than double!

---

I think I've talked myself out of this feature but would be good to know if there are any valid use cases that would benefit from suporting multiple daemonsets.

[alexellis]

Here's an idea. Make it configurable, then the operator decides?

[hholst80]

FWIW, I think that creating DaemonSets as a side effect of creating Deployments would cause operational complexity. I would never go with that unless I had a very specific need for it and for now I can not imagine a use case.