Implement best practices for coredns app

[LolloneS]

Coming from here.

The goal of this issue is to have the App eventually adhere to standard cloud-native best-practices. The points below are to be considered non-exhaustive. Feel free to add some that make sense and remove some that don't. If this issue does not make sense (e.g. is a duplicate or covers a deprecated app) please close it.

Expected outcome:

• the app has properly set PDBs (as percentages)
• the app can scale horizontally based on Custom Metrics (if it makes sense)
• the app runs with more than 1 replica by default (if it needs HA)
• the app does not use emptyDir volumes if not strictly necessary

[mcharriere]

the app has properly set PDBs (as percentages)

This app already has PDB as percentages. Reviewing the chart I noticed that the PDB and the updateStrategy are using the same values for its configuration.

This is not really desired and could potentially cause problems. I created a PR to decouple the values, leaving PDB maxUnavailable in 25%.

[mcharriere]

the app can scale horizontally based on Custom Metrics (if it makes sense)

I also added the option to use custom metrics for scaling in a new PR. Also created this issue https://github.com/giantswarm/giantswarm/issues/26942 for research, because scaling based on cluster size rather than usage might make more sense.

[mcharriere]

the app runs with more than 1 replica by default (if it needs HA)

It does already.

the app does not use emptyDir volumes if not strictly necessary

It does use emptyDir, and it seems to be well justified

[mcharriere]

released v1.17 with some improvements 🎉