Introduce etcd-druid upgrade tests

[shreyas-s-rao]

How to categorize this issue?

/area quality testing
/kind test enhancement

What would you like to be added:

I would like etcd-druid to introduce druid-upgrade tests, with its own make target, which allow testing of upgrade of druid from one version to another. Possible steps/definitions are as follows:

• previous druid version (default: master branch)
• pre-upgrade system config: what kind of Etcds to be deployed and reconciled by old druid - single-node awake, multi-node hibernated, etc
• upgrade step, which deploys new druid version
• post-upgrade test steps:
  • check status reconciliation of new druid against all Etcds
  • modify Etcd spec if required (change replicas, change peer URL TLS, etc)
  • reconcile each Etcd
  • check for reconciliation success for each Etcd

Why is this needed:

The current e2e tests are designed to test the operations of a running druid instance, but do not test how a new version of druid would behave and operate on Etcd resources created and reconciled by previous versions of druid.