k8s logs dump/assertion

[pepoviola]

In the k8s provider we rely on the kubectl logs cmd for both log line assertion and dumping the logs and most of the time this works as expected, but k8s internally read the pod log file from the host (vms in our case) and this file is managed by the log rotation settings so could be the case that the cmd kubectl logs... doesn't return the full log of the pod since the command only read the current file. This can cause false errors in logs assertions (e.g https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4420770)

❌ bob: count of log lines containing "Block history download is complete" is 1 within 60 second…
✅ bob: reports block height is at least 12136 within 90 seconds

(but the log line is present https://grafana.teleport.parity.io/goto/rFFuUAHSg?orgId=1 )

And also cause issues with the log dumps that we store as artifacts in gitlab, since we rely in the same command and we receive the partial log and not the full one.

After consider a couple of options, like store the process log locally or use an external log collator (e.g loki) to assert/dump, we decide to change the log reader fn to read the log line directly from the host. For doing this we should change the pod manifest to mount the path where the host store the logs from pods (same way that promtail does) and add the logic to read the logs of the pod not only in the current file but also in the rotated files.
We should also introduce a new key in the settings section to allow users to customize the path where the logs are stored in the host and we mount in the pods.

cc: @michalkucharczyk