feat(loki-canary): Add support to push logs directly to Loki.

[kavirajk]

What this PR does / why we need it:
Add additional push mode to loki-canary, which pushes the logs directly to given Loki URL as it generates logs.

The real function of Loki Canary is to act like a tenant and help us know the whether Loki installation is working as perceived by a real tenant.

Main rationale for this additional push mode is to make canary more standalone without needing for promtail (or grafana-agent) to scrape it's logs and send to loki, with this change, Loki canary happily tests Loki behavior without needing any other dependencies.

NOTE:
If you run Loki behind any proxy that has different authorization policies to READ or WRITE to Loki, then important change that canary operator need to be aware of it, now the user credentials they pass via -user and -pass to access loki endpoints need to have both read' and write` permissions (previously canary just query the logs where as promtail is the one pushes the logs, so just READ permissions was sufficient).

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
There will be follow up PR(s) to cleanup, particularly reader and comparitor component in terms of logging with proper logger. Rationale is to keep the changes small per PR to make it easy to review.

This changes were tested it in one of the internal Loki dev cell.

This PR is a no-op if this new push flag is disabled (it's disabled by default)

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[dannykopping]

Raised a couple points here, let's address before we merge because I see you already have one approval.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0.4%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[dannykopping]

Thanks @kavirajk! One comment about logging, then I think we're good to go

[dannykopping]

LGTM!

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%