Loki: Do not store exact log line duplicates (now including unordered inserts)

[slim-bean]

What this PR does / why we need it:

The behavior for Loki handling of log lines has been like this:

• Multiple log lines at the exact same timestamp to the ns precision which have different content are accepted and stored
• Multiple log lines at the exact same timestamp to the ns precision which have the same content are deduplicated upon insertion into a chunk.

This behavior was changed when we added support for receiving and storing unordered logs.

This PR partially addresses the issue by making sure that inserts into the unordered head block are deduped according to those rules.

However, I believe it's still possible to insert exact duplicates if it's done so across multiple blocks in a chunk, and this is very difficult to solve for, instead I think we should accept these and de-dupe them at query time.

At query time we de-duplicate exact lines across chunks but not within a single chunk (because the rules were such that we would never have exact dupes in a single chunk), but I think to completely handle this we would need to have exact de-duplication within a chunk at query time also.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[trevorwhitney]

Are we missing a test for the unordered case, or am I missing something?

[kavirajk]

@slim-bean we may not need this workaround if we make /push safe to retry
#6650

[owen-d]

I prefer this behavior. It should also handle deduping across blocks as-is because we rebuild out of order chunks before flushing to ensure that once flushed to storage, all chunks maintain internal ordering for performance: https://github.com/grafana/loki/blob/main/pkg/chunkenc/memchunk.go#L914

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0.1%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0.1%
+              logql	0%
+               loki	0%