Bypass duplicate message filtering on mapper startup with config flag

[Ruadhri17]

Proposed changes

TODO:

• fix tests
• add test
• replace sync_on_startup with better name

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#2605

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
390	0	3	390	100	1h1m28.051s

[albinsuresh]

Suggested change

	pub fn new<P>(log_dir: P, sync_on_startup: bool) -> Result<MessageLogWriter, std::io::Error>
	pub fn new<P>(log_dir: P, truncate: bool) -> Result<MessageLogWriter, std::io::Error>

From the MessageLogWriter, there is no notion of sync. The argument just decides whether to start fresh or append to the existing.

[codecov]

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (8181668) 75.9% compared to head (a1c8bda) 75.9%.
Report is 15 commits behind head on main.

Additional details and impacted files

Files	Coverage Δ
crates/core/c8y_api/src/json_c8y.rs	90.5% <100.0%> (+<0.1%)	⬆️
crates/extensions/c8y_mapper_ext/src/converter.rs	81.1% <100.0%> (+<0.1%)	⬆️
...s/extensions/c8y_mapper_ext/src/service_monitor.rs	95.3% <100.0%> (+<0.1%)	⬆️
crates/extensions/c8y_mapper_ext/src/tests.rs	91.8% <100.0%> (+<0.1%)	⬆️
.../tedge_config/src/tedge_config_cli/tedge_config.rs	81.1% <75.0%> (-0.1%)	⬇️
...tes/core/tedge_agent/src/software_manager/actor.rs	55.6% <0.0%> (ø)
...tes/core/tedge_agent/src/state_repository/state.rs	92.7% <0.0%> (ø)
crates/core/tedge_api/src/message_log.rs	84.8% <90.9%> (+0.8%)	⬆️
crates/extensions/c8y_mapper_ext/src/config.rs	45.8% <50.0%> (+<0.1%)	⬆️
crates/core/tedge_api/src/entity_store.rs	92.6% <72.2%> (-0.4%)	⬇️

... and 2 files with indirect coverage changes

[didier-wenzek]

This is working as expected. However, the code can be a bit simpler.

[didier-wenzek]

Passing a log_dir and clean_start flag telling to truncate or not the log.

I would consider to pass only a MessageLogWriter, built from the log using MessageLogWriter::new(log_dir) or MessageLogWriter::new_truncated(log_dir):

Suggested change

	log_dir: P,
	clean_start: bool,
	message_log: MessageLogWriter,

For that to work the MessageLogWriter has to be improved to build a MessageLogReader on the same log dir. So the entity store can use its self.message_log for reads and writes.

It might be better to address this only a follow-up PR after the 1.0 release.

[albinsuresh]

One suggestion is to have a MessageLog abstraction, which is passed to the entity store, out of which the MessageLogWriter and MessageLogReader instances can be retrieved.

[albinsuresh]

Could you align the @id values used in this test along the lines of #2657 ?

[albinsuresh]

Suggested change

	Re-enable Clean start
	Enable clean start

Just to make things clearer, it'd be better to explicitly do tedge config set c8y.entity_store.clean_start true, as the unset might be confusing to a reader who's not familiar with the default value.

[albinsuresh]

Since the timestamp assertion is already done in the previous test, we can either skip this whole block or at least avoid the looping and limit the check to only once.

[albinsuresh]

Why was this sleep required? That too at the end of this test. The reason isn't clear from the comment.

[Ruadhri17]

I noticed that sometimes tests just fail because of that test case:
Should Have MQTT Messages c8y/s/us message_contains=101,${prefix}plc date_from=${timestamp} minimum=2 maximum=2. It returns more messages than expected (It can be noticed for example in this run). The only solution I found to this was adding this sleep, but maybe someone else has better solution to that.