fix(pt): do not overwrite disp_file when restarting training (#3985)

New contents should be appended when restarting

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Improved training process by conditionally appending to files based on
the `restart_training` flag.
- Added functionality to create a record file when `SAMPLER_RECORD` is
true.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

deepmd/pt/train/training.py

@@ -630,7 +630,13 @@ def warm_up_linear(step, warmup_steps):
 
     def run(self):
         fout = (
-            open(self.disp_file, mode="w", buffering=1) if self.rank == 0 else None
+            open(
+                self.disp_file,
+                mode="w" if not self.restart_training else "a",
+                buffering=1,
+            )
+            if self.rank == 0
+            else None
         )  # line buffered
         if SAMPLER_RECORD:
             record_file = f"Sample_rank_{self.rank}.txt"