Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yieldthought/lt persist state #14891

Merged
merged 6 commits into from
Nov 8, 2024
Merged

Conversation

yieldthought
Copy link
Contributor

@yieldthought yieldthought commented Nov 8, 2024

Ticket

n/a

Problem description

Using lt for multiple runs of the same input would share log files, exiting lt by mistake would lose output and state

What's changed

Unique log index for each run, log files removed when they are removed from lt, lt state persisted between runs

Checklist

Copy link
Contributor

@cglagovichTT cglagovichTT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! Had one question. Approved

# Restore saved attributes
entry.status = (
"Cancelled"
if entry_data["status"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should status be "cancelled" if the status is in this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - this is to account for cases in which lt was terminated without its usual shutdown (e.g. ird timeout, connection lost etc). Those processes are gone and are not coming back.

@yieldthought yieldthought merged commit 8374441 into main Nov 8, 2024
117 of 119 checks passed
@yieldthought yieldthought deleted the yieldthought/lt-persist-state branch November 8, 2024 14:27
ct-clmsn pushed a commit to ct-clmsn/tt-metal that referenced this pull request Nov 12, 2024
* #0: Tie name of each log to a slot index

* #0: Detach log id's from visible index as this changes on deletion

* #0: remove logs when deleted from list

* #0: Persist lt status in logs/status.json

* #0: always reset on error code unless cancelled

* #0: Add X to clear all
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants