Skip to content

Commit

Permalink
Always close backend device before program exit (#258)
Browse files Browse the repository at this point in the history
If there's a host side crash or exception there is some device cleanup /
shutdown that doesn't automatically happen and will leave the device in
a hung state for the next run.

This change registers an atexit handler so that in most situations a
host side crash will not result in a hung device.
  • Loading branch information
nsmithtt authored Jul 31, 2024
1 parent 5d7fa3a commit cb68afd
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion runtime/tools/python/ttrt/common/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from pkg_resources import get_distribution
import sys
import shutil
import atexit

from ttrt.common.util import *

Expand Down Expand Up @@ -171,11 +172,11 @@ def run(args):

system_desc, device_ids = ttrt.runtime.get_current_system_desc()
device = ttrt.runtime.open_device(device_ids)
atexit.register(lambda: ttrt.runtime.close_device(device))
for loop in range(arg_loops):
ttrt.runtime.submit(device, fbb, 0, total_inputs[loop], total_outputs[loop])
print(f"finished loop={loop}")
print("outputs:\n", torch_outputs)
ttrt.runtime.close_device(device)

# save artifacts
for binary in binaries:
Expand Down

0 comments on commit cb68afd

Please sign in to comment.