Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix n300 perf #648

Closed
nsmithtt opened this issue Sep 8, 2024 · 2 comments · Fixed by #668
Closed

Fix n300 perf #648

nsmithtt opened this issue Sep 8, 2024 · 2 comments · Fixed by #668
Assignees

Comments

@nsmithtt
Copy link
Contributor

nsmithtt commented Sep 8, 2024

Docker fails to initialize:
https://github.com/tenstorrent/tt-mlir/actions/runs/10760903111/job/29839914911

Temporarily disabled with this PR:

@nsmithtt
Copy link
Contributor Author

nsmithtt commented Sep 8, 2024

In case the job link falls off, during the "Starting job container" step:

 /usr/bin/docker --config /home/ubuntu/actions-runner/_work/_temp/.docker_973b7022-4ae3-40c8-a61b-5c08b46f0ec4 login ghcr.io -u nsmithtt --password-stdin
  /usr/bin/docker --config /home/ubuntu/actions-runner/_work/_temp/.docker_973b7022-4ae3-40c8-a61b-5c08b46f0ec4 pull ghcr.io/tenstorrent/tt-mlir/tt-mlir-ci-ubuntu-22-04:latest
  latest: Pulling from tenstorrent/tt-mlir/tt-mlir-ci-ubuntu-22-04
  857cc8cb19c0: Pulling fs layer
  6466f0a577d8: Pulling fs layer
  8ff923add0ad: Pulling fs layer
  67d3f3e9ee23: Pulling fs layer
  66a75da9d3d2: Pulling fs layer
  31027ed44b38: Pulling fs layer
  2d0738d2d2f6: Pulling fs layer
  4f4fb700ef54: Pulling fs layer
  897fcff62162: Pulling fs layer
  9a316c1f5da0: Pulling fs layer
  ffd586f61e55: Pulling fs layer
  87bb63c25e12: Pulling fs layer
  cfc59818bf6b: Pulling fs layer
  c4cc7922b9ec: Pulling fs layer
  67d3f3e9ee23: Waiting
  da0ef11a4966: Pulling fs layer
  66a75da9d3d2: Waiting
  2d0738d2d2f6: Waiting
  31027ed44b38: Waiting
  4f4fb700ef54: Waiting
  897fcff62162: Waiting
  9a316c1f5da0: Waiting
  ffd586f61e55: Waiting
  87bb63c25e12: Waiting
  cfc59818bf6b: Waiting
  da0ef11a4966: Waiting
  c4cc7922b9ec: Waiting
  857cc8cb19c0: Verifying Checksum
  857cc8cb19c0: Download complete
  67d3f3e9ee23: Verifying Checksum
  67d3f3e9ee23: Download complete
  66a75da9d3d2: Verifying Checksum
  66a75da9d3d2: Download complete
  31027ed44b38: Verifying Checksum
  31027ed44b38: Download complete
  8ff923add0ad: Verifying Checksum
  8ff923add0ad: Download complete
  4f4fb700ef54: Verifying Checksum
  4f4fb700ef54: Download complete
  2d0738d2d2f6: Download complete
  9a316c1f5da0: Verifying Checksum
  9a316c1f5da0: Download complete
  6466f0a577d8: Verifying Checksum
  6466f0a577d8: Download complete
  857cc8cb19c0: Pull complete
  ffd586f61e55: Verifying Checksum
  ffd586f61e55: Download complete
  cfc59818bf6b: Verifying Checksum
  cfc59818bf6b: Download complete
  c4cc7922b9ec: Verifying Checksum
  c4cc7922b9ec: Download complete
  da0ef11a4966: Verifying Checksum
  da0ef11a4966: Download complete
  897fcff62162: Verifying Checksum
  897fcff62162: Download complete
  87bb63c25e12: Verifying Checksum
  87bb63c25e12: Download complete
  failed to register layer: lsetxattr security.capability /usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper: operation not supported
  Warning: Docker pull failed with exit code 1, back off 8.341 seconds before retry.

nsmithtt added a commit that referenced this issue Sep 8, 2024
@vmilosevic
Copy link
Contributor

Looks like the mounted volume dosnt support extended attibutes and cant be used for docker cache.

87bb63c25e12: Download complete
  failed to register layer: lsetxattr security.capability /usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper: operation not supported
  Warning: Docker pull failed with exit code 1, back off 8.341 seconds before retry.

As a part of tenstorrent/tt-forge-fe#211 I tried setting up additional storage to VMs and using that as docker cache. Mounted volume doesn't support extended attributes, so this won't work.

ubuntu@n300-2:/mnt/dockercache$ touch testfile
ubuntu@n300-2:/mnt/dockercache$ setfattr -n user.testattr -v "testvalue" testfile
setfattr: testfile: Operation not supported

I'm disabling n300-2 by turning off "in-device" label until this is fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants