Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilize night and perf pipelines for FD2 merge #7502

Closed
tt-rkim opened this issue Apr 16, 2024 · 2 comments
Closed

Stabilize night and perf pipelines for FD2 merge #7502

tt-rkim opened this issue Apr 16, 2024 · 2 comments
Assignees
@tt-rkim @TT-billteng
Labels
ci-bug bugs found in CI infra-ci infrastructure and/or CI changes P0

Comments

@tt-rkim
Copy link
Collaborator

tt-rkim commented Apr 16, 2024

@jliangTT @davorchap to track today's work on stabilizing CI for FD2 merge.
@tt-rkim tt-rkim added infra-ci infrastructure and/or CI changes P0 ci-bug bugs found in CI labels Apr 16, 2024
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Get rid of duplicate entries and put some extra notes, re-add …
6b7cf76 
…bert because we're on 800MHz
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Put models nightly into direct workflow call to help w/ splitting
ab36d29
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Skip vit, issue 1508
3724738
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Convert some common models calls to use the file rather than u…
64183a2 
…se -k
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Split into 3 models runs scripts for better triaging
9f6e14a
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: models 2 - Convert most models in models 2 to use file path, e…
769b313 
…xcept for some falcon and bert tests, converting soon
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Add executable permissions to nightly model scripts, fix pytes…
2a96e7a 
…t import in vit, and separate test_demo.py out into separate file for metal_BERT_large_11
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Remove metal BERT large calls in performance pipeline because …
ad2375e 
…we run that nightly already
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Move test_bert_batch_dram.py totally into models 2 script beca…
6892b6d 
…use we run on both archs with some skips
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Add specific configs for falcon for single card and still enab…
c038038 
…le multi-chip falcon calls
tt-rkim added a commit that referenced this issue Apr 16, 2024
@tt-rkim
#7502: Skip mistral7b attention and bert batch dram tests on WH b0 be…
6637548 
…cause they fail / hang respectively
vtangTT added a commit that referenced this issue Apr 17, 2024
@vtangTT
#7502: models 1 - all clean GS-only model tests with skip_for_wormhole
506a5f9
vtangTT added a commit that referenced this issue Apr 17, 2024
@vtangTT
#7502: models 1 - all failing tests, with corresponding ticket #
46a1305
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Skip test_demo.py for wormhole_b0 for bert large 11
64a8096
vtangTT added a commit that referenced this issue Apr 17, 2024
@vtangTT
#7502: models 3 - skip 2 mamba + 1 mistral test
5a5a575
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Revert "#7502: Skip test_demo.py for wormhole_b0 for bert larg…
d945706 
…e 11"

This reverts commit 64a8096.
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Get rid of duplicate entries and put some extra notes, re-add …
607bd21 
…bert because we're on 800MHz
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Put models nightly into direct workflow call to help w/ splitting
7e98bb9
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Skip vit, issue 1508
7cb5668
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Convert some common models calls to use the file rather than u…
eb6be48 
…se -k
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Split into 3 models runs scripts for better triaging
13f0411
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: models 2 - Convert most models in models 2 to use file path, e…
570a2dc 
…xcept for some falcon and bert tests, converting soon
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Add executable permissions to nightly model scripts, fix pytes…
7db50a4 
…t import in vit, and separate test_demo.py out into separate file for metal_BERT_large_11
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Remove metal BERT large calls in performance pipeline because …
eafbbc0 
…we run that nightly already
tt-rkim added a commit that referenced this issue Apr 17, 2024
@tt-rkim
#7502: Move test_bert_batch_dram.py totally into models 2 script beca…
309484e 
…use we run on both archs with some skips
dimitri-tenstorrent pushed a commit that referenced this issue Apr 19, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Revert "#7502: REVERT ME Add --collect-only to test scripts"
35876f3 
This reverts commit 59ceb6d.
dimitri-tenstorrent pushed a commit that referenced this issue Apr 19, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Separate ttnn nightly into its own job
3b41962
tt-rkim added a commit that referenced this issue Apr 19, 2024
@tt-rkim
#7502: Revert "#7599: Fix resnet on GS, fixup padding to align shard …
8b34e79 
…and grid shape" because it slowed down device perf significantly

This reverts commit b005e21.
tt-rkim added a commit that referenced this issue Apr 19, 2024
@tt-rkim
#7502: Skip resnet 18 ttnn functional nightly test for now because we…
c117b2f 
… can't fix it in time
tt-rkim added a commit that referenced this issue Apr 19, 2024
@tt-rkim
#7502: Skip resnet 18 ttnn functional nightly test for now because we…
07d114d 
… can't fix it in time
kevinmiTT11 pushed a commit that referenced this issue Apr 20, 2024
@tt-rkim @kevinmiTT11
#7502: Skip resnet 18 ttnn functional nightly test for now because we…
2203515 
… can't fix it in time
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
266200f 
…730 because we seem to have a real gain
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
d80aea1 
…730 because we seem to have a real gain
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
7bd5b44 
…730 because we seem to have a real gain
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
402f4b8 
…730 because we seem to have a real gain
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
23366b6 
…730 because we seem to have a real gain
tt-rkim added a commit that referenced this issue Apr 22, 2024
@tt-rkim
#7502: Increase lofi batch 16 (old) resnet device perf threshold to 6…
77ec48b 
…730 because we seem to have a real gain
@TT-billteng
Copy link
Collaborator

we have stabiliized nightly and perf pipelines in preparation for FD2 merge

dimitri-tenstorrent pushed a commit that referenced this issue Apr 26, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Put models nightly into direct workflow call to help w/ split…
0806702 
…ting
dimitri-tenstorrent pushed a commit that referenced this issue Apr 26, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Split into 3 models runs scripts for better triaging
a6d2ee9
dimitri-tenstorrent pushed a commit that referenced this issue Apr 26, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Take out run_models.sh from tests path and re-add api tests
a938caa
dimitri-tenstorrent pushed a commit that referenced this issue Apr 26, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Consolidate all FD nightly tests into one job
ea9fb2c
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Put models nightly into direct workflow call to help w/ split…
1bf108a 
…ting
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Split into 3 models runs scripts for better triaging
c62fd2a
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Take out run_models.sh from tests path and re-add api tests
03f0858
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Consolidate all FD nightly tests into one job
c99ea8c
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Put models nightly into direct workflow call to help w/ split…
7504748 
…ting
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Split into 3 models runs scripts for better triaging
5c4a920
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Take out run_models.sh from tests path and re-add api tests
dd708bb
dimitri-tenstorrent pushed a commit that referenced this issue Apr 30, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Consolidate all FD nightly tests into one job
819b8d2
dimitri-tenstorrent pushed a commit that referenced this issue May 1, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Put models nightly into direct workflow call to help w/ split…
aaf8b24 
…ting
dimitri-tenstorrent pushed a commit that referenced this issue May 1, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Split into 3 models runs scripts for better triaging
4dfd523
dimitri-tenstorrent pushed a commit that referenced this issue May 1, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Take out run_models.sh from tests path and re-add api tests
7f6a1b0
dimitri-tenstorrent pushed a commit that referenced this issue May 1, 2024
@tt-rkim @dimitri-tenstorrent
#7502: Consolidate all FD nightly tests into one job
b4bb3fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tt-rkim tt-rkim

@TT-billteng TT-billteng

Labels
ci-bug bugs found in CI infra-ci infrastructure and/or CI changes P0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tt-rkim @TT-billteng