Support Canary parallel inference #9517

karpnv · 2024-06-21T13:27:34Z

What does this PR do ?

Support Canary at transcribe_speech_parallel.py script

Collection: ASR

#python3 ./examples/asr/transcribe_speech_parallel.py model=./canary-1b.nemo predict_ds.manifest_filepath=./manifest.json output_path=/tmp trainer.devices=-1

PR Type:

[ V] New Feature
Bugfix
Documentation

Who can review?

@pzelasko

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…llel

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

examples/asr/transcribe_speech_parallel.py

pzelasko · 2024-06-24T14:47:21Z

examples/asr/transcribe_speech_parallel.py

@@ -109,7 +112,9 @@ class ParallelTranscriptionConfig:
    # att_context_size can be set for cache-aware streaming models with multiple look-aheads
    att_context_size: Optional[list] = None

-    trainer: TrainerConfig = TrainerConfig(devices=-1, accelerator="gpu", strategy="ddp")
+    trainer: TrainerConfig = TrainerConfig(
+        devices=-1, accelerator="gpu", strategy="ddp", use_distributed_sampler=False


be careful with distributed sampler setting here: non-lhotse datasets still likely require True. it might be better to just override this for EncDec model?

ok, rm use_distributed_sampler=False

pzelasko · 2024-06-24T14:51:07Z

nemo/collections/asr/data/audio_to_text_lhotse_prompted.py

@@ -72,7 +72,7 @@ def __getitem__(self, cuts: CutSet) -> tuple[torch.Tensor, torch.Tensor, torch.T
            prompts = None
            prompts_lens = None

-        return audio, audio_lens, prompts_with_answers, prompts_with_answers_lens, prompts, prompts_lens
+        return audio, audio_lens, prompts_with_answers, prompts_with_answers_lens, prompts, prompts_lens, cuts


this is not ideal as returning cuts here will transfer the data held in-memory across dataloading worker subprocesses to the main training/inference loop process. we should return cuts.drop_recordings() instead.

pzelasko · 2024-06-24T15:59:06Z

+ like we talked offline, we have to work around global_rank being set incorrectly before trainer.predict() is called - the dataloader has to be initialized with the correct global_rank (and world_size) in order for lhotse's distributed sampler to work correctly

titu1994 · 2024-06-24T19:23:39Z

trainer.global_rank is apriori set by PTL in slurm environment. It does not require model to be built or it's functions to be called c

Is this a case where PTL cannot apriori detect global rank ?

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…llel

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…karpnv/canary_parallel

…llel

github-actions · 2024-07-16T01:51:03Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2024-07-24T01:50:36Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…llel

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…llel

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

…llel

github-actions · 2024-09-08T01:58:52Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

github-actions · 2024-09-15T02:01:16Z

This PR was closed because it has been inactive for 7 days since being marked as stale.

…llel

karpnv · 2024-09-27T15:47:48Z

Jenkins

github-actions · 2024-09-27T15:54:42Z

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a CICD pipeline for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day!

//cc @ko3n1g

github-actions · 2024-09-27T18:08:13Z

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a CICD pipeline for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day!

//cc @ko3n1g

github-actions · 2024-09-27T21:28:12Z

[🤖]: Hi @karpnv 👋,

I just wanted to let you know that, you know, a CICD pipeline for this PR just finished successfully ✨

So it might be time to merge this PR or like to get some approvals 🚀

But I'm just a 🤖 so I'll leave it you what to do next.

Have a great day!

//cc @ko3n1g

* add Canary cats Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler=False Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * update lhotse Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * fix global_rank Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * OmegaConf.set_struct Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * review fix Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * predict_step return Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> --------- Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: karpnv <karpnv@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: karpnv <karpnv@users.noreply.github.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com>

* add Canary cats Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler=False Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * update lhotse Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * fix global_rank Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * OmegaConf.set_struct Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * review fix Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * predict_step return Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> --------- Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: karpnv <karpnv@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: karpnv <karpnv@users.noreply.github.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com> Signed-off-by: Lifu Zhang <tomzhanglf@gmail.com>

* add Canary cats Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler=False Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * rm use_distributed_sampler Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * update lhotse Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * fix global_rank Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * OmegaConf.set_struct Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * review fix Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> * Apply isort and black reformatting Signed-off-by: karpnv <karpnv@users.noreply.github.com> * Apply isort and black reformatting Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> * predict_step return Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> --------- Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com> Signed-off-by: karpnv <karpnv@users.noreply.github.com> Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: pzelasko <pzelasko@users.noreply.github.com> Co-authored-by: karpnv <karpnv@users.noreply.github.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Co-authored-by: pzelasko <pzelasko@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

karpnv added 2 commits June 21, 2024 06:21

add Canary cats

0064392

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

e9cfcb6

…llel

github-actions bot added the ASR label Jun 21, 2024

karpnv requested a review from pzelasko June 21, 2024 13:28

Apply isort and black reformatting

c1317d2

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

github-advanced-security bot found potential problems Jun 21, 2024

View reviewed changes

examples/asr/transcribe_speech_parallel.py Fixed Show fixed Hide fixed

pzelasko requested changes Jun 24, 2024

View reviewed changes

karpnv and others added 6 commits July 1, 2024 09:51

rm use_distributed_sampler=False

fe731b0

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

merge

2530414

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

6f670ff

…llel

Apply isort and black reformatting

c1bc97d

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

rm use_distributed_sampler

f574488

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'karpnv/canary_parallel' of github.com:NVIDIA/NeMo into …

d872dae

…karpnv/canary_parallel

karpnv requested a review from pzelasko July 1, 2024 17:07

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

98eed54

…llel

github-actions bot added the stale label Jul 16, 2024

github-actions bot closed this Jul 24, 2024

karpnv added 3 commits August 1, 2024 05:48

merge main

aaec974

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

update lhotse

39022b8

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

1fd1052

…llel

karpnv reopened this Aug 2, 2024

karpnv and others added 4 commits August 2, 2024 14:33

Apply isort and black reformatting

77b38cd

Signed-off-by: karpnv <karpnv@users.noreply.github.com>

fix global_rank

ff858a2

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

1e4e0ee

…llel

merge

f93bf12

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

pzelasko added the Run CICD label Aug 2, 2024

Merge branch 'main' into karpnv/canary_parallel

9101b0c

pzelasko added Run CICD and removed Run CICD labels Aug 21, 2024

karpnv added 2 commits August 24, 2024 07:45

predict_step return

1fef840

Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

8f606ff

…llel

karpnv dismissed pzelasko’s stale review via 8f606ff August 24, 2024 14:45

github-actions bot added the stale label Sep 8, 2024

github-actions bot closed this Sep 15, 2024

karpnv reopened this Sep 16, 2024

Merge branch 'main' into karpnv/canary_parallel

f98f01c

pzelasko added Run CICD and removed stale Run CICD labels Sep 16, 2024

karpnv added 2 commits September 17, 2024 07:25

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

04d8c02

…llel

Merge branch 'main' of github.com:NVIDIA/NeMo into karpnv/canary_para…

1f1d164

…llel

karpnv added Run CICD and removed Run CICD labels Sep 27, 2024

pzelasko approved these changes Sep 27, 2024

View reviewed changes

karpnv merged commit cbb1344 into main Sep 27, 2024
295 of 302 checks passed

karpnv deleted the karpnv/canary_parallel branch September 27, 2024 22:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Canary parallel inference #9517

Support Canary parallel inference #9517

karpnv commented Jun 21, 2024

pzelasko Jun 24, 2024

karpnv Jul 1, 2024

pzelasko Jun 24, 2024

karpnv Jul 1, 2024

pzelasko commented Jun 24, 2024 •

edited

Loading

titu1994 commented Jun 24, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Sep 8, 2024

github-actions bot commented Sep 15, 2024

karpnv commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

Support Canary parallel inference #9517

Support Canary parallel inference #9517

Conversation

karpnv commented Jun 21, 2024

What does this PR do ?

Who can review?

pzelasko Jun 24, 2024

Choose a reason for hiding this comment

karpnv Jul 1, 2024

Choose a reason for hiding this comment

pzelasko Jun 24, 2024

Choose a reason for hiding this comment

karpnv Jul 1, 2024

Choose a reason for hiding this comment

pzelasko commented Jun 24, 2024 • edited Loading

titu1994 commented Jun 24, 2024

github-actions bot commented Jul 16, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Sep 8, 2024

github-actions bot commented Sep 15, 2024

karpnv commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

github-actions bot commented Sep 27, 2024

pzelasko commented Jun 24, 2024 •

edited

Loading