Question

[naymaraq]

Hi,

I have a couple of questions about the challenge:

1. Can we apply spelling correction before using a speaker correction model?
2. Is there a leaderboard available?
3. Can we use non-open source LLMs like GPT-4?
4. Do we need to change only the speaker tags, or can we change the words as well?

[naymaraq]

Also, I am unable to achieve any improvements using the baseline beam-search that was provided compared to cpWER where no correction was made

[tango4j]

1. Yes, you could make any type of correction before feed the text into LLMs or Beam search decoder.
   This is due to the fact that it is very challenging to force LLMs to only fix the speaker tagging without changing the word tokens. However, you should be responsible for the WER degradation from the correction.

2. We haven't set up a leaderboard yet, but if there is demand, we can consider opening it.

3. Yes. You could use any type of LLMs, if you can state the prompt method in the technical paper.

4. I guess this is similar to Q1. You are allowed to change words. This is also already mentioned in the description.
   You are even allowed to use Track1's system to correct the ASR errors.
   However, you should also be aware that some ground truth files are only tagged to have correct speakers rather than fixing the words so fixing the words could sometimes damage cpWER. Having said that, there is no restriction on correcting the words.

[tango4j]

The challenge organizers made decision to exclude the audio source in this challenge after we choose the baseline system.
We also realized that the base line does not improve the given dataset.

We will be releasing the subset list of files that baseline system improves. Also, we are also planning to upload another baseline. Until then, please think of the baseline code as a tool for checking input/output.

[naymaraq]

Thanks for answers @tango4j

[tango4j]

@naymaraq
We have created a leaderboard today, and reduced the size of dev/eval set to 10~13 files for .

https://huggingface.co/spaces/GenSEC-LLM/task2_speaker_tagging_leaderboard

A few teams are preparing to submit.
It is good time to check the performance of your Speaker tagging corrector on this.

@naymaraq Let me know if there is question..! Thank you.

[naymaraq]

@tango4j
Thank you for sharing updates. We submitted our solution on the reduced dataset. Do we need to send the output on a whole dataset? Also, do we need to do other actions besides uploading seglist.json into a huggingface eval tool?

[tango4j]

@naymaraq
Thank you so much for putting this effort on this.
Your submission seems like performing far better than the baseline for both dev/eval.

If there is no need for tie-breaker, we might note request the participants to evaluate other bigger splits of datasets.
In terms of the technical descriptions, let me ask about it and get back to you.

[tango4j]

@naymaraq

Hi, the committee says that you are encouraged to submit a paper. Minimum 2 page technical details are required, max 6 page. You may include fine-tuning, prompting and data processing for train/inference, parameter tuning. Also, model information should be included in the technical paper.
You should register your paper by June/20th/2024 then you can modify it until June/27th/2024.
The paper will be in proceedings of SLT 2024.

https://sites.google.com/view/gensec-challenge/home

Technical Papers
Please submit a challenge submission paper through [CMT system]. Minimum 2 page - Max 6 page is allowed.�

For templates, detailed requirements, please visit https://2024.ieeeslt.org/paper_submission/
June 20, 2024 : Paper submission deadline�
June 27, 2024: Paper update deadline

[naymaraq]

Hi @tango4j

We posted the technical details of our submission here: link. Unfortunately, it didn't pass the review stage.

[tango4j]

Hi Dav. I appreciate your effort on publications. We, as challenge organizers, also have some disatisfactions that challenge submission papers do not get the slots for publications. However, if we publish further study, we will make sure that this paper is cited.

…

________________________________ From: Dav Karamyan ***@***.***> Sent: Thursday, September 12, 2024 11:25 AM To: tango4j/llm_speaker_tagging ***@***.***> Cc: Taejin Park ***@***.***>; Mention ***@***.***> Subject: Re: [tango4j/llm_speaker_tagging] Question (Issue #1) Hi @tango4j<https://github.com/tango4j> We posted the technical details of our submission here: link<https://arxiv.org/pdf/2409.00151>. Unfortunately, it didn't pass the review stage. — Reply to this email directly, view it on GitHub<#1 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADEZOW333HBOJHNMXWPZCCDZWHMCDAVCNFSM6AAAAABH7L7LHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBWHE3DKOJYHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>