Add CCAligned Multilingual Dataset

[gchhablani]

Hello,

I'm trying to add CCAligned Multilingual Dataset. This has the potential to close #1756.

This dataset has two types - Document-Pairs, and Sentence-Pairs.

The datasets are huge, so I won't be able to test all of them. At the same time, a user might only want to download one particular language and not all. To provide this feature, load_dataset's **config_kwargs should allow some random keyword args, in this case -language_code. This will be needed before the dataset is downloaded and extracted.

I'm expecting the usage to be something like -
load_dataset('ccaligned_multilingual','documents',language_code='en_XX-af_ZA'). Ofcourse, at a later stage we can provide just two character language codes. This also has an issue where one language has multiple files (my_MM and my_MM_zaw on the link), but before that the required functionality must be added to load_dataset.

It would be great if someone could either tell me an alternative way to do this, or point me to where changes need to be made, if any, apart from the BuilderConfig definition.

Additionally, I believe the tests will also have to be modified if this change is made, since it would not be possible to test for any random keyword arguments.

A decent way to go about this would be to provide all the options in a list/dictionary for language_code and use that to test the arguments. In essence, this is similar to the pre-trained checkpoint dictionary as transformers. That means writing dataset specific tests, or adding something new to dataset generation script to make it easier for everyone to add keyword arguments without having to worry about the tests.

Thanks,
Gunjan

Requesting @lhoestq / @yjernite to review.

[lhoestq]

Hi !

We already have some datasets that can have many many configurations possible.
To be able to support that, we allow to subclass BuilderConfig to add as many additional parameters as you may need.
This way users can load any language they want. For example the bible_para dataset is a dataset for translation and therefore users should be able to provide any language pair. You can check how the subclass of BuilderConfig is defined here.

For testing, only the configurations defined in the BUILDER_CONFIGS class attribute are used.
All the other configs combinations are not tested, but they can be used by users. If a config doesn't already exist in BUILDER_CONFIGS, then it is created on the fly.
For example in bible_para, only 6 configs are defined in BUILDER_CONFIGS.

So what I would do in your case is have something like

class CCAlignedConfig(datasets.BuilderConfig):
    def __init__(self, *args, documents_or_sentences=None, language_code=None, **kwargs):
        super().__init__(
            *args,
            name=f"{documents_or_sentences}-{language_code}",
            **kwargs,
        )
        self.documents_or_sentences = documents_or_sentences
        self.language_code = language_code

And of course, feel free to change/rename things if you want to. In particular I think we can improve the name of the parameter documents_or_sentences

[gchhablani]

Hi @lhoestq,

Thanks a lot! I don't know why I didn't think about that. :P
I'll make these changes and update.

[gchhablani]

Hi @lhoestq,

I have tested and added dummy files. Request you to review.

Also, does this mean BUILDER_CONFIGS is only needed while testing?

[gchhablani]

Hi @lhoestq,

Any changes required on this one?

Thanks,
Gunjan

[lhoestq]

Hi, Sorry for the delay ^^'

That's awesome thanks ! Good job with the dataset config
I left a few comments.

Also could you try to reduce the size of the dummy data for documents-ak_GH please ? It's currently 3.8MB and it would be awesome to have something less than 20KB

[gchhablani]

Hi @lhoestq,

Sorry for the delay, I have added the changes from the review. For the ISO format language codes, I just selected the first two characters from the names, hoping those are correct. Let me know if you want me to verify :P

Thanks for taking the time to add such a detailed review. I'll keep all these changes in mind the next time I'm adding a dataset.

Thanks,
Gunjan

[lhoestq]

Thanks for the changes :)

I added just a few comments in the dataset card to take into account the change to the Translation feature type.

Also it looks like the dummy_data.zip file for the ak_GH configuration is quite big (3MB), can you try to reduce its size ? Ideally it should be just a few KB like for the other configs.

[lhoestq]

Can you update this example now that we're using the Translation feature type please ?

[gchhablani]

Yes. I missed the README. Sorry.

[lhoestq]

same here

[lhoestq]

The Source_Content and Target_Content fields are now replaced by a field translation with two subfields.
One subfield is en_XX and the other one is the other language code.

[lhoestq]

same here

[gchhablani]

Hi @lhoestq,

I have changed the README, and added a single example per config. Even one example is long enough to make the files heavy. Hope that isn't an issue.

Thanks,
Gunjan

[lhoestq]

Thanks for the changes !
LGTM :)

[gchhablani]

Hi @lhoestq,

Thanks for approving.