Accuracy and web interface

[a455bcd9]

Hi Hamza,

Thanks a lot for your amazing work!

I wondered:

• When will "Simple extensible deep learning model for automatic Arabic diacritization" be published?
• Which datasets did you use?
• What accuracy rate (DER? WER?) did you achieve?
• Is it possible to download the model already trained?
• Or is it possible to try the model somewhere?

I asked these questions because I would love to use (or build) a Chrome extension that automatically adds diacritics to Arabic text online (for instance if the page or some paragraphs are tagged as lang=ar).

Thanks for any help you can provide.

Kind regards,

Antoine

[Hamza5]

Hi Antoine,

Thanks for your interest.

When will "Simple extensible deep learning model for automatic Arabic diacritization" be published?

All I know currently is that my paper was accepted on the 6th of August, 2021, in ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP). I don't know when they will publish it. I hope that will be by the end of this year. But I have no guarantee.

Which datasets did you use?

Tashkeela processed. I used only a subset of it as I described in the article (10 first files for training, first file for both validation and testing). I wanted to use it all but due to the COVID-19 outbreak I have lost access to the server that was available to me before, so I had to continue my research with limited resources.

What accuracy rate (DER? WER?) did you achieve?

On the testing file, when counting all Arabic letters at all positions, DER = 3% and WER = 8.99%

Is it possible to download the model already trained?

The code contained in this repository is already trained, otherwise, the diacritization will be completely random, which is not the case if you have tried the web interface. The default weights file is multilevel-diacritizer/params/MultiLevelDiacritizer-E128L128W150S30.h5, and is already in this repository.

Or is it possible to try the model somewhere

I have added all the required instructions in the README.md file on how to run this project. If you use Docker, it is basically just build then run commands, and you are done. Even without Docker, it is not hard to get it running. Check the README.md for that.

I hope my comments are helpful.

Best regards,

Hamza.

[a455bcd9]

Hi Hamza,

Thanks for your swift reply.

Tashkeela processed. I used only a subset of it as I described in the article (10 first files for training, first file for both validation and testing). I wanted to use it all but due to the COVID-19 outbreak I have lost access to the server that was available to me before, so I had to continue my research with limited resources.

Too bad... :( What kind of server would you need to run the training on the whole dataset?

I have added all the required instructions in the README.md file on how to run this project. If you use Docker, it is basically just build then run commands, and you are done. Even without Docker, it is not hard to get it running. Check the README.md for that.

Thanks. Have you already deployed it on your own Heroku account? What I would love is a link that I could send to friends (who don't know coding at all) so that they can try it/play with it. So I thought you may have already done that.

[Hamza5]

Hi Antoine,

Too bad... :( What kind of server would you need to run the training on the whole dataset?

I can't tell you what is the minimum requirements, because it can even train on a laptop if you use too small batch sizes, but it will take forever to complete!

However, the server that I had access to in the past, as far as I remember, has 32 vCPU, 123 Go of RAM, and 4 GPUs of NVIDIA 1080Ti. That's an expensive configuration that I could never afford outside my university that gives it for free for the students who are inside the campus. Since the beginning of the COVID-19, I couldn't return to my university and I don't have any access to it anymore.

Because of that, I moved to Google Colab to train it, which was not convenient at all, but at least I could get some results to publish.

Thanks. Have you already deployed it on your own Heroku account? What I would love is a link that I could send to friends (who don't know coding at all) so that they can try it/play with it. So I thought you may have already done that.

Yes, I did. It is currently here: https://multilevel-diacritizer.herokuapp.com/

But don't rely on it, because I will most likely delete it in the future since it takes credits from my account when it is run, and I may need these credits for other projects, so I will delete it to save them. To avoid this problem, just make an account on Heroku and deploy it using the button that is available in the README.md file. It should be a 5-minutes task and doesn't require any credit cards or any annoying information.

[a455bcd9]

Hi Hamza,

I can see that on AWS the g4dn.12xlarge costs $3.912/hour on-demand and with a reserved instance the cost goes down to $1.564/hour. How long would it take to train on the whole dataset? Could the Wuhan University of Technology finance that by any chance? I would be happy to help a bit if it's not crazy expensive :) [the g4dn.12xlarge is definitely not the best one in terms of performance, others such as p3.8xlarge are ~3x more expensive, but if they reduce the training time by more than 3x it may be worth using them instead?)

Indeed I don't want to use your credits so I won't use this app. Random idea: you could maybe monetize this app with micro-payments on the Bitcoin Lightning network, for instance for each API call users would have to pay $0.01 (or even less). Because payments on Lightning are instant and with almost 0 fees that would work.

Antoine

[Hamza5]

Hi,

I can see that on AWS the g4dn.12xlarge costs $3.912/hour on-demand and with a reserved instance the cost goes down to $1.564/hour. How long would it take to train on the whole dataset? Could the Wuhan University of Technology finance that by any chance? I would be happy to help a bit if it's not crazy expensive :) [the g4dn.12xlarge is definitely not the best one in terms of performance, others such as p3.8xlarge are ~3x more expensive, but if they reduce the training time by more than 3x it may be worth using them instead?)

Certainly, my university won't accept. In the past, at the end of 2019, I had VPN access to the network of the university, and it was possible to access their network from anywhere. The problem is when the winter vacation came (January and Febrary 2020) they shut down the server during this period (this is ordinary as they think no one would work during the vacation!). During this period, the COVID-19 spread quickly so they didn't return to school to turn on the servers again! It was until August that they came back. However, when I got access again, the internet connection of the server didn't work at all. After many tries and contacting some persons, I found that the problem is that this server was isolated completely from the public network so it won't have access to the internet anymore! I didn't get an equivalent server. Best of all, they have even stopped my VPN account at the end of 2020. I have complained a lot about these problems but they seem to ignore us (the foreign students). This is the China policy!

Anyway, I have completed the minimum requirements of two accepted papers, I just need to write the thesis before doing the Ph.D. thesis defense. So I am not interested in continuing to work on this project, because it is just a demonstration of the research only, it is not built to be a real production service. That's why I am not looking to monetize it. I may build a maintained production-ready diacritization system in the future, but I don't think it will be in the near future, as I am already busy with another blockchain-based project outside my Ph.D.

[Hamza5]

Hi @a455bcd9 , the link to the paper is now available in the README.md, in case you need it.

[a455bcd9]

Thanks a lot, Hamza!

Out of curiosity, what's the state-of-the-art in Arabic diacritization? I found this Apple paper from Sept 2021 claiming to be SOTA: https://arxiv.org/abs/2109.14150

[Hamza5]

I still didn't check that paper and I am taking a break from research in this scholar year, so I am a bit outdated. But in the topic of Arabic diacritization, everyone can claim to have SOTA results since there is no standard dataset with exact partitions, nor even accurately defined metrics. So it is generally hard to say whether the results are really SOTA or not.