[ Enhancements ] : Rewrite in a compiled langauge

[baseplate-admin]

Hi there,

It seems that i need this library for a hobby project of mine (Java|Kotlin based). I was wondering if you guys want to merge and contribute to one single repo.

Requirements:

1. Move this to an organisation ( i am bad at naming things )
2. Rewrite this in a compiled langauge and make it a CPython extension module.

Language choices:

1. Nim ( this can compile to JavaScript, C and a number of other langauge | Very pythonic )
2. Zig ( A new Language that i am seeing getting popular )
3. Rust
4. Go
5. C++ ( another good choice because i come from a JS background )

Reasoning :

1. Development efforts will be placed under one umbrella. Meaning efforts like @TanimSk put into reverse parsing can be put into one place instead of rewriting everywhere | For example it has to be rewritten for avro.go
2. Faster performance.
3. I have a number of ways to make this library much faster in what it does.

Considerations:

1. Do not break what we already have. All tests should pass without the need to modify them.

Chore :

1. You have to release a new version Everytime theres an update to the main project. ( this can be automated if you are willing to move this repo to the organisation and then use some form of CI/CD )

[hitblast]

Seems like a good idea. Besides the organization and CI/CD, do you have a draft roadmap to perform this in your mind? I'd like to have some thoughts.

[baseplate-admin]

I have finished half of the work done. This Friday my exam will finally be finished. After that expect 1 week to get everything working.

Now some insights :

I am converting everything into Variables. I dont think IO operations in a embedded system is a good way to handle stuff. ( this means JSON has to go)

I am trying to find @TanimSk to get some of his inner braincells to help me understand how he does stuff.

Other than that i have a git repo inside my harddrive that has the boilerplates.. ( i am trying RuST ATM but this is debatable )

[furtidev]

When it comes to Nim & Zig, Nim is a overall slower language compared to Zig. Zig is a low-level programming language with Rust like syntax. And, for C++, I don't think it's a good idea, when there are more modern options. I have some experience with Rust, but won't call myself an expert, because I sometimes forget even the basics of Rust. Go, is fast while being simple and efficient.

[baseplate-admin]

The speed difference is probably not applicable to us because we are not doing ahead of time compilation for ZIG. So Zig and Nim should theocratically perform at the same speed

Problem is Zig to Python bindings. Zig is not that much mature in this regard. Also you can compile Nim to C as a standalone script.

Also Zig has maybe AOT. This is why Lua is out of the equation. We want bare functions. Not some JIT type codes.

Go is a worse choice because of its GoRoutines. Everytime you execute the Go Code you have to effectively load Go into memory and then execute stuff. Which means you have 2 garbage collectors fighting over memory. My goal is JVM which has the hotspot VM. This does JIT code compilation. JITing a Garbage collector is a whole level of madness that i am sure most android phones will not be able to handle properly.

Rust is a good choice. But thing with Rust is its just a bit harder compared to other choices we have here.

As for C++. The reason i put it there is because of the demograph. Most of our Universities teach C/C++. We might get more help from picking something that people are familiar with.

[baseplate-admin]

I have a few more questions. Since this is an RFC lets take some votes on which langauge should we code this in.

My vote is on Rust/Nim ( Rust is just time consuming )

---

cc @TanimSk

---

The repo ia available at : https://github.com/baseplate-admin/avro-rs/

Should you guys want to move to an organisation lemme know. I will transfer the repo.

As for progress, i will comment here when its finished

[baseplate-admin]

use once_cell::sync::Lazy;
use std::collections::HashMap;

pub static VOWELS: &str = "aeiou";
pub static CONSONANTS: &str = "bcdfghjklmnpqrstvwxyz";

pub static CASE_SENSITIVE: &str = "oiudgjnrstyz";
pub static SORBORNO: &str = "অআইঈউঊঊএঐওঔ";
pub static KAR: Lazy<Box<[&str]>> =
    Lazy::new(|| Box::new(["া", "ি", "ী", "ু", "ূ", "ৃ", "ে", "ৈ", "ো", "ৌ"]));
pub static IGNORED_PUNCTUATION_MARKS: Lazy<Box<[&str]>> =
    Lazy::new(|| Box::new(["ঁ", "।", "?", ".", "-", ";"]));

pub static REMAPPED_WORDS: Lazy<HashMap<&str, &str>> = Lazy::new(|| {
    return HashMap::from([("ফেসবুক", "Facebook"), ("গুগল", "Google")]);
});

pub static NUMBERS: &str = "0123456789";

@hitblast this is how i am planning to handle stuff

[hitblast]

@baseplate-admin This iteration looks pretty good. Just gotta figure out how to rewrite it entirely in Rust. I have a fair share of experience but just like @furtidev, I sometimes struggle with the syntax.

[baseplate-admin]

Umm i have already completed 75% of the migration. I will make the repo public in Friday...

[baseplate-admin]

Till then will you guys mind making a public organisation?

[hitblast]

@baseplate-admin I guess we'll think about it.

[hitblast]

@baseplate-admin We'll be creating a public organization within a pretty short time. I'm just facing some irl issues related to time management so can't really focus on GitHub. Till then, can you give us an update on your progress?

Note: I'm also currently working on a Nim equivalent of avro.py.

[hitblast]

I mean we already know that Mugli is Mehedi himself. I guess you can proceed to work on adding the newer implementation from libavrophonetic. 🚀

@furtidev has also tried creating his own Go iteration of libavrophonetic, although it needs more development.

[baseplate-admin]

So my goal now is to reach feature parity with Avro.py then move onto porting the libavrophonetic.

I still have some exams left. But the parse function is nearly complete. ( just need to figure out the look behind logic )

[hitblast]

So my goal now is to reach feature parity with Avro.py then move onto porting the libavrophonetic.

I still have some exams left. But the parse function is nearly complete. ( just need to figure out the look behind logic )

Best of luck with your exams! Implementing libavrophonetic is something avro.py could benefit from avro.rs as well.

[baseplate-admin]

Best of luck with your exams!

Thank you so much. Though i very much hate studies. But oh well our country requires a degree to have a place in society sigh

Implementing libavrophonetic is something avro.py could benefit from avro.rs as well.

I hate the way ridmik keyboard is going on with its NFT type stickers. I plan to recreate ridmik actually with avro.rs

[hitblast]

I hate the way ridmik keyboard is going on with its NFT type stickers. I plan to recreate ridmik actually with avro.rs

We could recreate a bunch of Avro applications with our upcoming fleet of libraries.

[hitblast]

@baseplate-admin Would you like to give us a sneak-peek into the progress of avro-rs if you don't mind? :P

[baseplate-admin]

I currently am outside of Bangladesh. So for that reason i am doing maintainance works.

So heres my take : I have found openbangla keyboard. That has a submodule called riti. Thats what i plan to use on avro.py implementation.

[hitblast]

@baseplate-admin Would you mind noting down which features you'd like to implement with Riti which will put avro-rs on par with avro.py?

[baseplate-admin]

Would you mind noting down which features you'd like to implement with Riti which will put avro-rs on par with avro.py?

Hi there,
I have returned to my home. Now back to topic,

Riti is a phonetic engine. Essentially it does what avro.py does. ( I have not had enough time to check out Riti )

It fulfills my goal of avro-rs ( which is aimed to be a phonetic engine )

[hitblast]

@baseplate-admin Cool to hear back from you! I'm also working on the Nim implementation as of now. The dictionary is still incomplete but most of the code is done.

[hitblast]

@baseplate-admin Let us know about any updates you might have. 😶‍🌫️

In the case of avro.nim, I paused development due to my exams and studies and I won't be able to resume soon (I plan to do so before the end of May). I also have a rather experimental idea to fully replace avro.py's backend with a compiled one (as you had mentioned before). I'm also thinking about a new organization for these libraries.

[baseplate-admin]

Hii, I have an rust implementation for avro called riti that will do the same thing avro.py is doing.

if you want i can take a look into this.

( that being said i am busy these days working on coreproject XD | There's almost 3 people working on this project ATM )

[hitblast]

Seems reasonable. That being said, I also seemingly kinda forgot about this discussion because of some other side-projects.

If you're free then feel free to have a look into riti as needed. Thanks for your reply, much appreciated. :D

[hitblast]

@baseplate-admin I've recently made avro.nim public (still work-in-progress) and I'm kinda suffering from an issue related to text parsing. Would you like to have a look?

Mentioned issue: https://github.com/hitblast/avro.nim/issues/1

Currently avro.nim is a somewhat exact replica of avro.py and probably that is why most of the functions don't work properly.

[baseplate-admin]

So basically Nim starts index at 1. In order to fix that error you have to do -1 on the index range

[baseplate-admin]

Hey uh, is avro.nim goone?

[hitblast]

@baseplate-admin I guess I'll have to restart writing it since I couldn't really finish the older one and eventually the build failed too.

[baseplate-admin]

Ah, I see, I thought it was the next step for avro in nim

[hitblast]

@baseplate-admin Well it definitely was. It just needs a better execution :')