Create project 'exiv2' on Crowdin

[clanmills]

Prior discussion: #1466 (comment)

I've received an email from Crowdin.

Hello Robin,
Thanks for asking for a free open-source license
I encourage you to create your translation project and upload files that should be localized
Please let us know when a project is created, so we can approve your request

And my reply is:

Thank You for your email. A couple of Team Exiv2 members recommended that we use Crowdin and have asked me as the Lead Engineer to “sign” our request. I have asked them to create the project on your platform.

We conform with your other criteria (regular releases, no commercial projects, active community, regular collaborators etc).

@lbschenkel Can you set up the project. I can forward the details to Crowdin.

[clanmills]

Their reply is:

Dear Robin,

Thank you for choosing Crowdin, it's much appreciated! So we'll wait when your team will create the project and let us know about it, then we'll be able to grant you the open-source license on our platform

Let's keep in touch,

To which I have replied:

Thanks. That’s really good. I’ve opened an issue for our project and we’ll get the project set up. #1510

I’m really pleased to hear about this. We provide Localisation support in 18 languages. You can imagine that in the rush to make a new release, localisation doesn’t get the attention it deserves. I’m optimistic that Crowdin will remedy that situation.

[clanmills]

And they followed that with:

Dear Robin,

Thank you sharing your plans with us! Once your team will create a project in Crowdin - please let us know the username of the Crowdin account and will happy to connect free open-source license for you

Looking forward to your reply,

So, everything looks good to me.

[lbschenkel]

I set up the project and uploaded the PO files from 0.27 maintenance branch.
It is available at https://crowdin.com/project/exiv2/

I configured the project to be open, so anybody can contribute translations but everybody must have a Crowdin account.

In Crowdin there are 4 roles of members: https://support.crowdin.com/modifying-project-participants-roles/
I am currently the owner. What I recommend is that you @clanmills or somebody that you trust create an account that I can transfer ownership to. If I'm added with the role "manager" I can volunteer to keep managing the Crowdin project, at least for a while.

I have my own suggestions of workflow but I want you to play with this a little bit first.

Note that this project is currently under a trial of 14 days, so we have 2 weeks to transfer ownership and request the open-source license. We should ask for clarification if we can apply the open-source license now and that is not affected by a later transfer of ownership, or if we should transfer ownership first and then apply for the open-source license.

[lbschenkel]

And by the way: just let me know of any accounts that I should add to the Crowdin project (and which role).

[clanmills]

@lbschenkel Because I know nothing about Crowdin, I don't understand the options and possibilities. I'll be happy to forward you the emails from Crowdin and you can discuss directly with them.

I will setup an account and give you the details. @1div0 and @alexvanderberkel will probably do this also.

This seems like a really good way to proceed with localisation. Since I took on the maintenance of Exiv2 (about 2012 or so), I've never had time to investigate how to get this work done for every release. We've had the occasion update (to German, Dutch and Catalonian). I suspect most of the translation files are about 10 years old.

In 2018 I thoroughly investigated how the code works and documented localisation in README.md. Last year, I added code to the test harness to test that the localisation works (in French and Spanish).

As you'll see in the project plan for v1.00, I've assessed two topics as "Fair". Localisation and Previews. I assess the other 30 top-level topics as "Good" to be modest. I actually mean "Rock-solid and working well.". #1466 (comment)

Thanks for getting involved with this. Your contribution is appreciated.

[lbschenkel]

No problem. By the way, I was the lead developer globally for Globalization for a former employer and built most of the tooling and processes, so I have a lot of experience in this domain – including gettext.

I spent some minutes trying to translate to Brazilian Portuguese and I already identified a number of "localization bugs" in the codebase that make some text untranslatable. A few examples:

• If you have a tagged text like "Preview" in the source code, that could be a verb or a noun and the translator has no clue what it is. And depending on what it is, it needs to be translated differently because they won't be identical in the target language. Strings like this will require "contexts" being added in the source code, so they will not be wrongly deduplicated and they can be correctly translated because they will be visible to the translator.
• The code also seems to be declaring different "sub-strings" that end up being concatenated together to form a full sentence. This usually breaks down completely on translations, because of different word order and words that have to agree with each other. Usually you have to forbid concatenation and declare strings with placeholders instead. Same applies with plurals. Anything that gets resolved at runtime.

I don't mean to be critical to the project. Those are super-common issues that I was dealing with every day. I can help by submitting a few localized PRs to show by example how it can be done "right".

[clanmills]

Well, you're the expert. For sure, I'm a typical english-speaking engineer and hide when anybody says "localisation".

When I documented localisation, I was of the impression that every string to be translated has a marker/reference back to the code. So every translation of the same word can be modified by context. Maybe not!

I didn't write any of the localisation code, so I'm not going to defend it. Even if I wrote it, I'm not going to defend anything that's wrong. If it needs fixed, let's fix it!

I've been working on Exiv2 since 2008. I've written about half of Exiv2 and fixed issues in the other half. It's astonishing how users can report errors anywhere and everywhere. Few issues about localisation have been opened.

I know there are "substrings" used in the the error messages in src/error.cpp and I think they are easy to understand. If you discover puzzling "substrings" I'll be happy to investigate and suggest changes.

About the schedule. I am about to declare Code Freeze on 0.27.4. So, please don't submit PRs to 0.27-maintenance. Next week, I intend to set up a new default branch called 'main'. That's the branch which will ship as Exiv2 v1.00 in December 2021.

[clanmills]

@lbschenkel I emailed Crowdin this morning:

We’re up and running. My co-worker Leonardo Schenkel has opened an account and created a project ‘exiv2’. I’ve opened an account ‘clanmills’ and can access Leonardo’s project.

Leonardo is a very experienced localisation engineer. I believe he’s going to “try a few things out”. I’m a typical english speaking development engineer. I’m a curious spectator in this activity.

I’m unclear about where we go from here. Presumably, you update the ‘exiv2’ to make it permanent. The next release of Exiv2 will be on 2021-04-30. The release notes will invite users to update the localisation resources on your platform and changes will ship in our release planned for 2021-12-15.

Thanks for your courtesy and assistance.

Right. You're from Brazil and live in Malmö. I guess there are not many Brazilians in Malmö. I spent 3 months in Copenhagen as a student in 1972, so been to Malmö (and Lund) a couple of times. I was 70 in January this year. We retired in England in 2014 after 15 tough years in Silicon Valley, California. I was a Senior Computer Scientist at Adobe Systems Incorporated in San Jose (PostScript, ExtendScript, Acrobat).

I learned about metadata by working on Exiv2 since 2008. To escape and retire, I've written a book Image Metadata and Exiv2 Architecture. https://clanmills.com/exiv2/book/ I bought a Euphonium a couple of years ago and practice for about an hour every day.

[clanmills]

DannyBoy.mp4

[clanmills]

I've received the following email from Crowdin:

Thanks for such a detailed reply!

Hopefully, both of your and Leonardo's projects will work smoothly and efficient :)

If you'd like you can submit a request for an open source license (or Leonardo can submit it as well). So, in case you have open and non-profit product, it'd be possible for you to use Crowdin as long as you'd like without any limits. If it sounds interesting for you, you're welcome to check the form below:
https://crowdin.com/page/open-source-project-setup-request

In case you'd have any additional questions - please feel free to reach out to us at any time.

I've filled in the open-source request again.

I see you've granted me manager rights on the Crowdin project. Everything seems fine.

[clanmills]

And they've replied:

Dear Robin,

Just got your open source license, and I'm glad to say that your request was approved 🎉

As I've mentioned previously - it is free for Open Source. Forever :)

You can freely work on your project here and invite community - we'd be happy to take care of onboarding and share some tips for your translators as well (in case needed).

New Crowdin version is also right here to give a try: https://enterprise.crowdin.com

Also, feel free to share any feedback or questions. We'll be glad to hear them!

Have a nice day once more :)

[clanmills]

I've announced the Crowdin project on the Exiv2 Facebook Page: https://www.facebook.com/exiv2/posts/10158647258659091

When I release v0.27.4 RC2 next week, this will be documented in the release notes and README.md. You're welcome to comment on my words. Exiv2 v0.27.4 GM is scheduled to ship on 2021-04-30.

[clanmills]

I've moved this issue to Milestone v0.27.4 to remind me to update README.md concerning Crowdin. When the documentation has been updated, we can close this issue. If we start a discussion about using Crowdin, we should open a new issue for the v1.00 milestone.

[clanmills]

@lbschenkel It looks as though the Crowdin project is up and running. I clicked the "Download" button and got a directory with po files. They'll have to be renamed and submitted as a PR into 0.27-maintenance.

I intend to release Exiv2 v0.27.4 RC2 on Tuesday. I hope to release that code as v0.27.4 GM on 2021-04-30.

I only change version numbers and release notes when I promote RCfinal to GM. So, if you want to release those modified files in v0.27.4, the PR has to be merged by Tuesday lunch-time. Can you do that, please?

I'd also like to know more about your commitment to Exiv2. You can do as little or as much as you wish. However, I'd like to know your thoughts to avoid mis-understanding.

It's only in the last week that I have realised that we can move Exiv2 to v1.00 in 2021. I'm really pleased and surprised to realise this goal is within reach. I can retire with the satisfaction of completing the project. Others may, or may not, decide to add more features in future. Exiv2 v1.00 is a complete and comprehensive metadata toolkit.

[lbschenkel]

@lbschenkel It looks as though the Crowdin project is up and running. I clicked the "Download" button and got a directory with po files. They'll have to be renamed and submitted as a PR into 0.27-maintenance.

Yes, but I can set up a custom mapping in Crowdin so they'll have the expected names and no rename is necessary. Unfortunately Crowdin does not use the standard IETF/BCP 47 codes for all locales.

Also keep in mind that just as an experiment, I have enabled "pre-translation" via machine translation. Most files had very low coverage, way below 50%, so most the translations you are going to get aren't vetted. I'm not sure if that's what you want. If we want to keep everything manually translated, I need to fix this first.

I'm not sure if you have any specific translation workflow in mind. I can make some suggestions.

I'd also like to know more about your commitment to Exiv2. You can do as little or as much as you wish. However, I'd like to know your thoughts to avoid mis-understanding.

To be honest, I'm not sure myself. I recently had a baby daughter so my free time is limited and a bit unpredictable. I think it's feasible for me for now to help setting up/administering the Crowdin project and doing translation/proofreading of pt-BR locale on my own pace (there are more than 6000 strings, so that will take quite a while).

[clanmills]

If you're willing to provide Crowdin support to Team Exiv2 for 9 months to get us to v1.00, that would be really great. Peter (@1div0) has volunteered Slovak and Czech. Alex (@alexvanderberkel) has volunteered German. Luis (@piponazo) is Spanish. Toni (@bellaperez) does Catalan. Gilles (@cgilles) does French. I don't expect a sudden rush of contributors to localisation. I hope those folks will have a look at their languages in Crowdin.

I'm happy with machine translation. If it's awful, people will complain and I'll invite them to fix it! It's their language after all!

The situation I want to avoid is dealing with this on my own. I don't need any more work. I'm trying to retire! So, I'd like to say in the release notes that you will support and manage Crowdin for the v1.00 project. Tomorrow or Monday, I will submit the PR with the documentation changes concerning this and invite you to review.

This custom mapping sounds useful and will simplify the work-flow. It's probably possible for me to modify the CMake scripts to build directly from the Crowdin files. It's too late to change CMake code for v0.27.4.

I'm OK about changes to localisation and documentation going directly into the GM Release. I avoid changes to C++, CMake and the test framework after RCfinal. v0.27.2 (June 2019) required RC3 because we discovered a security issue after RC2 had been published and therefore shipped 4 weeks behind schedule. I like to ship "spot on" the schedule date. At the moment, Exiv2 v0.27.4 will ship 3 weeks ahead of schedule if there is no painful discovery in April.

[piponazo]

Hi! That Crowdin tool seems to be quite nice. I took a look to the Spanish translations and I corrected few of them. I could take care of double check all the items for that language.

I could also check @lbschenkel what would be the best way to integrate the tool with out CI pipelines, so that we can generate releases in an automatic way.

[lbschenkel]

I have changed the project settings, so now when you download the .zip with the translations the files will match the names used by this project so you can simply unzip in the po directory:

• Settings for exiv2.pot under "Files":
  
• Language mappings under "General":
  

This will at least simplify the "manual" workflow. In the manual workflow somebody has to manually upload exiv2.pot every time strings in the source change, and to refresh the translations you need to download all translations from Crowdin as a zip, and overwrite the .po files.

Regarding an automated workflow, Crowdin also has GitHub integration which I want to test with my fork first.

[lbschenkel]

I have played with GitHub integration on my own fork. You can see how it works in the project settings under "Integrations".
The way this works is that it'll automatically pick up changes to exiv2.pot [1] and upload that file to Crowdin. Then for new translations, it'll automatically submit a PR against the repo like this: lbschenkel#1

We need to have a crowdin.yaml committed to the project: https://github.com/lbschenkel/exiv2/blob/0.27-maintenance/crowdin.yml

[1] Now I have noticed that this file is 4 years old so it doesn't reflect the state of the source. To make the GitHub integration work we need to make changes to the CI build process, because you want to extract the strings from the source code via gettext to the .pot file and then commit and push that file (even if in a different branch).

If we can't push that file (CI having no credentials, for example) then we have to integrate differently. For example, we can use the Crowdin-provided command-line tool to upload the .pot file to Crowdin and call that from the Makefile. The problem is that this tool is Java-based, so it'll add a Java dependency to build the project (or at least to the target that builds the .pot).

I didn't try, but I have taken a look in the API and since we just want to do a simple upload of a single file it should be doable with curl to avoid the Java dependency.

Irrespective of mechanism, if we take this route we should take precautions to hide the Crowdin API key. It shouldn't be committed to the project for obvious reasons, so it'll need to be manually configured in the CI and exported via an environment variable.

[clanmills]

@lbschenkel Excellent. And great work as always by @piponazo
Leonardo: Please visit the chat server and I will introduce you to the gang:

(https://matrix.to/#/#exiv2-chat:matrix.org)

I've remembered that @pmatth2 updated the Dutch localisation and might be very interested in the new Crowdin tools.

[clanmills]

I don't think the Java dependency will cause a problem. That's much the same as needing various tools to generate the on-line docs. Here are the build options:

1417 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance $ grep -i option CMakeLists.txt 
# options and their default values
option( BUILD_SHARED_LIBS             "Build exiv2lib as a shared library"                    ON  )
option( EXIV2_ENABLE_XMP              "Build with XMP metadata support"                       ON  )
option( EXIV2_ENABLE_EXTERNAL_XMP     "Use external version of XMP"                           OFF )
option( EXIV2_ENABLE_PNG              "Build with png support (requires libz)"                ON  )
option( EXIV2_ENABLE_NLS              "Build native language support (requires gettext)"      OFF )
option( EXIV2_ENABLE_PRINTUCS2        "Build with Printucs2"                                  ON  )
option( EXIV2_ENABLE_LENSDATA         "Build including lens data"                             ON  )
option( EXIV2_ENABLE_VIDEO            "Build video support into library"                      OFF )
option( EXIV2_ENABLE_DYNAMIC_RUNTIME  "Use dynamic runtime (used for static libs)"            ON  )
option( EXIV2_ENABLE_WIN_UNICODE      "Use Unicode paths (wstring) on Windows"                OFF )
option( EXIV2_ENABLE_WEBREADY         "Build webready support into library"                   OFF )
option( EXIV2_ENABLE_CURL             "USE Libcurl for HttpIo (WEBREADY)"                     OFF )
option( EXIV2_ENABLE_SSH              "USE Libssh for SshIo (WEBREADY)"                       OFF )
option( EXIV2_ENABLE_BMFF             "Build with BMFF support"                               OFF )
option( EXIV2_BUILD_SAMPLES           "Build sample applications"                             ON  )
option( EXIV2_BUILD_EXIV2_COMMAND     "Build exiv2 command-line executable"                   ON  )
option( EXIV2_BUILD_UNIT_TESTS        "Build unit tests"                                      OFF )
option( EXIV2_BUILD_DOC               "Add 'doc' target to generate documentation"            OFF )
option( EXIV2_TEAM_EXTRA_WARNINGS     "Add more sanity checks using compiler flags"           OFF )
option( EXIV2_TEAM_WARNINGS_AS_ERRORS "Treat warnings as errors"                              OFF )
option( EXIV2_TEAM_USE_SANITIZERS     "Enable ASAN and UBSAN when available"                  OFF )
option( EXIV2_TEAM_PACKAGING          "Additional stuff for generating packages"              OFF )
option( BUILD_WITH_CCACHE             "Use ccache to speed up compilations"                   OFF )
option( BUILD_WITH_COVERAGE           "Add compiler flags to generate coverage stats"         OFF )

You'll see we have EXIV2_ENABLE_NLS to build NLS support into the library. When you do $ sudo make install some gettext magic runs to install localisation files.

In the case of the docs, the doxygen commands are generated in the makefile as follows:

$ cmake .. -DEXIV2_BUILD_DOC=On

And those commands are run by:

$ make doc

I'm sure a similar model can be used for localisation. I know enough cmake to be dangeous. Luis (@piponazo) is the expert.

Let's not disturb the cmake code on 0.27-maintenance. Downloading the .po files and submitting them in a PR will be fine for 0.27.4 RC2 and GM. You can polish this on branch 'main' which I plan to create on Tuesday.

It's amazing what can be achieved when an expert joins the team. Localisation has been unloved and neglected. Not any more. Thank You Very Much. I think a round of applause for Leonardo is deserved.

[clanmills]

@lbschenkel I have updated the documentation about Crowdin in PR: #1526. I've updated GitHub to send you an invitation to have write access to the Exiv2 repository. When you have accepted that invitation, I will be able to invite you to review/approve the documentation change. You are of course entitled to request changes to my update.

[clanmills]

Having submitted PR #1526, I've changed the Milestone for this issue to v1.00. This issue should not be closed when #1526 merges.

[lbschenkel]

Thanks for inviting me to the project. Although I will have commit rights, I don't intend to do any direct changes without submitting a PR and having another pair of eyes approving it.

[clanmills]

@lbschenkel You have write to the repos and your branches.

However, you will have to use a PR to submit changes to important branches such as 0.27-maintenance. All PRs require on review/approval. There is an "admin" level of access (which Luis and Dan have) which enables you to mess with those settings, invite/remove users AND provides a big red button to "merge anyway" for unapproved PRs.

You can guess the informal rule: if you think you're going to cause trouble, STOP and discuss it on the chat server.

[lbschenkel]

Yes, understood. I have commit rights to other open source projects as well and this is already my regular modus operandi.