diff --git a/CHANGES.rst b/CHANGES.rst index 2d9151691..112c6cf22 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -7,6 +7,17 @@ Documentation Updates: * `New ACL header configuration `_ +* `Locaalization / Multi-lingual Support Guide `_ + + +Localization Improvements: (`#647 `_) + +* Support for extracting, updating, listing and removing localizable commands via ``wb-manager i18n`` command. + +* UI: Add language switch header to all UI templates. + +* Mark localizable strings in translatable in existing templates. + Access Control Improvements: diff --git a/babel.ini b/babel.ini new file mode 100644 index 000000000..3396a5870 --- /dev/null +++ b/babel.ini @@ -0,0 +1,2 @@ +[jinja2: pywb/templates/**.html] +extensions=jinja2.ext.i18n,jinja2.ext.autoescape,jinja2.ext.with_ diff --git a/config.yaml b/config.yaml index 01827eb27..5522c6851 100644 --- a/config.yaml +++ b/config.yaml @@ -17,3 +17,6 @@ enable_memento: true # Replay content in an iframe framed_replay: true +locales: + - en + - es diff --git a/docs/index.rst b/docs/index.rst index 5dd508d49..af80b7445 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,6 +18,7 @@ A subset of features provides the basic functionality of a "Wayback Machine". manual/configuring manual/access-control manual/ui-customization + manual/localization manual/architecture manual/apis manual/owb-transition diff --git a/docs/manual/localization.rst b/docs/manual/localization.rst new file mode 100644 index 000000000..f54f453df --- /dev/null +++ b/docs/manual/localization.rst @@ -0,0 +1,147 @@ +.. _localizaation: + +Localization / Multi-lingual Support +------------------------------------ + +pywb supports configuring different language locales and loading different language translations, and dynamically switching languages. + +pywb can extract all text from templates and generate CSV files for translation and convert them back into a binary format used for localization/internationalization. + +(pywb uses the `Babel library `_ which extends the `standard Python i18n system `_) + +Locales to use are configured in the ``config.yaml``. + +The command-line ``wb-manager`` utility provides a way to manages locales for translation, including generatin extracted text, update translated text. + +Adding a Locale and Extracting Text +=================================== + +To add a new locale for translation and automatically extract all text that needs to be translated, run:: + + wb-manager i18n extract + +The ```` can be one or more supported two-letter locales or CLDR language codes. To list available codes, you can run ``pybabel --list-locales``. + +Localization data is placed in the ``i18n`` directory, and translatable strings can be found in ``i18n/translations//LC_MESSAGES/messages.csv`` + +Each CSV file looks as follows, listing source string and an empty string for the translated version:: + + "location","source","target" + "pywb/templates/banner.html:6","Live on","" + "pywb/templates/banner.html:8","Calendar icon","" + "pywb/templates/banner.html:9 pywb/templates/query.html:45","View All Captures","" + "pywb/templates/banner.html:10 pywb/templates/header.html:4","Language:","" + "pywb/templates/banner.html:11","Loading...","" + ... + + +This CSV can then be passed to translators to translate the text. + +(The extraction parameters arae configured to load data from ``pywb/templates/*.html`` in ``babel.ini``) + + +For example, the following will generate translation strings for ``es`` and ``pt`` locales:: + + wb-manager i18n extract es pt + + +The translatable text can then be found in ``i18n/translations/es/LC_MESSAGES/messages.csv`` and ``i18n/translations/pt/LC_MESSAGES/messages.csv``. + + +The CSV files should be updated with a translation for each string in the target column. + +The extract commannd add any new strings without overwriting existing translations, so it is safe to run multiple times. + + +Updating Locale Catalog +======================= + +Once the text has been translated, and the CSV files updated, simply run:: + + wb-manager i18n update + +This will parse the CSVs and compile the translated string tables for use with pywb. + + +Specifying locales in pywb +========================== + +To enable the locales in pywb, add one or more locales can be added to the ``locales`` key in ``config.yaml``, ex:: + + locales: + - en + - es + +Single Language Default Locale +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +pywb can be configured with a default, single-language locale, by setting the ``default_locale`` property in ``config.yaml``:: + + + default_locale: es + locales: + - es + + +With this configuration, pywb will automatically use the ``es`` locale for all text strings in pywb pages. + +pywb will also set the ```` so that the browser will recognize the correct locale. + + +Mutli-language Translations +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If more than one locale is specified, pywb will automatically show a language switching UI at the top of collection and search pages, with an option +for each locale listed. To include English as an option, it should also be added as a locale (and no strings translated). For example:: + + locales: + - en + - es + - pt + +will configure pywb to show a language switch option on all pages. + + +Localized Collection Paths +========================== + +When localization is enabled, pywb supports the locale prefix for accessing each collection with a localized language: +If pywb has a collection ``my-web-archive``, then: + +* ``/my-web-archive/`` - loads UI with default language (set via ``default_locale``) +* ``/en/my-web-archive/`` - loads UI with ``en`` locale +* ``/es/my-web-archive/`` - loads UI with ``es`` locale +* ``/pt/my-web-archive/`` - loads UI with ``pt`` locale + +The language switch options work by changing the locale prefix for the same page. + +Listing and Removing Locales +============================ + +To list the locales that have previously been added, you can also run ``wb-manager i18n list``. + +To disable a locale from being used in pywb, simply remove it from the ``locales`` key in ``config.yaml`` + +To remove data for a locale permanently, you can run: ``wb-manager i18n remove ``. This will remove the locale directory on disk. + +To remove all localization data, you can manually delete the ``i18n`` directory. + + +UI Templates: Adding Localizable Text +===================================== + +Text that can be translated, localizable text, can be marked as such directly in the UI templates: + +1. By wrapping the text in ``{% trans %}``/``{% endtrans %}`` tags. For example:: + + {% trans %}Collection {{ coll }} Search Page{% endtrans %} + +2. Short-hand by calling a special ``_()`` function, which can be used in attributes or more dynamically. For example:: + + ... title="{{ _('Enter a URL to search for') }}"> + + +These methods can be used in all UI templates and are supported by the Jinja2 templating system. + +See :ref:`ui-customizations` for a list of all available UI templates. + diff --git a/extra_requirements.txt b/extra_requirements.txt index 4bea8f804..5e5860de4 100644 --- a/extra_requirements.txt +++ b/extra_requirements.txt @@ -5,3 +5,4 @@ uwsgi ujson pysocks lxml +translate_toolkit diff --git a/pywb/apps/rewriterapp.py b/pywb/apps/rewriterapp.py index 9df559cd9..1b3004de2 100644 --- a/pywb/apps/rewriterapp.py +++ b/pywb/apps/rewriterapp.py @@ -72,7 +72,8 @@ def __init__(self, framed_replay=False, jinja_env=None, config=None, paths=None) self.jinja_env.init_loc(self.config.get('locales_root_dir'), self.config.get('locales'), - self.loc_map) + self.loc_map, + self.config.get('default_locale')) self.redirect_to_exact = config.get('redirect_to_exact') diff --git a/pywb/manager/locmanager.py b/pywb/manager/locmanager.py new file mode 100644 index 000000000..c031ac14c --- /dev/null +++ b/pywb/manager/locmanager.py @@ -0,0 +1,109 @@ +import os +import os.path +import shutil + +from babel.messages.frontend import CommandLineInterface + +from translate.convert.po2csv import main as po2csv +from translate.convert.csv2po import main as csv2po + + +ROOT_DIR = 'i18n' + +TRANSLATIONS = os.path.join(ROOT_DIR, 'translations') + +MESSAGES = os.path.join(ROOT_DIR, 'messages.pot') + +# ============================================================================ +class LocManager: + def process(self, r): + if r.name == 'list': + r.loc_func(self) + elif r.name == 'remove': + r.loc_func(self, r.locale) + else: + r.loc_func(self, r.locale, r.no_csv) + + def extract_loc(self, locale, no_csv): + self.extract_text() + + for loc in locale: + loc_dir = os.path.join(TRANSLATIONS, loc) + if os.path.isdir(loc_dir): + self.update_catalog(loc) + else: + os.makedirs(loc_dir) + self.init_catalog(loc) + + if not no_csv: + base = os.path.join(TRANSLATIONS, loc, 'LC_MESSAGES') + po = os.path.join(base, 'messages.po') + csv = os.path.join(base, 'messages.csv') + po2csv([po, csv]) + + def update_loc(self, locale, no_csv): + for loc in locale: + if not no_csv: + loc_dir = os.path.join(TRANSLATIONS, loc) + base = os.path.join(TRANSLATIONS, loc, 'LC_MESSAGES') + po = os.path.join(base, 'messages.po') + csv = os.path.join(base, 'messages.csv') + + if os.path.isfile(csv): + csv2po([csv, po]) + + self.compile_catalog() + + def remove_loc(self, locale): + for loc in locale: + loc_dir = os.path.join(TRANSLATIONS, loc) + if not os.path.isdir(loc_dir): + print('Locale "{0}" does not exist'.format(loc)) + return + + shutil.rmtree(loc_dir) + print('Removed locale "{0}"'.format(loc)) + + def list_loc(self): + print('Current locales:') + print('\n'.join(' - ' + x for x in os.listdir(TRANSLATIONS))) + print('') + + def extract_text(self): + os.makedirs(ROOT_DIR, exist_ok=True) + + CommandLineInterface().run(['pybabel', 'extract', '-F', 'babel.ini', '-k', '_ _Q gettext ngettext', '-o', MESSAGES, './', '--omit-header']) + + def init_catalog(self, loc): + CommandLineInterface().run(['pybabel', 'init', '-l', loc, '-i', MESSAGES, '-d', TRANSLATIONS]) + + def update_catalog(self, loc): + CommandLineInterface().run(['pybabel', 'update', '-l', loc, '-i', MESSAGES, '-d', TRANSLATIONS, '--previous']) + + def compile_catalog(self): + CommandLineInterface().run(['pybabel', 'compile', '-d', TRANSLATIONS]) + + + @classmethod + def init_parser(cls, parser): + """Initializes an argument parser for acl commands + + :param argparse.ArgumentParser parser: The parser to be initialized + :rtype: None + """ + subparsers = parser.add_subparsers(dest='op') + subparsers.required = True + + def command(name, func): + op = subparsers.add_parser(name) + if name != 'list': + op.add_argument('locale', nargs='+') + if name != 'remove': + op.add_argument('--no-csv', action='store_true') + + op.set_defaults(loc_func=func, name=name) + + command('extract', cls.extract_loc) + command('update', cls.update_loc) + command('remove', cls.remove_loc) + command('list', cls.list_loc) diff --git a/pywb/manager/manager.py b/pywb/manager/manager.py index dc36be64d..bb079690e 100644 --- a/pywb/manager/manager.py +++ b/pywb/manager/manager.py @@ -441,6 +441,17 @@ def do_acl(r): ACLManager.init_parser(acl) acl.set_defaults(func=do_acl) + # LOC + from pywb.manager.locmanager import LocManager + def do_loc(r): + loc = LocManager() + loc.process(r) + + loc_help = 'Generate strings for i18n/localization' + loc = subparsers.add_parser('i18n', help=loc_help) + LocManager.init_parser(loc) + loc.set_defaults(func=do_loc) + # Parse r = parser.parse_args(args=args) r.func(r) diff --git a/pywb/rewrite/templateview.py b/pywb/rewrite/templateview.py index 1c9f194a0..122e18d51 100644 --- a/pywb/rewrite/templateview.py +++ b/pywb/rewrite/templateview.py @@ -98,6 +98,8 @@ def __init__(self, paths=None, assets_env.resolver = PkgResResolver() jinja_env.assets_environment = assets_env + self.default_locale = '' + def _make_loaders(self, paths, packages): """Initialize the template loaders based on the supplied paths and packages. @@ -117,16 +119,19 @@ def _make_loaders(self, paths, packages): return loaders - def init_loc(self, locales_root_dir, locales, loc_map): + def init_loc(self, locales_root_dir, locales, loc_map, default_locale): locales = locales or [] + locales_root_dir = locales_root_dir or os.path.join('i18n', 'translations') + default_locale = default_locale or 'en' + self.default_locale = default_locale if locales_root_dir: for loc in locales: - loc_map[loc] = Translations.load(locales_root_dir, [loc, 'en']) + loc_map[loc] = Translations.load(locales_root_dir, [loc, default_locale]) #jinja_env.jinja_env.install_gettext_translations(translations) def get_translate(context): - loc = context.get('env', {}).get('pywb_lang') + loc = context.get('env', {}).get('pywb_lang', default_locale) return loc_map.get(loc) def override_func(jinja_env, name): @@ -160,6 +165,7 @@ def quote_gettext(context, text): self.jinja_env.globals['locales'] = list(loc_map.keys()) self.jinja_env.globals['_Q'] = quote_gettext + self.jinja_env.globals['default_locale'] = default_locale @contextfunction def switch_locale(context, locale): diff --git a/pywb/static/default_banner.js b/pywb/static/default_banner.js index 8777244a9..33dcdf04a 100644 --- a/pywb/static/default_banner.js +++ b/pywb/static/default_banner.js @@ -182,7 +182,7 @@ This file is part of pywb, https://github.com/webrecorder/pywb ancillaryLinks.appendChild(calendarLink); this.calendarLink = calendarLink; - if (typeof window.banner_info.locales !== "undefined" && window.banner_info.locales.length) { + if (typeof window.banner_info.locales !== "undefined" && window.banner_info.locales.length > 1) { var locales = window.banner_info.locales; var languages = document.createElement("div"); @@ -317,4 +317,4 @@ This file is part of pywb, https://github.com/webrecorder/pywb } } -})(); \ No newline at end of file +})(); diff --git a/pywb/templates/base.html b/pywb/templates/base.html index 716ff388e..0bb02c270 100644 --- a/pywb/templates/base.html +++ b/pywb/templates/base.html @@ -9,6 +9,7 @@ + diff --git a/pywb/templates/error.html b/pywb/templates/error.html index c0745395c..9b2f22975 100644 --- a/pywb/templates/error.html +++ b/pywb/templates/error.html @@ -1,5 +1,5 @@ {% extends "base.html" %} -{% block title %}Pywb Error{% endblock %} +{% block title %}{{ _('Pywb Error') }}{% endblock %} {% block body %}
@@ -8,22 +8,22 @@

Pywb Error

{% if err_status == 451 %} -

Access Blocked to {{ err_msg }}

+

{% trans %}Access Blocked to {{ err_msg }}{% endtrans %}

{% elif err_status == 404 and err_details == 'coll_not_found' %} -

Collection not found: {{ err_msg }}

+

{% trans %}Collection not found: {{ err_msg }}{% endtrans %}

-

See list of valid collections

+

{{ _('See list of valid collections') }}

{% elif err_status == 404 and err_details == 'static_file_not_found' %} -

Static file not found: {{ err_msg }}

+

{% trans %}Static file not found: {{ err_msg }}{% endtrans %}

{% else %}

{{ err_msg }}

{% if err_details %} -

Error Details:

+

{% trans %}Error Details:{% endtrans %}

{{ err_details }}
{% endif %} {% endif %} diff --git a/pywb/templates/index.html b/pywb/templates/index.html index 9157368bc..338232a3f 100644 --- a/pywb/templates/index.html +++ b/pywb/templates/index.html @@ -3,7 +3,7 @@

{{ _('Pywb Wayback Machine') }}

-

This archive contains the following collections:

+

{{ _('This archive contains the following collections:') }}

    diff --git a/pywb/templates/not_found.html b/pywb/templates/not_found.html index 727cbdf71..89e671d0d 100644 --- a/pywb/templates/not_found.html +++ b/pywb/templates/not_found.html @@ -1,6 +1,6 @@ {% extends "base.html" %} -{% block title %}URL Not Found{% endblock %} +{% block title %}{{ _('URL Not Found') }}{% endblock %} {% block body %}
    @@ -13,7 +13,7 @@

    {% trans %}URL Not Found{% endtrans %}

    {% if wbrequest and wbrequest.env.pywb_proxy_magic and url %}

    - Try Different Collection + {{ _('Try Different Collection') }}

    {% endif %} diff --git a/pywb/templates/search.html b/pywb/templates/search.html index 151e89d25..08331327c 100644 --- a/pywb/templates/search.html +++ b/pywb/templates/search.html @@ -13,7 +13,7 @@

    - Collection {{ coll }} Search Page + {% trans %}Collection {{ coll }} Search Page{% endtrans %}

    @@ -27,9 +27,9 @@

    + title="{{ _('Enter a URL to search for') }}" type="search" required/>
    - Please enter a URL + {% trans %}'Please enter a URL{% endtrans %}

@@ -37,7 +37,7 @@

- +
@@ -47,51 +47,51 @@

- Date/Time Range + {{ _('Date/Time Range') }}

- +
-
From:
+
{% trans %}From:{% endtrans %}
- Please enter a valid From timestamp. Timestamps may be 4 <= ts <=14 digits + {% trans %}Please enter a valid From timestamp. Timestamps may be 4 <= ts <=14 digits{% endtrans %}
- +
-
To:
+
{% trans %}To:{% endtrans %}
- Please enter a valid To timestamp. Timestamps may be 4 <= ts <=14 digits + {% trans %}Please enter a valid To timestamp. Timestamps may be 4 <= ts <=14 digits{% endtrans %}
@@ -99,41 +99,41 @@

-

Filtering

+

{% trans %}Filtering{% endtrans %}

- +
- +
- + @@ -141,7 +141,7 @@

    -
  • No Filter
    • +
  • {% trans %}No Filter{% endtrans %}
@@ -151,7 +151,7 @@

{% if metadata %}
-

Collection Metadata

+

{{ _('Collection Metadata') }}