add japanese documentation

[rajveer43]

part of #18413

What does this PR do?

Fixes #18413
Fixes #26355

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[amyeroberts]

cc @younesbelkada @stevhliu

[younesbelkada]

Thanks a lot for this great addition, content-wise it looks great to me! I just left one comment (I am not sure) about the title! Thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[stevhliu]

Thanks for your translation!

Currently, it looks like the Japanese translations are being built/shown in the docs so I think we need to add ja to the language codes here:

transformers/.github/workflows/build_documentation.yml

Line 18 in 05de038

languages: de en es fr it ko pt zh

and here:

transformers/.github/workflows/build_pr_documentation.yml

Line 17 in 05de038

languages: de en es fr it ko pt zh

[rajveer43]

@stevhliu I have added documentation for preprocessing.md as well kindly review it and added language code for it as well in the worflows.

[rajveer43]

also will add for most of the files

[stevhliu]

Wow nice, thanks for translating all those additional docs! 👏

[rajveer43]

will add for all!

[rajveer43]

Locally It is not passing I got the following error while building doc locally

doc-builder build transformers docs/source/ja/ --build_dir ~/tmp/test-build

I am running above command and the logs are. do I need to add that files to _toctree.yml, Its contradicts to PR_doc_build workflow

Building docs for transformers docs/source/ja/ /home/codespace/tmp/test-build/transformers/main/en
Building the MDX files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:00<00:00, 210.89it/s]
Traceback (most recent call last):
  File "/home/codespace/.python/current/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/build_doc.py", line 369, in build_doc
    sphinx_refs = check_toc_integrity(doc_folder, output_dir)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/build_doc.py", line 427, in check_toc_integrity
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- debugging
- notebooks
- add_new_pipeline
- sagemaker
- contributing
- tasks/visual_question_answering
- tasks/audio_classification
- tasks/summarization
- tasks/document_question_answering
- tasks/image_classification
- tasks/sequence_classification
- tasks/masked_language_modeling
- tasks/video_classification
- tasks/question_answering
- tasks/semantic_segmentation
- tasks/idefics
- tasks/zero_shot_image_classification
- tasks/language_modeling
- tasks/asr
- tasks/image_captioning
- tasks/monocular_depth_estimation
- tasks/multiple_choice
- tasks/translation
- tasks/text-to-speech
- tasks/object_detection
- tasks/token_classification
- tasks/zero_shot_object_detection
- main_classes/output
- main_classes/configuration
- main_classes/deepspeed
- main_classes/keras_callbacks
- main_classes/onnx
- main_classes/feature_extractor
- main_classes/pipelines
- main_classes/image_processor
- main_classes/trainer
- main_classes/callback
- main_classes/agent
- main_classes/logging
- main_classes/quantization
- main_classes/text_generation
- main_classes/optimizer_schedules
- main_classes/model
- main_classes/tokenizer
- main_classes/data_collator
- main_classes/processors
- internal/file_utils
- internal/modeling_utils
- internal/pipelines_utils
- internal/image_processing_utils
- internal/tokenization_utils
- internal/time_series_utils
- internal/audio_utils
- internal/generation_utils
- internal/trainer_utils
- model_doc/swiftformer
- model_doc/mpnet
- model_doc/auto
- model_doc/cpm
- model_doc/persimmon
- model_doc/mobilevitv2
- model_doc/switch_transformers
- model_doc/roc_bert
- model_doc/flan-ul2
- model_doc/owlvit
- model_doc/barthez
- model_doc/vision-text-dual-encoder
- model_doc/ul2
- model_doc/phobert
- model_doc/xlm-roberta
- model_doc/realm
- model_doc/convnextv2
- model_doc/gptj
- model_doc/git
- model_doc/xlm-prophetnet
- model_doc/xclip
- model_doc/segformer
- model_doc/vilt
- model_doc/pix2struct
- model_doc/clap
- model_doc/speech-encoder-decoder
- model_doc/dpr
- model_doc/canine
- model_doc/herbert
- model_doc/tapas
- model_doc/bigbird_pegasus
- model_doc/deformable_detr
- model_doc/roberta-prelayernorm
- model_doc/marian
- model_doc/pegasus_x
- model_doc/bridgetower
- model_doc/vision-encoder-decoder
- model_doc/bark
- model_doc/encoder-decoder
- model_doc/rembert
- model_doc/oneformer
- model_doc/esm
- model_doc/bert-generation
- model_doc/llama2
- model_doc/dinov2
- model_doc/wav2vec2-conformer
- model_doc/convbert
- model_doc/layoutlmv2
- model_doc/instructblip
- model_doc/trocr
- model_doc/lxmert
- model_doc/vits
- model_doc/vivit
- model_doc/matcha
- model_doc/deta
- model_doc/dinat
- model_doc/cvt
- model_doc/mra
- model_doc/retribert
- model_doc/mobilevit
- model_doc/van
- model_doc/conditional_detr
- model_doc/gpt_neox
- model_doc/squeezebert
- model_doc/data2vec
- model_doc/ernie_m
- model_doc/sam
- model_doc/vit_mae
- model_doc/mbart
- model_doc/opt
- model_doc/informer
- model_doc/visual_bert
- model_doc/mpt
- model_doc/jukebox
- model_doc/convnext
- model_doc/xlnet
- model_doc/gpt_bigcode
- model_doc/code_llama
- model_doc/umt5
- model_doc/transfo-xl
- model_doc/bert-japanese
- model_doc/nystromformer
- model_doc/roformer
- model_doc/fnet
- model_doc/sew-d
- model_doc/musicgen
- model_doc/mobilenet_v1
- model_doc/videomae
- model_doc/rwkv
- model_doc/flan-t5
- model_doc/timesformer
- model_doc/swin
- model_doc/roberta
- model_doc/whisper
- model_doc/camembert
- model_doc/plbart
- model_doc/vit_msn
- model_doc/megatron_gpt2
- model_doc/bort
- model_doc/m2m_100
- model_doc/bloom
- model_doc/codegen
- model_doc/deplot
- model_doc/gpt_neox_japanese
- model_doc/nat
- model_doc/speech_to_text_2
- model_doc/mega
- model_doc/mobilebert
- model_doc/wav2vec2
- model_doc/blip-2
- model_doc/beit
- model_doc/t5v1.1
- model_doc/gpt-sw3
- model_doc/reformer
- model_doc/clip
- model_doc/funnel
- model_doc/resnet
- model_doc/poolformer
- model_doc/regnet
- model_doc/bertweet
- model_doc/fsmt
- model_doc/mask2former
- model_doc/flava
- model_doc/tapex
- model_doc/dialogpt
- model_doc/mvp
- model_doc/longformer
- model_doc/xmod
- model_doc/ernie
- model_doc/idefics
- model_doc/focalnet
- model_doc/rag
- model_doc/pop2piano
- model_doc/layoutlmv3
- model_doc/chinese_clip
- model_doc/xlsr_wav2vec2
- model_doc/trajectory_transformer
- model_doc/layoutxlm
- model_doc/bros
- model_doc/xlm-v
- model_doc/speecht5
- model_doc/unispeech-sat
- model_doc/prophetnet
- model_doc/mgp-str
- model_doc/bart
- model_doc/nezha
- model_doc/table-transformer
- model_doc/upernet
- model_doc/bit
- model_doc/mctct
- model_doc/hubert
- model_doc/graphormer
- model_doc/vit_hybrid
- model_doc/mluke
- model_doc/t5
- model_doc/swin2sr
- model_doc/ctrl
- model_doc/perceiver
- model_doc/led
- model_doc/vitdet
- model_doc/llama
- model_doc/mobilenet_v2
- model_doc/unispeech
- model_doc/dpt
- model_doc/luke
- model_doc/splinter
- model_doc/clipseg
- model_doc/efficientnet
- model_doc/yoso
- model_doc/decision_transformer
- model_doc/donut
- model_doc/nllb
- model_doc/lilt
- model_doc/wav2vec2_phoneme
- model_doc/falcon
- model_doc/blenderbot
- model_doc/bert
- model_doc/gpt_neo
- model_doc/efficientformer
- model_doc/groupvit
- model_doc/imagegpt
- model_doc/nougat
- model_doc/openai-gpt
- model_doc/electra
- model_doc/longt5
- model_doc/megatron-bert
- model_doc/deit
- model_doc/encodec
- model_doc/biogpt
- model_doc/time_series_transformer
- model_doc/mt5
- model_doc/xglm
- model_doc/flaubert
- model_doc/distilbert
- model_doc/pvt
- model_doc/audio-spectrogram-transformer
- model_doc/qdqbert
- model_doc/deberta
- model_doc/wavlm
- model_doc/xlm
- model_doc/glpn
- model_doc/swinv2
- model_doc/gptsan-japanese
- model_doc/blenderbot-small
- model_doc/markuplm
- model_doc/tvlt
- model_doc/speech_to_text
- model_doc/yolos
- model_doc/align
- model_doc/gpt2
- model_doc/xlm-roberta-xl
- model_doc/levit
- model_doc/pegasus
- model_doc/bartpho
- model_doc/autoformer
- model_doc/open-llama
- model_doc/layoutlm
- model_doc/albert
- model_doc/big_bird
- model_doc/detr
- model_doc/ibert
- model_doc/vit
- model_doc/vitmatte
- model_doc/altclip
- model_doc/dit
- model_doc/nllb-moe
- model_doc/maskformer
- model_doc/blip
- model_doc/deberta-v2
- model_doc/mistral
- model_doc/cpmant
- model_doc/sew
- model_doc/mms
- model_doc/xls_r
- model_doc/byt5
Add them to docs/source/ja/_toctree.yml.

[stevhliu]

Locally It is not passing I got the following error while building doc locally

It's probably because all those files are in the directory, so you can just remove them, and it should be built locally. But it's ok since the CI tests are passing now. There are still some changes (like this one) that need to be addressed.

[rajveer43]

things I learned:

• Work module wise
• documentation is as much as important as coding.

[rajveer43]

@stevhliu approve the workflow.

[stevhliu]

Thanks again for translating and contributing such a significant amount of the docs to Japanese! 🇯🇵🥳 Hope you had fun!

One last thing before we merge, let's revert the changes to the English md files. The date should reflect when the doc was written so there's no need to update them to 2023.

[rajveer43]

Thanks again for translating and contributing such a significant amount of the docs to Japanese! 🇯🇵🥳 Hope you had fun!

One last thing before we merge, let's revert the changes to the English md files. The date should reflect when the doc was written so there's no need to update them to 2023.

sure

[rajveer43]

@stevhliu I reverted the changes you told!