easy contributable internationalization process with sphinx @ pyconmy2015
Post on 09-Feb-2017
850 Views
Preview:
TRANSCRIPT
Easy contributable internationalization process with Sphinx
Takayuki Shimizukawa
Sphinx co-maintainer
Sphinx-users.jp
1
こんにちはKon-Nichi-Wa
2
Who am I
@shimizukawa
1. Sphinx co-maintainer
2. Sphinx-users.jp
3. PyCon JP committee
BePROUD co, ltd.3
http://pycon.jp/
4
Agenda
1. Sphinx introduction & Setup for i18n
2. A easy way and Non easy ways to translate
3. Sphinx i18n feature
4. Automated translation process with several services
5. Tips, tricks, traps 5
Question
6
Question1
How many people do you use Open Source
Software?
7
Question2
How many people may have contributed to the
OSS?
8
Question3
Does the translation into familiar language
contribute to other developers?
9
10
Translator
Product Author
Translated Docs
Readers
English Docs
Translators
Tools & Services
Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for
Transifex
Services5. Transifex - translation support service6. Drone.io - continuous integration service
11
1. Sphinx introduction& Setup for i18n
12
What is Sphinx?
Sphinx is a documentation generator
Sphinx generates doc as several output formats from the reST text
markup
13
1. Sphinx introduction & Setup for i18n
Sphinx
reSTreSTreStructuredText
(reST)reST
Parser
HTML Builder
ePub Builder
LaTeX Builder texlive
HTMLtheme
Favorite Editor
The history of Sphinx (short ver.)
14
1. Sphinx introduction & Setup for i18n
The father of Sphinx
Too hard to maintenance
~2007
Easy to writeEasy to
maintenance2007~
Sphinx Before and After
Before
There was no standard ways to write documentsSometime, we need converting markups into other
formats
After
Generate multiple output format from single sourceIntegrated html themes make read docs easierAPI references can be integrated into narrative docsAutomated doc building and hosting by ReadTheDocs
15
1. Sphinx introduction & Setup for i18n
Many docs are written by SphinxFor examples
Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …
Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish
16
1. Sphinx introduction & Setup for i18n
Many docs are written by SphinxFor examples
Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …
Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish
17
1. Sphinx introduction & Setup for i18n
SPHINXdocutils
HTML Builder HTML theme(Jinja2)
gettext Builder
*.pot
*.po
I18N
*.mo OmegaT
Pootle
Transifex
Translation Tools, Services
Favorite Editor
Sphinx i18n feature (built-in)
18
1. Sphinx introduction & Setup for i18n
reST Parser(directive / role)
doctree(Intermediat
e)
reSTreSTreStructuredText(reST)
$ pip install sphinx
Support tools for translation
How to install Sphinx with i18n tools
To build a sphinx document
$ pip install sphinx-intl$ pip install transifex-client=0.8
19
1. Sphinx introduction & Setup for i18n
$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.
"make html" command generates html files into _build/html.
make html once
20
1. Sphinx introduction & Setup for i18n
Files & setup conf.py$ tree /path/to/deepthought+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.py| +- make.bat| +- Makefile+- setup.py
21
1. Sphinx introduction & Setup for i18n
Document source
Document build output
Target library for doc1. ...2. 3. language = 'ma'4. locale_dirs = ['locale']5.
doc/conf.py
2. Easy contributable internationalization process with Sphinx
22
What is internationalization?
23
2. Easy contributable internationalization process with Sphinx
I18N
What is easy contributable?
24
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 1/3)
The manual are provided only in the HTML files
Rewriting reST files
25
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 2/3)
The manual are generated fromreST files and
docstrings in the source files
Rewriting reST filesRewriting docstrins in the source files.
26
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 3/3)
OK, we are engineers. We can use git!
Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to
do!)git commitgit pullSometimes resolve conflictgit push
27
2. Easy contributable internationalization process with Sphinx
Not easy vs easy
Not a easy way Easy way1. Learn git2. Learn GitHub3. git clone to get the
code4. Translation5. git commit, pull,
push6. Resolve conflict7. Build html your self
1. No git2. No Github3. No file4. Translation5. Update
Automatically6. No conflict7. You can get a HTML
output w/o hand-build.
28
2. Easy contributable internationalization process with Sphinx
3. Sphinx i18n feature
29
Two i18n features of SphinxOutput pot files:
from reST
Input po files:to generate translated HTML
30
3. Sphinx i18n feature
reST pot
reST
htmlpo
Translation flowGenerate pot
Translate pot into po
Generate Translated HTML
31
3. Sphinx i18n feature
reST pot
reST
htmlpo
pot poTranslator
Pleasetranslate
Translate!Thanks for yourContribution!
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""
msgid "For example:"msgstr ""
32
Output pot files 3. Sphinx i18n feature
$ make gettext...Build finished. The message catalogs are in _build/gettext.
$ ls _build/gettextapi.pot examples.pot generated.pot index.pot
generated.pot
reST pot
Preparing po files to translate
doc+- _build/| +- gettext/| +- api.pot| +- examples.pot| +- generated.pot| +- index.pot+- locale/
doc+- _build/+- locale/ +- zh_cn/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/
copy and renamefor each langs
Translate them
Translator
pot po
33
3. Sphinx i18n feature
34
Preparing po files to translate
$ sphinx-intl update -p _build/gettext -l zh_cnCreate: locale/zh_cn/LC_MESSAGES/api.poCreate: locale/zh_cn/LC_MESSAGES/examples.poCreate: locale/zh_cn/LC_MESSAGES/generated.poCreate: locale/zh_cn/LC_MESSAGES/index.po
At first, sphinx-intl copy pot files and rename them
pot po
sphinx-intl
$ make gettext$ sphinx-intl update -p _build/gettext -l zh_cnNot Changed: locale/zh_cn/LC_MESSAGES/api.poUpdated: locale/zh_cn/LC_MESSAGES/examples.po +3, -1Not Changed: locale/ma/LC_MESSAGES/generated.po
After change the document, sphinx-intl update differences
3. Sphinx i18n feature
Translate po files
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""
generated.po
pot po
sphinx-intl
Translator
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr " 序列 ``obj`` 以 JSON 格式 :class:`str` 。 "
generated.po
Translate by using Vim, Emacs, OmegaT, ...
35
3. Sphinx i18n feature
Input po files
36
3. Sphinx i18n feature
reST
htmlpo
Translated
$ make html...Build finished. The HTML pages are in _build/html.
Big picture
37
3. Sphinx i18n feature
reST pot
html
po
make gettext
sphinx-intl
Translator
make html
Doc Author
TranslationCatalog
Translatedcatalog
4. Automated translation process with several services
38
Entire process to translate sphinx docs
39
4. Automated translation process with several services
reST pot
html
po
make gettext
sphinx-intl
make html
Doc Author
TranslationCatalog
Translatedcatalog
Translator
Translator
Translator
Autor / Translator
upload
Translator
clone
Translator
To Be
40
4. Automated translation process with several services
reST pot
html
po
make gettext
sphinx-intl
make html
Doc Author
TranslationCatalog
Translatedcatalog
upload
Translators
To BeAutomated
To BeParallel
clone
Translation tool typesVim / Emacs (Editors)
Edit local filesTranslation support plugins are available.
OmegaT (Translation Tools)Edit local files.Translation support features.
Transifex (Services)Edit onlineTranslation support
features.
41
4. Automated translation process with several services
po
Translators
To BeParallel
Be Parallel by using Transifex
Transifex provides...As API:Upload potDownload po
As Web console:GlossaryTranslation memoryRecommendationAuto-translation
42
4. Automated translation process with several services
po
Translators
Parallel
pot
Upload
pot
Auto Update
sphinx-intltransifex-client
po transifex-clientDownload
Translation on Transifex web console
43
4. Automated translation process with several services
Original Text
Translated Text(you should keep reST syntax)
Suggestions fromTranslation Memory (TM)
Original(pot)
Translation(po)
Copy orig to translationMachine translation
SaveReview (if needed)
Translators
Parallel1
2
4
3
56
To Be Automated
44
4. Automated translation process with several services
po
Translators
Parallel
pot
Upload
pot
Auto Update
sphinx-intltransifex-client
po transifex-clientDownload
reST
html
make gettext
make html
Doc Author
upload
To BeAutomated
clone
To Be Automated
45
4. Automated translation process with several services
pot
Upload sphinx-intltransifex-client
po transifex-clientDownload
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomated
The procedure for automation
1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html
46
4. Automated translation process with several services
pot
Upload sphinx-intltransifex-client
po transifex-clientDownload
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomate
d
1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources
# update .tx/config8. tx push -s # push pot files to
transifex9. tx pull -l zh_cn # pull po files from
transifex10. make html SPHINXOPTS="-D language=ma"
Shell commands for automationrun.sh
$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh
47
4. Automated translation process with several services
The drone.io
48
WebHook
Deploy
Clone repositoryRun shell script
The drone.io is a continuous integration service.
4. Automated translation process with several services
GitHub + drone.io + S3
49
GitHub
Amazon S3
Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html
4. Automated translation process with several services
2
1
3
Be Automated
50
pot
Upload sphinx-intltransifex-client
po transifex-clientDownload
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomated
UploadpotDownload
poupload
WebHookclone
1
5
6make html
make gettext2
3
4
1WebHook
Automated
4. Automated translation process with several services
Automated by drone.io
51
UploadpotDownload
poupload
WebHookclone
1
56
make html
make gettext2
3
4
1WebHook
Automated
4. Automated translation process with several services
View from doc author
Doc Author doesn't require annoying procedure.
UpdateTranslation Source
Doc Author
Notify
See
Commit
52
4. Automated translation process with several services
View from doc translatorsNo gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-
build.
Translators
ParallelSee
TranslateTranslated Pages
53
4. Automated translation process with several services
The entire automated process
54
UpdateTranslation Source
Doc Author
Translators
Parallel
Notify
See
See
Publish
Translate
Commit
DownloadTranslations
Notify
Automated
4. Automated translation process with several services
Summary
Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for
Transifex
Services5. Transifex - translation support service6. Drone.io - continuous integration service
55
4. Automated translation process with several services
5. Tips, tricks, traps
56
TIP: Drone.io limits 15mins for a build
Drone.io limits 15mins for a build.Install from wheel package may help you
57
5. Tips, tricks, traps
1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -
2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh
ex. https://bitbucket.org/sphinxjp/docutils-translation/
TRAP: Version of transifex-client
transifex-client 0.11b3 is not stable (for me)Especially for Windows users
If you have encountered trouble with using newer version, please try:
58
5. Tips, tricks, traps
$ pip install "transifex-client=0.8"
Drone.io only list-up repositories which you have admin permission.
Prepare a empty repository to create a drone.io project.
59
TRICK: Preparing Drone.io project5. Tips, tricks, traps
Examples
Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin
Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin
Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin
60
5. Tips, tricks, traps
Questions?@shimizukawa
Grab me after the session :)
61
Thanks :)
62
top related