Regression in 1.3.1 due to #407 when scripts and notebooks are in a separate folder.

[lesteve]

The directory structure looks like this:

├── ./notebooks
│   ├── ./notebooks/01_tabular_data_exploration.ipynb
├── ./python_scripts
│   ├── ./python_scripts/01_tabular_data_exploration.py

I am running jupytext like this:

jupytext --set-formats python_scripts//py:percent,notebooks//ipynb python_scripts/01_tabular_data_exploration.py

Output:

[jupytext] Reading python_scripts/01_tabular_data_exploration.py
[jupytext] Updating notebook metadata with '{"jupytext": {"formats": "python_scripts//py:percent,notebooks//ipynb"}}'
Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/scikit-learn-tutorial/bin/jupytext", line 10, in <module>
    sys.exit(jupytext())
  File "/home/lesteve/miniconda3/envs/scikit-learn-tutorial/lib/python3.7/site-packages/jupytext/cli.py", line 271, in jupytext
    exit_code += jupytext_single_file(nb_file, args, log)
  File "/home/lesteve/miniconda3/envs/scikit-learn-tutorial/lib/python3.7/site-packages/jupytext/cli.py", line 464, in jupytext_single_file
    for alt_path, alt_fmt in paired_paths(nb_file, fmt, formats)[::-1]:
  File "/home/lesteve/miniconda3/envs/scikit-learn-tutorial/lib/python3.7/site-packages/jupytext/paired_paths.py", line 126, in paired_paths
    short_form_multiple_formats(formats)))
jupytext.paired_paths.InconsistentPath: Paired paths 'python_scripts/python_scripts/01_tabular_data_exploration.py','python_scripts/notebooks/01_tabular_data_exploration.ipynb' do not include the current notebook path 'python_scripts/01_tabular_data_exploration.py'. Current format is 'py:percent', and paired formats are 'python_scripts//py:percent,notebooks//ipynb'.

If I try to change the format to avoid the error then jupytext --sync does not seem to create the notebook in the right place. Maybe I was doing something fishy all along in which case I would appreciate advice how to use jupytext better ...

Here is a way to reproduce:

pip install jupytext==1.3.1
cd /tmp && git clone https://github.com/lesteve/scikit-learn-tutorial && cd scikit-learn-tutorial
jupytext --set-formats python_scripts//py:percent,notebooks//ipynb python_scripts/01_tabular_data_exploration.py

Same thing works fine with jupytext 1.3:

pip install jupytext==1.3
cd /tmp && git clone https://github.com/lesteve/scikit-learn-tutorial && cd scikit-learn-tutorial
jupytext --set-formats python_scripts//py:percent,notebooks//ipynb python_scripts/01_tabular_data_exploration.py

git bisecting points to #407.

[mwouts]

Thanks @lesteve for reporting this. I think you have reported every bit of useful information (👍 👍 ), so I'll soon add a test to reproduce this, and then fix it.

[mwouts]

This should be fixed in version 1.3.5 (available on pip). Thank you @lesteve for reporting this!

[lesteve]

Great thanks a lot for this! I just tested with 1.3.5 and it does fix the issue.

[lesteve]

By the way, we are trying to have a nice workflow where we synchronise python scripts and Jupyter notebooks with Jupytext. Some people prefer to write Python scripts and execute them cell by cell with Visual Studio Code. Some people prefer to write notebooks instead so there is no primary "source of truth". On top of that we are trying to use yapf in order to have short lines because we want to use the notebooks for tutorials (you often need to increase the font and we want the lines to fit on the screen)

If you know of some projects who have similar workflows with Jupytext and have some nice scripts or Makefile or something better to automatically sync everything nicely, that would be greatly appreciated!

At the moment I started with a horrible Makefile and I'd rather start afresh from something that already works rather than poorly reinvent the wheel.

[mwouts]

Hello @lesteve , well I am not aware yet of a project which has solved this sync/reformatting challenge, but I think we'll soon see one - maybe yours?

Let me add that

• I don't know much about Makefile unfortunately, but yours look nice!
• maybe there's something to be done with jupytext + pre-commit hooks
• we are also discussing a similar question at How to clean/tests notebooks? #432

When we find a convincing way to address the sync/correctedness/reformatting issues, I'd be interested in including this (or a link to a blog post) in the documentation.

[lesteve]

I don't know much about Makefile unfortunately, but yours look nice!

I don't know much either about Makefiles and just to be explicit, nobody should use my Makefile because it is horrible and it does not really work!

OK if I end up doing something not too hacky, I will let you know. My feeling is that it will be very hacky though, at least the first iteration.