remove obsolete option --comment-magics

[FSpanhel]

--comment-magics is not available in version 1.4.1

[codecov]

Codecov Report

Merging #465 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #465   +/-   ##
=======================================
  Coverage   98.88%   98.89%           
=======================================
  Files          79       80    +1     
  Lines        7823     7861   +38     
=======================================
+ Hits         7736     7774   +38     
  Misses         87       87           

Impacted Files	Coverage Δ
tests/test_using_cli.py	100.00% <100.00%> (ø)
tests/test_read_simple_python.py	100.00% <0.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0676d7...cda2661. Read the comment docs.

[mwouts]

Thanks for pointing this out... We really should have a test that executes all these commands!

What happened is that the comment_magics option was moved to the format options, cf. this extract of jupytext --help:

  --format-options FORMAT_OPTIONS, --opt FORMAT_OPTIONS
                        Set format options with e.g.
                            --opt comment_magics=true
                            --opt notebook_metadata_filter=-kernelspec.

So maybe I'd prefer to keep the example with --opt comment_magics=true... what do you think?

[FSpanhel]

I agree with your suggestion.
It's better to update it than to remove it 👍

I think it is useful if this option is explicitly mentioned in the docs. The feature that magic commands are commented out by default confused me at the beginning. (I use VS Code and did not read the docs in a linear fashion but skipped directly to the command line usage)

Thank you for providing such a great package!

[mwouts]

Thank you @FSpanhel . I have added a test on the documentation, we'll see how it goes...

There's one more thing I'd be curious to know... are you able to run the magic commands in VS Code?

[FSpanhel]

Okay. Commands that are specific to IPython, such as shell or magic commands, are commented out by default.
If they are not commented out (and I now know how to do it), we have the problem that the Python script raises an error if it is run (at least if the classical Python interpeter aka 'pure' Python is used).
So in both cases, the script might not produce the same results as the Jupyter notebook.
I think that is the reason of your follow-up question? (:

Instead of deactivating these commands I would convert them so that the script can be run with 'pure' Python.
This is exactly what VS Code does.

The conversions would be as follows:

!dir -> IPython.get_ipython().system('dir')
%magic a b -> IPython.get_ipython().run_line_magic('magic', 'a b')
For cell magic, use IPython.get_ipython().run_cell_magic()

So you would also have to add 'import IPython' at the beginning of the script (or from IPython import get_ipython, which I would prefer). You could probably call this format the 'vs code' format?

You can also see the 'pure' Python equivalent of the IPython commands using %hist -t.
E.g., start an IPython/Jupyter console/Jupyter Notebook (using ipython/jupyter console/jupyter notebook in the terminal), type some IPython comamnds and then run %hist -t.

I would also propose to use IPython and not Jupyter in sentences like the second sentence in following paragraph (and related paragraphs):
''Jupyter notebooks often include magic commands like %load_ext or %matplotlib inline. These commands are Jupyter specific and cannot be executed by the classical interpreter.''
The reason is that beginners often get confused about the difference between IPython and Jupyter Notebooks. So I think it is better to be more specific here (:

[mwouts]

Hello @FSpanhel , well that's an interesting suggestion. I think translating the IPython code to plain Python would be difficult, so I'd prefer not to implement that in Jupytext.

Anyway, if that can help, when I want the same document to be both a valid notebook and a valid script, I use the active-ipynb or active-py cells to create cells that are executed only in one or the other environment - this is briefly documented here.

Also, did you notice the new 'Python interactive' mode in Code?

It seems to be using IPython to execute the code. By the way, if you want to use that mode, you may want to use the hydrogen format, named after Hydrogen, the Atom plugin, which was one of the first IDE to allow executing a script with IPython.

[FSpanhel]

To make a long story short: Have a look at https://github.com/FSpanhel/jupytext_run_shell_magic

Excuse my late reply. I was busy and also wanted to take a deeper look at my suggestion.
Well, running a script that has been created by converting a notebook with the classical Python interpreter is a little bit more difficult than I thought. I think it might be possible if you embed an IPython session, but for simplicity I would suggest to use the IPython interpreter right from the start.
I think one has to distinguish two use cases:

1. Convert a notebook to a *.py script and then use ipython *.py to run the program
2. Launch an interactive IPython shell and use your favorite IDE to edit *.py and run some parts of the script

Moreover, we have to differentiate between a) shell commands “!” and b) magic commands “%”.

I think that 1) and 2) are easy to accomplish for shell commands. I think that 1) would work with some magic commands out-of-the box but not with all. Finding out what would work directly and where other solutions are required would definitely be some effort. I do not recommend following this route. However, 2) might be relatively easy to accomplish for all magic commands.

So let’s assume I’ve created a notebook *.ipynb and I want to edit my code. I would convert the notebook to *.py script and then use my favorite IDE and the IPython shell to investigates my edits. I typically do not use magic commands, but I often invoke the shell using “!”. For example, see the notebooks mentioned below. Right now, these shell commands are commented out (or do not work) when the notebook is converted to a .py script using jupytext.

Let’s have a look at my demonstration: https://github.com/FSpanhel/jupytext_run_shell_magic
In particular, run the notebooks shell.ipynb and magic.ipynb and have a look at them.

I’ve written a very simple program (activate_magic.py) that makes shell commands runnable from IPython (as a script or interactively) and that also makes (almost all) magic commands that do not require a kernel available for an interactive IPython session.
To accomplish this, I’ve done the following steps:

1. Convert *.ipynb to *.py using !jupytext --to py:percent -o *.py *.ipynb (it should work for all formats)
2. (i)python activate_magic.py *.py
   a. Adds “from IPython get ipython” after the jupytext metadata
   b. To support shell commands “# !my_shell_cmd” is replaced by “get_ipython().getoutput(“my_shell_cmd”)”
   c. To support magic commands # %magic_cmd arg1 arg2 is replaced by “get_ipython().run_line_magic(“magic_cmd”, “arg1 arg2”)

Note that I have not considered cell magics. That is because jupytext does not comment out all cell magics (e.g., %latex becomes # %% language="latex") and I have no time to consider this right now.

All in all, I hope this demonstrates that it makes sense to consider shell commands for 1) and 2). Maybe one could also allow magic commands to be used for 2). However, I do not know whether this works for all magic commands. Probably it will work for all magics that are not specific to a kernel such as %connect_info. So it could be easy to allow magic commands for 2) or it could also take some time. Apart from that, allowing magic commands for 1) is very though. For instance, using %autocall 1 allows us to write “print 3” instead of “print(3)”. This works if only print 3 is send to the terminal. However, if you send “print 3\n print 2” to the terminal an error is thrown. This also happens if you execute “print 3\n print 2” in a Jupyter notebook cell. That is the reason why ipython magic_with_runnable_magic_cmds.py throws an error.

I hope my comments are interesting and useful for you.

PS: I’ve found that magic and shell commands are not always commented out., e.g., my_host = !hostname in shell.ipynb is not commented out when using jupytext (maybe because you only check for % as first non-whitespace character?).

[FSpanhel]

About the 'Python interactive' mode in Code.
I've known it for some time :)
I think it does not use IPython, because it starts a kernel. I don't know what it exactly is, but I think it is quite similar to the jupyter console which also starts a kernel.
I prefer to use IPython and the jupyter console though. I don't like the formatting of the 'Python interactive' window.

But doesn't vs code use the percent format? I haven't taken a look into the hydrogen format. Will do.

[FSpanhel]

So py:hydrogen seems to be the same as py:percent --opt comment_magics=false?
I think this works with new native Juypter notebook support that has been released last October.