Use project / system line endings

[Zeckie]

Previously, blackd always used \n (Unix / Linux style) line endings, even on Windows.

When I am using this plugin, and make changes that result in black changing the formatting, I get this warning in PyCharm;

I am using these PyCharm settings (on Windows):

This plugin should use the PyCharm / IDEA line separator setting, and either pass that configuration to blackd, or convert the line endings when black makes any changes.

[lensvol]

Interesting idea, thank you. We will try and add something like this in the near future.

[KotlinIsland]

Currently, blackd always uses \n (Unix / Linux style) line endings, even on Windows.

psf/black#3257

[Zeckie]

Currently, blackd always uses \n (Unix / Linux style) line endings, even on Windows.

psf/black#3257

That will help, but this issue proposes checking that the line endings match the project settings, not just that they are consistent within the file.

[lensvol]

It should not be very difficult to accommodate that, I will look into it. Or someone else may also participate, it is basically the issue off replacing any existing line endings in a file with a project specific ones.

[Zeckie]

I'll have a go at implementing this, though haven't worked on PyCharm / Intellij plugins before

[lensvol]

@Zeckie Did you make any progress with this? Or can I implement this proposal by myself?

[Zeckie]

Yes, happy for you to do that.

I didn't have much success trying to narrow down what exactly was causing the issue. From looking at the code, it seems as though the responsibility for line ending type was with the intellij code, not this plugin.

I also found that I couldn't reproduce this issue with just pycharm and this plugin, I needed to set up a filewatcher (eg running isort) too. It seems as though the issue occurs when both black and isort are trying to change the same file.

[lensvol]

@Zeckie Interestingly, I tried to reproduce this using PyCharm 2022.2.3 with the current plugin and failed to trigger those inspections. I was using \r\n as a line separator on my Mac OS X machine.

While code returned from blackd indeed has \n as a line separator, after we pass it to PyCharm no error is triggered.

Can you please share your PyCharm version?

[Zeckie]

@lensvol, I was able to reproduce this fairly reliably, but have installed some updates (and potentially made some minor changes to settings), and it does not seem to happen any more.

I've installed some updates since originally reporting, now I'm using:

• IDE: PyCharm (build #PC-223.8214.51, 20 Dec 2022 21:21)
• Windows 10 (10.0, amd64)
• JVM: 17.0.5+1-b653.14 (OpenJDK 64-Bit Server VM)

Plugins:

• BlackConnect 0.5.0
• File Watchers 223.7571.113

Python dependencies (pip freeze):

aiohttp==3.8.3
aiosignal==1.3.1         
async-timeout==4.0.2     
attrs==22.2.0            
black==22.12.0           
charset-normalizer==2.1.1
click==8.1.3             
colorama==0.4.6          
frozenlist==1.3.3        
idna==3.4                
isort==5.11.4            
multidict==6.0.4         
mypy-extensions==0.4.3   
pathspec==0.10.3         
platformdirs==2.6.0      
tomli==2.0.1             
yarl==1.8.2

Configuration:

• line separator settings set to to windows (\r\n), and Inconsistent line separators inspection enabled
• Mark modified enabled, so I can see when the file has been saved
  
• BlackConnect using mostly default settings (configured path to blackd, and ensured it is running and connected)
• one file watcher configured:
  

Steps that I was previously able to use to reproduce the issue:

1. Start with a python file that has correct line endings open in editor
2. Paste some code into editor that deliberately has some issues to be fixed by black, and some to be fixed by isort, like:

from  typing import  (
Any, Any)
import  re
a: Any = re.match(
"foo", 
"bar")

3. Save (Ctrl + S)
4. Wait for black and isort to both run (black to fix the formatting, isort to swap the import lines, and remove the duplicate Any. The warning about inconsistent lines was usually displayed.

Now, it seems as though something has changed at step 4 - after saving, it looks like only black runs, and the editor shows up as though it has not saved (the asterisk over the icon):

Pressing Ctrl+S again runs isort.

If I turn off the Trigger when saving changed files setting in BlackConnect, it no longer shows as being unsaved after pressing Ctrl+S once.

I saw this in idea.log, and it looks like it might be related:

2022-12-29 19:33:30,735 [111016548]   INFO - #c.i.o.f.i.MemoryDiskConflictResolver - reload test.py from disk?
2022-12-29 19:33:30,735 [111016548]   INFO - #c.i.o.f.i.MemoryDiskConflictResolver -   documentStamp:3502311
2022-12-29 19:33:30,735 [111016548]   INFO - #c.i.o.f.i.MemoryDiskConflictResolver -   oldFileStamp:3502310
2022-12-29 19:34:15,643 [111061456]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking vfs changes] start aborted: already running
2022-12-29 19:34:15,643 [111061456]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking vfs changes] start aborted: already running
2022-12-29 19:34:15,643 [111061456]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking unsaved documents] start aborted: already running
2022-12-29 19:34:15,643 [111061456]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking vfs changes] start aborted: already running
2022-12-29 19:34:15,644 [111061457]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking unsaved documents] stopped
2022-12-29 19:34:15,644 [111061457]   INFO - #c.i.p.w.FwProjectChangeTracker - [tracking vfs changes] start aborted: already running

[Zeckie]

I have also been able to get the same error (from Inconsistent line separators inspection) without BlackConnect installed, by doing the following:

1. Start with a python file that has correct line endings open in editor
2. Paste some code into editor that deliberately has some issues to be fixed by black, and some to be fixed by isort, like:

from  typing import  (
Any, Any)
import  re
a: Any = re.match(
"foo", 
"bar")

3. Save (Ctrl + S)
4. While isort watcher is running, type something (eg. a space)

[lensvol]

@Zeckie Given your findings that this can be reproduced without BlackConnect, I would assume the issue is not really related to plugin?

[Zeckie]

@lensvol, yes that makes sense - looks to me as though it is either an issue with File Watchers plugin or some internals of PyCharm (https://youtrack.jetbrains.com/issue/PY-58254/Line-endings-incorrectly-changed-on-conflict-between-changes-in-editor-and-file-watcher)

There does still appear to be an issue with this plugin, but I think it is different enough to warrant a new issue:

Now, it seems as though something has changed at step 4 - after saving, it looks like only black runs, and the editor shows up as though it has not saved (the asterisk over the icon):