Problems integrating parinfer into Cursive

Here's some general information about the problems of integrating parinfer into Cursive/IntelliJ. This is partly to answer a question @shaunlebron asked about why it was so hard, possibly to help anyone who wants to integrate parinfer into a similarly complicated editor (perhaps Visual Studio, Eclipse or something of that nature) and also to potentially guide future parinfer changes to try to avoid these problems. This is mostly a brain dump right now.

Race conditions break your code by not running Paren Mode in time

The main issue that people have encountered with the Cursive integration is that it will very occasionally break their code. This is due to the need to initially run paren mode on a file before running indent mode. For most editors this seems trivial - when the user opens a file, you run paren mode and you're done. However IntelliJ has a much more complicated architecture which makes this non-trivial. I suspect that the most common case Cursive users encounter is a simple race condition I can hopefully fix, but in general it's impossible to do this reliably for several common use cases.

Document object has zero or more Editor views

IntelliJ represents file content using Document objects. Unlike most editors, Document objects aren't tied to Editor instances, and they're shared across open projects (i.e. every project uses the same Document object for a particular file, on disk or in a jar etc). IntelliJ also has Editor objects representing open editors, and each one is a view on a Document. This architecture means that there can be multiple Editors viewing and editing a single Document, and there are also Document objects with no associated editor.

Run Paren Mode after Document or Editor creation

The main issue is determining when to run paren mode. I could do this on Document creation, but that will modify files even when the user doesn't do anything e.g. when indexing. Currently I do this on Editor creation, but Documents are frequently modified in the background even when not associated with an editor at all. This will commonly happen with refactorings, for example a project-wide rename.

Run Paren Mode before Editor action

Another thing I tried is running paren mode right before an action is executed. This would be desirable because then file indentation would not be modified by just viewing a file, the user would actually have to edit it to provoke this change. Unfortunately this is hard - I can get a notification before an action is invoked. Unfortunately before the action is invoked there's no way to know if it will modify the file or not, and since basically everything in IntelliJ is an action even the act of opening a file or navigating away from it would trigger this. Additionally, there's no way to know which documents an action might affect, even if I keep a whitelist of mutating actions.

Run Paren Mode after first Document action

Cursive's parinfer implementation is driven by a listener of modifications to Document contents. I tried to run paren mode on the first modification I receive, but this is prohibited - you can't modify a Document during one of its listeners. This makes sense, since an action will often read the file contents, calculate offsets and related changes it's going to make and then apply them. If the document were to be moved around during the application of the changes, the action's pre-calculated values would be wrong and the action would corrupt the document.

Additional outstanding complications

1. Caret positions are associated with Editors, not Documents. Since the Editor/Document mapping can be 1-1, 0-1 or many-1, it's also hard to reliably get caret position information for a Document when calling parinfer. Currently I have a hack which doesn't work well in some cases (e.g. editing a file in an IntelliJ diff view). I may need to modify the algorithm to potentially accept multiple caret positions, but I suspect that will lead to complicated cases when e.g. multiple carets are on the same line simultaneously.
2. Document modifications happen asynchronously on the Swing EDT thread, which I suspect is the cause of the race condition. I'm currently not sure how to solve this.
3. Actions can create multiple overlapping changes during processing. In particular, some of the paredit actions can create changes in unpredictable orders at unpredictable locations. This complicates the calculation of the changes to be passed in to parinfer, and would also complicate the partial application of parinfer discussed below.

Conclusion

So, this is very hard. I'm going to try to fix the race condition, but even if I manage that I cannot reliably run paren mode before indent mode. In my ideal world, parinfer would only fix up the sexps affected by a particular action, but that is a surprisingly complicated thing to do correctly. I'd also settle for simply refusing to apply parinfer if the document isn't formatted correctly and showing a warning to the user - this may turn out to be my best option. However this may nag the user a lot if they're working on a file which is indented incorrectly in some distant part of the file. Ideally this nagging would also be local, but as above this is difficult.