Julia hangs on shell command in macro body

[stevengj]

The following command hangs for me (Debian GNU/Linux x86_64):

macro foo(ex)
   run(`echo hello`)
   ex
end
@foo nothing

(Why am I doing this, you ask? I have a @pyimport macro that imports a Python module, which needs to initialize Python, which needs to run the python executable to obtain the shared-library name.)

[JeffBezanson]

This fixes it, @loladiro and/or @vtjnash can review:

--- a/base/client.jl
+++ b/base/client.jl
@@ -89,6 +89,7 @@ function eval_user_input(ast::ANY, show_value)
 end

 function readBuffer(stream::AsyncStream, nread)
+    @async begin
     global _repl_enough_stdin::Bool
     while !_repl_enough_stdin && nb_available(stream.buffer) > 0
         nread = int(search(stream.buffer,'\n')) # never more than one line or readline explodes :O
@@ -109,6 +110,7 @@ function readBuffer(stream::AsyncStream, nread)
         #println(STDERR,stream.buffer.data[stream.buffer.ptr-nread:stream.buffer.ptr-1])
         ccall(:jl_readBuffer,Void,(Ptr{Void},Int32),ptr,nread)
     end
+    end
     return false
 end

The problem is that the run is blocking in the same task that parses user input, which is an event handler. Wasn't expecting blocking operations there.

I do suggest making macros pure functions. With this change, the code should work for now, but I can imagine strange problems where things don't get expanded when or as often as you want.

[Keno]

That seems extremely weird to me, but I have no real objection to it either. It seems to me however that the only general fix for this is to never parse code inside an event handler, which I don't think is feasible.

[vtjnash]

I'll have to think on this some more. I don't like the implications of making this async (it's called on every keystroke). However, I hadn't contemplated the possibility that this could block, since seemed it shouldn't be executing code until it was back in the main thread. I guess we could set up a dedicated stdin reader task. However, I'm inclined to think code compilation needs to be delayed.

[JeffBezanson]

I agree, for the repl at least, it would be better to delay expansion to when the input is executed.

[stevengj]

Conceptually, it is much easier to write macros if they can do anything a function can do, i.e. if they act just like a function that returns an expression, albeit evaluated at parse time (or better yet, compile time). Restrictions that the macro body be pure may be hard to enforce; in my real situation, for example, the run was being invoked by a function called by a function called by a function called by the macro.

In my particular case of @pyimport, I can require the user to initialize Python before using the macro (so I can avoid this particular bug), but I can't avoid side effects entirely because the macro body needs to import a Python module (which has side effects within the Python interpreter, although they don't hang Julia) in order to figure out what symbols are in it before it can generate the wrapper expressions. (In the longer run, if #1974 is ever addressed, this issue will go away because I will be able to turn the macro into a regular runtime function call.)

[stevengj]

Update: I found a workaround for my @pyimport use case. The macro now generates a call to a function pywrap which generates and evaluates the expression at runtime (rather than parse-time). This way the @pyimport macro body is a pure function, which I agree is much nicer for many reasons.

Of course, then I immediately ran into another bug (#2386), but that bug I could work around with global variables.