help Julia get further in bootstrapping this branch · JuliaLang/julia@e5afe78

vtjnash · 2014-08-16T03:52:40Z

this is mostly sufficient to get julia through bootstrapping, with some comments:

.1. because of some implementation details, defining the call method twice like this populates the method tables better (see below). this is necessary since start(Array{Any,1}) isn't defined until much later, but we can't define it earlier because it needs support for numbers, which is defined in int.jl, after this would fails):


julia> call(f::Function, args...; kws...) = f(args...; kws...)
call (generic function with 1 method)
julia> call(f::Function, args...; kws...) = f(args...; kws...)
call (generic function with 1 method)

julia> code_lowered(call,(Function,))
1-element Array{Any,1}:
 :($(Expr(:lambda, {:f,:(args::(top(apply_type))(Vararg,Any))}, {{symbol("#s105"),symbol("#s90")},{{:f,:Any,0},{:args,:Any,0},{symbol("#s105"),:Any,18},{symbol("#s90"),:Any,18}},{}}, :(begin $(Expr(:line, 1, :none, symbol("")))
        #s105 = (top(Array))(top(Any),0)
        #s90 = #s105
        return (top(apply))(__call#2__,(top(tuple))(#s90,f),args)
    end))))

julia> call(f::Function, args...) = f(args...)
call (generic function with 1 method)

julia> code_lowered(call,(Function,))
1-element Array{Any,1}:
 :($(Expr(:lambda, {:f,:(args::(top(apply_type))(Vararg,Any))}, {{},{{:f,:Any,0},{:args,:Any,0}},{}}, :(begin  # none, line 1:
        return (top(apply))(f,args)
    end))))

julia> code_lowered(getfield(current_module(),symbol("__call#1__")),(Array{Any,1},Function,))
1-element Array{Any,1}:
 :($(Expr(:lambda, {:kws,:f,:(args::(top(apply_type))(Vararg,Any))}, {{symbol("#s115"),symbol("#s112"),symbol("#s111"),symbol("#s110"),symbol("#s109"),symbol("#s107"),symbol("#s114"),symbol("#s106"),symbol("#s113"),symbol("#s108")},{{:kws,:Any,0},{:f,:Any,0},{:args,:Any,0},{symbol("#s115"),:Any,18},{symbol("#s112"),:Any,18},{symbol("#s111"),:Any,2},{symbol("#s110"),:Any,18},{symbol("#s109"),:Any,18},{symbol("#s107"),:Any,18},{symbol("#s114"),:Any,18},{symbol("#s106"),:Any,18},{symbol("#s113"),:Any,18},{symbol("#s108"),:Any,2}},{}}, :(begin $(Expr(:line, 1, :none, :call))
        #s115 = (top(Array))(top(Any),0)
        #s112 = kws
        #s111 = (top(start))(#s112)
        unless (top(!))((top(done))(#s112,#s111)) goto 1
        2: 
        #s110 = (top(next))(#s112,#s111)
        #s109 = (top(tupleref))(#s110,1)
        #s108 = (top(start))(#s109)
        #s107 = ((top(getfield))(top(Base),:indexed_next))(#s109,1,#s108)
        #s114 = (top(tupleref))(#s107,1)
        #s108 = (top(tupleref))(#s107,2)
        #s106 = ((top(getfield))(top(Base),:indexed_next))(#s109,2,#s108)
        #s113 = (top(tupleref))(#s106,1)
        #s108 = (top(tupleref))(#s106,2)
        #s111 = (top(tupleref))(#s110,2)
        (top(ccall))(:jl_cell_1d_push2,Void,(top(tuple))(Any,Any,Any),#s115,0,(top(typeassert))(#s114,top(Symbol)),0,#s113,0)
        3: 
        unless (top(!))((top(!))((top(done))(#s112,#s111))) goto 2
        1: 
        0: 
        unless (top(isempty))(#s115) goto 4
        return (top(apply))(f,args)
        4: 
        return (top(apply))(top(kwcall),(top(tuple))(f,0,#s115),args)
    end))))

.2. the system image can't be re-saved from an old image because generic functions aren't really allowed in Core (since it populates the method table with references to the old base module, and serialize in dump.c will not like that). maybe it is possible to drop any method in the method table call.env that reference the old module?

.3. dispatching call{T}(Type{T},x) to convert(T,x) fails to take into account that a type may be constructed by calling its constructor with multiple arguments (e.g., bootrapping fails with a no method error on call(Complex{Bool}, false, true)). not dispatching call to convert (as I have here) fails to take into account that some types can't be constructed by calling their constructors (e.g. Int(1) / call(Int,1) doesn't work here). having both defined is just ugly and inconsistent. perhaps you could just define const convert = call as follows:

diff --git a/base/base.jl b/base/base.jl
index 7e5bfc5..4d469a2 100644
--- a/base/base.jl
+++ b/base/base.jl
@@ -10,14 +10,15 @@ const NonTupleType = Union(DataType,UnionType,TypeConstructor)

 typealias Callable Union(Function,DataType)

-convert(T, x) = convert_default(T, x, convert)
-
-convert(::(), ::()) = ()
-convert(::Type{Tuple}, x::Tuple) = x
-
 # general definition for call (order is important for these two definitions)
 call(f::Callable, args...; kws...) = f(args...; kws...)
 call(f::Callable, args...) = f(args...)
+const convert = call
+
+convert(T::Type, x) = convert_default(T, x, convert)
+
+convert(::(), ::()) = ()
+convert(::Type{Tuple}, x::Tuple) = x

 argtail(x, rest...) = rest
 tupletail(x::Tuple) = argtail(x...)

However, this doesn't work since we want convert(String, "HI") to be a no-op, but String("HI") maybe not. This implies that we would do need to have a unique convert method that is used internally. I'm not sure how to handle deprecation, but it seems that the bitstypes constructors will need to be migrated over to extending call, not convert. then convert can be modified to try the default call constructor as a fallback, rather than the reverse.

cross ref: #8008

@stevengj

JeffBezanson · 2014-08-16T04:24:36Z

A good approach to the bootstrapping problem is to have Core.call and Base.call be separate functions. Code in Core uses Core.call, and everything else uses Base.call. Base.call can fall back to Core.call. This is the "layered" kind of separate compilation julia can handle.

I don't imagine convert(String,"HI") and String("HI") ever being different.

It's not correct for convert to fall back to call.

Constructors will be changed to add methods to call, so the call(T::Type, x) = convert(T,X) fallback should be neatly shadowed in those cases.

vtjnash · 2014-08-16T04:41:47Z

A good approach to the bootstrapping problem is to have Core.call and Base.call be separate functions. Code in Core uses Core.call, and everything else uses Base.call. Base.call can fall back to Core.call. This is the "layered" kind of separate compilation julia can handle.

Agreed, this is just also slightly complex because codegen then has multiple copies of call that it can call, and it needs to be able to pick the right one

I don't imagine convert(String,"HI") and String("HI") ever being different.

for String this is probably true, but for container types, it might not. The only example I can think of right now is in Gtk, where:
Frame(Frame()) should get you a Frame containing a Frame, whereas convert(Frame, Frame()) should not.

It's not correct for convert to fall back to call.

I think I am in agreement now, just had to think through it more.

Constructors will be changed to add methods to call, so the call(T::Type, x) = convert(T,X) fallback should be neatly shadowed in those cases.

taking this into account, I was trying to decide if a rename of convert would be helpful (and defining const convert = call) would help with the transition, and for writing code to work with both versions. I don't know.

JeffBezanson · 2014-08-16T16:35:39Z

The Frame example is perfect --- I don't know why it never occurred to me. For nested or stateful objects, constructing is clearly different from converting.

StefanKarpinski · 2014-08-16T18:49:56Z

Yes, that's a great example. Conflating constructing and converting has always struck me as fishy, but this really nails why.

JeffBezanson · 2014-08-16T20:06:06Z

This also reminds me we need to be careful: while Int(1.0) is a perfectly good way to convert 1.0 to 1, it's not at all clear whether Array([1]) converts the argument to an array (no-op) or makes a new array of arrays.

stevengj · 2014-08-18T17:11:46Z

dispatching call{T}(Type{T},x) to convert(T,x) fails to take into account that a type may be constructed by calling its constructor with multiple arguments

No, this isn't a problem, because the constructor will eventually add methods to call as Jeff mentioned. The point is that defining a convert method should always define a single-argument constructor, but the converse is not true. And if you explicitly define both a single-argument constructor and a convert method, the former will shadow the latter when called as a constructor.

stevengj · 2014-08-18T17:47:41Z

@vtjnash, when I try to make clean && make from this branch I still get an error:

...
graphics.jl
profile.jl
precompile.jl
*** This error is usually fixed by running 'make clean'. If the error persists, try 'make cleanall'. ***
make[1]: *** [/Users/stevenj/Code/julia/usr/lib/julia/sys.o] Error 1

Where is it going wrong?

vtjnash · 2014-08-20T04:03:02Z

The Frame example is perfect --- I don't know why it never occurred to me. For nested or stateful objects, constructing is clearly different from converting.

I could define that Frame(Frame()) is a no-op – and that nesting objects always requires a separate function call – such that constructing and converting are always equivalent, but that is just inconvenient in this case. I considered giving Array([1]) as an example, but since that is currently undefined, I decided not to use it.

This also reminds me we need to be careful: while Int(1.0) is a perfectly good way to convert 1.0 to 1, it's not at all clear whether Array([1]) converts the argument to an array (no-op) or makes a new array of arrays.

This also makes a decent counter-example to above. While convert(Array,1) seems to obviously return Int[1], Array(1) might return either Float64[] or Int[1], depending on which behavior you want to consider to be the base case.

Where is it going wrong?

@stevengj see my 2nd bullet point in my initial comment, and Jeff's response in the next comment

Original file line number	Diff line number	Diff line change
Expand Up		@@ -244,4 +244,4 @@ typealias ByteString Union(ASCIIString,UTF8String)

		include(fname::ByteString) = ccall(:jl_load_, Any, (Any,), fname)

		call(f::Function, args...; kws...) = f(args...; kws...)
		call() = nothing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

9 comments on commit `e5afe78`

vtjnash commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

vtjnash commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

StefanKarpinski commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

stevengj commented on `e5afe78` Aug 18, 2014

stevengj commented on `e5afe78` Aug 18, 2014

vtjnash commented on `e5afe78` Aug 20, 2014

Commit

There are no files selected for viewing

9 comments on commit e5afe78

vtjnash commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

JeffBezanson commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

vtjnash commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

JeffBezanson commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

StefanKarpinski commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

JeffBezanson commented on e5afe78 Aug 16, 2014

Choose a reason for hiding this comment

stevengj commented on e5afe78 Aug 18, 2014

Choose a reason for hiding this comment

stevengj commented on e5afe78 Aug 18, 2014

Choose a reason for hiding this comment

vtjnash commented on e5afe78 Aug 20, 2014

Choose a reason for hiding this comment

9 comments on commit `e5afe78`

vtjnash commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

vtjnash commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

StefanKarpinski commented on `e5afe78` Aug 16, 2014

JeffBezanson commented on `e5afe78` Aug 16, 2014

stevengj commented on `e5afe78` Aug 18, 2014

stevengj commented on `e5afe78` Aug 18, 2014

vtjnash commented on `e5afe78` Aug 20, 2014