RFC: marking private and public APIs

[Roger-luo]

This is based on #42117 but is a bit different proposal that might be breaking for Julia 1.x based on different considerations thus I decided to open a separate issue to make reading a bit easier.

In short, I'd like a similar macro/keyword that marks an object:

• public, accessible via <module>.<name> or <name> after explicitly imported by import/using
• unmarked names are considered private, which means one cannot access them at all from outside

the syntax may vary due to compatibility concerns, e.g

Keep things non-breaking

because we currently do not hide things by default, we will need a marker for things we want to make private

@public <something>
@private <something>

Breaking but simpler

if we choose something more breaking, it could be a public/export keyword combined with #39235, where export/public could just mean "public" accessible APIs, and things are not accessible from the outside module directly.

On the other hand, using XXX and import XXX needs to be private by default, unless marked with export/public so we can prevent someone using a long module path to access some deep dependencies from a package (see the point 2 in Why).

Additionally

I'd like a macro that marks certain API's stability status, this is something I find quite nice from the rust community, where they have things like

#[cfg_attr(not(test), rustc_diagnostic_item = "IpAddr")]
#[stable(feature = "ip_addr", since = "1.7.0")]
#[derive(Copy, Clone, Eq, PartialEq, Hash, PartialOrd, Ord)]
pub enum IpAddr {
    /// An IPv4 address.
    #[stable(feature = "ip_addr", since = "1.7.0")]
    V4(#[stable(feature = "ip_addr", since = "1.7.0")] Ipv4Addr),
    /// An IPv6 address.
    #[stable(feature = "ip_addr", since = "1.7.0")]
    V6(#[stable(feature = "ip_addr", since = "1.7.0")] Ipv6Addr),
}

It marks certain things' stability at the same place where it's defined programmatically so that a linter can warn users based on their current toolchain version.

In Julia, we only have a poor docstring saying "use at your own risk", which is something I think could be improved by this. Having this shouldn't break, it could be one macro marks struct fields and function arguments about their availability and stability. This could make the experimental feature easier to provide and play with in downstream.

Why?

#42117 has overlapped with this proposal thus the reasons @DilumAluthge listed also apply here, I'd like to provide a few other motivations tho

1. potential reduction on pkgimage/cache/sysimg size, because many functions in a package are used by the package rather than the downstream user, the methods corresponding to these functions that are not used by downstream can be deleted in the downstream package cache/sysimg in principle, but we currently cannot do it because users are allowed access them by a deep chain of module path (e.g A.B.C.<a private function>. I think quite a few AOT languages have a similar mechanism to mark things private so they can be tree-shake away. I'm not an expert on package cache or system images, but I think this might be one of the low-hanging fruit that can be improved by changing the semantics a bit, so please correct me if this is not what can be improved in Julia's case.

A demonstration of this can be resolving the issue that https://expronicon.rogerluo.dev/intro/bootstrap and https://github.com/thautwarm/DevOnly.jl trying to solve automatically:

MLStyle is an extreme example of this, what @match generates only depends on Base, but the only reason why downstream packages will still load MLStyle is only that users are allowed to access MLStyle's @match via AAA.BBB.CCC.@match if CCC contains using MLStyle: @match, and one can get a rough estimation in this extreme case on loading time improvements

julia> @time using MLStyle
  0.041998 seconds (148.07 k allocations: 10.553 MiB)

julia> @time using ExproniconLite
  0.020682 seconds (53.83 k allocations: 3.991 MiB)

here you can see even Expronicon depends on MLStyle, by removing MLStyle from loading entirely we can get twice faster loading time than depending on MLStyle.

2. prevent downstream users hacking unstable things, this is more or less an effect of having 1, but I want to argue it has certain advantages, one example is the non-public APIs from Base, we currently have a very vague way of distinguishing them by whether the function has a docstring or not, IMO even functions only made for developers deserves a docstring in the dev docs. If we have a mark to distinguish such APIs, then the reference page can be generated automatically for manual and dev docs of the corresponding functions. And APIs like Base.print_matrix etc. can be more clear to people that whether they should use and whether this is maybe broken in future versions.

But I'd like to mention one real-world example which is the usage of DiffEq, most of the time one only uses one ODE solver from that giant package, thus in principle downstream package should not be loading the whole thing, but that ODE solver code only. But because currently you are allowed to access other solvers by something like MyPackage.DiffEq.OrdinaryDiffEq.Vern8 we will have to load the whole thing which is super slow.

Ideally, the compiler should cache the corresponding solver code into the downstream package image and only load that piece when using MyPackage without explicit using DiffEq. But I think this is not allowed because users can technically do MyPackage.DiffEq to access anything inside DiffEq

[Seelengrab]

I don't think preventing people from accessing a given function entirely via some form of @private is a good idea, but I do think that we need some means of explicitly marking an unexported thing as part of the SemVer API guarantees.

But I'd like to mention one real-world example which is the usage of DiffEq, most of the time one only uses one ODE solver from that giant package, thus in principle downstream package should not be loading the whole thing, but that ODE solver code only.

I also don't think such an @private macro would help with the woes of all of DiffEq needing to be loaded, because there's no definite line that can be drawn for where that boundary of accessible/non-accessible needs to be drawn in general (I've hacked around too many too restrictive private modifiers in java for my share of problems..). As far as I'm aware, the trend for these has also been going more towards allowing more extension from the outside, rather than disallowing more (such as favoring protected over private). Do you have an example of a language that uses the access modifiers for tree-shaking purposes?

In addition to this, I think having a way to mark objects public/private will also resolve the issue

In regards to smaller package images - in an AOT setting, where a static binary without compilation is the goal, you don't need an additional layer of @private to delete things. Just delete things that aren't reachable from a given (list of) static entry point(s) (which you need either way, else you can't start the binary/access the library). Currently, I think that would be exported names, but extending that with an @public list of unexported names should be sufficient, since there's no new code that may be generated at runtime that may access new names.

I don't see how @private helps with that in a non-AOT setting, since the presence of eval (or any macro, really) may mean that new functions accessing internals can exist, without necessarily being visible to static analysis (which is required for removing unused things from a package image).

---

I really like having an @public (or @api?) marker for marking an unexported thing explicitly as SemVer API. This should be queryable in code, to allow doc systems like the help-mode in the REPL or Documenter.jl to automatically pick it up & display the warning (possibly with a version of when it was added, which can generate a fitting admonition in the docstring automatically).

---

As a point of reference (but please don't feel the need to post), there's also this lengthy discourse thread about this very topic, with lots of good arguments for & against various versions of this (including my quite strong stance about why I personally think a private modifier is a bad idea).

[Roger-luo]

@Seelengrab I probably should emphasize on the improvement of code loading times more, for the API definitions part I think it should be in the scope of #42117, I think the comments on private totally make sense in the discourse post, I wasn't being clear that the above-proposed syntax is only made for compatibility reasons, and there could be a better way of making things inaccessible. However, I don't think there has been anyone discussed the impact of loading in discourse posts or issues before (unless I searched the wrong keyword).

In regards to smaller package images - in an AOT setting, where a static binary without compilation is the goal, you don't need an additional layer of @Private to delete things. Just delete things that aren't reachable from a given (list of) static entry point(s) (which you need either way, else you can't start the binary/access the library).

Yes, that's exactly what I'm trying to propose here actually, a more "static" marker, the proposed public/private marker is mainly for compatibility, if we break the semantics of export we can totally do what you described here.

As far as I'm aware, the trend for these has also been going more towards allowing more extension from the outside, rather than disallowing more (such as favoring protected over private). Do you have an example of a language that uses access modifiers for tree-shaking purposes?

I don't have an example right now, since I thought this was obvious. But I could be wrong on this. But let me explain two things here:

• the proposed private marker is not necessary if we let things not exported inaccessible by default, I propose that mainly for compatibility reasons
• for a Julia package A that depends on B.foo, and A does not re-export B, e.g

module A
# package A

using B: foo
end # module A

B.foo will be an entry for package B, and if B is not accessible from outside A, it will be safe to delete everything unrelated to foo and only save the definition of foo in the compiled cache of A? Because no one can access anything else from B except A.foo.

But eval is a good point, I didn't think about it, but I think in this case A.eval(:(foo)) will still work even if other function definitions in B are not loaded? and to make sure something is accessible you can always do

A.eval(quote
   using B: goo
   goo
end)

and by calling the loader at runtime throw eval will invalidate A's cache and load the whole cache of B in this case.

Basically, here we are treating exported or public APIs as the static entry point, and removing things not reachable, which is why we need to distinguish what is accessible and what is not. The simplest solution is treat exported APIs as static entry points, and do not treat other APIs as an entry points, but that would be breaking because all other APIs can be accessed via the dot operator.

This is basically why I'm saying if we want this in 1.x, we will need something like a private, but

1. we potentially combine this with 2.0: just one import keyword #39235 and break the convention of export, no extra keyword is needed,

2. or if we break the convention that everything is accessible from a module using a dot, then only a public marker is sufficient.

I don't see how @Private helps with that in a non-AOT setting, since the presence of eval (or any macro, really) may mean that new functions accessing internals can exist, without necessarily being visible to static analysis (which is required for removing unused things from a package image).

I don't think you have access to all the internals even for the current module loading system? if the only module you have access to above is A? You cannot access to A.B.goo without importing B to the scope like using B: B.

And in my proposal, if the loading statement is written as public using B: B, goo then you will have access to B and we should load B entirely as well, otherwise, because this is not an entry point for A anymore, we can just delete them.

---

@Seelengrab also maybe you didn't see the example I provided for the MLStyle case above, which is an extreme case because it doesn't have any dependencies, if we make MLStyle inaccessible from outside, we can safely not load MLStyle which will have an improvement on loading (the ExproniconLite package, which I created using a hacky way). This is a bit hard to demonstrate using a macro, so I'm not able to provide something more complicated, but I hope this demonstrates the idea.

[Roger-luo]

As far as I'm aware, the trend for these has also been going more towards allowing more extension from the outside, rather than disallowing more (such as favoring protected over private).

I guess extensions are not exactly the same as this - extensions are lazily loaded, but here it's about reducing downstream package loading time by reducing unnecessary cache. The effect is similar - much faster loading time, but both should be used in practice to reduce loading time further.

On the other hand, I'd doubt if converting say every ODE solver into an extension makes sense or practical... mark things inaccessible in downstream packages could be easier to adapt.

[Seelengrab]

Because no one can access anything else from B except A.foo.

I don't think you have access to all the internals even for the current module loading system? if the only module you have access to above is A? You cannot access to A.B.goo without importing B to the scope like using B: B.

That is not true:

julia> module B
           foo() = "hi"
           bar() = "bar"
       end
Main.B

julia> module A
       # package A

       using ..B: foo
       end # module A
Main.A

julia> (A.foo |> parentmodule).bar()
"bar"

So unless we break the notion of accessibility of unexported names, you can currently get everything in a dependency via reflection on anything that module gave us. If you want to get this into a 1.x release, breaking here does not look like an option because there's no safe state to fall back to in general here - we don't actually track the API surface of anything in a programmatic way right now, and "unexported" is not an accurate proxy variable here. To me that means the only option left is to actually make it queryable what is API, by placing explicit @api markers (and export, if the name needs to be exported).

Basically, here we are treating exported or public APIs as the static entry point, and removing things not reachable, which is why we need to distinguish what is accessible and what is not.

Yes, but that is not sufficient. What is accessible vs. what is API are orthogonal concepts, and the current (unspoken & unqueryable) convention is that unexported names are (usually) not API, and documented names are (usually) API. It's this fuzziness around what exactly constitutes the API of a package that is the problem, not whether you can access the given name at all. Regardless of solving that, using any sort of API marker as a boundary for tree shaking is bound to lead to problems because that's not the job of an API marker - their job is to inform a developer/user about what is considered safe/stable to access across versions, and accessing anything else does not have any guarantees associated with it.

I guess extensions are not exactly the same as this

On the other hand, I'd doubt if converting say every ODE solver into an extension makes sense or practical

To be very clear here - with "extension" I was not referring to the weak dependencies/package extensions we now have on master, but referring to the practice of extending the interface of a library in java with a new type/object. Unless that library has all of its boundary marked as protected or public, new code implementing e.g. an abstract class cannot change the behavior of some things (which there are very legitimate reasons to want to do).

[Roger-luo]

That is not true:

you are mixing up modules ported to Main and modules defined in Base.__toplevel__ module, they are not exactly the same: for packages, they are evaluated in a hidden Base.__toplevel__ which is why if B is a package, and using B: foo will not port B into the scope.

I didn't see the parentmodule part, yeah, I think you are right then, if we want to enable tree-shaking a package, we need to break the convention of what is accessible from a package then. Perhaps by making packages more special.

---

To me that means the only option left is to actually make it queryable what is API, by placing explicit @api markers (and export, if the name needs to be exported).

Yeah, or that, I'm open to anything that marks an API, a publicly accessible name, or whatever we name it. But I'm just saying this can be helpful in reducing loading time if we make things inaccessible.

Regardless of solving that, using any sort of API marker as a boundary for tree shaking is bound to lead to problems because that's not the job of an API marker - their job is to inform a developer/user about what is considered safe/stable to access across versions, and accessing anything else does not have any guarantees associated with it.

OK, you get me confused now, are you proposing we treat the @api marked object as a static entry, and delete unreachable? Or what you are trying to propose in terms of tree-shaking unused definition from a "package" (not a module inside Main)?

[BioTurboNick]

1. prevent downstream users hacking unstable things, one example is the non-public APIs from Base, we currently have a very vague way of distinguishing them by whether the function has a docstring or not, IMO even functions only made for developers deserves a docstring in the dev docs. If we have a mark to distinguish such APIs, then the reference page can be generated automatically for manual and dev docs of the corresponding functions. And APIs like Base.print_matrix etc. can be more clear to people that whether they should use and whether this is maybe broken in future versions.

I'm against hard barriers to access. Hacking "private" methods is honestly a very useful feature of Julia. If you want there to a little extra friction for access to "private" methods, that would be a nice addition: make the consuming code self-document that it's using a "private" method. There would have to be a very compelling reason to strip it entirely, IMO. Which leaves...

2. potential reduction on pkgimage/cache/sysimg size[...]

Which I don't know enough about to comment on the merits.

[Roger-luo]

If you want there to a little extra friction for access to "private" methods, that would be a nice addition: make the consuming code self-document that it's using a "private" method. There would have to be a very compelling reason to strip it entirely, IMO. Which leaves...

I kinda regret I put this as my first point now... so basically, if we want to delete these methods later, we need a way to prevent downstream users from accessing them, otherwise, we cannot delete them. I think this makes a lot of sense for an AOT language, but it's tricky for a JIT language... as people's expectation is different

I kinda understand why you are against hard barriers to access as a developer, I think if we can trade being able to hack private methods with faster loading time I'd choose later, but let me change my post a bit to be more clear on this...

[Seelengrab]

To be clear, I think the claim that a public/private or accessible/inaccessible distinction can improve load times is dubious - at best, you can delete code that isn't API and isn't used by the package internally, at which point, why is that code in the package in the first place? The load times are not a result of functions that noone calls after all, but of all those that are called (one way or another).

[Seelengrab]

weakdeps/Requires give us the support of a), which can be done when the package is very modular itself, but we don't have b), and not having b) will make things transit to a) a lot harder.

I don't follow why a distinction between public/private makes the transition to sub-modules/-packages per Solver easier? It's a change in workflow - add the Solver package you want to use, add the DiffEq package and you're done. Or are you suggesting that you want to keep the big, central DiffEq package and have julia figure out on its own what a user might need? (again, that's not really possible at package install time)

Thus I don't see why this is dubious?

I'm not saying that the usecase is dubious, I'm saying that I don't think having public/private API markers makes loading DiffEq faster. The reason for that is that, a priori, the package loading mechanism doesn't know which part of DiffEq a user actually wants to use when installing/adding DiffEq and such a public/private distinction from the POV of DiffEq & the package loading mechanism doesn't inform the compiler about which of the numerous functions that are available end up being used once the code runs.

Loading stuff into RAM is not the slow part of loading a package - it's all the compilation & invalidations that happen that are the slow part (which you can't speed up with a public/private distinction).

To be clear, this does not improve the load time of a package like DiffEq but improves the load time of a package depending on a framework package like DiffEq/MLStyle/...

IF (and that's a big if, I'm not a DiffEq dev after all) DiffEq moves to a weakdeps approach, depending on DiffEq should get faster just from not loading all the stuff it doesn't need anymore, because it then needs to explicitly load the solver it cares about. Again, that's a difference in how the DiffEq API (and the solvers) are presented to the user and is not something that julia can do for you.

But we have to do this by writing a bunch of code manually to expand the macro, which is way less stable if the compiler can figure out that MLStyle is not needed after precompile, and thus not loading it. But it cannot do this because we need MLStyle module to be accessible via Expronicon.MLStyle.

That sounds more like a request for an explicit "build this package" step than a need for a public/private distinction, because your use of the macros in Expronicon/MLStyle happen (semantically at least) when your code is called/parsed. I also don't follow how public/private would help you with MLStyle, since presumably the main @match macro would be marked public, and hence everything it calls internally would need to be shipped anyway (irrespective of this shipping/loading not being the slow part of MLStyle, as I understand it).

and as a more realistic and worse case in our package Bloqade.jl, we only need a few non-stiff ODE solvers, but now we have to load all the ODE solvers every time our users load Bloqade. Making DiffEq/OrdinaryDiffEq as a weakdeps will not full resolve this problem (because it will still be loaded if user need to load that one specific solver), what would your solution be if it's not this?

I'm not sure I follow - what do you mean with weakdeps not solving that issue? If the various solvers in DiffEq are weakdeps, it is Bloqade that adds the solver package explicitly as a dependency, not DiffEq. You don't end up loading solvers you don't explicitly depend on.

[Roger-luo]

It seems like the case of needing something during precompilation and not during normal execution may suggest a need for precompilation-specific environments to allow something to be loaded during precompilation but otherwise not available?

Kinda, yes, I think limiting accessibility in a precompile/AOT context makes more sense, but this only works for macros, not normal functions? Unless there is something like a module saying this module is special and will strip out things during precompile?

Basically, the effect should be equivalent to copy-paste relevant code from a large dependency.

I think because Julia is dynamic, and this is mostly an AOT feature, so it doesn't have to be exactly the same as an AOT semantic. but now you guys should get what I want here (finally...)

If all the solvers are part of the public API of DiffEq, they'd all have to be loaded even if they're not part of your package's public API.

I guess I probably don't get this part, if my users are disallowed to access DiffEq via my package, how do they know they can use DiffEq through my package? Then we can treat this as an optimization and not load the full DiffEq? It won't change the program's behavior strictly.

Or what's the reason that we still need to load the full DiffEq public API in this case?

[Roger-luo]

IF (and that's a big if, I'm not a DiffEq dev after all) DiffEq moves to a weakdeps approach, depending on DiffEq should get faster just from not loading all the stuff it doesn't need anymore, because it then needs to explicitly load the solver it cares about. Again, that's a difference in how the DiffEq API (and the solvers) are presented to the user and is not something that julia can do for you.

Yes, that's something I'm trying to approach in the current release version. But I'm looking for something more automatic. I'm not asking Julia to do this at package loading, but instead asking for a static compilation stage (thanks @BioTurboNick for pointing out this), that happens either during precompile or pre-registration (that'd be the best, but harder). So that when we load the package, there will be less work to compile (because of fewer functionalities are included without user noticing).

I'm not sure I follow - what do you mean with weakdeps not solving that issue? If the various solvers in DiffEq are weakdeps, it is Bloqade that adds the solver package explicitly as a dependency, not DiffEq. You don't end up loading solvers you don't explicitly depend on.

I mean if currently directly putting DiffEq as a weakdeps won't solve this issue because it's still a giant dependency, but (a big) if someone pay a lot of time splitting that package into many small ones, then yes, even without weakdep, loading time will be improved.

But as you say, this is a big IF, because there will be a lot of extra boring work to make it happen (I believe), and I think can in principle be done automatically by looking at the function call tree from the API get called in upstream? And DiffEq is not the only giant package in the ecosystem... I understand splitting things into smaller packages could be a good thing, but giant packages will just keep appearing because they are collections of tools and will keep accumulating new functionalities and people are lazy, which is why I think if the compiler can help with this things will be much easier.

[BioTurboNick]

In order for Julia to load your package, it first has to load its dependencies. And when it loads the dependency, it has no idea what you need. So you're always paying the cost of loading the full dependency.

But once it's loaded, you say you want to remove from memory those parts of DiffEq your package doesn't need. How does Julia (currently) know that no more methods in DiffEq will need compilation during runtime? What if a user has a new type they want to pass into your package, and rely on Julia's dispatch for it to just work?

[Roger-luo]

Thanks, @BioTurboNick's explanation, yeah I agree that this shouldn't happen at code loading time, but precompile time only. Then it makes sense to not have things inaccessible but instead have something marking a piece of code that happens only at precompile time.

What if a user has a new type they want to pass into your package, and rely on Julia's dispatch for it to just work?

I think even if we delete things because we are not deleting relevant code, but only irrelevant Julia's definitions, the methods in the API call tree are still loaded. Thus new methods can still be generated, it's just we do not need to precompile other unrelated things at loading anymore.

E.g a way to demonstrate this we copy and paste the definition of Vern8 solver and functions in its call tree from OrdinaryDiffEq to my own package, loading this package will not require us to load DiffEq, but require a separate build stage, thus precompile won't happen for other solvers. And if you put a Dual number, it will still generate new methods specialized to Dual number.

I think this is basically just equivalent to allowing the compiler to copy-paste relevant code from the dependency to your own package's cache at a build time by guaranteeing that unused APIs from upstream will not be used by your users.

But considering this needs to happen before precompile (because precompile happens after loading I assume?), the above proposal probably needs to be revised now, I'll probably come back with something else considering this.

[BioTurboNick]

From my comment on Slack: a symbol followed by a question mark without a space foo? is available syntax (currently errors) as an alternative to foo. for "what can I do with this" lookup.

A motivation for this would be if we want to avoid the association with OOP-style lookup, making people think they could actually write a function call with that form.

[timholy]

Closing now that we have public