Proposal: wildcard suffix for using statements

[daveaglick]

Namespaces work great for deconflicting types across module boundaries. However, in C# they’ve also been adopted as a de facto organizational construct. Library authors often put types under nested namespaces because it’s organizationally convenient during development, not because of any collision or scoping concerns, a practice which is further reinforced by Visual Studio’s default behavior of mingling folder structure with namespace hierarchy. While loosely related, namespace scoping and organizational structure are mostly orthogonal concerns in my opinion.

We can observe the challenge this presents on the producer side in extension methods. It’s often organizationally convenient to place them in sub-folders, but because they’re harder to discover and consume in a child namespace classes that provide extension methods are often placed in the top-most namespace unlike all other classes in that sub-folder. As we move towards “extension everything” I suspect this mixing of folders, namespaces, and extensions will only become more pronounced.

The practice of using namespaces for organization leads to confusion, or at least extra work, on the part of library consumers too. It’s not unusual to have to hunt for a specific type in a child namespace so that you can bring it into scope, or to rely on Visual Studio to add using statements for extension methods that you know should exist. There’s concrete evidence of this dichotomy in the way Visual Studio is currently adding features specifically to break out of the currently scoped namespaces to discover types. Often all a library consumer wants is semantics like “bring this library into scope.”

That said, the use of namespaces for organization still does have some advantages. For example, it’s a convenient way for automated tools like documentation generators to infer organization of code. While it may not cause collisions, flattening an entire library into a single namespace can introduce problems of it’s own.

This was a long-winded introduction to a language proposal for wildcard using statements. Specifically, I’d like to be able to represent “bring this library into scope” semantics with the following syntax:

using Something.*;

The behavior of this statement is simple. It brings the requested namespace and all known child namespaces into scope.

• This deals with the problem of namespaces and scoping vs. organization at the consumer, allowing them to flatten a namespace hierarchy if it’s appropriate to do so. For example, if the new syntax becomes conventional then it would allow extension definitions to stay in their appropriately nested namespace organizationally while still bringing them into scope in consumers.
• The wildcard can only go at the end of the using statement. This isn’t a proposal to introduce some sort of complex syntax or pattern matching. It would be sufficient to check only for a .* suffix.
• I'm aware other languages use similar syntax with different semantics (I.e., to bring all types in a given namespace into scope). Personally I don't see that as a concern, but I guess it's open for discussion.
• The same scoping rules as any using declaration apply and the inferred namespace(s) are only in scope for the file, block, etc.
• As far as I know this change is fully backwards compatible. The syntax is new and has no chance of impacting existing code and given that scoping is a compiler concern I don't think this has an impact on generated IL. It's opt-in as well so even if there are side effects, you have explicitly enable the functionality to see them.
• Using a wildcard does increase the possibility of type ambiguity since more types will be in scope. If via a wildcard you end up importing too much and create ambiguity, then it’s up to the consumer to go back to single namespace imports or deconflict via fully specifying the ambiguous type. I don't see this as a complication since it's no different than bringing an entire namespace tree into scope manually through multiple using statements.

[PathogenDavid]

I agree that the problem you're trying to solve is a real one, but I don't think this is the right solution.

I feel like we need better tooling around non-namespace organization, rather than ways to tolerate libraries that leak their internal structure through their namespaces.

They aren't without their own set of flaws, but I think even filters (as they work in Visual Studio with C++ projects) would be better than this.

[daveaglick]

I agree that the problem you're trying to solve is a real one

Thanks, at least I know it’s not just me :). This is a real problem that I currently encounter. As a library producer, I want my types to be easily discoverable by my users and the best way I’ve found to do that right now is by flattening the namespace, either wholly or in part (I.e., extension methods) which isn’t ideal organizationally. As a consumer I want to easily discover and use the types from libraries, and that’s not straightforward today in deeply nested organizationally-oriented namespace hierarchies without extra tooling (I.e., VS).

I feel like we need better tooling around non-namespace organization

I agree that the ideal solution would solve this at on the producer side and allow library authors to better indicate their organizational intent. However, any such approach would probably require new language constructs, tooling, and/or introduce compatibility concerns. This proposal has the benefit of being fully opt-in, requiring minimal language changes, and not impacting generated IL. As a 80% or 90% mitigation to the underlying problem it’s pretty decent and is much more likely to be championed and approved (though still not a given).

[yaakov-h]

As a consumer I want to easily discover and use the types from libraries, and that’s not straightforward today in deeply nested organizationally-oriented namespace hierarchies without extra tooling (I.e., VS).

It's not possible without tooling. The language only defines what's in scope. Intellisense and friends are provided by tooling... and VS includes an Object Explorer for this kind of purpose.

[daveaglick]

It's not possible without tooling.

True, but that tooling at least partly relies on the developer expressing their intent with regard to the subset of things it has access to. The disconnect is that many times a developer actually intends “make this whole library available” but doesn’t have the functionality to say that so they rely on extra tooling to bridge the gap (like the new VS Intellisense-from-outside-your-scope features, which themselves have been a bit controversial since they can rewrite your usings).

[HaloFour]

@daveaglick

As a library producer, I want my types to be easily discoverable by my users and the best way I’ve found to do that right now is by flattening the namespace

You can do this today. Why do you need the language to flatten the namespaces for you? What would the point of namespaces be at all? This would encourage developers to do using System.*; which not only adds a ton of ambiguity but puts a huge burden on the compiler due to the massive number of types that are forced into scope and similar burden on any developer trying to read that code.

[daveaglick]

This would encourage developers to do using System.*;

Would it? I certainly wouldn’t import such a large parent namespace, but maybe there’d be some education needed. Not to mention, if they really want to and understand the consequences (I.e., lots of fully-specified types) why not give the option?

More likely I’d use the feature to do something like using Newtonsoft.Json.*; and then happily use JObject and friends without having to think too hard about them being in Newtonsoft.Json.Linq.

Why do you need the language to flatten the namespaces for you?

That’s sort of the point - this proposal makes it easier for me not to flatten my namespaces as a library author because it allows my users to bring the whole library into scope or not depending on what they want.

[glennawatson]

I imagine it could have similar globbing rules to the dotnet extensions has for file systems. Obviously all the pattern matching wouldn't be necessary but the logic would be similar.

[HaloFour]

@daveaglick

Would it? I certainly wouldn’t import such a large parent namespace, but maybe there’d be some education needed. Not to mention, if they really want to and understand the consequences (I.e., lots of fully-specified types) why not give the option?

A feature that requires education specifically to avoid the developer following their natural tendency to "eliminate verbosity" sounds like a footgun to me. I'd expect that bringing in up to tens of thousands of types into scope would be quite a burden on the compiler and tooling. I'd expect Intellisense to become completely unusable.

More likely I’d use the feature to do something like using Newtonsoft.Json.*; and then happily use JObject and friends without having to think too hard about them being in Newtonsoft.Json.Linq.

And all of the extension methods that come along with? That would also make it much more difficult for Newtonsoft to add new implementations of some of those extension methods as adding them under a new namespace isn't safe for them to do anymore.

That’s sort of the point - this proposal makes it easier for me not to flatten my namespaces as a library author because it allows my users to bring the whole library into scope or not depending on what they want.

And my point is that if you expect the consumers of your library to be flattening the namespace then there's no reason for you to structure the types arbitrarily under namespaces. Those namespaces exist specifically to aid the consumer, not the author. And the language should be optimizing for the more common use cases, which would be the library consumer who will be reading that code in the future.

[daveaglick]

A feature that requires education specifically to avoid the developer following their natural tendency to "eliminate verbosity" sounds like a footgun to me.

On the scale of footguns in C#, this ranks pretty low IMO 😆. The behavior is easy to understand, it'll produce errors at compile-time if there are problems, and it's easy to back off and be more explicit about imports.

I'd expect that bringing in up to tens of thousands of types into scope would be quite a burden on the compiler and tooling. I'd expect Intellisense to become completely unusable.

This is probably a bigger concern to me. If importing a large namespace tree makes tooling unusable then we'll either need to figure out a way to prevent that (compile error if you import too big a tree?) or improve the tooling to support. I don't like the idea of exposing a language feature that could actually break the tooling. We need some metrics first to see how the compiler, VS, Intellisense, etc. handle large amounts of namespaces (shouldn't be too tough to get those numbers just by scripting out large namespace blocks at the top of existing code files).

And all of the extension methods that come along with?

Yes, exactly.

That would also make it much more difficult for Newtonsoft to add new implementations of some of those extension methods as adding them under a new namespace isn't safe for them to do anymore.

Maybe? Don't the same concerns already exist though? If I'm developing a library for public consumption, I'm going to be mindful of type name collisions across my own namespaces regardless because I don't know when someone is going to bring both into scope. IMO this doesn't introduce any extra burden on the part of library authors over what they should already be watching out for.

if you expect the consumers of your library to be flattening the namespace then there's no reason for you to structure the types arbitrarily under namespaces

Yes!. We're totally in agreement here, and in fact I've started doing just this: flattening the namespaces in my libraries. But this creates a bunch of problems on it's own - documentation tooling assumes namespaces == organizational hierarchy, VS assumes the same out of the box (or at least the default new class template does), some stuff like tests also keys of namespaces, and on the occasion where I want child namespaces to reduce scope of more specific functionality it becomes awkward with a mostly-flat-but-not-quite-everywhere convention.

[HaloFour]

@daveaglick

Maybe? Don't the same concerns already exist though? If I'm developing a library for public consumption, I'm going to be mindful of type name collisions across my own namespaces regardless because I don't know when someone is going to bring both into scope. IMO this doesn't introduce any extra burden on the part of library authors over what they should already be watching out for.

Probably. Given that namespaces exist specifically to prevent the collisions of names, and that this proposal seeks to largely dispose of the purpose of namespaces (that's how it'll be used), type collisions will probably become a real concern. The follow-up proposal would have to be to have wildcard namespaces with exclusions.

Sure, it would be nice to have better discoverability, but I feel that's the job of the tooling, not the language. Flooding the scope with more types won't help the tooling, it'll just give the developers considerably more noise in Intellisense. The fact that namespaces are limiting the type in scope is namespaces doing the job for which they were designed. I don't find it a burden to have to add more than one in order to work with a library.

[MgSam]

@HaloFour

You can do this today. Why do you need the language to flatten the namespaces for you? What would the point of namespaces be at all? This would encourage developers to do using System.*; which not only adds a ton of ambiguity but puts a huge burden on the compiler due to the massive number of types that are forced into scope and similar burden on any developer trying to read that code.

I don't know how the "did you mean to add this using?" features are implemented, but I'd guess they already essentially import every possible namespace in order to provide this analysis.

Also, using wildcards for importing/using is common in other languages.

[HaloFour]

@MgSam

I don't know how the "did you mean to add this using?" features are implemented, but I'd guess they already essentially import every possible namespace in order to provide this analysis.

I'd wager that's only invoked on fallback when no other token is found and that it wouldn't work well when applied to all of Intellisense. And I think that tool provides the discoverability that the OP is looking for without requiring to language changes at all.

Also, using wildcards for importing/using is common in other languages.

Which languages import recursively? I'm only familiar with languages where imports only bring in one level, e.g. Java, Scala, Kotlin, Python, EcmaScript.

[theunrepentantgeek]

While the original problem is a real one (that regularly causes me pain), I can see some real drawbacks to this behavior.

• The load on the compiler of bringing in all those unused namespaces would likely have a measurable (and human perceivable) impact on compilation times.

• Some heavily used libraries (such as NHibernate) use different namespaces as pay-for-play, so that you don't pull in optional features that you aren't using. This also serves to ensure you don't accidentally start using features just because they're in scope.

• Some libraries deliberately segregate vNext features in a different namespace (often .Experimental) as a way to solicit feedback from users with an appetite for change without affecting mainstream users who desire stability. Automagically pulling in such prerelease namespaces would no doubt cause issues for those users.

• Some libraries have implementation types with extremely common names (e.g. Result<T>, Option<T>, Parameter<T>, Logger) that are used across multiple assemblies (and are hence public) but which are not a part of the public API.

• There would inevitably be someone asking for using * ...

I'd suggest that a better solution would be to provide a feature for library publishers instead, though I'm not quite sure what shape that would take.

[daveaglick]

I'd suggest that a better solution would be to provide a feature for library publishers instead

I don't disagree. However, such a feature would almost certainly introduce new syntax, possibly modify IL to include additional metadata, etc. The more changes a language feature introduces, the longer the tail to get through approval and implementation, and the less likely the feature is to see daylight. IMO this is a workable middle-ground that solves the problem well enough and has a greater chance of success.

Some heavily used libraries (such as NHibernate) use different namespaces as pay-for-play...some libraries deliberately segregate vNext features in a different namespace...Some libraries have implementation types with extremely common names...

These (and other concerns in previous issues) all seem like variations on the same theme to me: "what if developers bring too much or the wrong things into scope?" I don't want to dismiss that concern, but in the interest of defending the proposal, I'm not sure it's a particularly strong argument against this feature.

Yes, the potential exists to shoot yourself in the foot by improperly importing namespace hierarchies that are designed to be used independently. Perhaps there's something to consider here around library authors somehow indicating a namespace is excluded from wildcard inclusion or hierarchies that can't be recursively included, but I don't think it's worth the effort and added complexity.

Developers have all sorts of ways they can make mistakes and screw things up. It comes with the job. As with any other language feature, the onus is on the user to understand the implications and use the feature as appropriate (given adequate guidance of course). Unless the implication of the feature is confusing or ambiguous, I'm not a fan of rejecting language features solely because they could be misused. And the implication of this feature is clear and easy to understand: it's equivalent to a bunch of explicit using statements - nothing more or less. This feature in particular also has a low risk profile because at least any poor usage is easy to detect (compile-time errors) and easy to back out (don't import that namespace with a wildcard).

[HaloFour]

@daveaglick

However, such a feature would almost certainly introduce new syntax, possibly modify IL to include additional metadata, etc.

Why couldn't it be accomplished via attributes? If the goal is to feed more information to tooling to aid in discoverability that would require no changes at all to the language or the runtime. I think that the debugger display attributes are a great example at how custom attributes and tooling can work together without requiring compiler support.

Developers have all sorts of ways they can make mistakes and screw things up.

Which is why language features should be catering to the wider audiences, that being the library consumers. As a library author you already have the power to improve the discoverability of your types. Any language feature has to be weighed against the potential of accidental misuse and the ramifications of doing so.

This feature in particular also has a low risk profile because at least any poor usage is easy to detect (compile-time errors)

Adding thresholds to the feature only complicates it. Who decides on that threshold and when does it apply? What options does the developer have and how can the compiler/analyzers offer to fix it? You couldn't approach converting that to individual using statements without completely evaluating all of the potentially imported types and extension methods. It sounds much more complicated to solve than adding more discoverability to auto-importing namespaces.

[glennawatson]

At the risk of further complicating the proposal, what about some sort of Attribute like syntax for ignoring the namespace as part of wildcard inclusions, eg for experimental or vnext based operations.

[HaloFour]

As an alternative I'd suggest an attribute that can be applied to types within a namespace that offers related types that an IDE can use to offer targeted suggestions:

namespace Newtonsoft.Json
{
    [RelatedTypes(typeof(Newtonsoft.Json.Linq.JObject))]
    public static class JsonConvert { ... }
}

So if you have already imported the Newtonsoft.Json namespace the IDE would offer JObject in Intellisense automatically, and if the developer chose it that namespace would then also be imported. The attribute could offer specific types or potentially an entire namespace by name:

namespace Newtonsoft.Json
{
    [RelatedNamespaces("Newtonsoft.Json.Linq.JObject")]
    public static class JsonConvert { ... }
}

The tooling could prioritize the offered suggestions based on whether the type has been only imported in scope or is actually being used in the current source file (arguments as to how this impacts muscle memory, but any changes to tooling here has that potential problem).

[glennawatson]

Interesting thought, that'd be a lot of extra metadata the API developer would have to be aware of. Eg every single object in your class you'd have to decorate in the often useful case.

[HaloFour]

@glennawatson

Interesting thought, that'd be a lot of extra metadata the API developer would have to be aware of. Eg every single object in your class you'd have to decorate in the often useful case.

That depends on how it's implemented. Consider what I suggested spaghetti thrown at the wall. Perhaps the tooling could consider any type currently in scope rather than any one specific type. Perhaps the attribute could be applied to the module/assembly and advertise all of the interesting types. There are numerous possibilities, and whatever it is could be pretty flexible and even controlled by configuration given that it wouldn't impact the compiler at all.