Request for feedback: Discontinuing support for dart:mirrors #44489
Comments
mit-mit
added
the
area-meta
|
To make sure the info gets to you, here is my response to dart-misc mailing list: I might be reading too much into this but discontinuing If anything I'd request to add mirrors to AOT snapshots too, instead of discontinuing, so that we'd get to deploy our code as a single binary with all the benefits and drawbacks it means. But... Let me address the concerns I have:
I'm sure you have a list of pros and cons for mirrors vs. code generation too. Having said that, I'm not against the idea completely if there are good enough reasons. There have been several precedents where such a change was preceded by preparing users, crafting new tools or giving some extra attention to alternatives. Most recent one is the NNBD tooling to this. I hope it is the case here as well, in which case feedback on why I'm nervous about moving to My recent incursions to writing builders were ... trying because:
I've had some steep learning curves over the years. I hope that this is the kind of feedback you were looking for to help you make the right decisions. |
|
To those downvoting, it would be great if you could also post a comment with what concerns you. As mentioned, we'd like to understand the use cases where you use mirrors today, so that we can investigate if there is an alternate way by which we can support those use cases without mirrors. |
|
I work with @daniel-v and @gothamgasworks - I can second Daniel's input that removing runtime reflection would impact us quite negatively. Our product actively uses runtime reflection in server-side Dart code, both in production and in development, and our preference would be to retain the ability to introspect code rather than rely on build-time code generation, both in the interest of development cycle times and ease of understanding of deployed code. |
|
@daniel-v 's summary is pretty good, I agree with them. Some things I want to emphasize:
|
|
We use dart:mirrors in:
Of course we could work around it with builders, I would just like to be able to not work with the builder/analyzer API, which requires a serious amount of maintenance as opposed to the mirrors API. |
|
The use cases I had in the past month:
My wish-list, which would probably make the lack of
Summary: To replace
|
|
The Dart Analyzer team uses |
|
My use case of reflection is docs generation via iterating over class members and invoking them to get sample outputs. The code using |
|
I have mentioned this elsewhere but worth mentioning here for posterity - build_runner uses dart:mirrors in order to discover all of the libraries that exist in a program. In our case that is the build script itself, and we use that to know if we need to invalidate the snapshot. |
|
I think that a new framework for reflection should exists. For now we should kill dart:mirrors. In the future, after NNBD, a new framework should exists with another name. One where you can mark classes that should have their full description after compilation (the main information needed for runtime reflection). The marked classes could be by annotation or passing a list of classes for the "refletor factory" that should be tracked, allowing any class to be tracked for reflection. With that approach,Dart tree shaking won't be affected, unless you really need a reflected class for your project. |
|
It's worth noting that @isoos's proposed solution would likely not be sufficient for Grinder, since it's infeasible to have a code generation step as part of a task runner workflow. |
|
This (a) doesn't solve all the problems with dart:mirrors, and (b) would only cater to specific use cases, but leaving the idea here nonetheless... what if we were to still tree-shake as normal even in the presence of dart:mirrors, and rely on developers marking methods as |
|
My use case is... well, hold on just a dang minute. I don't think I should have to lay out my thinking process to express the fact that I value the presence of
So allocate the dev resources to get on that. "Significantly behind" is a prioritization issue. This is plainly technical debt. Defenestrating a sub-section of your server-side users building ORMs (oops, there's my use case..) would be an easy way out, and clearly a cheaper path forward. But seriously, cheaper over better? Not very Googley, imho. Do the right thing. Strive for excellence (and make the one thing you're doing really well be creating a language that can be used everywhere, including places where reflection makes sense). Keep an eye on the goals (both short and long term). Be proactive (and bring dart:mirrors up to par). Go the extra mile... Well, you know all this. You're the ones that work there. I've been one of the earliest, most enthusiastic, and staunchest allies of both Dart and Flutter, but the bloom is off the rose with several prioritization choices that make me just, well, nervous. I'm tired of building things in AngularDart, or Polymer.dart (oy..), and then getting downstream just enough to feel the aging, unfinished edges and have to admit to myself that it's not a viable path forward. I have had to abandon chunks of my work because of hitting these dead ends and faded photos of what was planned. It's turning my attention to other places, and mine has teh sad. It took this thread for me to even glance back at AngularDart and see that it received an update in October of this year. Yay! But.. I can't trust it; already tried that, and got left behind. TL;DR: this decision will impact the trust of your user base regarding your team's intentions in general; trust that is already flying on one engine. If we smell even a whiff of "Flutter doesn't need it so it's getting de-funded", even the features you do nurture will be untrusted for any other use case, and weaken your entire value proposition. Many of us, even your biggest fans, will simply walk. Not a rage-quit, but a pragmatic step out of what would appear to be a populism-driven-dev world and into other environments that focus on remaining well-rounded. Over time. For everyone. The End |
Not without a replacement being widely available and adopted we shouldn't! Evidently, If the team moves forward with deprecating and removing this, it should only be done after they provided a replacement framework/solution, and the majority of the ecosystem, or at least its core components, have transitioned to the new way of doing things. |
|
The more I think about code generation versus reflection/introspection, the more convinced I am that codegen as a whole, regardless of language, is a very double-edged sword (and that's leaving aside the fact that most generated code is fugly, but the saying is usually that you don't need to read it anyway - whether that's true is another thing entirely). Either you persist the generated code, in which case you get the benefit of short build times, but you trade storage for it (you're persisting essentially redundant code) and you risk the generated code drifting from the handwritten code it's supposed to work with; or you ignore it (as in Either way I slice it, I don't really see upsides to code generation as opposed to runtime reflection/introspection. |
|
I think original post by @mit-mit was a bit unclear: we have no plans to deprecate and remove For me an ideal outcome would be that we end up staffing an effort to build a solution that covers Another possible outcome is that we just invest some effort in bringing Figuring out where we go requires feedback from current users of the library, so thanks to everybody who has provided it. |
I would very much like that, yes. |
|
Thanks for all the feedback so far! Let me elaborate a bit, and answer a few questions: @mraleph wrote:
Agree, the goal of this feedback issue is for us to better understand a) what critical uses mirrors have today, and b) based on those use cases, if we could design an alternate mechanism that could satisfy those same uses, but potentially be better aligned with our compiler goals. @daniel-v wrote:
Correct, we don't have such a proposal yet. As I wrote above, we're hoping to learn how mirrors are being used today, and from that we'll potentially design an alternate mechanism, which we could then share for feedback. |
@mit-mit Why ? Are you implying that if reflection is used, code that has nothing to do with what it was used with can't be tree shaken ? If I use reflectClass(SomeClass) , my whole codebase can still be tree shaked, minus that specific class. Alternatively, you could still tree shake that class and the reflection happens on the tree shaked class (assuming properties are tree shaked). |
Not with the current And actually, its worse than that because you can also use the This is just a part of the problem with the current |
|
To expand on what @scheglov mentioned about test_reflective_loader: it uses a small piece of mirrors, but there is not another (codegen-free) solution: The system is fed test classes with |
|
I use mirrors for ORM, JSON serialization, documentation.. Reflection is a really big thing and should be a core part of Dart. Also it should be able to use in dart AOT. Build_runner is not an option. If mirror is "experimental" build_runner is still a "thought" on production level. Too complicated to use and too low level to have actual value for programmers not interested in how the analyzer works. I mean, "I just want to get the properties' names and types, not learn how the dart analyzer works!". Also, it lacks documentation and greatly increases compile time. I don't know how complicated the relation tree shaking-reflection is, and woudl love to read more about it, but since reflection could potentially access parts of code that was shaken off, it could have a property entitled "isShaken", so that devs can expect a specific element to exist but was shaken off, and trying to access it would raise an exception. To avoid exceptions, another method for reflection could be implemented like "reflectSafe()" which doesn't reference any part of dead code at all. I don't like the idea to annotate members to be able to reflect, because it seems a downgrade for the current lib, but it's better than abandon reflection altogether. |
|
Just as a thought: With the advances in the Dart ecosystem on the Flutter side I think it is save to say that Dart will also have more adoption on the server side. |
|
@mit-mit Is there any news about this topic? |
|
Thanks for all the feedback so far, it's very helpful! We're currently doing some very initial brainstorming and exploration of some potential features in the meta-programming space. We speculate that they might support the kinds of use cases we're seeing in this thread, but it's too soon to tell with any certainty yet. We're going to explore that a bit more, and then share. For now, we're not making any decisions about mirrors until we understand the space a bit better. |
|
I want to add my voice to the rest of the voices, my use case is to load modules dynamically and use the functions they provide. As many people have said, marking classes that should support reflection is great way to solve many problems, and having a flag to enable reflection on the whole project when the value of mirrors outweigh the costs. |
|
I think that the biggest problem with reflection is the collateral effect with the tree shaking. To minimize the problem, I think that the Reflector (the class that will use reflection, that will use use a Class X by reflection), should be the one that defines the classes that will be reflected (accessed by reflection), and not the opposite. I saw in the past some frameworks where the class that will be reflected is self marked (usually with an annotation in the class). Actually reflection should exists to access any class, in the project or in other packages! This is the main mistake of the reflection frameworks in Dart, the miss conception that a reflected class should be marked in the class sources code itself. If the reflector is the one that defines, statically, the classes to reflect, the side effect in the tree shaking will exists only in the marked classes for reflection. A well defined interaction of reflection and tree shaking is the most important base for the whole framework, but no one is discussing this. |
|
@gmpassos I very much agree with you :). I have even hand coded something to mimic a theoretical "const reflection" proposal in this package https://github.com/jakemac53/static_reflection (sorry for lack of readme etc there). You can see that it does indeed not hinder tree shaking in the same ways that dart:mirrors or reflectable do. It is also a bit more cumbersome to use but I think the tradeoffs are worth it. However this issue is more about just gathering use cases than it is brainstorming solutions :). Primarily we want to understand if a feature such as const reflection objects (note this is far from a real proposal) would solve all of peoples use cases. |
@jakemac53 I'm not looking to output it to a file 🤔 I'm looking to do an in memory revive. I think I might not be understanding what you mean. Can you expand a bit? |
For each constant you are trying to revive (each class used as an annotation), you could run a separate code generator (or eventually, macro), which would generated the code for reviving that particular class. Similar to generating a toJson/fromJson for a class. |
|
Ahhh I see what you're saying. In this case I don't think that would be viable. It seems that way to me because:
While there are lots of amazing use cases for generating source code this doesn't feel like one. We have the source and can rebuild the object in memory based on the DartObject and I would rather see an updated and stabilized API over the removal because there are valid use cases. |
If you consider a sufficiently well integrated code generation mechanism (part of the compiler) I think most of these concerns go away.
But if we were able to remove dart:mirrors, that required data would no longer necessarily be present.
The "source code" in this case is actually a very abstract representation, not really any different from JSON or some other serialization format. The types you see are not even necessarily available in the current isolate and thus impossible to rehydrate at all. So, I would actually say it is less hacky to rely on the explicitly generated code to rehydrate very specific known types.
I agree this is the advantage of mirrors in general - you can write something very general purpose and it has a decent chance of succeeding. When it fails though its going to be a lot harder to debug, and then all the other downsides of mirrors regarding tree shaking and performance (although it can be possible to get decent performance if you are very tricky, most of the time its pretty bad in practice).
Mirrors are one way to solve metaprogramming, but not the only way. So most of this issue is about understanding the use cases, and what the tradeoffs would look like if we went with a different approach. I do agree moving your example to some form of code generation, however well integrated, would be a somewhat degraded development experience. It would be possible, and more statically safe, but would involve some extra boilerplate as well. |
|
@jakemac53
I would be inclined to disagree. While yes the types aren't necessarily available in the current isolate, the library and it's declarations are. With knowing the library and declaration name (which we currently get from
/// Default [RemoteSpecHeaderDelegate] used when retrieving a remote OAS spec.
class RemoteSpecHeaderDelegate {
const RemoteSpecHeaderDelegate();
Map<String, String>? header() => null;
}Defined and exposed in our library. The consumer would define a subclass like: // within my_spec_delegate.dart
import 'package:spec_builder/spec_builder.dart';
class MyRemoteSpecHeaderDelegate extends RemoteSpecHeaderDelegate {
const MyRemoteSpecHeaderDelegate(): super();
@override
Map<String, String>? header() => {'Authorization':'someAuthToken'};
}
// within spec_dfn.dart
import 'package:spec_builder/spec_builder.dart';
import 'path/to/my_spec_delegate.dart';
@SpecBuilder( // Imported from builder library
delegate: MyRemoteSpecHeaderDelegate(),
// some other properties
)
class SpecDef {}In this case the consuming repo is building off of the base we delegate we provide, but they aren't implementing the builder themselves, and are having us use it in a predictable manner. I'm not sure how this would be achieved outside of
True, but I'm providing a use case for leaving the library which this comment is ignoring.
Correct and that is what I'm trying to point out. This is a valid use case and is only something that really needs to be done on the developer side (builders are dev_dependencies, unless you're building; at least I would expect that to be the common case, I could be wrong here). I would love to hear another potential solution on how that may be applied to a builder or how that might be achieved without a build (obviously nothing concrete). (I've started to read dart-lang/language#1482 but I'm gonna need to make a couple passes before I can sus everything out and understand it more indepth) |
Only if the build script actually includes the library and declaration though. I don't understand how those are being included in your case, without the user going through all the other builder related config to set up their own builder? My guess is that in your case, you are actually all within one package, and your This is all mostly orthogonal though - I think the use case you are describing is well understood - it is effectively identical to the serialization/deserialization use case. With mirrors you can create a general purpose mechanism for "rehydrating" (or "dehydrating" :P) objects. If we do add expression level macros, you could get something similar. But it would be based on the static type of an object and not its runtime type, which can be a pretty big distinction. |
|
I use mirrors in utilities like |
|
For us, mirrors are crucial for many tests so we can inspect/manipulate values of private/protected properties. |
|
I use dart:mirrors in several backend and CLI projects. I think that even if expression level macros were implemented in Dart there are always use cases where reflection can be much more useful for the developer |
|
I don't know if macros can do this? but I want to get all annotated top level members: @Injectable(providedIn: 'root')
final class UserService {}
@RootInjector() // macro generated constant class
external Injector get rootInjector;
T inject<T>() {
return rootInjector.get<T>();
}
final userService = inject<UserService>(); |
I don't know how the If your example above is searching a directory for files with certain annotations that portion of it would not be possible, but you could use a slightly less magic API such as the example above. |
|
@jakemac53, |
This test has flaked on our CI but actually it's consistently failing because we do not represent `typedef`s in a reflective way in the VM since switching to kernel (because they used to be desguared on kernel level). So this has not been working for a long time, let's remove this test. Issue #44489 Change-Id: I36e8d8aee7a7736e8afc5741b01d10429ac0d7a9 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/331941 Commit-Queue: Martin Kustermann <kustermann@google.com> Reviewed-by: Slava Egorov <vegorov@google.com>
|
I started to learn mirrors so I want to check here is really been obsolete/deprecated? I was planning to create a small MVC framework. What can I use instead for example loop a user defined 'controller' files in a directory and instantiate object calling users defined controller class/methods from those files. |
Not yet, though we heavily discourage the use of mirrors as they are not really compatible with closed-world AOT compilation. They don't work on the Web or in Flutter so you can only use them in CLI in JIT mode. We have not been updating the library with respect to new language features: e.g. you can reflect on a record but you can't get record's shape (because there is no corresponding So my recommendation would be: don't use mirrors for anything critical. |
Edit - I missed the obvious API for this on |
It is: |
|
Just my 2 cents: I agree that use of mirrors for main application should be discouraged, due to incompatibility with AOT compilation. HOWEVER, mirrors play a critical role in our automated testing suites allowing us to verify functionality of things otherwise inaccessible (since tests are outside the test-target lib, tests can't see private members/methods, and therefore can't exercise them (or verify they were called as expected)). Deprecating mirrors would be a massive regression for us. |
|
Good afternoon 👋! Sorry to bother, but I'm trying to understand a little bit more about on the situation of dart:mirrors. I understood from this issue that right now dart:mirrors shouldn't be used in any critical case. I'm really fond of using reflection both for backend and object serialization (dartson-like) but I'm a little bit concerned about the current state of the library since as stated by @mraleph there is not a way to get the shape of a record for example. Can you give a more detailed update on what plans dart has for this specific library for example if it will be discontinued in a more permanent way in the near future or if the team is trying to provide an alternative? |
|
I can't speak with authority, I do not control such things, but I believe it is unlikely it would be removed in a non-breaking Dart language release (so, Dart 4 at the earliest). But, it also is unlikely to get many improvements such as knowledge about record types. It has always been an experimental/unstable API and never graduated to full support. We strive mostly not to break the existing use cases, until such time as there is a viable replacement. This issue exists primarily as a way to track those use cases that exist, such that we can make decisions in the future about the support of this API, as well as the features needed to make any replacement a viable one. |
|
I wouldn't say that That was at a significant cost, though, which was why one was strongly discouraged from actually using it on the web, and if you did, there were annotations that allowed some tree-shaking to happen anyway. It still broke tree-shaking in general and required extra metadata, so if just one corner of your web app uses it, your deployment size could grow a lot. Rather than worry about that, it was discontinued on the web, and never supported in AoT, and because of that Using reflection in tests means that those tests cannot be run AoT compiled, which means they are not testing the code that's actually going to run in production. (Can't do You can use I do hope that macros can give users what they need to replace existing uses of reflcection. Like a |
https://codereview.chromium.org//61793006 marked it as "unstable" - that was 2013 The documentation (today) says: This library is only supported by the Dart VM and only available on some platforms. |
|
True, I guess the API was never considered stable, which makes sense if it has to match an evolving language. The API at the time was supported everywhere. Mirrors, as such, were not considered experimental in Dart 1. The concrete API was, possibly, conisdered a little experimental, but after surviving a few releases, that disclaimer didn't mean much. |
Right, macros can definitely enable these class based test runners. It could pretty easily even just be a small layer on top of package:test, since tests/groups can be added programatically. |
|
Macros will need to be able to inform the annotations of classes and methods or use them as filters and input parameters for generators. |
|
All right, I think I have the answer. We just need a simple way to have a class extend Then the compiler can see this info and add custom code accordingly. I tried using Reflectable, and it works, but you have to do an extra build step that takes a long time, and it doesn't work well with hot reload, and you have to import the generated file. So this method is better because the compiler can see it and do the work for us! In Swift, you just extend I know it may be a lot of work to implement, but I'm telling you guys, this is a super important feature, and should be prioritized for the sake of dart. |
|
Automatically generating code is what macros is intended to answer. Dart is not JavaScript, not everything can be converted to JSON, or a map. In some cases, choosing a good representation is non-trivial, and a naive conversion could be disasterous for performance. One size will not fit all. |
and others
The Dart mirrors core library (
dart:mirrors) is currently in a sub-optimal state:It is marked "experimental" in the list of core libraries
It is marked "unstable" in the API docs
It is unsupported in the Dart web platform
Is is disabled from use in the Flutter framework (and has been so for years)
We've investigated potentially making mirrors a stable, fully supported offering, however in its current form this has a few large challenges:
Dart has powerful "tree shaking" for ensuring we can produce apps with small code sizes. This relies on the ability to detect unused code, which doesn't work if reflection can use any of it dynamically (it essentially makes all code implicitly used).
The APIs supported by mirrors are significantly behind the current Dart language, for example they do not support extension methods or null safety.
We believe that one of the core uses of mirrors is to do various forms of code generation and meta-programming. Before we potentially fully discontinue mirrors, we'd like to better understand the desired use cases here so that we could have alternative solutions and migration path ready for current users before we make any changes to
dart:mirrors. If you have any such use cases, please leave a comment.The text was updated successfully, but these errors were encountered: