Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify <Foo> and <Foo>Impl classes that are effectively simple enums or data classes #1604

Merged
merged 8 commits into from
Sep 27, 2024

Conversation

mkustermann
Copy link
Member

@mkustermann mkustermann commented Sep 26, 2024

The API of package:native_assets_cli uses currently a one-size-fits-all approach:

  • almost every API class is duplicated between a <Foo> and <Foo>Impl
  • package:native_assets_cli/native_assets_cli.dart exposes <Foo>
  • package:native_assets_cli/native_assets_cli_internal.dart exposes <Foo>Impl

This has a number of issues

  • The _internal.dart is actually a public API: It's used e.g. by flutter tools and breaking changes to it need major version bumps.

  • A user of the <Foo> API doesn't know that such objects are for example

    • hashable (e.g. go-to-definition to <Foo> there's no indication of that), e.g. DynamicLoadingSystem makes no mention of it, no get hashCode override, ... only in DynamicLoadingSystemImpl there is.
    • have a toString() that code can rely on the returned string not changing
    • ...
  • It causes code to have to choose between e.g. OS and OSImpl - which is very confusing as it's just one concept.

  • It causes code to have to downcast e.g. os as OSImpl.
    => This is a very bad smell, as generally it's an unsafe operation to downcast.

  • It's code duplication across the api/ and model/ folders, make it harder to navigate in codebase (as one may have to constantly switch between the files), ...

One original intention for this seperation may have been to hide all traces of the json protocol in the non-internal version, but I think this is not needed at all, because the protocol itself will be more stable than the APIs offered by the package itself (much easier to break the dart API and publish and new version of the package than to break the CLI protocol).

=> So this PR starts to merge some <Foo> and <Foo>Impl classes into one <Foo>.
=> This simplifies the code, removes as ...Impl downcasts, etc

One extra thing this CL does is remove some members from e.g. OS. This is part of disentangling the concepts. The build protocol may always expose the target operating system, but it may only expose the target ABI if the building code supports code assets.

=> So one should be able to use OS without OS.dylibFileName / OS.architectures / ...

=> We may even decide to move code asset related code into it's own package, making the core build protocol package not need the concept of Architecture at all.

Copy link

PR Health

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
native_assets_cli Breaking 0.9.0-wip 0.9.0-wip 0.9.0-wip ✔️
Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbols
License Headers ✔️
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/ffigen/example/libclang-example/generated_bindings.dart
pkgs/ffigen/example/shared_bindings/generate.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_shared_b_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/base_gen.dart
pkgs/ffigen/example/simple/generated_bindings.dart
pkgs/ffigen/lib/src/header_parser/clang_bindings/clang_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_decl_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_symbol_address_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_type_name_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_reserved_keyword_collision_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_comment_markup_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_dart_handle_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_enum_int_mimic_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_forward_decl_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_functions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_imported_types_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_native_func_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_opaque_dependencies_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_packed_structs_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_regress_384_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_struct_fptr_fields_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_unions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_varargs_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_cjson_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_libclang_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_sqlite_bindings.dart
pkgs/ffigen/test/native_test/_expected_native_test_bindings.dart
pkgs/jni/lib/src/third_party/generated_bindings.dart
pkgs/jni/lib/src/third_party/global_env_extensions.dart
pkgs/jni/lib/src/third_party/jni_bindings_generated.dart
pkgs/jnigen/android_test_runner/lib/main.dart
pkgs/jnigen/example/in_app_java/lib/android_utils.dart
pkgs/jnigen/example/kotlin_plugin/example/lib/main.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_bindings.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/pdfbox_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocument.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocumentInformation.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/_package.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/PDFTextStripper.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/_package.dart
pkgs/jnigen/lib/src/bindings/descriptor.dart
pkgs/jnigen/lib/src/elements/elements.g.dart
pkgs/jnigen/test/jackson_core_test/third_party/bindings/com/fasterxml/jackson/core/_package.dart
pkgs/jnigen/tool/command_runner.dart
pkgs/native_assets_builder/test_data/native_dynamic_linking/bin/native_dynamic_linking.dart
pkgs/swift2objc/lib/src/config.dart
pkgs/swift2objc/lib/src/generate_wrapper.dart
pkgs/swift2objc/lib/src/generator/_core/utils.dart
pkgs/swift2objc/lib/src/generator/generator.dart
pkgs/swift2objc/lib/src/generator/generators/class_generator.dart
pkgs/swift2objc/lib/src/parser/parsers/declaration_parsers/parse_initializer_declaration.dart
pkgs/swift2objc/lib/src/parser/parsers/declaration_parsers/parse_property_declaration.dart
pkgs/swift2objc/lib/src/transformer/_core/unique_namer.dart
pkgs/swift2objc/lib/src/transformer/_core/utils.dart
pkgs/swift2objc/lib/src/transformer/transformers/transform_property.dart
pkgs/swift2objc/test/unit/parse_initializer_param_test.dart
Package publish validation ✔️
Package Version Status
package:ffi 2.1.3 already published at pub.dev
package:ffigen 15.0.0-wip WIP (no publish necessary)
package:jni 0.12.0-wip WIP (no publish necessary)
package:jnigen 0.12.0-wip WIP (no publish necessary)
package:native_assets_cli 0.9.0-wip WIP (no publish necessary)
package:objective_c 2.1.0-wip WIP (no publish necessary)
package:swift2objc 0.0.1-wip WIP (no publish necessary)
package:swiftgen 0.0.1-wip WIP (no publish necessary)

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@coveralls
Copy link

coveralls commented Sep 26, 2024

Coverage Status

coverage: 92.22% (-0.1%) from 92.361%
when pulling e480b33 on impl-removal
into 5d79761 on main.

Copy link
Collaborator

@dcharkes dcharkes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API of package:native_assets_cli uses currently a one-size-fits-all approach:

  • almost every API class is duplicated between a <Foo> and <Foo>Impl
  • package:native_assets_cli/native_assets_cli.dart exposes <Foo>
  • package:native_assets_cli/native_assets_cli_internal.dart exposes <Foo>Impl

This has a number of issues

  • The _internal.dart is actually a public API: It's used e.g. by flutter tools and breaking changes to it need major version bumps.

I believe this is a common and established pattern.

  • A user of the <Foo> API doesn't know that such objects are for example

    • hashable (e.g. go-to-definition to <Foo> there's no indication of that), e.g. DynamicLoadingSystem makes no mention of it, no get hashCode override, ... only in DynamicLoadingSystemImpl there is.
    • have a toString() that code can rely on the returned string not changing

This could be alternatively addressed by adding those methods to the API.

  • It causes code to have to choose between e.g. OS and OSImpl - which is very confusing as it's just one concept.

Yes, prioritizing hook writer API cleanliness for package:native_asset_cli makes the use of package:native_asset_cli/internal.dart in native_assets_builder, flutter_tools, and dartdev more ugly. That's the cost of the trade-off here.

I'm on the fence whether that cost is worth it.

  • It causes code to have to downcast e.g. os as OSImpl.
    => This is a very bad smell, as generally it's an unsafe operation to downcast.

For final classes the casts are guaranteed to succeed. The package owns all possible implementations. I believe this to be an accepted pattern. (Agreed it's ugly in the code!)

  • It's code duplication across the api/ and model/ folders, make it harder to navigate in codebase (as one may have to constantly switch between the files), ...

One original intention for this separation may have been to hide all traces of the json protocol in the non-internal version

And any other things needed by any native_assets_builder, flutter_tools and dartdev that does not fit well in those packages and is more conveniently unit tests in native_assets_cli.

The JSON serialization/deserialization is probably the most important one, but some are not related to serialization:

  • Architecture.dartPlatform is now public
  • Architecture.fromString is now public
  • Architecture.fromAbi is now public (not JSON serializable related)
  • OS.dartPlatform is now public
  • OS.fromString is now public
  • OS.architectures is now public (not JSON serializable related)
  • OS.osCrossCompilationDefault is now public (not JSON serializable related)
  • BuildMode.fromString is now public
  • IOSSdk.xcodebuildSdk is now public
  • LinkModePreference.fromString is now public

Can the non-JSON ones be made private or moved to native_assets_builder without breaking native_assets_builder, flutter_tools and dartdev?

but I think this is not needed at all, because the protocol itself will be more stable than the APIs offered by the package itself (much easier to break the dart API and publish and new version of the package than to break the CLI protocol).

The fromString/toString methods are not at all obvious that they are meant for json serialization and deserialization. So if we expose them, we should document accordingly that this is what they are meant for.

Ditto for Architecture.dartPlatform, OS.dartPlatform, IOSSdk.xcodebuildSdk. It's not obvious that these are the unique keys for JSON serialization.

=> So this PR starts to merge some <Foo> and <Foo>Impl classes into one <Foo>. => This simplifies the code, removes as ...Impl downcasts, etc

There's an inherent trade-off between having a clean user API without clutter and a simpler implementation.

I believe this PR clutters the user API more. But indeed (after rolling) it makes our own job easier in native_assets_builder, flutter_tools and dartdev. So, I'm on the fence on whether we want to move this direction.

If we want to move this direction:

  • All newly exposed public APIs should have doc comments. The doc comments should explain that this is their valid JSON serialization. And should explain that the name is unique and stable.
  • Some of the members should maybe be renamed: Architecture.dartPlatform -> Architecture.name? Ditto for OS.dartPlatform and IOSSdk.xcodebuildSdk. (Or can they be made private and we only expose toString(). But maybe having a name is cleaner.)
  • Move the non-JSON serialization related things to extension methods in native_assets_builder or make them private if only used inside the implementation.

pkgs/native_assets_cli/lib/src/api/native_code_asset.dart Outdated Show resolved Hide resolved
pkgs/native_assets_cli/CHANGELOG.md Outdated Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/architecture.dart Outdated Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/link_mode.dart Outdated Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/link_mode.dart Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/link_mode.dart Outdated Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/link_mode.dart Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/target.dart Outdated Show resolved Hide resolved
@mkustermann
Copy link
Member Author

/cc @dart-lang/interop-team Any reservation against this?

@HosseinYousefi
Copy link
Member

/cc @dart-lang/interop-team Any reservation against this?

I like the change.

@HosseinYousefi
Copy link
Member

There's an inherent trade-off between having a clean user API without clutter and a simpler implementation.

Could we add @internal to all elements that are not supposed to be used by hook authors and ignore lints in flutter_tools,...

@mkustermann
Copy link
Member Author

mkustermann commented Sep 26, 2024

@dcharkes Had offline conversation with @lrhn and it seems he agrees with me

  • we should not have a separation between {BuildMode,OS,...} and {BuildModeImpl,OSImpl,...}
  • if we don't intend something to have more than one implementation it doesn't have to be an interface
  • it's fine for those classes to have a fromJson / toJson and operator== / .hashCode (if needed)
  • its not good to have downcasts like os as OSImpl

Copy link
Member

@lrhn lrhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Feel free to ignore my comments for now, since this CL is just about combining existing code, it's not adding anything new. I'll be back! 😉)

@mkustermann
Copy link
Member Author

Could we add @internal to all elements that are not supposed to be used by hook authors and ignore lints in flutter_tools,...

I could mark it as @internal. But future refactoring will intentionally start exposing more of the underlying json protocol to make this package more extensible from the outside (so flutter can add flutter-specific configurations & asset types without touching dart-lang/native).

Also: Anything flutter_tools uses is part of the public API. Even hook writers may want to use package:logging and dump some of these values via .toJson() if they get into some build issues, so I don't think it's harmful to have those methods.

But if someone feels strongly about this, let me know and I can put the annotation on.

@mkustermann
Copy link
Member Author

  • Rebased on top of 46527a8.
  • Rebasing allows also unifying CCompilerConfig{Impl,}
  • Addressed comments

PTAL

Copy link
Collaborator

@dcharkes dcharkes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: Anything flutter_tools uses is part of the public API. Even hook writers may want to use package:logging and dump some of these values via .toJson() if they get into some build issues, so I don't think it's harmful to have those methods.

I don't believe this statement is true. It's perfectly fine to use internal.dart in flutter_tools. So what Flutter tools and dartdev need I don't consider part of the public API. What hook writers need I consider part of the public API.

Changes in general LGTM! 🚢

pkgs/native_assets_cli/lib/src/c_compiler_config.dart Outdated Show resolved Hide resolved
pkgs/native_assets_cli/lib/src/c_compiler_config.dart Outdated Show resolved Hide resolved
pkgs/native_toolchain_c/CHANGELOG.md Outdated Show resolved Hide resolved
@dcharkes
Copy link
Collaborator

I could mark it as @internal. But future refactoring will intentionally start exposing more of the underlying json protocol to make this package more extensible from the outside (so flutter can add flutter-specific configurations & asset types without touching dart-lang/native).

But if someone feels strongly about this, let me know and I can put the annotation on.

We could consider marking these things @internal (and @nodoc) until we expose the complete JSON protocol, now it's kind of half exposed, only the leaf nodes in the JSON tree. And then we can review how we expose the JSON protocol in the PR that aims to fully make the JSON protocol available in building blocks to users. (But I don't feel very strongly about this.)

@mkustermann
Copy link
Member Author

mkustermann commented Sep 27, 2024

We could consider marking these things @internal (and @Nodoc) until we expose the complete JSON protocol

If this is a concern I can remove all the {OS,Architecture, ...}.{fromJson,toJson,fromString,toString} and make them top-level private methods that aren't exported via package:native_asset_cli/native_asset_cli.dart / package:native_asset_cli/native_asset_cli_internal.dart. I just thought it's nice to have them together with the class (one could even use package:json_serializable to generate such classes, ...)

@dcharkes @HosseinYousefi let me know what you think!

@mkustermann
Copy link
Member Author

Also: Anything flutter_tools uses is part of the public API. Even hook writers may want to use package:logging and dump some of these values via .toJson() if they get into some build issues, so I don't think it's harmful to have those methods.

I don't believe this statement is true. It's perfectly fine to use internal.dart in flutter_tools. So what Flutter tools and dartdev need I don't consider part of the public API. What hook writers need I consider part of the public API.

I guess it's terminology. We could have package:webdev start using this build system and using the ..._internal.dart import (as it's a building/bundling tool). So the API surface exposed via ..._internal.dart is subject to semantic versioining, breaking it requires major version bumps.

(I guess currently one could argue there's sort of an exception because Dart SDK uses this package from source and not from pub and flutter tools uses a specific pinpointed version, but this is also supposed to be usable from 3rd parties)

@HosseinYousefi
Copy link
Member

HosseinYousefi commented Sep 27, 2024

@dcharkes @HosseinYousefi let me know what you think!

If the problem is only with toString and JSON stuff, it's fine to keep it as is. User generally shouldn't rely on toString of some object being consistent between the same major version.

@dcharkes
Copy link
Collaborator

Also: Anything flutter_tools uses is part of the public API. Even hook writers may want to use package:logging and dump some of these values via .toJson() if they get into some build issues, so I don't think it's harmful to have those methods.

I don't believe this statement is true. It's perfectly fine to use internal.dart in flutter_tools. So what Flutter tools and dartdev need I don't consider part of the public API. What hook writers need I consider part of the public API.

I guess it's terminology. We could have package:webdev start using this build system and using the ..._internal.dart import (as it's a building/bundling tool). So the API surface exposed via ..._internal.dart is subject to semantic versioning, breaking it requires major version bumps.

(I guess currently one could argue there's sort of an exception because Dart SDK uses this package from source and not from pub and flutter tools uses a specific pinpointed version, but this is also supposed to be usable from 3rd parties)

Yeah, if we have two APIs (one for hook writers, and one for embedders/launchers) we would want two semantic versions. Maybe we get the desired result if we export all types from internal.dart in package:native_assets_builder and use that package for versioning the API for embedders/launchers.

@dcharkes
Copy link
Collaborator

We could consider marking these things @internal (and @Nodoc) until we expose the complete JSON protocol

If this is a concern I can remove all the {OS,Architecture, ...}.{fromJson,toJson,fromString,toString} and make them top-level private methods that aren't exported via package:native_asset_cli/native_asset_cli.dart / package:native_asset_cli/native_asset_cli_internal.dart. I just thought it's nice to have them together with the class (one could even use package:json_serializable to generate such classes, ...)

@dcharkes @HosseinYousefi let me know what you think!

Yeah, that's why I had the impl classes in the first place, I didn't want to have the methods not on the class, because that prevented using dynamic dispatch for any variants. So lets keep them in the class.

No strong opinions, just do what you think is best w.r.t. annotations and privateness.

…e data classes

The API of `package:native_assets_cli` uses currently a
one-size-fits-all approach:

* almost every API class is duplicated between a `<Foo>` and `<Foo>Impl`
* `package:native_assets_cli/native_assets_cli.dart` exposes `<Foo>`
* `package:native_assets_cli/native_assets_cli_internal.dart` exposes `<Foo>Impl`

This has a number of issues

* The `_internal.dart` is actually a public API: It's used e.g. by
  flutter tools and breaking changes to it need major version bumps.

* A user of the `<Foo>` API doesn't know that such objects are for
  example
    * hashable (e.g. go-to-definition to `<Foo>` there's no
      indication of that), e.g. `DynamicLoadingSystem` makes no mention
      of it, no `get hashCode` override, ... only in
      `DynamicLoadingSystemImpl` there is.
    * have a `toString()` that code can rely on the returned string not
      changing
    * ...

* It causes code to have to choose between e.g. `OS` and `OSImpl` -
  which is very confusing as it's just one concept.

* It causes code to have to downcast e.g. `os as OSImpl`.
  => This is a very bad smell, as generally it's an unsafe operation to
  downcast.

* It's code duplication across the api/ and model/ folders, make it
  harder to navigate in codebase (as one may have to constantly switch
  between the files), ...

One original intention for this seperation may have been to hide all traces
of the json protocol in the non-internal version, but I think this is not
needed at all, because the protocol itself will be *more* stable than the APIs
offered by the package itself (much easier to break the dart API and publish and
new version of the package than to break the CLI protocol).

=> So this PR starts to merge some `<Foo>` and `<Foo>Impl` classes into
one `<Foo>`.
=> This simplifies the code, removes `as ...Impl` downcasts, etc

One extra thing this CL does is remove some members from e.g. `OS`. This
is part of disentangling the concepts. The build protocol may always
expose the target operating system, but it may only expose the target
ABI if the building code supports code assets.

=> So one should be able to use `OS` without `OS.dylibFileName` /
`OS.architectures` / ...

=> We may even decide to move code asset related code into it's own
package, making the core build protocol package not need the concept of
`Architecture` at all.
@mkustermann
Copy link
Member Author

Yeah, that's why I had the impl classes in the first place, I didn't want to have the methods not on the class, because that prevented using dynamic dispatch for any variants. So lets keep them in the class.

The dynamic dispatch will go away (if we don't know all asset types, we could call .toJson() on a base class for 3rd party assets, but that doesn't work for .fromJson() - so I chose a structure that removes the dynamic dispatch). More in upcoming CLs.

No strong opinions, just do what you think is best w.r.t. annotations and privateness.

I'll leave it like this for now. We have plenty of time to revisit and polish the APIs before declaring 1.0. For now my goal is to get the rough structure into a place that allows easy extension from flutter with custom asset types and custom confinguration.

Thanks for the chiming in, everyone!

@mkustermann mkustermann merged commit 1aae9da into main Sep 27, 2024
32 checks passed
@mkustermann mkustermann deleted the impl-removal branch September 27, 2024 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants