Cleanup DDC files copied to the released SDK for Dart 3.0 #50700

nshahan · 2022-12-13T00:49:13Z

I want to highlight and coordinate some technically observable breaking changes to the files packaged with the SDK that I want to make for Dart 3.0. My intention is that no user (using a supported DDC build) will notice any changes. The intent is to only change the files copied to the packaged SDK that we ship externally, not the files that we build on our test bots or in google3.

With the removal of weak null safety we no longer need to ship the precompiled weak .dill files for the SDK libraries. In the process I would also like to cleanup the names of the sound null safe .dill files shipped for DDC. This will make the names consistent across dart2js, dart2wasm, and ddc.

@mit-mit

I'm not sure exactly when we consider it safe to break compiling with --no-sound-null-safety.

@domesticmouse

Do we still publish a version of "DDC DartPad" that supports weak null safety? Where are the scripts that compile the sound null safe modules for "DDC DartPad" and what .dill files do they use for the SDK?

@annagrin

I don't believe the Flutter web builds rely on these .dill files but could you help me confirm? Also what about the debugger?

@jakemac53

build_web_compilers will need to update the .dill file paths with this change lands.

DDC SDK .dill Files

Description	Current path/file	Proposed path/file
Sound null safety outline	dart-sdk/lib/_internal/ddc_outline_sound.dill	dart-sdk/lib/_internal/ddc_outline.dill
Sound null safety full	dart-sdk/lib/_internal/ddc_platform_sound.dill	dart-sdk/lib/_internal/ddc_platform.dill
Weak null safety outline	dart-sdk/lib/_internal/ddc_sdk.dill	DELETED
Weak null safety full	dart-sdk/lib/_internal/ddc_platform.dill	DELETED

I also want to remove the precompiled SDK JavaScript files. We never shipped the sound null safe versions and instead updated all the integrations of DDC to compile the SDK as needed. We only left the files in the SDK for backwards compatibility which we are breaking with Dart 3.0 anyway. AFAIK we don't use any of the files I want to delete in any supported use case.

@domesticmouse @jakemac53 @annagrin

The one file I think we need to keep shipping is the module loading library require.js. AFAIK all of our external uses (webdev, flutter web, DartPad) all rely on a copy of that file. I'm proposing to remove a pointless directory from the path but I'm open to your thoughts on this one.

DDC JavaScript Files

Description	Current path/file	Proposed path/file
Weak null safety compiled SDK (amd format)	dart-sdk/lib/dev_compiler/kernel/amd/dart_sdk.js	DELETED
SDK source map	dart-sdk/lib/dev_compiler/kernel/amd/dart_sdk.js.map	DELETED
Require.js module loader	dart-sdk/lib/dev_compiler/kernel/amd/require.js	dart-sdk/lib/dev_compiler/amd/require.js
Weak null safety compiled SDK (common format)	dart-sdk/lib/dev_compiler/kernel/common/dart_sdk.js	DELETED
SDK source map	dart-sdk/lib/dev_compiler/kernel/common/dart_sdk.js.map	DELETED
Common module loader	dart-sdk/lib/dev_compiler/kernel/common/run.js	DELETED
Weak null safety compiled SDK (es6 format)	dart-sdk/lib/dev_compiler/kernel/es6/dart_sdk.js	DELETED
SDK source map	dart-sdk/lib/dev_compiler/kernel/es6/dart_sdk.js.map	DELETED

The text was updated successfully, but these errors were encountered:

nshahan · 2022-12-13T00:57:37Z

cc @sigmundch

domesticmouse · 2022-12-13T02:30:28Z

There is no null-unsafe version of DartPad at this point. We just run null safe versions of stable, old (stable - 1), beta, and master. (There is a hidden one that has the name dev, which is a clone of stable with different package config)

For the code that Dart Services uses to build DDC artifacts with the dill files, please see https://github.com/dart-lang/dart-services/blob/master/tool/grind.dart#L240

domesticmouse · 2022-12-13T02:31:45Z

On the require.js front, please chat with @johnpryan as that is a concern for the DartPad front end, which is something John knows a lot better than I do.

mit-mit · 2022-12-13T09:55:43Z

I think this very reasonable, and if the only changes are behind the legacy flag, then I do not consider it formally breaking.

jakemac53 · 2022-12-13T15:26:06Z

My only comment would be it might be best to avoid re-using any pre-existing file paths. The errors are likely to be more confusing for the users that are affected, than if we used a totally new path so the old file doesn't exist at all.

sigmundch · 2022-12-13T16:30:55Z

/cc @christopherfujino may be able to help answer some of your questions regarding flutter web

nshahan · 2022-12-16T18:20:49Z

@mit-mit

I think this very reasonable, and if the only changes are behind the legacy flag, then I do not consider it formally breaking.

I just want make sure I know what you mean by "the legacy flag". Do you mean --no-sound-null-safety? I'm essentially removing the files that will support that flag in DDC from the released SDK.

mit-mit · 2022-12-19T13:11:30Z

I should have said "related to the legacy flag". I have no problem with removing that flag, and the associated support from DDC.

nshahan · 2023-01-04T23:23:16Z

Update: This work is in progress but we have found tests for package:dwds that expect to run with weak null safety. They need to continue running for now to prevent any regressions. The deletion of the unsound .dill files from the packaged SDK will not happen until after we move those tests to run in a different environment. For now they will be renamed in preparation for the eventual deletion:

Description	Current path/file	Proposed path/file
Sound null safety outline	dart-sdk/lib/_internal/ddc_outline_sound.dill	dart-sdk/lib/_internal/ddc_outline.dill
Sound null safety full	dart-sdk/lib/_internal/ddc_platform_sound.dill	dart-sdk/lib/_internal/ddc_platform.dill
Weak null safety outline	dart-sdk/lib/_internal/ddc_sdk.dill	dart-sdk/lib/_internal/ddc_outline_unsound.dill
Weak null safety full	dart-sdk/lib/_internal/ddc_platform.dill	dart-sdk/lib/_internal/ddc_platform_unsound.dill

domesticmouse · 2023-01-04T23:31:55Z

For now they will be renamed in preparation for the eventual deletion

When is this scheduled to happen? This change will break dart-lang/dart-services I believe. Given Dart Services supports everything from master to stable - 1 this is going to be an interesting thing to manage through the waterfall.

nshahan · 2023-01-05T00:01:43Z

I'd like to land this change as soon as possible, but I don't think this will actually break dart-lang/dart-services. Looking at this line it looks like the .dill file being used comes from the Flutter SDK which is built separately.
https://github.com/dart-lang/dart-services/blob/53ace61b517e689fe2bd0d17e2afc24fc58f29d4/lib/src/compiler.dart#L191

I don't see any .dill file being passed that comes from the Dart SDK which is what I am changing.

nshahan · 2023-01-05T00:12:53Z

@johnpryan Do you know where require.js is being pulled from to load DDC compiled code? That is another file in the Dart SDK that I'm planning on moving in another change.

nshahan · 2023-01-06T18:59:55Z

Discussed via chat, DartPad has its own copy of require.js checked in so my change should not be a problem.
https://github.com/dart-lang/dart-pad/blob/master/web/scripts/require.js

annagrin · 2023-01-10T01:15:26Z

@nshahan Summarizing our discussion offline: we don't want to break webdev tests that are currently running in weak mode, so the incremental plan is in dart-lang/webdev#1878:

dwds tests fixtures: Use special branch of build_web_compilers that supports weak null safety
webdev, dwds test infrastructure: Do not pass --(no)-sound-null-safety flags to build_web_compilers
dwds test infrastructure (using build_web_compilers): Generate js and dill files from sources for weak version of SDK
dwds test infrastructure (using build_web_compilers): Handle old and new names for the SDK assets
SDK: Then we can remove the weak sdk files from the sdk releases.

- Removes all versions of `dart_sdk.js` and `dart_sdk.js.map` from the packaged SDK. All supported build systems compile their own versions of these files. - Removes `dart-sdk/lib/dev_compiler/kernel/common/run.js`. No supported build system uses the "common" module system. - Moves `dart-sdk/lib/dev_compiler/kernel/amd/require.js` to `dart-sdk/lib/dev_compiler/amd/require.js` cleaning up the unnecessary `kernel/` sub-directory. Issue: #50700 Change-Id: Id4173ddec31a6c0260009924bf1e1ae3d3f32abf Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/275482 Reviewed-by: Mark Zhou <markzipan@google.com> Commit-Queue: Nicholas Shahan <nshahan@google.com>

Remove entirely the build_vm_compilers package, this should be marked discontinued on pub. Drop all code relating to sound/unsound null safety. Update build_web_compilers, build_vm_compilers, and build_runner to only support 3.x sdks. Point at the new DDC resources in the SDK based on dart-lang/sdk#50700.

mit-mit · 2023-01-22T11:58:04Z

@nshahan I think these changes may have broken our basic webdev workflow? Try these with a recent master (I'm using dart from flutter master):

dart create -t web webapp
cd webapp
dart pub global activate webdev
dart pub global run webdev serve

=>

[SEVERE] build_web_compilers:sdk_js on asset:build_web_compilers/$package$: Error compiling dartdevc module:build_web_compilers|lib/src/dev_compiler/dart_sdk.sound.js

Unhandled exception:
PathNotFoundException: Cannot open file, path = '/Users/mit/dev/flutter/bin/cache/dart-sdk/lib/_internal/ddc_platform_sound.dill' (OS Error: No such file or directory, errno = 2)

mit-macbookpro7:webapp1 mit$ dart pub global run webdev --version
2.7.12

jakemac53 · 2023-01-23T16:02:15Z

@mit-mit you must be on the latest build_runner/build_web_compilers/build_modules for things to work.

nshahan · 2023-01-23T18:18:15Z

@mit-mit You are right, the dart create -t web webapp command is currently broken. I have an update to the new project template prepared but I've been waiting for all versions of the necessary packages webdev, dwds, build_web_compilers to be published before I updated the template. Those are done or very close but I believe some are blocked on a release of the Dart SDK at or after 3.0.0-134.0.dev which is the first version to include my changes.

nshahan · 2023-01-23T21:35:03Z

@mit-mit I opened a new issue to track the progress: #51106

nshahan added web-dev-compiler area-web Use area-web for Dart web related issues, including the DDC and dart2js compilers and JS interop. labels Dec 13, 2022

nshahan self-assigned this Dec 13, 2022

nshahan added the P2 A bug or feature request we're likely to work on label Dec 13, 2022

mit-mit mentioned this issue Dec 13, 2022

Dart 3 tools changes #50707

Closed

16 tasks

jakemac53 mentioned this issue Dec 13, 2022

Drop support for unsound null safety in Dart 3 dart-lang/build#3424

Merged

nshahan mentioned this issue Jan 6, 2023

[web] Update needed for the updated directory structure in the Dart SDK flutter/flutter#118119

Closed

annagrin mentioned this issue Jan 10, 2023

Prepare for dart 3.0.0 alpha release dart-lang/webdev#1878

Closed

13 tasks

annagrin mentioned this issue Jan 14, 2023

Prepare for dart 3.0 alpha changes: use sdk directory copy for tests dart-lang/webdev#1895

Closed

parlough mentioned this issue Jan 25, 2023

Dill file missing when serving web project in Dart 3 dart-lang/webdev#1918

Closed

copybara-service bot closed this as completed in d56b9b3 Jan 26, 2023

elliette mentioned this issue Jan 30, 2023

Rename dill files in Flutter's Dart SDK flutter/flutter#119552

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup DDC files copied to the released SDK for Dart 3.0 #50700

Cleanup DDC files copied to the released SDK for Dart 3.0 #50700

nshahan commented Dec 13, 2022

nshahan commented Dec 13, 2022

domesticmouse commented Dec 13, 2022

domesticmouse commented Dec 13, 2022

mit-mit commented Dec 13, 2022

jakemac53 commented Dec 13, 2022

sigmundch commented Dec 13, 2022

nshahan commented Dec 16, 2022

mit-mit commented Dec 19, 2022

nshahan commented Jan 4, 2023

domesticmouse commented Jan 4, 2023

nshahan commented Jan 5, 2023

nshahan commented Jan 5, 2023

nshahan commented Jan 6, 2023

annagrin commented Jan 10, 2023 •

edited

Loading

mit-mit commented Jan 22, 2023 •

edited

Loading

jakemac53 commented Jan 23, 2023

nshahan commented Jan 23, 2023

nshahan commented Jan 23, 2023

Cleanup DDC files copied to the released SDK for Dart 3.0 #50700

Cleanup DDC files copied to the released SDK for Dart 3.0 #50700

Comments

nshahan commented Dec 13, 2022

nshahan commented Dec 13, 2022

domesticmouse commented Dec 13, 2022

domesticmouse commented Dec 13, 2022

mit-mit commented Dec 13, 2022

jakemac53 commented Dec 13, 2022

sigmundch commented Dec 13, 2022

nshahan commented Dec 16, 2022

mit-mit commented Dec 19, 2022

nshahan commented Jan 4, 2023

domesticmouse commented Jan 4, 2023

nshahan commented Jan 5, 2023

nshahan commented Jan 5, 2023

nshahan commented Jan 6, 2023

annagrin commented Jan 10, 2023 • edited Loading

mit-mit commented Jan 22, 2023 • edited Loading

jakemac53 commented Jan 23, 2023

nshahan commented Jan 23, 2023

nshahan commented Jan 23, 2023

annagrin commented Jan 10, 2023 •

edited

Loading

mit-mit commented Jan 22, 2023 •

edited

Loading