Skip to content

Latest commit

 

History

History
199 lines (125 loc) · 15.1 KB

DEP-pkgspec.md

File metadata and controls

199 lines (125 loc) · 15.1 KB
Raw

Package Resolution Configuration File v1.1

Contact information

Name: Lasse R.H. Nielsen
E-mail: lrn@google.com
DEP Proposal Location
Further stakeholders:

Summary

Specify Dart tools' "package:" URI resolution in a separate package resolution configuration file.

Motivation

Dart tools currently resolve package-URIs using the "--package-root" command line parameter to specify a directory, and resolving the path-part of a package URI against this directory (defaulting to the "packages" sub-directory of the current working directory if no "--package-root" parameter is given).

This approach has some shortcomings.

First and foremost, the packages must all be represented in a centralized location of a single file system. This is currently accomplished by creating a "packages" directory and creating a symbolic link to the "pub" cache location for each package.

Symbolic links are not working well on Windows systems, and they do not interact well with most version control systems.

Copying the contents instead of using symbolic links does not scale, with developing mutually dependent packages as a problematic use case.

Also, symbolic links can't always span multiple file systems, making it impossible to have some imports from local disk and other imports from, e.g., a HTTP server.

Using a package resolution configuration file that associates a base directory to each package name, it becomes possible to reference packages from different sources directly. It avoids creating extra directories with symbolic links, and instead puts the same information into a single text file. Text files work flawlessly with version control systems and they can be edited with a simple text editor if needed.

Examples

An example .packages file generated by the pub tool could be:

# This file has been generated by the Dart tool pub on Apr 14 09:14:14 2015.
# It contains a map from Dart package names to Dart package locations.
# Dart tools, including Dart VM and and Dart analyzer, rely on the content.
# AUTO GENERATED - DO NOT EDIT
unittest:/home/somebody/.pub/cache/unittest-0.9.9/lib/
async:/home/somebody/.pub/cache/async-1.1.0/lib/
quiver:/home/somebody/.pub/cache/quiver-1.2.1/lib/

This package configuration will allow a program run using it to import package:unittest/unittest.dart and receive the file /home/somebody/.pub/cache/unittest-0.9.9/lib/unittest.dart.

At the same time, other libraries can be fetched from the net every time they are needed, or from another project that is still in development.

Proposal

The solution proposed here is:

  • Dart tools can load their "package"-URI resolution from a single file.
  • They must still support the "packages" directory and "--package-root" argument for backwards compatibility.
  • The proposed default name is .packages.
  • The file uses a simple line-based key/value format similar to Java properties files or Windows ini files. This format is deliberately kept so simple that parsing it is trivial.
  • Tools that now support a "--package-root" parameter must also support a "--packages" parameter which takes a file name as argument.

The file itself contains a list of package name/package location pairs, separated by a : character. The syntax is:

  • File must be valid UTF-8 text (no overlong encodings). It can only contain non-ASCII characters in comments, so parsing can treat the file as ASCII.
  • Lines are separated by CR (U+000D) or NL (U+000A) characters.
  • Empty lines are ignored (so CR+NL can be used as line separator).
  • Lines starting with a # character (U+0023) are comments, and are otherwise ignored.
  • The remaining lines are key/value entries. They must contain a : character.
  • The characters before the first : are the package name and the characters after are the package location.
  • The package name is is any sequence of valid URI path characters (RFC 3986 pchar) except for percent encodings and colon (':', U+003A) - that is, the RFC characters corresponding to unreserved / sub-delims / '@', and it must contain at least one non-'.' character. Rationale: The name must be usable both as a directory name and a URI path segment, preferably without conversion. By using a subset of pchar without percent encodings, the name can be used directly in a URI. By disallowing '/', '' and ':' (where only colon is a pchar), as well as the names '.' and '..', the name can be used as a directory name on most common file systems.
  • If the same package name occurs twice in the file, it is an error. The tool may fail immediately when detecting the duplicate definition, or it may give a warning and continue running and not fail until the package name is actually used in an import (similarly to when the same name is imported from two different libraries).
  • The package location is a URI reference. It may be a relative URI, in which case it is resolved against the location of the package resolution configuration file. That is, a line like homebrew:../../homebrew/lib will be resolved relative to the location of the package file. This must specify a directory, so if the path does not end in a slash ('/'), then one is added automatically. If the resolved package location URI is itself a package: URI, it won't work - it's not intended to be resolved repeatedly, and if the platform can load package:-locations directly it doesn't need a .packages file.

After loading and resolving the package-name/package-location pairs from the package resolution configuration file, the tool will resolve "package" URIs using this information.

For example, the import import 'package:unittest/unittest.dart'; is resolved by first case- and path-normalizing the URI (to avoid spurious .. path segments and to get the package name into a canonical form), then splitting it into the package name, unittest and the remainder of the path, unittest.dart.

If the unittest package was specified as:

unittest:../../packages/unittest-0.9.9/lib

in the configuration file file:///home/somebody/dart/project/smarty/.packages, then the base path of the package unittest is file:///home/somebody/dart/packages/unittest-0.9.9/lib/. The remaining path "unittest.dart" is resolved against this, getting file:///home/somebody/dart/packages/unittest-0.9.9/lib/unittest.dart.

As another example, the import import 'package:unittest/../../bar/something.dart'; is first normalized to import 'package:bar/something.dart' before it's resolved. This avoids clever URIs from escaping from the specified package locations and reading arbitrary files on the same system.

Resolving package: URIs

If a tool gets neither a "--packages" or a "--package-root" command line parameter, it should look for a way to resolve package URIs as follows:

  1. Look for a .packages file next to the program entry point (which can then not be given using a package: URI). For example running an application like:
    dart http://example.com/smarty/main.dart
    will cause the dart stand-alone VM to check for the existence of http://example.com/smarty/.packages, and if that URI returns a file, use the content for resolving package URIs in the application.

  2. If a tool does not find a .packages file in the previous step, it should look for a packages directory next to the entry point. If the entry point is a file: URI this means checking if a packages directory exists, in all other cases, the directory is assumed to exist. The tool then resolves package URIs as if the packages directory had been specified using --package-root.

  3. If the entry point was file: URI and a packages directory was not found in the previous step, then the tool checks for the presence of a .packages file in the parent directory of the entry point, and recursively check each parent directory up to the root directory until a .packages file is found. That .packages file is used to resolve package URIs.

  4. Otherwise, when no .packages file was found in the previous step, the tool must refuse to handle package URIs.

The reason that step 3 is only taken for file: URIs is that there is no simple and safe way to check whether a directory exists on an HTTP server. Fetching http://example.com/app/packages/ may fail even if http://example.com/app/packages/foo/foo.dart would succeed.

Loading a package: URI using a .packages file

A .packages file is read and converted to a map from package name to location URI.

Resolving a URI like package:foo/bar/baz.dart is then performed by:

  1. Looking up foo in the location map. If it is not there, resolution fails and the package file cannot be loaded.
  2. If lookup succeeds and finds a location URI, the relative path bar/baz.dart is resolved wrt. that location.
  3. If the result has an unknown or unsupported scheme, then loading fails. This includes URIs with a package: scheme - those are resolved again.
  4. Otherwise the platform tries to load the file.

Loading a package: URI using a packages/ directory

A packages/ directory is given by a URI.

Resolving a URI like package:foo/bar/baz.dart is then performed by:

  1. Resolving the path "foo/bar/baz.dart" against the packages directory URI.
  2. If the result has an unknown or unsupported scheme, then loading fails. This includes URIs with a package: scheme - those are not resolved again. This can typically be detected earlier because the directory URI has the same scheme.
  3. Otherwise the platform tries to load the file.

Alternatives and Variants

There are a few tweaks that can be applied to the behavior above if it is deemed advantageous, but which should probably not be part of a first implementation.

Formatting tweaks

The simple line-based syntax may be tweaked slightly.

  • Simplifying the format by dropping comments. Comments are very useful for both manually written files and for adding extra information, and as currently specified, detecting a comment is only a matter of reading the first character of a line.

  • Allowing extra white-space in the format at the start and end of a line and around the :. This allows, for example, aligning entries, but requires defining "whitespace" and (very) slightly increases the complexity of the parser. It's not necessary, but might be convenient. Just accepting space and tab is likely sufficient for most users, but it's also annoying to have other white-space characters not allowed if they are not visually distinguishable from allowed spaces.

Allow more than one package resolution configuration file.

It could be possible to add more than one file on the command line, and/or allow imports in configuration files. This would allow reuse of existing files, and patching together a configuration from partial configurations. Again this will increase start-up latency, and imports will require a more complex format.

Multiple locations for the same package

Allow the same package name to occur more than once, associating it to more than one target location. Resolving a file in that package then checks each possible target location in order until it finds one that holds the requested file. The use of this feature is highly speculative, but could allow some parts of a package to reside in a different location than the rest, without having to copy the files to a common location.

Implications and limitations

Adding the package-spec as the way to configure package-URI resolution will affect start-up time for the VM. The VM will then need to load and parse the file before it can import any library through a package-URI, where it can now just check for the package's directory in the package-root when the package is first used. Unused packages cost to load the configuration for, even if they are never used. The format was picked to be quick for the VM to parse.

The Isolate.spawnUri function has a packageRoot parameter. It should probably be extended with a packageResolution parameter of type Map<String,Uri>. If both parameters are passed, the spawnUri function should fail.

There should be a Dart package for reading and writing .packages files, converting to and from Map<String, Uri>. This can be used by all Dart based tools that need to read or write the package resolution configuration file.

The "per package name" configuration enforces that package names are special, they are not just the first segment of path of the package: URI. Nothing currently prevents placing a file in the "package" directory, say "trick.dart" and then importing "package:trick.dart". With the package-spec file, that is no longer possible, because "trick.dart" would be seen as an (invalid) package name and the file path is missing, so it would not find any file.

Making package names special is a practical change. It allows for a later change where package URIs are normalized before using, so that package:unittest will be normalized to package:unittest/unittest.dart, effectively allowing a shorthand for the default-named library of a package. With the current resolution, that is not possible - "package:unittest" may be referring to the "unittest" file in the package root directory.

Deliverables

Language specification changes

The Dart specification currently says that:

A URI of the form package:s is interpreted as a URI of the form packages/s relative to an implementation specified location.

It should be changed to something like:

A URI of the form packages:s is interpreted in an implementation specific way by tools.

A working implementation

An example/initial package for reading and writing package resolution configuration files has been created as package:package_config. This can be used by Dart based tools to read and write the package configuration.

Creating .packages files.

As part of the implementation of this proposal, the "pub" tool should be changed to allow writing a .packages file where it currently creates a "package" directory in a package's root directory. Pub should avoid automatically creating duplicate .packages files in other locations.

Consuming .packages files.

Tools that need to support the "--packages" parameter includes the standalone VM, dart2js, the development compiler and the dart-analyzer.

A Dart package is developed with support for reading and writing .packages files and determining resolution strategy.

Tests

No tests have been written.

Patents rights

TC52, the Ecma technical committee working on evolving the open Dart standard, operates under a royalty-free patent policy, RFPP (PDF). This means if the proposal graduates to being sent to TC52, you will have to sign the Ecma TC52 external contributer form and submit it to Ecma.

History

Version 1.1: Changed format to allow some non-identifiers as package names. Version 1.0: Approved by DEP committee.