All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- Update jsonwebtoken dependency.
- [BREAKING] Tachometer is now an ES module binary, so it requires Node 12 or higher.
- Upgraded dependencies and removed unnecessary ones.
-
[BREAKING] Benchmark and
root
paths are now interpreted relative to the location of the config file, instead of the current working directory.For example, a config at path
<repo>/benchmarks/foo/tachometer.json
that used to look as follows, and had to be run with<repo>
as the cwd:{ "root": ".", "benchmarks": [ { "url": "benchmarks/foo/index.html" } ] }
Should now look like this, and can be run from any cwd:
{ "root": "../..", "benchmarks": [ { "url": "foo.html" } ] }
-
The
--horizons
flag andhorizon
config setting has been replaced with--auto-sample-conditions
andautoSampleConditions
.--horizon
will continue to work for backwards compatibility, but please do update to the new name. -
Copyright notice owner changed from "The Polymer Project Authors" to "Google LLC". Trivial reformatting for
LICENSE
file to match spdx.org version. Source license headers replaced with concise SPDX-style. -
Fix bug where log files would be created with '' backslash names instead of nested directories.
-
Fix bug where
browser.addArguments
JSON config setting did not work for Firefox. -
Add
browser.profile
JSON config setting that sets the browser profile directory. Currently supported in Chrome and Firefox. -
Upgrade dependencies.
-
Fix bug where git URLs like
git@github.com/MyOrg/my-repo.git
were treated as local paths. -
Bump dependencies.
- Upgrade
systeminformation
dependency with security vulnerability alert.
- Add
trace
config to capture performance logs from browsers (currently only Chromium based browsers).
-
Fix bug in dependency swapping where local file paths in
git
dependencies would fail to install if they were relative. -
Fix "should be object" exception during result table formatting.
- A warning is now displayed when there are multiple performance marks or measurements being retrieved with the same name.
- Fix bug where a unique browser would not be launched if the only difference
between browser configurations was
binary
,addArguments
,removeArguments
,cpuThrottlingRate
orpreferences
.
-
Fix
git checkout
errors when using advanced git-based dependency swapping. -
When using dependency swapping, a fresh install will now be performed whenever any dependency version has changed (either in the original
package.json
, or in the dependency-swap configuration). Thelabel
field is no longer significant in this respect. -
When using advanced git-based dependency swapping
{kind: 'git', ...}
, a query will now always be made to the remote git repo to determine if the configuredref
is still up to date. If it is stale, a fresh install will be performed. -
When using dependency swapping, temp directories will be deleted when there is an installation failure, so that they are not re-used in a broken state.
- Add advanced configuration for cloning a git repo as a dependency swap for
monorepos and other git layouts where the
package.json
is not at the root. - Add support for
name
property to measurement config - Include
measurement
object for each benchmark in JSON results output - Fix errors with auto-installing WebDriver modules using Yarn (#186).
- Fix
Package "chromedriver" cannot be installed on demand
and similar errors affecting Node versions 10 and below (#182).
- Change WebDriver-related packages for Chrome, Firefox and IE to be installed
on-demand. This saves a lot of time during the intial installation of the
tachometer
package. See the README for more details.
-
Add ability to specify multiple measurements from the same page load by setting the
measurement
property in the JSON config file to an array. For example, you can now use the performance API to define two intervals on the same page, and compare them to each other or to other pages. -
Add ability to pull measurements from the browser performance measurement API, e.g.:
"benchmarks": [ { "measurement": { "mode": "performance", "entryName": "foo" } } ]
-
Add new syntax for specifying benchmarks:
"benchmarks": [ { "measurement": { "mode": "callback" } }, { "measurement": { "mode": "expression", "expression": "window.tachometerResult" } } ]
-
Fix
main
entry in package.json to point tolib/cli.js
. -
Added more fields to JSON output file to more closely match table printed to the console
-
Fix default benchmark name on Windows to replace all
\\
with/
. For example, previously a benchmark name might have been"src/test\\data\\for-loop.html"
. With this fix, the benchmark name will now be"src/test/data/for-loop.html"
.
-
Added
--csv-file-raw
flag, which outputs a CSV file containing all raw measurement results for each benchmark. Columns correspond to benchmarks, rows correspond to sample measurements in milliseconds. The first row is a header row containing the name of each benchmark. -
Added
samples
property to the JSON file emitted by--json-file
which contains all raw sample measurements in milliseconds.
- Fix failures to launch recent versions of Firefox and Safari.
- Remove unnecessary dependencies.
- When using custom package versions, Tachometer will resolve bare modules in the temporary npm install folders, instead of the root folder. This change fixes a bug where different package versions should have resolved to different files but Tachometer always resolved to the version in the root folder, not the temporary npm install folder.
- Added
preferences
property to Firefox browser config, which can be used to set any option that is usually set from the about:config page.
- Files are no longer cached by the server when using
--manual
mode. - Running
npm
commands is fixed on Windows - Improved Windows support in build scripts
- Chromedriver dependency is no longer pinned to a particular major version, so fresh installs should be compatible with the latest version of Chrome.
- Added
cpuThrottlingRate
option tobrowser
JSON config (Chrome only), which emulates slow CPUs by the given factor (1 for no throttle, 2 for 2x slowdown, etc).
-
Improved reliability of measurements by opening new tabs with
noopener=yes
. This change appears to reduce or eliminate shared code caching across benchmarks, removing effects such as the order of benchmarks reliably producing different results. -
Fixed bug where the
measurementExpression
setting was not being respected when passed via the config file (vs the command line flag).
- Added a
measurementExpression
option for overriding the default global measurement ofwindow.tachometerResult
to an arbitrary expression.
- It is now possible to control the binary location and arguments to the
browser by using the following settings in a
browser
section of the JSON config file:binary
(Chrome and Firefox) specifies a custom path to the browser binary.addArguments
(Chrome and Firefox) specifies additional arguments to pass to the binary.removeArguments
(Chrome) specifies arguments to omit that WebDriver would usually include by default.
-
A notification will now be printed if the current version of tachometer is outdated.
-
Added
--json-file
flag which outputs raw statistical results to a JSON file (similar to the--csv-file
flag). This supersedes the--save
flag, which will be removed in the next major version.
- Fix bug where the
--manual
,--csv-file
, and other flags were ignored if a config file was in use.
-
Fix bug where using
--config
always fails with an error about--resolve-bare-modules
. -
Fix bug where a JavaScript syntax error could result in an empty page being served (#106).
-
Added
--csv-file
flag which writes raw NxN results result table to a CSV file. (#88). -
During auto-sampling, the time remaining before the timeout will be hit is now displayed (#107).
-
--resolve-bare-modules
(with no value) no longer disables bare module resolution (#99). -
Fix bug where 404s results in e.g.
Unknown response type undefined for /favicon.ico
errors logged to the console. (#105).
-
Responses from the local server are now cached in-memory. This greatly improves performance when bare module resolution is enabled, because HTML and JS is now only parsed once per benchmark, instead of once per sample.
-
Do one throw-away warm-up run for each benchmark before starting measurement. This should help reduce measurement variation due to any cold-start effects that would have previously applied to the first sample.
-
Fix bug where timeouts in measuring the
window.tachometerResult
global (e.g. when the server is down) could cause a crash withReduce of empty array with no initial value
(#86). -
When using custom package versions, the temporary NPM install directories will now be re-used less aggressively across runs of tachometer. If any of the specified dependency versions have changed, or if the version of tachometer being used has changed, then a fresh NPM install will be performed. Additionally, the new
--force-clean-npm-install
flag can be used to force a clean NPM install every time. -
Fix bug where the
node_modules
directory could sometimes be mounted at the URL//node_modules
, causing benchmarks to fail to load dependencies. -
Don't show URL query parameters in the result table when an alias was specified.
-
Fix bug where browser in result table was displayed as
[object Object]
instead of its name.
-
Fix bug where
<html>
,<body>
, and<head>
tags could be removed from HTML files served by the built-in static server (via version bump tokoa-node-resolve
). -
Browsers in the JSON config file can now be specified as an object, e.g.
browser: { name: 'chrome', headless: true }
. The string format is still supported, though more options will be supported by the object form (e.g.windowSize
below). -
Added
--window-size
flag andbrowser:{ windowSize: {width, height} }
JSON config file property to control browser window size. Browsers will be resized to 1024x768 by default.
- Add support for Internet Explorer in Windows (
--browser=ie
).
-
Add support for Edge in Windows (
--browser=edge
). -
Add support for remote WebDriver with e.g.
--browser=chrome@http://<remote-selenium-server>
. SeeREADME
for more details. -
Add
--measure=global
mode, where the benchmark assigns an arbitrary millisecond result towindow.tachometerResult
, and tachometer will poll until it is found. -
Fix bug where no browser other than Chrome could be launched.
-
Fix bug where process did not exit on most exceptions.
- Fix
$schema
property URL automatically added to config files.
- Remove noisy debug logging for bare module import resolution.
-
Automatically update config files with a
$schema
property, pointing to the JSON schema for the file on unpkg. This will provide in-editor contextual help for many IDEs (like VS Code) when writing tachometer config files. -
Add
tachometer
bin alias, so thatnpx tachometer
can be used (previously the binary could only be invoked astach
).
-
Add
--config
flag to configure benchmarks through a JSON configuration file instead of flags. SeeREADME.md
for format details. -
JavaScript imports with bare module specifiers (e.g.
import {foo} from 'mylib';
) will now be automatically transformed to browser-compatible path imports using Node-style module resolution (e.g.import {foo} from './node_modules/mylib/index.js';
). This feature can be disabled with the--resolve-bare-modules=false
flag or theresolveBareModules: false
JSON config file property.
- A
label
can now be set in the GitHub check JSON object.
-
Benchmarks are now specified as arbitrary paths to local HTML files or directories (containing an
index.html
). Benchmarks are no longer required to be laid out in any particular directory structure. E.g. you can now invoke astach foo.html
ortach myalias=foo/bar/baz
. There is no longer a concept of implementations. -
Local benchmark files can now include URL query strings (e.g.
tach foo.html?a=b
) which will be included as-is in the launched URL. -
Variants no longer exist. Use URL query strings instead (see above).
-
Custom package versions are now installed to the system's temp dir, instead of into the project directory.
-
--manual
mode no longer shows benchmark names and other metadata (but it should only be used for testing the web server anyway since it has no statistical significance). -
Add
--version
flag.
-
Full URLs are now supported (e.g.
tach http://example.com
). Only first-contentful-paint measurement is supported in this case. -
Benchmarks are now specified as bare arguments (e.g.
tach foo
) instead of with the--name
flag. -
Fix race condition where benchmarks that returned results quickly might not be registered.
- Added support for measuring First Contentful Paint (FCP), enabled by setting
the
--measure=fcp
flag.
-
Result differences are now reported as an NxN matrix, so that every result can be compared to any other result. The
--baseline
flag has been removed, since it is no longer necessary. -
GitHub Check report is now formatted as HTML instead of ASCII.
-
Remove standard deviation column, as it is rarely useful to interpret directly.
- Initial release.