v0.10.0 / Firefox 77.0.1 release

CHANGELOG

@@ -1,8 +1,33 @@
-v0.10.0 - under development
+v0.10.0 - 2020-06-11
 =======
 
+This release is a long overdue release of OpenWPM, and contains too many
+changes to list here. Instead, we highlight the major architectural since the
+previous release. The instrumentation has been completely rewritten as part of
+this release to the new WebExtensions framework. Older versions of OpenWPM
+should not be used.
+
 Changes:
-  * Migrate to WebExtensions from the addon-sdk.
+  * Migrate instrumentation from the addon-sdk framework to WebExtensions.
+  * Migrate to unbranded builds of Firefox, and off of the ESR channel to the
+    Release channel
+  * Add support for MacOS development
+  * Add an S3Aggregator that saves data in Parquet format on S3
+  * Use conda for dependency management
+  * Disable stateful crawling due to intermittent loss of profiles and
+    geckodriver incompatibilities
+  * Refactor extension instumentation to live in a separate module
+  * Re-write logger and add support for logging to sentry
+  * Add a crawler.py crawl script that can be used for cloud deployments like
+    the type documented in https://github.com/mozilla/openwpm-crawler
+  * Add support for Firefox's native headless mode alongside XVFB
+  * Add Dockerfile and automatically deploy builds to dockerhub
+  * Add a Navigation instrument that records navigation events
+  * Drop support for Python 2
+  * Remove support for Flash
+  * Numerous stability and data saving improvements (particularly for cloud
+    crawls / the S3Aggregator)
+  * Numerous bugfixes + improved testing
 
 v0.9.0 - 2019-04-15
 ======

README.md

@@ -15,6 +15,7 @@ Table of Contents <!-- omit in toc -->
 
 * [Installation](#installation)
 * [Quick Start](#quick-start)
+* [Advice for Measurement Researchers](#advice-for-measurement-researchers)
 * [Instrumentation and Data Access](#instrumentation-and-data-access)
 * [Output Format](#output-format)
     * [Local Databases](#local-databases)
@@ -39,7 +40,6 @@ Table of Contents <!-- omit in toc -->
   * [Building the Docker Container](#building-the-docker-container)
   * [Running Measurements from inside the Container](#running-measurements-from-inside-the-container)
   * [MacOS GUI applications in Docker](#macos-gui-applications-in-docker)
-* [Disclaimer](#disclaimer)
 * [Citation](#citation)
 * [License](#license)
 
@@ -87,13 +87,23 @@ lint all the changes before you make a commit.
 
 ### Troubleshooting
 
-`make` / `gcc` may need to be installed in order to build the web extension. On Ubuntu, 
-this is achieved with `apt-get install make` on OSX the necessary packages are part of xcode
-`xcode-select --install`
-
-On a very sparse operating system additional dependencies may need to be installed. See
-the [Dockerfile](Dockerfile) for more inspiration, or open an issue if you are still having problems.
-
+1. `make` / `gcc` may need to be installed in order to build the web extension.
+   On Ubuntu, this is achieved with `apt-get install make`. On OSX the necessary
+   packages are part of xcode: `xcode-select --install`.
+2. On a very sparse operating system additional dependencies may need to be
+   installed. See the [Dockerfile](Dockerfile) for more inspiration, or open
+   an issue if you are still having problems.
+3. If you see errors related to incompatible or non-existing python packages,
+   try re-running the file with the environment variable
+   `PYTHONNOUSERSITE` set. E.g., `PYTHONNOUSERSITE=True python demo.py`.
+   If that fixes your issues, you are experiencing
+   [issue 689](https://github.com/mozilla/OpenWPM/issues/689), which can be
+   fixed by clearing your
+   python [user site packages directory](https://www.python.org/dev/peps/pep-0370/),
+   by prepending `PYTHONNOUSERSITE=True` to a specific command, or by setting
+   the environment variable for the session (e.g., `export PYTHONNOUSERSITE=True`
+   in bash). Please also add a comment to that issue to let us know you ran
+   into this problem.
 
 Quick Start
 -----------
@@ -113,11 +123,28 @@ tutorial, including a
 [platform demo](https://github.com/citp/OpenWPM/wiki/Platform-Demo)
 and a description of the
 [additional commands](https://github.com/citp/OpenWPM/wiki/Available-Commands)
-available. You can also take a look at two of our past studies, which use the
-infrastructure:
+available.
+
+
+Advice for Measurement Researchers
+----------------------------------
 
-1. [The Web Never Forgets](https://github.com/citp/TheWebNeverForgets)
-2. [Cookies that Give You Away](https://github.com/englehardt/cookies-that-give-you-away)
+OpenWPM is [often used](https://webtap.princeton.edu/software/) for web
+measurement research. We recommend the following for researchers using the tool:
+
+**Use a versioned [release](https://github.com/mozilla/OpenWPM/releases).** We
+aim to follow Firefox's release cadence, which is roughly once every four
+weeks. If we happen to fall behind on checking in new releases, please file an
+issue. Versions more than a few months out of date will use unsupported
+versions of Firefox, which are likely to have known security
+vulnerabilities. Versions less than v0.10.0 are from a previous architecture
+and should not be used.
+
+**Include the OpenWPM version number in your publication.** As of v0.10.0
+OpenWPM pins all python, npm, and system dependencies. Including this
+information alongside your work will allow other researchers to contextualize
+the results, and can be helpful if future versions of OpenWPM have
+instrumentation bugs that impact results.
 
 Instrumentation and Data Access
 -------------------------------
@@ -657,22 +684,6 @@ Or, run commands directly:
     ./run-on-osx-via-docker.sh python -m pytest
     ./run-on-osx-via-docker.sh python -m pytest -vv -s
 
-Disclaimer
------------
-
-Note that OpenWPM is under active development, and should be considered
-experimental software. The repository may contain experimental features that
-aren't fully tested. We recommend using a [tagged
-release](https://github.com/citp/OpenWPM/releases).
-
-Although OpenWPM is actively used by our group for research studies and we
-regularly use of the data collected, it is still possible there are unknown bugs
-in the infrastructure. We are in the process of writing comprehensive tests to
-verify the integrity of all included instrumentation. Prior to using OpenWPM
-for your own research we encourage you to write tests (and submit pull
-requests!) for any instrumentation that isn't currently included in our test
-scripts.
-
 Citation
 --------