packagekit: Add OS package update support #6724

martinpitt · 2017-05-29T14:41:05Z

Add a "Software Updates" page for classic dnf/yum/apt based systems,
to complement what "ostree" provides on Atomic. This uses the PackageKit
API for platform independence.

Set priority 0 to "hide" behind ostree if it is installed (which has
default priority 1), so that we don't try to mess around with packages
on OSTree/Atomic. Ship it in a new cockpit-packagekit rpm/deb as we
don't want to pull in the packagekit dependency in cockpit-system (at
least just yet).

Don't ship this on Ubuntu 16.04, as a PackageKit crash there
(https://launchpad.net/bugs/1689820) makes this functionality mostly
useless.

Specification:
https://github.com/cockpit-project/cockpit/wiki/Feature:-System-Updates-for-dnf,-yum,-apt-hosts

This is work in progress mostly due to missing tests. This has existed as a standalone project and got a lot of fixes and some third-party testing already, so it now becomes time to integrate it properly. I'd appreciate an initial coarse-grained review to point out code style and similar structural flaws, as this is by and large my first piece of React work.

Shouldn't block this PR, can be a followup:

finalize design of "applying updates" page with @garrett (header bar vs <h2>, the rest is implemented)
implement new "applying updates" design
add a button to go back from error page to update list?

martinpitt · 2017-05-30T12:31:13Z

Pushed initial basic tests, to see how far they get on all our images. Some of them don't have PackageKit installed and thus will have to be updated, and there might be other OS/release specific issues that I haven't considered/encountered yet.

We need that to test OS updates (cockpit-project#6724) and soon for AppStream based service installation. Closes cockpit-project#6736

We need that to test OS updates (#6724) and soon for AppStream based service installation. Closes #6736

martinpitt · 2017-06-01T09:29:02Z

@garrett: The "installing updates" screen is currently a bit bare-bone, but the only reliable pieces of information that we have are the currently processed package, and the overall percentage:

Is this okay for the first version, or do you have some suggestions about the design/layout here?

garrett · 2017-06-01T13:11:06Z

It's a bit tall, but here's an annotated mockup that covers the update steps.

martinpitt · 2017-06-01T14:17:20Z

OS package updates: interactive

martinpitt · 2017-06-01T14:17:22Z

OS package updates: interactive

martinpitt · 2017-06-02T07:37:11Z

@garrett: I implemented some parts of the new design: the width-limited progress bar, no inline label any more, spelled out action label (Updating/Verifying/Downloading etc.), showing version number. if packagekit gives a time estimate, it is now shown on the right hand side too, but as dnf doesn't, it's not currently visible:

Some issues I'd still like to discuss with you:

Using a header bar in some states (available updates, error loading/applying updates) but not during install seems a bit inconsistent. The above screenshot still keeps the current implementation of putting "Applying updates" into the header bar instead of a <h2>. If you prefer a <h2>, supposedly a failing update, or failure to check for updates should be shown in <h2> instead of the header bar too? What about "your system is up to date"?
We don't show the host name on other Pages, just in the nagivation bar; it is often called "localhost" which isn't really pretty. Should we just leave that out?

martinpitt · 2017-06-02T08:35:56Z

Splitting out the code into a new cockpit-pkgupdates binary package now requires adding PackageKit to at least Fedora-24 as well, for the selenium tests. I'll wait for the other test results and will then send a separate PR for that.

mvollmer · 2017-06-02T10:03:13Z

The commit message has a duplicated paragraph "Set priority to 0 ...".

stefwalter

This is amazing. Needs a few fixes ... but really great work.

stefwalter · 2017-06-09T06:10:26Z

tools/cockpit.spec

@@ -93,6 +93,7 @@ Recommends: %{name}-docker = %{version}-%{release}
 Suggests: %{name}-pcp = %{version}-%{release}
 Suggests: %{name}-kubernetes = %{version}-%{release}
 Suggests: %{name}-selinux = %{version}-%{release}
+Suggests: %{name}-pkgupdates = %{version}-%{release}


Are we sure about this name? Do you think we should make it be cockpit-packagekit ... to follow along with cockpit-ostree ... and with the assumption that other ways of updating a system will show up?

I don't like cockpit-packagekit TBH:

This is soon going to interact with dnf/yum/apt specific configuration for setting up automatic updates (as PackageKit doesn't cover that).

PackageKit is not solely intended for updates, but also for finding and installing new software. For example, the app install bits that @mvollmer is working on is also going to use PK. PK is a means to an end, not a functionality description.

If we are going to add other mechanisms to actually update packages (like calling yum directly instead of PackageKit), these would go into this module, not a different one.

Okay that makes sense. But still not an trivial decision.

Our packages typically have not been named after functionality but after technology. So that multiple things with competing or different technologies can cover the same functionality ... and thus share a package alias (the "name" field in manifest.json).

If we are going to add other mechanisms to actually update packages (like calling yum directly instead of PackageKit), these would go into this module, not a different one.

So does cockpit-ostree fould into this package or naming structure somehow? Or how does that fit into this picture? It's the same functionality with a different implementation.

Right, ostree's manifest name is "updates", so it should probably have been called "ostree-updates", and we could call this "packagekit-updates" or "pk-updates" for brevity. To compare, we wouldn't put the functionality of searching and installing new apps (what @mvollmer is working on) into ostree just because that's also going to work on ostree.

Naming packages entirely after the underlying interfaces that they use is tricky -- for example, lots of our code talks to systemd, not only pkg/systemd.

So, I'm not sure about the package name for this, I just have a strong feeling that "cockpit-packagekit" is really misleading and ambiguous, and it should be something else. Meeting topic?

Sure. Names a hard ... maybe writing a Feature page will help you figure that out.

https://github.com/cockpit-project/cockpit/tree/master/doc/guide

The feature pages are meant to help admins imagine how Cockpit is interacting with their system, and to give them clues on how to perform similar actions from the command line.

Once you think such a page through, maybe that'll help with figuring out a name. The package name should have a similar intent. Point toward the thing that we're interacting with, aid in diagnosis, quickly understanding how we're talking to the system.

For the scope of this PR, the CLI equivalent is little more than "pkcon refresh" and "pkcon update" (I'll create a page for it). It's going to grow with configuring automatic updates and reboot detection though, as these need backend specific (yum/dnf/apt) commands. But I suppose the main conflict here is "stories/workflow" (apply/configure operating system updates) vs. "backend" (packagekit/dnf/apt/yum).

stefwalter · 2017-06-09T06:10:44Z

Makefile.am

@@ -152,6 +152,7 @@ WEBPACK_PACKAGES = \
 	networkmanager \
 	ostree \
 	pcp \
+	pkgupdates \


See comment below about the package name. Likely 'packagekit'?

stefwalter · 2017-06-09T06:10:59Z

pkg/pkgupdates/index.html

+  <link href="../base1/cockpit.css" rel="stylesheet">
+  <link href="pkgupdates.css" rel="stylesheet">
+
+  <script src="../base1/jquery.js"></script>


Are you actually using jquery? If not, no need to include it.

I do use it for setting D-Bus signal handlers with $(myDbusProxy).on(...). The JSX already require()s it, but without this <script> here I get an Oops about "ReferenceError: jQuery is not defined".

In that case, I would suggest just using .addEventListener() and removing all use of jQuery.

Nice! Done. Worth updating http://cockpit-project.org/guide/latest/cockpit-dbus.html for that? (In a separate PR)

stefwalter · 2017-06-09T06:11:32Z

pkg/pkgupdates/index.html

+  <meta charset="utf-8">
+
+  <link href="../base1/patternfly.css" type="text/css" rel="stylesheet">
+  <link href="../base1/cockpit.css" rel="stylesheet">


Are you actually using cockpit.css? Typically all the css can be webpack bundled into pkgupdates.css rather than using this legacy file.

Yes I do, mostly the table.listing-ct bits.

Then you should require("listing.less"); and don't worry about including this CSS file. This is legacy. We should use "static-linking" for all non-patternfly CSS stuff.

Thanks, that works nicely. Fixed.

stefwalter · 2017-06-09T06:11:43Z

pkg/pkgupdates/index.html

+<html>
+
+<head>
+  <title translate>Software Updates</title>


Does this actually get translated at runtime?

I'm not sure how to verify that (i. e. how to create a PO template and update the translations). kubernetes, ostree, and systemd do the same, so I copied it from that.

I think you just need to set the window.title at runtime during start up of your react based app.

We don't set window.title anywhere in Cockpit. It's not really necessary either, as even without it the window title on http://localhost:9090/updates is fine, it's apparently built from the manifest's tools→updates→label plus the hostname. Do you know if setting it explicitly is useful for anything? Perhaps embedding as iframe in third-party pages or so? As every other pkg/* has a <title> I'm a bit hesitant to drop it.

You don't need to drop the title tag. Just make sure it gets translated. It's marked for translation ... just make sure it gets translated in your app.

For example:

window.title = cockpit.gettext(window.title)

This is the only translatable string outside of your React app. It makes sense to handle it properly.

Other apps that put everything in their HTML, use something like:

cockpit.translate()

to find and translate all these strings. I don't think you need that for this one string.

That makes perfect sense, thanks for the explanation. Fixed.

stefwalter · 2017-06-09T06:19:43Z

pkg/pkgupdates/pkgupdates.jsx

+                    }
+                });
+
+


Extra line.

stefwalter · 2017-06-09T06:20:06Z

pkg/pkgupdates/pkgupdates.jsx

+        // return type is 'u', but returns -1 if not supported; so ignore implausibly high values (> 10 years)
+        dbus_pk.call('/org/freedesktop/PackageKit', 'org.freedesktop.PackageKit', 'GetTimeSinceAction',
+                     [PK_ROLE_ENUM_REFRESH_CACHE], {timeout: 5000})
+            .done(seconds => (seconds > 315360000 || this.setState({timeSinceRefresh: seconds})))


This magic number here could use a constant.

stefwalter · 2017-06-09T06:21:17Z

test/verify/check-pkgupdates

+from testlib import *
+
+@skipImage("Image uses OSTree", "continuous-atomic", "fedora-atomic", "rhel-atomic")
+@skipImage("PackageKit crashes, https://launchpad.net/bugs/1689820", "ubuntu-1604")


This should be a known issue, not a skip.

It needs to be a skip as currently cockpit-pkgupdates isn't built at all for ubuntu-1604 due to this bug. (See below).

It can't build or just isn't building right now? We should build it even if an operating system bug prevents it from working ... and track it as a known issue.

It could build, but I added blacklisting to tools/debian/rules. I really disagree about shipping it in a known completely broken state. I'm fine with building it on ubuntu-1604 for our CI, but not on "the real thing" (on releases and in Ubuntu xenial-backports).

stefwalter · 2017-06-09T06:22:10Z

test/verify/check-pkgupdates

+        if self.machine.image in ["ubuntu-1604", "debian-stable"]:
+            # old PackageKit+NM on Debian/Ubuntu misdetect online status with ifupdown; work around
+            # https://launchpad.net/bugs/1694438
+            # TODO: use a naughty override instead?


Naughty override if it affects real users using the feature in Cockpit.

Can do, but this means that we wouldn't test OS updates on debian-stable at all. It wouldn't affect debian-stable users who use ifupdown (i. e. servers), or use NetworkManager without a static ifupdown config (i. e. desktops). This just affects the case of ifupdown plus NM that we use on our test images. So I thought it would be more useful to run the checks with a config that works than to skip them entirely due to our special setup.

Makes sense, if it doesn't affect most users, it's fine to work around it.

stefwalter · 2017-06-09T06:23:36Z

tools/debian/rules

@@ -8,6 +8,12 @@ ifneq ($(shell dpkg -s libpcp3-dev >/dev/null 2>&1 && echo yes),yes)
 	CONFIG_OPTIONS = --disable-pcp
 endif

+# PackageKit crashes on update information on Ubuntu 16.04, which makes
+# "Software Updates" useless (LP: #1689820)


This is not a reason not to ship the feature. That's a bug in Ubuntu. And we should track it as a known naughty issue.

Sure it is a bug in Ubuntu, and I filed it properly downstream. But this is a dealbreaker - I didn't find a workaround, and not being able to get updates makes this page entirely useless. I am not going to ship a completely broken new package, we can enable it later if/once the bug gets fixed. Note that pkgupdates does get shipped on ubuntu 17.04, just not on 16.04.

Some background: For a long time Ubuntu has used aptdaemon as a PackageKit compatible (-ish) implementation as PackageKit's apt support was rather poor in the past. In newer versions PackageKit itself has gotten much more attention, and Ubuntu moves away from aptdaemon.

stefwalter · 2017-06-09T07:54:18Z

pkg/pkgupdates/pkgupdates.jsx

+    loadUpdateDetails(pkg_ids) {
+        pkTransaction()
+            .done(transProxy => {
+                $(transProxy).on('UpdateDetail', (event, packageId, updates, obsoletes, vendor_urls,


This should just use transProxy.addEventListener('UpdateDetail', ... ) so we can drop jQuery. jQuery is really a legacy thing and not necessary in React apps.

stefwalter · 2017-06-09T08:41:02Z

This also needs a feature page in the documentation:

https://github.com/cockpit-project/cockpit/tree/master/doc/guide

martinpitt · 2017-06-09T09:09:20Z

I pushed the FIXUPs for @stefwalter's review comments, except for "add feature page" and "discuss package name" which I added to the TODO list in the description for now.

martinpitt · 2017-06-09T10:03:28Z

Feature page pushed too now, CI was happy with the previous FIXUPs.

mvollmer · 2017-06-12T09:39:02Z

For some reason, the table isn't full-width anymore. I couldn't quickly figure out what has changed.

mvollmer · 2017-06-12T08:06:32Z

pkg/pkgupdates/pkgupdates.jsx

+const PK_INFO_ENUM_SECURITY = 8;
+const PK_STATUS_ENUM_WAIT = 1;
+const PK_STATUS_ENUM_UPDATE = 10;
+const PK_STATUS_ENUM_WAITING_FOR_LOCK = 1;


PK_STATUS_ENUM_WAITING_FOR_LOCK is 30.

Fixed locally (didn't push yet), thanks!

mvollmer · 2017-06-12T08:07:01Z

pkg/pkgupdates/pkgupdates.jsx

+const PK_STATUS_ENUM_WAITING_FOR_LOCK = 1;
+
+const PK_STATUS_STRINGS = {
+    9: _("Downloading"),


PK_STATUS_ENUM_DOWNLOAD is 8, no?

Dito, thanks!

I have never seen the "Downloading" state, actually. I tried to provoke it, but failed. Maybe this is a PackageKit quirk on Fedora.

Yes, me neither, the download part is reported as INSTALL on Fedora. But I think it can't hurt to provide a human readable string for it anyway - it may get fixed in PackageKit one day, or already be used with other backends (yum, apt, etc.)

mvollmer · 2017-06-12T09:11:09Z

pkg/pkgupdates/pkgupdates.jsx

+
+            case "loadError":
+            case "updateError":
+                return this.state.errorMessages.map(m => <pre>{m}</pre>);


There is no way out of this state except by reloading the page. I think there should be a way to go back to 'loading'.

I'm not sure - if the upgrade fails, you should really investigate on the command line, as there is no interactivity in the PackageKit upgrade. The user might try to re-run the update, but that might make things worse (or at least not better). I'm okay with adding some "Acknowledge" or "Go back" button there in case a retry does help. @garrett, do you have some input on that?

I added it as a TODO item for now, but I don't think this necessarily needs to block this PR.

I added it as a TODO item for now, but I don't think this necessarily needs to block this PR.

I agree.

martinpitt · 2017-06-12T09:58:37Z

For some reason, the table isn't full-width anymore. I couldn't quickly figure out what has changed.

I figure what's changed is that this time you only have one update with a very short description, no bugs etc. On "design patterns" the listing-ct isn't full width either, and it would only make the table harder to read to add extra whitespace. I wouldn't like to divert from cockpit's standard listing-ct here. I have 40 updates (since Friday) and the table is full width, with line breaks.

mvollmer · 2017-06-12T10:05:55Z

I wouldn't like to divert from cockpit's standard listing-ct here.

Makes sense, thanks.

martinpitt · 2017-06-12T14:23:33Z

@mvollmer: In today's weekly meeting we discussed the package name again, between "cockpit-pkgupdates" and "cockpit-packagekit". Do you have some thoughts how this would relate to your upcoming "app discovery/install" page? That will use PackageKit, docker, and (on Atomic) possibly OSTree? Does it make sense for it to depend on "cockpit-packagekit" or even go into that same binary package? How about the module name in pkg/? @petervo and I would appreciate your thoughts here. Thanks!

garrett · 2017-06-12T15:18:33Z

@stefwalter: I agree. I didn't design an empty state yet — but definitely should. Thanks for pointing that out! 👍

da2x · 2017-06-12T22:45:35Z

cockpit-pkgupdates gets stuck on Loading available updates, please wait... forever. It should probably times out after a while and show and error. (pkcon refresh crashes on current f26.)

martinpitt · 2017-06-13T08:56:31Z

@da2x: I can easily reproduce the F26 PackageKit refresh crash, that was introduced only recently. This corresponds to the progress bar getting stuck at some 20% indeed. This doesn't send out an Error signal (quite naturally, as PK crashed), so indeed this needs to be handled at the D-Bus connection level.

I now have a FIXUP with corresponding test case that handles PackageKit crashes. I will push it in a bit as I also change the "empty state" page according to @garrett 's designs and don't want to trigger CI too often.

garrett · 2017-06-13T10:08:45Z

I made some empty state mockups, shared them with @martinpitt over IRC today, and we refined them iteratively (mainly for the current lack of auto-updates).

...and realized we should have a stale state, which automatically triggers a check for new packages if someone is visiting the page and there hasn't been a check in a while. (In the mockup, I have this at >= 1 day.)

The available packages header is a little awkward here — especially when there are no auto-updates, so @martinpitt and I decided to drop the header altogether in these states for now. We'll revisit the header (and the text it may contain... if it should be different) when the automatic updates feature is added in.

garrett · 2017-06-13T10:10:07Z

In IRC, @martinpitt mentioned that the update progress is actually known, so I'll look into modifying these mockups. They'll probably change inline to the latest versions, as I hotlinked them from their master version in the design repo.

martinpitt · 2017-06-13T10:19:29Z

I pushed four fixups now:

Fix wrong PK status enum values that @mvollmer spotted
Handle PackageKit crashes
Prettier "no updates" page (from @garrett)
Automatically trigger refresh when last checked ≥ 1 day ago (from @garrett above)

Add a "Software Updates" page for classic dnf/yum/apt based systems, to complement what "ostree" provides on Atomic. This uses the PackageKit API for platform independence. Set priority 0 to hide behind ostree if it is installed (which has default priority 1), so that we don't try to mess around with packages on OSTree/Atomic. Ship it in a new cockpit-packagekit rpm/deb as we don't want to pull in the packagekit dependency in cockpit-system (at least just yet). Don't ship this on Ubuntu 16.04, as a PackageKit crash there (https://launchpad.net/bugs/1689820) makes this functionality mostly useless. Specification: https://github.com/cockpit-project/cockpit/wiki/Feature:-System-Updates-for-dnf,-yum,-apt-hosts Closes cockpit-project#6724

martinpitt · 2017-06-13T14:40:01Z

Force-pushed again to rename from "pkgupdates" to "packagekit" as per team discussion. I also squashed all the fixups together.

martinpitt · 2017-06-14T06:08:18Z

@mvollmer, @stefwalter : This is now all green and should address all your feedback.

martinpitt added the bot label May 29, 2017

martinpitt force-pushed the os-updates branch 2 times, most recently from 9f16a08 to bae270d Compare May 30, 2017 12:29

martinpitt removed the bot label May 30, 2017

martinpitt force-pushed the os-updates branch from bae270d to aac7523 Compare May 30, 2017 13:34

martinpitt mentioned this pull request May 30, 2017

test: Add PackageKit to Fedora/RHEL/CentOS images #6736

Merged

4 tasks

martinpitt added the blocked Don't land until something else happens first (see task list) label May 30, 2017

martinpitt force-pushed the os-updates branch from b143e03 to 214136f Compare May 30, 2017 14:39

martinpitt added a commit to martinpitt/cockpit that referenced this pull request May 31, 2017

test: Add PackageKit to Fedora/RHEL/CentOS images

fd29cba

We need that to test OS updates (cockpit-project#6724) and soon for AppStream based service installation. Closes cockpit-project#6736

petervo pushed a commit that referenced this pull request May 31, 2017

test: Add PackageKit to Fedora/RHEL/CentOS images

9d4db81

We need that to test OS updates (#6724) and soon for AppStream based service installation. Closes #6736

martinpitt removed the blocked Don't land until something else happens first (see task list) label May 31, 2017

martinpitt force-pushed the os-updates branch from 214136f to c376836 Compare May 31, 2017 17:22

martinpitt force-pushed the os-updates branch from c376836 to 52ea1c4 Compare June 1, 2017 10:19

martinpitt changed the title ~~WIP: updates: Add OS package update support~~ WIP: pkgupdates: Add OS package update support Jun 2, 2017

martinpitt force-pushed the os-updates branch from ec01480 to c51164f Compare June 2, 2017 08:26

stefwalter suggested changes Jun 9, 2017

View reviewed changes

stefwalter reviewed Jun 9, 2017

View reviewed changes

martinpitt force-pushed the os-updates branch from 1693f76 to 2fb81be Compare June 9, 2017 09:08

martinpitt mentioned this pull request Jun 9, 2017

doc: Move to addEventListener() #6918

Merged

mvollmer requested changes Jun 12, 2017

View reviewed changes

martinpitt force-pushed the os-updates branch from ef5df3f to f24c1ee Compare June 13, 2017 10:18

martinpitt force-pushed the os-updates branch from f24c1ee to cab23fd Compare June 13, 2017 14:39

martinpitt changed the title ~~pkgupdates: Add OS package update support~~ packagekit: Add OS package update support Jun 13, 2017

mvollmer approved these changes Jun 14, 2017

View reviewed changes

mvollmer merged commit 45c2edb into cockpit-project:master Jun 14, 2017

martinpitt deleted the os-updates branch June 14, 2017 10:41

andreasn mentioned this pull request Jul 26, 2017

Yum updates #2386

Closed

packagekit: Add OS package update support #6724

packagekit: Add OS package update support #6724

Conversation

martinpitt commented May 29, 2017 • edited Loading

martinpitt commented May 30, 2017

martinpitt commented Jun 1, 2017

garrett commented Jun 1, 2017

martinpitt commented Jun 1, 2017

martinpitt commented Jun 1, 2017

martinpitt commented Jun 2, 2017 • edited Loading

martinpitt commented Jun 2, 2017

mvollmer commented Jun 2, 2017

stefwalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefwalter Jun 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefwalter Jun 9, 2017 • edited Loading

Choose a reason for hiding this comment

stefwalter commented Jun 9, 2017 • edited Loading

martinpitt commented Jun 9, 2017

martinpitt commented Jun 9, 2017

mvollmer commented Jun 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinpitt commented Jun 12, 2017

mvollmer commented Jun 12, 2017

martinpitt commented Jun 12, 2017

garrett commented Jun 12, 2017

da2x commented Jun 12, 2017

martinpitt commented Jun 13, 2017

garrett commented Jun 13, 2017

garrett commented Jun 13, 2017

martinpitt commented Jun 13, 2017

martinpitt commented Jun 13, 2017

martinpitt commented Jun 14, 2017

martinpitt commented May 29, 2017 •

edited

Loading

martinpitt commented Jun 2, 2017 •

edited

Loading

stefwalter Jun 9, 2017 •

edited

Loading

stefwalter Jun 9, 2017 •

edited

Loading

stefwalter commented Jun 9, 2017 •

edited

Loading