Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Collapse Docker build commands in Complement CI runs to make the logs easier to read. #13058

Merged
merged 4 commits into from
Jun 15, 2022
Merged

Collapse Docker build commands in Complement CI runs to make the logs easier to read. #13058

merged 4 commits into from
Jun 15, 2022

Conversation

reivilibre
Copy link
Contributor

@reivilibre reivilibre commented Jun 14, 2022 •
edited
Loading

apt-get is frequently one of the spammiest things when scrolling down into Complement logs. Let's quieten it a bit.

Note that nobody seems to have found the ultimate cure for apt-get noise other than /dev/null, which I'm reluctant to do...

EDIT: The entire Docker building process is basically spam, so this PR now folds that away.

@reivilibre reivilibre changed the title Make Complement CI logs easier to read. Quieten apt commands in Complement CI runs to make the logs easier to read. Jun 14, 2022
@richvdh
Copy link
Member

richvdh commented Jun 14, 2022

got a source/link for the -o=Dpkg::Use-Pty=0 incantation?

@richvdh
Copy link
Member

richvdh commented Jun 14, 2022

(how does it differ from DEBIAN_FRONTEND=noninteractive which is what gets used in Dockerfiles?)

@DMRobertson
Copy link
Contributor

Apologies for bikeshedding, but: IMO the problem here is that "Run Complement Tests" actually runs complement.sh, so includes all of the setup for complement.

Maybe we could do something like this?

  • make complement.sh accept a --setup-only flag which skips tests
  • use this flag in CI
  • add a new job/step afterwards which invokes go test ... | gotestfmt and nothing else

@reivilibre
Copy link
Contributor Author

reivilibre commented Jun 15, 2022 •
edited
Loading

got a source/link for the -o=Dpkg::Use-Pty=0 incantation?

It's one of those things I've picked up over the years. https://askubuntu.com/a/668859 seems to say that it's broken on modern Ubuntu versions though :-( — I'd agree, it still looks cluttered.

DEBIAN_FRONTEND=noninteractive AFAIK means that it won't pause to ask questions on stdin. -o=Dpkg::Use-Pty=0 used to hide a lot of the spammy (Updating xxxxxx ..... 5%) messages.

I'm tempted to switch track a bit:

  • Use DEBIAN_FRONTEND=noninteractive wherever, since it makes sense anyway. I don't know if it does much quietening but I have had apt block forever waiting for input without this before, so let's call it 'best practice'.
  • Collapse the setup part of complement.sh into a collapsed group in the logs. This means that the spam will still be there, but it'll be hidden away unless you want to see it.
  • make complement.sh accept a --setup-only flag which skips tests

This is OK, but I would be hesitant to duplicate the go test ... | gotestfmt line because it has quite a few env vars set up within it. I'd probably go for --test-only, but then I wonder if collapsing the logs would make more sense..?

@DMRobertson
Copy link
Contributor

DMRobertson commented Jun 15, 2022 •
edited
Loading

then I wonder if collapsing the logs would make more sense..?

That's a much better idea (edit: than mine of adding a new flag). Could even check for the existence of a githubby env var before outputting the magic collapse-me-please notation.

@reivilibre
Try to make apt less noisy
629688b 
Make apt-get less noisy in the Dockerfiles that Complement invokes
@reivilibre
Newsfile
6541082 
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
@reivilibre
Collapse the Docker builds
f18a906
@reivilibre reivilibre force-pushed the rei/complement_apt_spam branch from 151c1eb to f18a906 Compare June 15, 2022 11:12
@reivilibre reivilibre changed the title Quieten apt commands in Complement CI runs to make the logs easier to read. Collapse Docker build commands in Complement CI runs to make the logs easier to read. Jun 15, 2022
@reivilibre reivilibre marked this pull request as ready for review June 15, 2022 11:41
@reivilibre reivilibre requested a review from a team as a code owner June 15, 2022 11:41
DMRobertson
DMRobertson approved these changes Jun 15, 2022
View reviewed changes
Copy link
Contributor

@DMRobertson DMRobertson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running complement.sh looks great!

I'm not sure if -qq will mean we don't get error messages if apt-get fails due to e.g. dodgy network conditions. But it's probably fine and easily revertable.

@reivilibre
Copy link
Contributor Author

Running complement.sh looks great!

I'm not sure if -qq will mean we don't get error messages if apt-get fails due to e.g. dodgy network conditions. But it's probably fine and easily revertable.

In my experience it will still report issues with -qq e.g.:

root@lithium:/home/rei# apt-get -yqq install orasntisorantoiasrhdoasrdnr
E: Unable to locate package orasntisorantoiasrhdoasrdnr

Thanks for the review

@reivilibre reivilibre enabled auto-merge (squash) June 15, 2022 14:22
@reivilibre reivilibre merged commit 538044a into develop Jun 15, 2022
@reivilibre reivilibre deleted the rei/complement_apt_spam branch June 15, 2022 14:42
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 23, 2022
@Fizzadar
Merge tag 'v1.62.0' into beeper-master
96fb058 
Synapse 1.62.0 (2022-07-05)
===========================

No significant changes since 1.62.0rc3.

Authors of spam-checker plugins should consult the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.62/docs/upgrade.md#upgrading-to-v1620) to learn about the enriched signatures for spam checker callbacks, which are supported with this release of Synapse.

Synapse 1.62.0rc3 (2022-07-04)
==============================

Bugfixes
--------

- Update the version of the [ldap3 plugin](https://github.com/matrix-org/matrix-synapse-ldap3/) included in the `matrixdotorg/synapse` DockerHub images and the Debian packages hosted on `packages.matrix.org` to 0.2.1. This fixes [a bug](matrix-org/matrix-synapse-ldap3#163) with usernames containing uppercase characters. ([\matrix-org#13156](matrix-org#13156))
- Fix a bug introduced in Synapse 1.62.0rc1 affecting unread counts for users on small servers. ([\matrix-org#13168](matrix-org#13168))

Synapse 1.62.0rc2 (2022-07-01)
==============================

Bugfixes
--------

- Fix unread counts for users on large servers. Introduced in v1.62.0rc1. ([\matrix-org#13140](matrix-org#13140))
- Fix DB performance when deleting old push notifications. Introduced in v1.62.0rc1. ([\matrix-org#13141](matrix-org#13141))

Synapse 1.62.0rc1 (2022-06-28)
==============================

Features
--------

- Port the spam-checker API callbacks to a new, richer API. This is part of an ongoing change to let spam-checker modules inform users of the reason their event or operation is rejected. ([\matrix-org#12857](matrix-org#12857), [\matrix-org#13047](matrix-org#13047))
- Allow server admins to customise the response of the `/.well-known/matrix/client` endpoint. ([\matrix-org#13035](matrix-org#13035))
- Add metrics measuring the CPU and DB time spent in state resolution. ([\matrix-org#13036](matrix-org#13036))
- Speed up fetching of device list changes in `/sync` and `/keys/changes`. ([\matrix-org#13045](matrix-org#13045), [\matrix-org#13098](matrix-org#13098))
- Improve URL previews for sites which only provide Twitter Card metadata, e.g. LWN.net. ([\matrix-org#13056](matrix-org#13056))

Bugfixes
--------

- Update [MSC3786](matrix-org/matrix-spec-proposals#3786) implementation to check `state_key`. ([\matrix-org#12939](matrix-org#12939))
- Fix a bug introduced in Synapse 1.58 where Synapse would not report full version information when installed from a git checkout. This is a best-effort affair and not guaranteed to be stable. ([\matrix-org#12973](matrix-org#12973))
- Fix a bug introduced in Synapse 1.60 where Synapse would fail to start if the `sqlite3` module was not available. ([\matrix-org#12979](matrix-org#12979))
- Fix a bug where non-standard information was required when requesting the `/hierarchy` API over federation. Introduced
  in Synapse v1.41.0. ([\matrix-org#12991](matrix-org#12991))
- Fix a long-standing bug which meant that rate limiting was not restrictive enough in some cases. ([\matrix-org#13018](matrix-org#13018))
- Fix a bug introduced in Synapse 1.58 where profile requests for a malformed user ID would ccause an internal error. Synapse now returns 400 Bad Request in this situation. ([\matrix-org#13041](matrix-org#13041))
- Fix some inconsistencies in the event authentication code. ([\matrix-org#13087](matrix-org#13087), [\matrix-org#13088](matrix-org#13088))
- Fix a long-standing bug where room directory requests would cause an internal server error if given a malformed room alias. ([\matrix-org#13106](matrix-org#13106))

Improved Documentation
----------------------

- Add documentation for how to configure Synapse with Workers using Docker Compose. Includes example worker config and docker-compose.yaml. Contributed by @Thumbscrew. ([\matrix-org#12737](matrix-org#12737))
- Ensure the [Poetry cheat sheet](https://matrix-org.github.io/synapse/develop/development/dependencies.html) is available in the online documentation. ([\matrix-org#13022](matrix-org#13022))
- Mention removed community/group worker endpoints in upgrade.md. Contributed by @olmari. ([\matrix-org#13023](matrix-org#13023))
- Add instructions for running Complement with `gotestfmt`-formatted output locally. ([\matrix-org#13073](matrix-org#13073))
- Update OpenTracing docs to reference the configuration manual rather than the configuration file. ([\matrix-org#13076](matrix-org#13076))
- Update information on downstream Debian packages. ([\matrix-org#13095](matrix-org#13095))
- Remove documentation for the Delete Group Admin API which no longer exists. ([\matrix-org#13112](matrix-org#13112))

Deprecations and Removals
-------------------------

- Remove the unspecced `DELETE /directory/list/room/{roomId}` endpoint, which hid rooms from the [public room directory](https://spec.matrix.org/v1.3/client-server-api/#listing-rooms). Instead, `PUT` to the same URL with a visibility of `"private"`. ([\matrix-org#13123](matrix-org#13123))

Internal Changes
----------------

- Add tests for cancellation of `GET /rooms/$room_id/members` and `GET /rooms/$room_id/state` requests. ([\matrix-org#12674](matrix-org#12674))
- Report login failures due to unknown third party identifiers in the same way as failures due to invalid passwords. This prevents an attacker from using the error response to determine if the identifier exists. Contributed by Daniel Aloni. ([\matrix-org#12738](matrix-org#12738))
- Merge the Complement testing Docker images into a single, multi-purpose image. ([\matrix-org#12881](matrix-org#12881), [\matrix-org#13075](matrix-org#13075))
- Simplify the database schema for `event_edges`. ([\matrix-org#12893](matrix-org#12893))
- Clean up the test code for client disconnection. ([\matrix-org#12929](matrix-org#12929))
- Remove code generating comments in configuration. ([\matrix-org#12941](matrix-org#12941))
- Add `Cross-Origin-Resource-Policy: cross-origin` header to content repository's thumbnail and download endpoints. ([\matrix-org#12944](matrix-org#12944))
- Replace noop background updates with `DELETE` delta. ([\matrix-org#12954](matrix-org#12954), [\matrix-org#13050](matrix-org#13050))
- Use lower isolation level when inserting read receipts to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12957](matrix-org#12957))
- Reduce the amount of state we pull from the DB. ([\matrix-org#12963](matrix-org#12963))
- Enable testing against PostgreSQL databases in Complement CI. ([\matrix-org#12965](matrix-org#12965), [\matrix-org#13034](matrix-org#13034))
- Fix an inaccurate comment. ([\matrix-org#12969](matrix-org#12969))
- Remove the `delete_device` method and always call `delete_devices`. ([\matrix-org#12970](matrix-org#12970))
- Use a GitHub form for issues rather than a hard-to-read, easy-to-ignore template. ([\matrix-org#12982](matrix-org#12982))
- Move [MSC3715](matrix-org/matrix-spec-proposals#3715) behind an experimental config flag. ([\matrix-org#12984](matrix-org#12984))
- Add type hints to tests. ([\matrix-org#12985](matrix-org#12985), [\matrix-org#13099](matrix-org#13099))
- Refactor macaroon tokens generation and move the unsubscribe link in notification emails to `/_synapse/client/unsubscribe`. ([\matrix-org#12986](matrix-org#12986))
- Fix documentation for running complement tests. ([\matrix-org#12990](matrix-org#12990))
- Faster joins: add issue links to the TODO comments in the code. ([\matrix-org#13004](matrix-org#13004))
- Reduce DB usage of `/sync` when a large number of unread messages have recently been sent in a room. ([\matrix-org#13005](matrix-org#13005), [\matrix-org#13096](matrix-org#13096), [\matrix-org#13118](matrix-org#13118))
- Replaced usage of PyJWT with methods from Authlib in `org.matrix.login.jwt`. Contributed by Hannes Lerchl. ([\matrix-org#13011](matrix-org#13011))
- Modernize the `contrib/graph/` scripts. ([\matrix-org#13013](matrix-org#13013))
- Remove redundant `room_version` parameters from event auth functions. ([\matrix-org#13017](matrix-org#13017))
- Decouple `synapse.api.auth_blocking.AuthBlocking` from `synapse.api.auth.Auth`. ([\matrix-org#13021](matrix-org#13021))
- Add type annotations to `synapse.storage.databases.main.devices`. ([\matrix-org#13025](matrix-org#13025))
- Set default `sync_response_cache_duration` to two minutes. ([\matrix-org#13042](matrix-org#13042))
- Rename CI test runs. ([\matrix-org#13046](matrix-org#13046))
- Increase timeout of complement CI test runs. ([\matrix-org#13048](matrix-org#13048))
- Refactor entry points so that they all have a `main` function. ([\matrix-org#13052](matrix-org#13052))
- Refactor the Dockerfile-workers configuration script to use Jinja2 templates in Synapse workers' Supervisord blocks. ([\matrix-org#13054](matrix-org#13054))
- Add headers to individual options in config documentation to allow for linking. ([\matrix-org#13055](matrix-org#13055))
- Make Complement CI logs easier to read. ([\matrix-org#13057](matrix-org#13057), [\matrix-org#13058](matrix-org#13058), [\matrix-org#13069](matrix-org#13069))
- Don't instantiate modules with keyword arguments. ([\matrix-org#13060](matrix-org#13060))
- Fix type checking errors against Twisted trunk. ([\matrix-org#13061](matrix-org#13061))
- Allow MSC3030 `timestamp_to_event` calls from anyone on world-readable rooms. ([\matrix-org#13062](matrix-org#13062))
- Add a CI job to check that schema deltas are in the correct folder. ([\matrix-org#13063](matrix-org#13063))
- Avoid rechecking event auth rules which are independent of room state. ([\matrix-org#13065](matrix-org#13065))
- Reduce the duplication of code that invokes the rate limiter. ([\matrix-org#13070](matrix-org#13070))
- Add a Subject Alternative Name to the certificate generated for Complement tests. ([\matrix-org#13071](matrix-org#13071))
- Add more tests for room upgrades. ([\matrix-org#13074](matrix-org#13074))
- Pin dependencies maintained by matrix.org to [semantic version](https://semver.org/) bounds. ([\matrix-org#13082](matrix-org#13082))
- Correctly report prometheus DB stats for `get_earliest_token_for_stats`. ([\matrix-org#13085](matrix-org#13085))
- Fix a long-standing bug where a finished logging context would be re-started when Synapse failed to persist an event from federation. ([\matrix-org#13089](matrix-org#13089))
- Simplify the alias deletion logic as an application service. ([\matrix-org#13093](matrix-org#13093))
- Add type annotations to `tests.test_server`. ([\matrix-org#13124](matrix-org#13124))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@DMRobertson DMRobertson DMRobertson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@reivilibre @richvdh @DMRobertson