Merge tag 'v1.38.0' into babolivier/dinsic_1.41.0

Synapse 1.38.0 (2021-07-13)
===========================

This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information.

No significant changes since 1.38.0rc3.

Synapse 1.38.0rc3 (2021-07-13)
==============================

Internal Changes
----------------

- Build the Debian packages in CI. ([\#10247](matrix-org/synapse#10247), [\#10379](matrix-org/synapse#10379))

Synapse 1.38.0rc2 (2021-07-09)
==============================

Bugfixes
--------

- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](matrix-org/synapse#10336))

Improved Documentation
----------------------

- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](matrix-org/synapse#10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](matrix-org/synapse#10331))

Synapse 1.38.0rc1 (2021-07-06)
==============================

Features
--------

- Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\#9450](matrix-org/synapse#9450))
- Add support for evicting cache entries based on last access time. ([\#10205](matrix-org/synapse#10205))
- Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](matrix-org/synapse#10214))
- Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](matrix-org/synapse#10225), [\#10243](matrix-org/synapse#10243))
- Add SSO `external_ids` to the Query User Account admin API. ([\#10261](matrix-org/synapse#10261))
- Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](matrix-org/synapse#10263))
- Add metrics for new inbound federation staging area. ([\#10284](matrix-org/synapse#10284))
- Add script to print information about recently registered users. ([\#10290](matrix-org/synapse#10290))

Bugfixes
--------

- Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](matrix-org/synapse#10223))
- Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](matrix-org/synapse#10252))
- Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](matrix-org/synapse#10264), [\#10267](matrix-org/synapse#10267), [\#10282](matrix-org/synapse#10282), [\#10286](matrix-org/synapse#10286), [\#10291](matrix-org/synapse#10291), [\#10314](matrix-org/synapse#10314), [\#10326](matrix-org/synapse#10326))
- Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](matrix-org/synapse#10279))
- Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](matrix-org/synapse#10303))

Improved Documentation
----------------------

- Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](matrix-org/synapse#10166))
- Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](matrix-org/synapse#10242))
- Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](matrix-org/synapse#10258))
- Fix homeserver config option name in presence router documentation. ([\#10288](matrix-org/synapse#10288))
- Fix link pointing at the wrong section in the modules documentation page. ([\#10302](matrix-org/synapse#10302))

Internal Changes
----------------

- Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](matrix-org/synapse#10114))
- Add type hints to the federation servlets. ([\#10213](matrix-org/synapse#10213))
- Improve the reliability of auto-joining remote rooms. ([\#10237](matrix-org/synapse#10237))
- Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](matrix-org/synapse#10239))
- Fix type hints for computing auth events. ([\#10253](matrix-org/synapse#10253))
- Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](matrix-org/synapse#10256))
- Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](matrix-org/synapse#10268))
- Re-enable a SyTest after it has been fixed. ([\#10292](matrix-org/synapse#10292))

.github/workflows/docs.yaml

@@ -7,6 +7,8 @@ on:
       - develop
       # For documentation specific to a release
       - 'release-v*'
+      # stable docs
+      - master
 
   workflow_dispatch:
 
@@ -23,42 +25,42 @@ jobs:
           mdbook-version: '0.4.9'
 
       - name: Build the documentation
-        run: mdbook build
-
-      # Deploy to the latest documentation directories
-      - name: Deploy latest documentation
-        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          keep_files: true
-          publish_dir: ./book
-          destination_dir: ./develop
+        # mdbook will only create an index.html if we're including docs/README.md in SUMMARY.md.
+        # However, we're using docs/README.md for other purposes and need to pick a new page
+        # as the default. Let's opt for the welcome page instead.
+        run: |
+          mdbook build
+          cp book/welcome_and_overview.html book/index.html
 
-      - name: Get the current Synapse version
+      # Figure out the target directory.
+      #
+      # The target directory depends on the name of the branch
+      #
+      - name: Get the target directory name
         id: vars
-        # The $GITHUB_REF value for a branch looks like `refs/heads/release-v1.2`. We do some
-        # shell magic to remove the "refs/heads/release-v" bit from this, to end up with "1.2",
-        # our major/minor version number, and set this to a var called `branch-version`.
-        #
-        # We then use some python to get Synapse's full version string, which may look
-        # like "1.2.3rc4". We set this to a var called `synapse-version`. We use this
-        # to determine if this release is still an RC, and if so block deployment.
         run: |
-          echo ::set-output name=branch-version::${GITHUB_REF#refs/heads/release-v}
-          echo ::set-output name=synapse-version::`python3 -c 'import synapse; print(synapse.__version__)'`
+          # first strip the 'refs/heads/' prefix with some shell foo
+          branch="${GITHUB_REF#refs/heads/}"
 
-      # Deploy to the version-specific directory
-      - name: Deploy release-specific documentation
-        # We only carry out this step if we're running on a release branch,
-        # and the current Synapse version does not have "rc" in the name.
-        #
-        # The result is that only full releases are deployed, but can be
-        # updated if the release branch gets retroactive fixes.
-        if: ${{ startsWith( github.ref, 'refs/heads/release-v' ) && !contains( steps.vars.outputs.synapse-version, 'rc') }}
-        uses: peaceiris/actions-gh-pages@v3
+          case $branch in
+              release-*)
+                  # strip 'release-' from the name for release branches.
+                  branch="${branch#release-}"
+                  ;;
+              master)
+                  # deploy to "latest" for the master branch.
+                  branch="latest"
+                  ;;
+          esac
+
+          # finally, set the 'branch-version' var.
+          echo "::set-output name=branch-version::$branch"
+          
+      # Deploy to the target directory.
+      - name: Deploy to gh pages
+        uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           keep_files: true
           publish_dir: ./book
-          # The resulting documentation will end up in a directory named `vX.Y`.
-          destination_dir: ./v${{ steps.vars.outputs.branch-version }}
+          destination_dir: ./${{ steps.vars.outputs.branch-version }}

.github/workflows/release-artifacts.yml

@@ -0,0 +1,90 @@
+# GitHub actions workflow which builds the release artifacts.
+
+name: Build release artifacts
+
+on:
+  push:
+    # we build on develop and release branches to (hopefully) get early warning
+    # of things breaking
+    branches: ["develop", "release-*"]
+
+    # we also rebuild on tags, so that we can be sure of picking the artifacts
+    # from the right tag.
+    tags: ["v*"]
+
+permissions:
+  contents: write
+
+jobs:
+  # first get the list of distros to build for.
+  get-distros:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - id: set-distros
+        run: |
+          echo "::set-output name=distros::$(scripts-dev/build_debian_packages --show-dists-json)"
+    # map the step outputs to job outputs
+    outputs:
+      distros: ${{ steps.set-distros.outputs.distros }}
+
+  # now build the packages with a matrix build.
+  build-debs:
+    needs: get-distros
+    name: "Build .deb packages"
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        distro: ${{ fromJson(needs.get-distros.outputs.distros) }}
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          path: src
+      - uses: actions/setup-python@v2
+      - run: ./src/scripts-dev/build_debian_packages "${{ matrix.distro }}"
+      - uses: actions/upload-artifact@v2
+        with:
+          name: debs
+          path: debs/*
+
+  build-sdist:
+    name: "Build pypi distribution files"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - run: pip install wheel
+      - run: |
+          python setup.py sdist bdist_wheel
+      - uses: actions/upload-artifact@v2
+        with:
+          name: python-dist
+          path: dist/*
+
+  # if it's a tag, create a release and attach the artifacts to it
+  attach-assets:
+    name: "Attach assets to release"
+    if: startsWith(github.ref, 'refs/tags/')
+    needs:
+      - build-debs
+      - build-sdist
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download all workflow run artifacts
+        uses: actions/download-artifact@v2
+      - name: Build a tarball for the debs
+        run: tar -cvJf debs.tar.xz debs
+      - name: Attach to release
+        uses: softprops/action-gh-release@a929a66f232c1b11af63782948aa2210f981808a  # PR#109
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          files: |
+            python-dist/*
+            debs.tar.xz
+          # if it's not already published, keep the release as a draft.
+          draft: true
+          # mark it as a prerelease if the tag contains 'rc'.
+          prerelease: ${{ contains(github.ref, 'rc') }}