ImportError: cannot import name 'unquote' #2662
Labels
Comments
|
Same problem since yarl 1.0.0, downgrading to |
|
Same here, opened a ticket to yarl. |
|
Can we get someone to pin |
|
For reference, #2664 |
pfreixes
added a commit
that referenced
this issue
Jan 15, 2018
|
@asvetlov ^^ |
|
Just in case: I'm working on moving aiohttp internals towards public yarl interfaces. |
4 tasks
asvetlov
added a commit
that referenced
this issue
Jan 15, 2018
openstack-gerrit
pushed a commit
to openstack-infra/zuul
that referenced
this issue
Jan 15, 2018
Aiohttp had an open requirement specification on yarl which has released a 1.0 that is backwards incompatible. Pin to <1.0 until aio-libs/aiohttp#2662 is fixed. Change-Id: I4e750900501ed92bdbb616f5664f7e8ab7fa99c3
|
Damn that was quick! Good job everyone |
2 tasks
Jagrmi-C
added a commit
to Jagrmi-C/osm-validator
that referenced
this issue
Jan 18, 2018
move from requirements.txt to https://docs.pipenv.org/ keep locked dependencies on Pipfile.lock fix dependencies: aio-libs/aiohttp#2662
openstack-gerrit
pushed a commit
to openstack/openstack
that referenced
this issue
Jan 18, 2018
* Update zuul from branch 'master'
- Replace master with feature/zuulv3
Change-Id: I5d9e9a573e9bffd6e06c73b2412f12c92169d8a3
- Merge "Stop running tox-cover job" into feature/zuulv3
- Merge "Link to zuul-base-jobs docs from User's Guide" into feature/zuulv3
- Merge "Register term_handler for all zuul apps" into feature/zuulv3
- Merge "Make ZuulDaemonApp an abstract base class" into feature/zuulv3
- Document connection information for components
Add some information about how the different Zuul components
communicate with each other.
Change-Id: I637a949e089c8df75203c6e585d6aeb933c2cb98
- Stabilize git driver tests
These tests relied on sleeps which can cause races when running
the full test suite in parallel. Instead, wait for the events
we know will happen to happen.
Also remove the dependency on yarl now that aiohttp has made a
release which works with yarl 1.0 (however, it does not work with
<1.0 which is why this needs to be combined with this change to
fix tests).
Change-Id: Ib1c626cdd3f083dd1d23a3c6547bd7163b66567e
- Merge "Centrally register stack dump handler" into feature/zuulv3
- Merge "Fix dependency cycle false positive" into feature/zuulv3
- Merge "Remove unused method term_handler" into feature/zuulv3
- Fix dependency cycle false positive
This corrects a false-positive in the dependency cycle detection,
but only for the new URL-style depends-on headers. It does not
do so for the legacy gerrit headers.
We used a single history list to store all the changes we enqueued
ahead of a given change, but this meant that if there was more
than one path to a change, we would see it in the history on the
second traversal. Instead, when traversing the tree, use copies
of the history list at each stage so that it can be rewound when
going back up the tree. The second path to a change will not
trip the cycle detection, and will proceed on to the point where
it notices the change is already in the queue and return harmlessly.
Also, check whether the exact change is in the history, not just
the number, since numbers are no longer unique with multiple sources.
Also, fix a bug in the test_crd_cycle test which was causing the
test to always pass since the changes were never enqueued due to
missing approval requirements.
Change-Id: I3241f90a1d7469d433cfa176e719322203d4d089
Story: 2001427
Task: 6133
- Merge "Enable direct use of github driver in debug tool" into feature/zuulv3
- Merge "Add --strip option to encrypt_secret.py" into feature/zuulv3
- Merge "Handle sigterm in nodaemon mode" into feature/zuulv3
- Merge "Remove updateChange history from github driver" into feature/zuulv3
- Merge "Documentation changes for cross-source dependencies" into feature/zuulv3
- Merge "Add support for protected jobs" into feature/zuulv3
- Merge "Disambiguate with Netflix and Javascript zuul" into feature/zuulv3
- Merge "Use hotlink instead log url in github job report" into feature/zuulv3
- Merge "Really change patchset column to string" into feature/zuulv3
- Stop running tox-cover job
tox-cover frequently fails and is non-voting so is also frequently
ignored. Stop running the job until such a time as someone cares enough
to figure out how to make it stable enough that we might make it voting.
Change-Id: Icfffb8060c80a4cbaa834987654f754bc35fbea0
- Merge "Add cross-source tests" into feature/zuulv3
- Merge "Support cross-source dependencies" into feature/zuulv3
- Merge "Add skipped CRD tests" into feature/zuulv3
- Remove updateChange history from github driver
This is not necessary because updateChange in this driver
is no longer recursive.
Change-Id: If0322a5938b17000e9ef4106fa4b0ffcbfa80c89
- Documentation changes for cross-source dependencies
Change-Id: Idb6530df5c6e647acd201509d2f0d9d86803db14
Story: 2001334
Task: 5885
- Add cross-source tests
Change-Id: Iaf31211d12a2c8ce3b4a2860e079748f7e705aba
Story: 2001334
Task: 5885
- Support cross-source dependencies
Additional tests and docs in later patches.
Change-Id: I3b86a1e3dd507fa5e584680fb6c86d35f9ff3e23
Story: 2001334
Task: 5885
- Temporarily pin yarl while aiohttp is broken
Aiohttp had an open requirement specification on yarl which has
released a 1.0 that is backwards incompatible. Pin to <1.0
until https://github.com/aio-libs/aiohttp/issues/2662 is fixed.
Change-Id: I4e750900501ed92bdbb616f5664f7e8ab7fa99c3
- Merge "Remove need to start executor as root" into feature/zuulv3
- Add skipped CRD tests
Change-Id: I145617342b424397ae19cf92335b357e413559ba
- Merge "Move CRD tests to test_gerrit_legacy_crd" into feature/zuulv3
- Really change patchset column to string
The previous change I26ff56159c2710af1515955d27bf4e9ebfcf76e9 had a
small bug which rendered it basically a noop. Further it missed some
important parts and test changes.
Change-Id: I6cfb8a6af05e589140c2f7c8b1d7228f6d4d8fcb
- Remove need to start executor as root
Now that we have a finger gateway, we no longer need to start the
executor as root so that the finger streamer on the executor can
bind to port 79 (default port for the finger streamer is changed
from 79 to 7900). Remove that requirement.
Change-Id: I6df685044c4ce81fd263043adba832609da100af
- Merge "Add specific setup inventory" into feature/zuulv3
- Merge "Fix error handling for pidfile" into feature/zuulv3
- Move CRD tests to test_gerrit_legacy_crd
This new class holds the unit tests which verify the soon-to-be
"legacy" form of cross-repo-dependencies which use the gerrit
change-id in the commit message footer. The new form will use the
URL instead. We will maintain both forms for some time, so keep
these tests around. In a later change, this file will be duplicated
to exercise the same behavior with the new URL-based form.
Change-Id: I63a7398640bf82963fadf19acdcd2208c185da0b
- Fix error handling for pidfile
Zuul will now try to open the pidfile and raise an exception if there
is a permission / issue with the pidfile path.
Change-Id: I3276ffe2de0eeb99a4eb11ba7c0e12a62f1d9886
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Add alembic.ini
When adding a new migration 'alembic revision' required an alembic.ini
with at least the script location set.
Change-Id: I4704f48229a000b8283802f74d302b53ac5cd080
- Change patchset column to string
When running zuul with Github the sql reporter fails when trying to
report data [1]. The reason for this is that github uses the git sha of
the PR head as patchset which is no integer. Thus we must change this
column to string.
[1] Error message:
sqlalchemy.exc.DataError: (psycopg2.DataError) invalid input syntax for integer: "fa42019a0d4f2a4a99ed6704c3fd037b7812e08f"
LINE 1: ...b56049945e792082', 'check', 'foo/bar', 2, 'fa42019a0...
Change-Id: I26ff56159c2710af1515955d27bf4e9ebfcf76e9
- Merge "Reprime the installation map in getGitUrl if needed" into feature/zuulv3
- Merge "Add the project to GithubUser" into feature/zuulv3
- Merge "Fix github app authentication in _getNeededByFromPR" into feature/zuulv3
- Merge "Use configured github server in app mode" into feature/zuulv3
- Link to zuul-base-jobs docs from User's Guide
In the User's Guide section dealing with Git Repositories, link to
the zuul-base-jobs documentation to solve half of the TODO comment
there.
Change-Id: Ic333278f1f00184f033ab01ab4568e53b2e40fea
- Merge "Initialize github auth in getGitUrl if needed" into feature/zuulv3
- Merge "Correctly use project name in getGitUrl" into feature/zuulv3
- Use hotlink instead log url in github job report
This change make the job name in the job report comments of github
driver as a hotlink to instead of using a log url directly. This can
make the report comments more brief.
Change-Id: I19fb8ffbc153230b7f8eedfcd5ac15ec81a66c72
- Merge "Share a fake pull request database across connections" into feature/zuulv3
- Merge "Don't treat finger client disconnect as exception" into feature/zuulv3
- Merge "Fake more of the github3 api" into feature/zuulv3
- Share a fake pull request database across connections
Because connections can be recreated, ensure that the fake pull
request database (really a dictionary) is shared across instances
of connections and fake github classes.
Also, move the fake github3 classes to their own file -- they were
getting larger and unruly.
Change-Id: I471c1487039c8b25a0bab95d918f31b92b9cd32b
- Fake more of the github3 api
We are currently not testing several GithubConnection methods because
they are faked out. Instead, move the fake layer to the github3 api
so that we exercise our code.
This lets us test methods relating to getting pull requests and
searching.
Change-Id: Ife355299073e347f8b3270650fdefe55646f6455
- Don't treat finger client disconnect as exception
The fingergw currently logs client disconnects as exceptions.
This makes the log unnecessarily noisy. Just ignore them.
Change-Id: Ic28acabcb47359d4b7077a1eecddefe0f7094212
- Disambiguate with Netflix and Javascript zuul
There are two other projects named zuul. Let's make sure we disambiguate
so users aren't confused.
Change-Id: I6c459d062970e2abcbb890d626297595d979d324
- Don't remove builds that don't exist
We saw test_reconfigure_window_fixed test fail in the change to
switch to Ansible 2.4. This may have been a timing bug where we
try to remove a build that we've already removed.
Change-Id: Ic55eeb35589b6469b0dcce8267f49eda852a49f7
- Strip \r from build UUID in fingergw
Without stripping this, we won't be able to match UUIDs.
Change-Id: I06b98b7f883433313304bfc3bb21edd5725b94e6
- Handle invalid build UUID in finger gateway
The RPC call will return an empty dict if the build UUID
cannot be found. We should handle that gracefully.
Change-Id: Ie0fa49e08d9213bf7226c6301896507866c36e28
- Add --strip option to encrypt_secret.py
Sometimes you're storing a password in a file by editting it, but you
want it to be raw without newlines/whitespace/etc. This lets you do
that easily.
Change-Id: Idc961b89a5ec3fb639e70a321b4ea587cf743b9d
- Add specific setup inventory
By default, Zuul uses runAnsibleSetup on all inventory nodes prior
to running job playbooks.
This translates to doing an 'ansible -m setup' against all nodes, but
this won't work on nodes where Python is not available, like network
nodes.
This change adds a specific setup_inventory.yaml file, which will not contain
nodes where setup module cannot work.
Change-Id: Ieb02a19036854b8d9089bcd4cc9dd0b46e3ce2fc
- Register term_handler for all zuul apps
Almost all zuul apps use the method term_handler for SIGINT and
SIGTERM. Defining this centrally in ZuulDaemonApp makes this much
simpler and without repitition.
Change-Id: I68f8d1bf52b0e16340818d2bcc44cd9fc5868ca7
- Make ZuulDaemonApp an abstract base class
We use ZuulDaemonApp like an abstract base class with run() as an
abstract method so make that explicit. This creates the groundwork for
later refactorings like centralized signal handling.
Change-Id: I20f14274df27ab181711b2ca2b80251fa5b09938
- Centrally register stack dump handler
We want the stack dump handler to be present in all zuul apps so this
can be registered in a central place.
Change-Id: I0c4a97d6ee983aa4d57928682dfb6eeffd050197
- Remove unused method term_handler
This method seems to be superseeded by exit_handler and grep tells me
it is unused.
Change-Id: I5a4dc126acbbe1ac2f99153bc7757c4f6e46fc8c
- Merge "Add zuul-web tests equivalent to the webapp tests" into feature/zuulv3
- Merge "Fix misleading message about statsd not installed" into feature/zuulv3
- Merge "Fix indentation on debug statement" into feature/zuulv3
- Merge "Add implicit project name matching" into feature/zuulv3
- Enable direct use of github driver in debug tool
The github debugging tool proved to be quite useful for prototyping
new stuff using the github api. In the process I ended up copying half
of the GithubConnection into it. However for trying out new stuff it
would be great to implement and try out the features/fixes directly in
the Github driver.
And here it is! The script now starts up a standalone version of the
Github driver which can be used to query and debug isolated stuff
against Github.
Change-Id: Ifdad1e69dae009011847869d51ff24000c44adb8
- Fix indentation on debug statement
Change-Id: If8ee9a59e765590eec9a530d85213cc14d8df574
- Really fix canonical/non-canonical project merge
In change Icaf8fca3aa4577b009d691f9a67adcb43ea040f5 the merge of
canonical and non-canonical projects was fixed. However the fix was
not complete and only covered the static configuration workflow. The
dynamic configuration workflow is still broken.
Now really fix by canonicalize the project names in every case and
make the tenant parameter mandatory.
Change-Id: I5af74763fc9c4be395a341f28c6751d22bd46195
- Add implicit project name matching
Most project pipelines define jobs for the project they are defined
in. However they still need to name the project explicitly which makes
actions like repo renaming/movement very difficult.
Thus we want to add the project itself as an implicit fallback.
Change-Id: I273a2b3b1ba2a50565624553e3898be2da5611de
Depends-On: I5af74763fc9c4be395a341f28c6751d22bd46195
- Add zuul-web tests equivalent to the webapp tests
Change-Id: I5c6337d9f6525df7cbbb2b57c1f709d3e17e796a
- Remove github delay
The caching issues are solved now, so try with no delay again.
Change-Id: I109b88bd3bb2b9ff14f8909e6d03de247c797ca8
- Reprime the installation map in getGitUrl if needed
The getGitUrl is also used by the mergers and executors. In order to
work with github apps they need to know the mapping of projects to
installation ids. However as they don't receive webhook events they
currently only create this mapping on startup. Thus we need to refresh
the installation map if the project is not found.
Change-Id: I4e0f4473e0402ea0d619b83e56eb2a3f4fbe6698
- Merge "Add github debugging template" into feature/zuulv3
- Merge "Fix github caching" into feature/zuulv3
- Add github debugging template
This is a small hacky github debugging template script for testing
github issues.
Change-Id: I9b8d1e9c2ad85eec0ea81527e97c162e8695124a
- Merge "Use connection type supplied from nodepool" into feature/zuulv3
- Fix github caching
We do caching of the requests to github using cachecontrol which is
injected into the urllib3 session. The caching needs to be entirely
etag based as max-age based caching leads to working with stale data.
Unlike documented [1] cachecontrol doesn't priorize the etag caching
but doesn't even rerequest until max-age was elapsed.
Thus we need to add a custom caching heuristic [2] which simply drops
the cache-control header containing max-age. This way we force
cachecontrol to only rely on the etag headers.
[1] http://cachecontrol.readthedocs.io/en/latest/etags.html
[2] http://cachecontrol.readthedocs.io/en/latest/custom_heuristics.html
Change-Id: If47e1eaa942914a243fc03666b83cd896665bd71
- Merge "Use python3 for docs publication" into feature/zuulv3
- Use connection type supplied from nodepool
For supporting windows nodes we need the connection type to be
configurable. This adds the ansible_connection host variable if
nodepool defines it.
Change-Id: I6d2f81c7586ae0d533add95ea96a9ea8ce8c3ab5
- Merge "Git driver" into feature/zuulv3
- Fix docs building
We need graphviz for docs building, use "doc" tag in bindep for it.
This is needed with updated jobs that look for doc tags.
Change-Id: Idc329ab113761061678c9bd31485ac5e18273e23
- Fix misleading message about statsd not installed
If the stats object is none we actually didn't configure it. In case
statsd is configured but not installed the scheduler will crash
earlier with a stack trace.
Change-Id: Id9976d78ca41ed4d2ffb164942048273cff7e07f
- Handle sigterm in nodaemon mode
When running zuul within a container it normally runs in nodaemon mode
as pid 1. Currently in this mode zuul just ignores SIGTERM which is
used normally to stop containers. Thus when running within OpenShift
it waits for a timeout until it gets killed forcefully.
Fix this by handling SIGINT and SIGTERM equally.
Change-Id: I24bd8c953e734fdb9545714126d77cbcdc161bbd
- Use python3 for docs publication
Change-Id: I40ba4939c09d9f3f5105e37a23090dad6cabdbdb
Depends-On: I2d0222225f4ae6c45d09f9352c256062e2535d63
- Merge "Remove unused function validate_conf" into feature/zuulv3
- Merge "encrypt_secret: remove the trailing '/' when building url" into feature/zuulv3
- Add support for protected jobs
For some use cases protected jobs can be useful. Protected jobs can
only be inherited by jobs defined in the same project. This adds
support for these protected jobs.
Change-Id: I62a8ecbbfa9eec54ab599bb34148976dcabfd40a
- Merge "Fix doc typo" into feature/zuulv3
- Merge "Add debug project-pipeline option" into feature/zuulv3
- Merge "Add implied-branches pragma directive" into feature/zuulv3
- Merge "Update sphinx jobs to use python3" into feature/zuulv3
- Fix doc typo
Change-Id: If5eeae9eb87c849272d6dd316e2c174cc9169ef2
- Update sphinx jobs to use python3
The zuul sphinx jobs require python3 to run. Set the sphinx_python
variable so that it will.
Change-Id: Iee6d1f1003ba9464daffbc0009b82a21001e7739
Depends-On: I481e032834fdbf674157b2c9a8fa6f95fc570ddb
- Add the project to GithubUser
The data of the GithubUser object is queried lazy in order to save
requests. In the case of Github apps on GHE we need to supply the
related project in order to do the lazy requests authenticated as the
correct installation.
Change-Id: Ic35d870be17eaf99bfe398fadfc122375be0f963
- Fix github app authentication in _getNeededByFromPR
When running as a github app we must supply the project in order to do
proper authentication as the installation. Otherwise we end up as an
unauthenticated request which will be rejected by most GHE
installations.
A draw back currently is that the search will be scoped by the app
installation (which is either a single repository or an
organization). As a future improvement we might want to iterate over
all installation and do the query. However this would need to be
configurable as with many installations this quickly can become quite
expensive. A different optimization could be searching in the local
change cache.
Change-Id: Iee7f009693dc8afa4631d2e48a630bb6da689aec
- Use configured github server in app mode
Currently when running as github app the urls for the api access are
hard coded to github.com. We need to use the configured github server
for this if we want to use apps for github enterprise.
Change-Id: I9b98d45f37d7cb6842c95cdce6435b86d33e924d
- Initialize github auth in getGitUrl if needed
The executors and mergers don't initialize the github authentication
upfront. However they call getGitUrl which gets unauthenticated if we
run as a github app. So check and initialize the authentication and
installation map.
Change-Id: If05b9f8660cab1d78ac82ea0455aa24721f3a7e2
- Correctly use project name in getGitUrl
The method getGitUrl is called via the project object and not
string. While this doesn't matter when adding it to a string this
breaks lookup of the installation key.
Change-Id: I6632497af0b1e13f6e77b5b7b0a0ee105215a1f9
- Git driver
This patch improves the existing git driver by adding
a refs watcher thread. This refs watcher looks at
refs added, deleted, updated and trigger a ref-updated
event.
When a refs is updated and that the related commits
from oldrev to newrev include a change on .zuul.yaml/zuul.yaml
or zuul.d/*.yaml then tenants including that ref is reconfigured.
Furthermore the patch includes a triggering model. Events are
sent to the scheduler so jobs can be attached to a pipeline for
running jobs.
Change-Id: I529660cb20d011f36814abe64f837945dd3f1f33
- encrypt_secret: remove the trailing '/' when building url
Change-Id: Ie534063b85d333abfcc4116b4e3903299941f139
- Remove unused function validate_conf
Change-Id: I2a1abcbefb639e82949291f997f6946f136b3dd2
- Merge "Docs: group matchers together and explain them" into feature/zuulv3
- web: remove 'id' column from builds page
The builds controller doesn't return the 'id' field anymore, this change removes
the empty column from the web page
Change-Id: I34210c224a42c505795616f4fc6a90eaf80ba69c
- Merge "web: fix key request route" into feature/zuulv3
- Merge "doc: refine zuul_return instruction" into feature/zuulv3
- web: fix key request route
By default, aiohttp route identifier matches [^{}/]+, which doesn't
work for project named with a slash. This change fixes the route so
that it matches anything until the '.pub' extension.
Change-Id: Ia7d7c257fafd3959e8f13b4d26b7999a156fb5ee
- Add debug project-pipeline option
This may be set by a project to help debug why a job is or is not
running. It works speculatively, and so can be used to debug
a single change.
Change-Id: I1957d21fe7775f786935e5d7d4bdf65b86eb5e4d
- Add implied-branches pragma directive
When two projects have dissimilar stable branch names, for example,
stable/pike and stable/jewel, but should generally be used together
and therefore share job variants, it can be difficult to make that
happen. Currently, one must add explicit multi-branch matchers
to every such job and project-template.
This allows a user to add a pragma directive to indicate all the
jobs in a file should apply to multiple branches.
Change-Id: I57cf159992d8f501cbaf41aef19562951ef6b7ea
- Merge "Fix attribute syntax in docs" into feature/zuulv3
- Merge "Drop local fork of GitPython for 2.1.8 release" into feature/zuulv3
- Merge "Send open CORS header for jobs and builds" into feature/zuulv3
- Merge "Serve keys from canonical project name" into feature/zuulv3
- Merge "Fix nit in fingergw doc" into feature/zuulv3
- Merge "Add finger gateway" into feature/zuulv3
- Fix nit in fingergw doc
Change-Id: I8cff1bef9fd1b84472a8f1d413d7e4cafa9a94b2
- Send open CORS header for jobs and builds
We don't have any precious data yet, so keep builds and jobs open like
status. This lets us use built artifacts as preview from CI jobs.
Change-Id: I81cad00d2d8f0b763601fc67a4863faa96037fc7
- Docs: group matchers together and explain them
Change-Id: Ib14e8c81edc35e5900b89dd46ba5b3dc4a41fd08
- Add finger gateway
This adds the zuul-fingergw app that should be run as root (so that
it can connect to the standard finger port 79), but changes user privs
immediately after binding that port.
Common streaming functions have been moved to streamer_utils.py to
be shared among modules.
Support for CommandSocket has been included.
Change-Id: Ia35492fe951e7b9367eeab0b145d96189d72c364
- doc: refine zuul_return instruction
The zuul_return task needs to be executed on the executor node,
otherwise it fails with KeyError: 'ZUUL_JOBDIR'
Change-Id: Ib02648f40b1602c85b8c70ee0965f0fde2d227fe
- Merge "Fix line wrapping in github docs" into feature/zuulv3
- Merge "Support table prefix for sql reporter" into feature/zuulv3
- Merge "Remove unused setup_tables" into feature/zuulv3
- Drop local fork of GitPython for 2.1.8 release
2.1.8 incorporates the noted fixes.
Story: 2001393
Task: 5982
Change-Id: I828506bd7c1a1f7ce088e958361782b6cbc71f5a
- Merge "Error on duplicate YAML keys" into feature/zuulv3
- Merge "Add more job selection debug lines" into feature/zuulv3
- Error on duplicate YAML keys
This should help avoid a situation where a dictionary value is
silently overwritten. This is in accordance with the YAML spec,
and it is a great mystery why pyyaml doesn't do this already.
Change-Id: I130f2b7e9c9fb8bd4176fd658f378e249a6a3b3f
- Add more job selection debug lines
Even if a global variant matches, a project pipeline variant must
still match. This is important information to know when debugging.
Change-Id: Iaaff183c62457f9d264111002371f5f44ced867b
- Merge "Debug job selection" into feature/zuulv3
- Debug job selection
Add some (well, a lot, more than likely) debug lines for each
job variant which is considered when freezing the job graph.
This will help admins determine why a job doesn't run.
Change-Id: Ia3b140db2a5d5117f96c2fec9ae1cc9ddd28b2d0
- Fix attribute syntax in docs
The extra column was causing sub-items to render as:
"tenant.untrusted-projects.<project>:.include".
Change-Id: Ica71a4638a654d6ac140a83ec3bb4a52ca6d5242
- Fix line wrapping in github docs
Change-Id: I96706bfc87dfccb1b619d0def3b895949dbdddb8
- Remove unused setup_tables
Setup tables was called twice.
Change-Id: Idd937e1844689edfadbfeb2e180a8ab497c29ec9
- Support table prefix for sql reporter
In some environments zuul operators may have to rely on external
database providers. In this case it can be cumbersome to get extra
databases for each test environment. Adding an optional prefix to the
table names makes it possible to gracefully run several zuul
deployments against the same database and ensure they're still
isolated against each other.
Change-Id: Ib9948d6d74f4dc2453738f5d441e233e39e7f944
- Merge "Add stackdumphandler to zuul-web" into feature/zuulv3
- Merge "tox: remove validate-layout" into feature/zuulv3
- Merge "Make all zuul-web urls relative" into feature/zuulv3
- Remove large status header and tagline
With the dashboard navbar above, the words "Zuul Status" aren't really
neeed. Also, "Real-time status monitor of Zuul, the pipeline manager
between Gerrit and Workers." seems like text that's a little out of
place now.
Change-Id: I5741f1e7edf721eec3a445c054f036a1d17bb172
- Web: Drop some columns from build list
The newrev, node name, and end time are not as useful as the other
columns, and can be found by following links to the change or logs.
In the future, we can redesign the display so that the user can select
extra details for a change, but for now, improve the ability to
scan the results quickly by removing these less-used columns.
Change-Id: I17179178b5486101c9134f8684c6e69f35e1ea7c
- Serve keys from canonical project name
Rather than asking users to know the 'source' name in order to
retrieve the project's public key, use the tenant and canonical
project name. The tenant already appears in most web urls, and
the project name should be known to the user.
Change-Id: Icd1269ffdd8879bd177fd452978a2c88b2f1b205
- Make all zuul-web urls relative
Currently some static files are retrieved with relative and some with
absolute paths. This requires hacks (e.g. ProxyHTMLURLMap) if zuul
runs under a sub url and makes url rewriting much harder. Switch them
all to relative paths.
Change-Id: I3350faf760af7b5bd92dbc344a1110dde3a35863
- Merge "Remove implied playbook extensions" into feature/zuulv3
- Merge "Rename ssh_port to connection_port" into feature/zuulv3
- Merge "Add command socket support to zuul-scheduler" into feature/zuulv3
- Merge "Move send_command() into ZuulApp" into feature/zuulv3
- Merge "Add command_socket setting to executor section" into feature/zuulv3
- Merge "Add command socket support to zuul-merger" into feature/zuulv3
- Merge "Correctly stream the remaining buffer" into feature/zuulv3
- Add command socket support to zuul-scheduler
Bring online commandsocket support for the scheduler.
Change-Id: Ia1719650623e79d40f239776eb770550bb73169b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Move send_command() into ZuulApp
We can start to remove some duplicate code and make it easier to add
commandsocket support to other processes.
Change-Id: I605ca36b6627ac4fbea960205b9a88abdeed67f5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Add command_socket setting to executor section
Like the merger, add support for setting command_socket path via
zuul.conf.
Change-Id: I88aa47870d98b0906dfb733f68af663c2dc00993
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Merge "Update playbook paths with extension" into feature/zuulv3
- Add command socket support to zuul-merger
Like we have in zuul-executor, add command socket support for
zuul-merger.
Change-Id: I66a2cb2ba3f55bdd03e884f47648278e30d2f6ab
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Merge "Remove file extension when building SimpleLayout" into feature/zuulv3
- Rename ssh_port to connection_port
With the upcoming windows support we don't have ssh as the only
connection type. As a preparation for this generalize ssh_port to
connection_port.
Change-Id: Ie505fedea7b641a860b7ac4f834d96ce5351dd74
Depends-On: Ic1939054f0604411e0122db8dbd7e9886ceaa974
- Fix zuul.projects type in docs
There was a second location where the old "list" type was used in the
docs that was missed in the change to dict of dicts. Fix this minor
documentation detail to say dictionary instead of list.
Change-Id: I4c9f90f201040e3044e818b839fc715e906f049d
- Merge "github: add integration documentation" into feature/zuulv3
- Merge "Add support for shared ansible_host in inventory" into feature/zuulv3
- Merge "Convert zuul.projects to a dict" into feature/zuulv3
- Merge "Changes for Ansible 2.4" into feature/zuulv3
- Merge "Fix mixed canonical/non-canonical project merge" into feature/zuulv3
- Add stackdumphandler to zuul-web
The stackdumphandler could also be useful for zuul-web from times to
times.
Change-Id: Icf788391fcf7df94b136ea2d78f72c8d7e5e5778
- web: add /{source}/{project}.pub route
This change adds key:get function to the scheduler gearman worker so
that zuul-web can request project's public key.
Change-Id: I7ae16f27ccd9efd68c8cf10df7ff0ff5b7b7aa13
- web: make console-stream tenant scoped
This change replaces /console-stream route by /{tenant}/console-stream.
Change-Id: Ib5ecbaf2e8a115bb1edebe71696c565b72e7c481
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
- web: add /{tenant}/builds route
This change adds a SqlHandler to query the sql reporter database from
zuul-web through the /{tenant}/builds.json controller.
This change also adds a /{tenant}/builds.html basic web interface.
Change-Id: I423a37365316cc96ed07ad0895c7198d9cff8be5
- web: add /{tenant}/jobs route
This change adds the 'job:list' job to the scheduler gearman worker
to expose the tenant jobs list.
This change also adds the /{tenant}/jobs.json endpoint to the zuul-web as well
as a /{tenant}/jobs.html web interface and command line client:
zuul show jobs $tenant
Change-Id: I950cb6a809a360867b2daccded9a8a45ac46359c
- Correct exception for missing job dependencies
When a job dependency is not configured to run, the emitted error
should be clear that's the problem.
Change-Id: I8f696e6d3e82fc88853a7b00f191cc7d458e6a81
Co-Authored-By: James E. Blair <corvus@inaugust.com>
- Remove implied playbook extensions
We deprecated this a while ago, and most changes to OpenStack
projects implementing the change have landed.
Change-Id: Ie2519f2b4144761b8e4d17ec3299cf4238e16f08
- Update playbook paths with extension
These playbook paths were missing ".yaml" extentions which fails when
implied matching goes away (Ie2519f2b4144761b8e4d17ec3299cf4238e16f08).
Change-Id: Iff04d894fffdcec0a0861d2460635b4c44a15ae4
- Remove file extension when building SimpleLayout
When using simple_layout, the setup function is adding ".yaml" to the
file names it is creating (meaning playbooks get names like
"playbooks/run/integration.yaml.yaml").
This is hidden by the implied playbook extension matching, which
searches for files with an added ".yaml" so finds them
anyway. However, simple_layout unit tests then break when this
matching is removed (Ie2519f2b4144761b8e4d17ec3299cf4238e16f08).
Change-Id: Ibfacd73840b995873156f1687860ea6792b30b26
- Fix implied branch matchers and tags
Adding an implied branch matcher to jobs on in-repo project defs
works great when an item *has* a branch. But some items, such as
tags, don't. With recent changes, it is now impossible for a
project to add a job in-repo that runs in a tag pipeline.
To correct this, we need to drop some of the optimizations which
assumed we could match the implied branch against existing branch
matchers, and instead, when adding a job in-repo, simply add a new
kind of branch matcher, an ImpliedBranchMatcher, that is evaluated
in a boolean 'and' with any existing branch matchers.
The ImpliedBranchMatcher only fails if the item has a branch, and
the branch doesn't match. If the item doesn't have a branch, it
always succeeds.
This means that when a project adds a job to a tag pipeline in-repo,
it will most likely only have the ImpliedBranchMatcher, which will
simply succeed.
It also means that the multiple project configurations present in
the project's multiple branches can all add jobs to tag pipelines,
and so to remove such a job, changes may need to be made to all
branches of a project. However, there's not much that can be done
about that at the moment.
Change-Id: Id51ddfce7ef0a6d5e3273da784e407ac72a669db
- Fix complex branch matchers in project configs
A recent change to always apply an implied branch matcher to in-tree
project configs on projects with branches had an optimization to
avoid adding unecessary jobs which was too simple. It assumed that
if the raw text of the matcher on the job did not match the name
of the branch, then there was no need to add the job. That only
works for branch matchers that are simple branch names. One that
is a regex may or may not match the implied branch.
To correct this, when building the job list, run the implied branch
name against the branch matcher regex (if there is one) for the job
to determine whether it will not match.
Change-Id: Ibbe097801a45928bc9942991d868a78f5f441887
- tox: remove validate-layout
This is no longer valid
Change-Id: Ie2acf3f0ba982a746c700880614143e6ea9a8ec5
- Merge "Fix scheduler reconfiguration handler" into feature/zuulv3
- Merge "Don't set job var override_checkout if null" into feature/zuulv3
- Fix scheduler reconfiguration handler
This method rename was missed in a previous change.
Change-Id: Idfcbc2a600b0933ac1158c612ac754932bd12950
- Merge "Don't shrink windows on reconfiguration" into feature/zuulv3
- Don't set job var override_checkout if null
It's been suggested this variable would be more convenient to use
if it were simply omitted rather than set to null if the job does
not set an override_checkout value.
Also, remove a stray TODO which is done.
Change-Id: Iededec3007857f4aebabcd027c40e9e0fc101ced
- Don't shrink windows on reconfiguration
The window size supplied by the user on a pipeline is really
the starting window size. It can grow without bound, and so
there is no reason to shrink a window on reconfiguration.
For that matter, it's not necessary to increase it either,
unless the user has raised the floor higher than the current
value.
Therefore, use the existing window value on static (ie, dependent)
change queues during reconfiguration, or the new floor if it is
higher.
Dynamic queues will get new values from the pipeline (whose value
is also conditionally updated in the same manner).
In review, we discovered that some folks may be using the expotential
window type to create a static window. To make sure this doesn't
break that case, add a test for it. Since that can be used to
intentionally reduce the window size on reconfiguration, this new
test mirrors the test for window shrinkage added in the previous
change which must be altered now that we don't auto-shrink on
reconfiguration.
Change-Id: Iaba25788ebe51ced919dc896aa20aa90675d7773
- Merge "Fix branch checkout order" into feature/zuulv3
- Merge "Print a message when we start the Zuul console" into feature/zuulv3
- Print a message when we start the Zuul console
The link for the job console is made available before Ansible starts
printing things. If the console is empty, users will be greeted with
an "END OF STREAM" message and will need to keep refreshing until there
is actual content.
Writing a minimal message to "start" the console will prevent that.
Change-Id: I4200c91b468b1c2fcccee23dda19962b30335ef5
- Convert zuul.projects to a dict
This follows-on from I4476b9d4915d107e29b91229287865bff0ada305 where
we are converting zuul.projects to a dict for easier access.
With the dependent changes, there should be no in-tree users of
zuul.projects so we can do this switch. We will then convert
zuul._project users back to zuul.projects for the final removal
(I283e66293110aa3bc99acb1cbc6eb3e0cc11f750).
This updates documentation, and also gives some samples of how to use
the variables (the "| list" is a bit of a gotcha trick -- python3's
values() returns a view, so it is necessary for iteration).
Depends-On: Id9a7c137ca5bed25d81087201091157c8401576a
Depends-On: I9d88f405f34d1c5f75ebf4f52cedfaaab20c3bda
Change-Id: I3c011f72933e98ccbf8badf0e9197c8659766c51
- Remove nodesets from builds canceled during reconfiguration
We observed errant behavior in the configuration covered by
test_reconfigure_window_shrink in production when a reconfiguration
shrunk the active window to less than the current value when
jobs had already completed.
Correct the underlying issue by removing the nodeset associated with
a build from the buildset when the reconfiguration routine cancels it.
Then, if we later launch the same job for some reason, we will obtain
a new nodeset.
If the build is running at the time it's canceled, we will still need
the scheduler to return the nodeset to nodepool. Since it currently
relies on the value in the buildset to find the nodeset, attach the
nodeset to the build directly, so that even if we have removed the
nodeset from the buildset, the scheduler will still have a pointer
to the nodeset when the build completes.
Having said all that, we really don't want to waste resources by
shrinking the window on reconfiguration. A future change is likely
to correct that and very likely invalidate the test just added. The
only other time a build is likely to be canceled during reconfiguration
yet used again later is if a job is removed, then added back to a
project while changes are in the queue. So that we continue to have
a test which covers this case, add a second test based on that scenario.
Both of these tests fail without the included fix.
Change-Id: If61b34e0f1464cb69d9d0b9053e05f1af996a67b
- Changes for Ansible 2.4
Squashed changes:
- Use 'inventory' instead of 'hostfile' in ansible.cfg.
'hostfile' is deprecated.
- Use 'os.environ.copy()' in zuul_return.py since this causes 2.4 to
throw an exception now deep within module.exit_json().
Change-Id: I0a52c9e169a54d24a7b361010045fb10211418b7
- Merge "Add inventory variables for checkouts" into feature/zuulv3
- Merge "Normalize daemon process handling" into feature/zuulv3
- Fix mixed canonical/non-canonical project merge
When defining multiple project stanzas for the same project the
resulting job graphs get merged. This should also work if they are
given as a mixture of canonical and non-canonical project names like
the following stanza.
- project:
name: review.example.com/org/project1
check:
jobs:
- common-config-job
- project:
name: org/project1
check:
jobs:
- project1-job
However currently only one of the stanzas are effective depending on
which was processed last.
This can be fixed by canonicalizing the project names when extending
the unparsed config.
Change-Id: Icaf8fca3aa4577b009d691f9a67adcb43ea040f5
- Normalize daemon process handling
Adopt some of the structure from nodepool to make daemon process
handling more consistent. Handle some argument parsing centrally.
Change the default pid file structure to match nodepool:
/var/run/zuul/<processname>
Attempt to use the pidfile before daemonizing so that errors are
immediately reported.
Drop the config validation test since it is almost useless at this
point.
Change-Id: I4a9d9473ce028e0b0cd32a8c48598c1682e1c329
- Merge "Remove unused function toList from scheduler" into feature/zuulv3
- Merge "Re-enable test_zuul_trigger_project_change_merged" into feature/zuulv3
- Merge "web: add Cache-Control to static files" into feature/zuulv3
- Merge "Use username from node information if available" into feature/zuulv3
- Fix branch checkout order
Since we're deprecating override-branch in favor of override-checkout,
favor override-checkout when deciding which to use.
Also, change an if to elif to make sure we only do one of these.
Change-Id: I55a94188d9894df27d99416cd3cefff8f3a64718
- github: add integration documentation
Change-Id: Ie3f8f9b68319d3f0c420b8c0770e8fce37f750e1
- web: add Cache-Control to static files
This change add the Cache-Control header to static files' response.
Change-Id: Ibdf1c35bad378507162d807cf5acdf13fc3fab88
- Combine branch templates and pipeline branch matchers
In a situation where a projec-template is defined on multiple branches
and then used by a project on multiple branches, the behavior was not
as one might expect.
Currently, assuming no explicit branch matchers are in play, all the
jobs listed in the project-template will get implied branch matchers
attached. So the resulting template looks like
- job @ master
- job @ stable/newton
- job @ stable/ocata
Then when that template is applied to the project, since the jobs within
already have branch matchers, the implied branch matcher for the project
pipeline definition is not applied. When the template is added to the
project on the ocata branch, all 3 of those jobs are added to the project,
and when it is added on the newton branch, all 3 jobs are added to the
project again. That's invisible to the user, until they attempt to remove
the template from one of the branches. Because the other branches still
add the template, which contains all the jobs, they still run.
To correct this, either replace the branch matcher obtained from the job
in the project template (if it is a simple match of a single branch), or
combine it using a boolean "and" (if it is something more complex) with a
branch matcher for the project definition. When the above template is added
to a project on the newton branch, only the following job will be added to
the project-pipeline:
- job @ stable/newton
When the template is added on the ocata branch, likewise only the ocata job
will be added.
If, instead, the project template had an explicit branch matcher, the
resulting template might be:
- job @ ^(?!stable/diablo).*$
After adding that to the project on the newton branch, the resulting project
pipeline would be:
- job @ { ^(?!stable/diablo).*$ AND stable/newton }
Ensuring that since the template was only added to the project on newton,
its jobs only run on newton changes.
Change-Id: I1969d588bc47b8ab5a54a885a68f98178b16b9d5
- Correctly stream the remaining buffer
In case ansible crashes while executing a command module we loose the
last line of the log. This is caused because we don't stream the
remaining buffer but the last line we split out of the buffer.
I wasn't able to construct a test case which makes the original code
break thus this change is without a test case.
Change-Id: I10ffe248c756e0a77f39f6e075dd3615c53f1b03
- Use username from node information if available
Nodepool knows the username that you should ssh with at image build time
and includes this information in the node data. Zuul should use this
username for the executor target.
Change-Id: I1e677061b9fd495b192d25a5825362c81e40d0c6
Depends-On: Ife0daa79f319aea04ed32513f99c73c460156941
- Add inventory variables for checkouts
Add override_checkout to the inventory variables, along with the
final checkout used for individual projects.
This can aid certain jobs like devstack which need to alter their
behavior in complex ways based on the branch the user intends to
test.
Change-Id: I58d94330e18670052fcd2d92b67a1a091d9249e6
- web: add /static installation instructions
Until a javascript client framework thing such as webpack is there, this change adds
the instructions to setup the /static directory
Change-Id: Ia2f899af97c3a8d430d2b1712ef26fb040fbbf6c
- Add support for shared ansible_host in inventory
Today it is possible to create the following ansible inventory file:
[foo]
foo01 ansible_host=192.168.1.1
[bar]
bar01 ansible_host=192.168.1.1
Which allows a user to create multiple host aliases for a single
connection. This could be done with ansible groups, however there is
some functional differences on how ansible runs in that configuration.
We could also request 2 nodes from nodepool, however in this case, it
would be a waste of CI resources because every alias would need a new
node from nodepool.
Now, a user is able to alias multiple host names to a single node from
nodepool by doing the following:
nodeset:
nodes:
- name:
- foo
- bar
label: ubuntu-xenial
This would result in a single node request from nodepool, but create
an inventory file with 2 alaises sharing and single ansible_host
variable.
Change-Id: I674d6baac26852ee1503feb1ed16c279bf773688
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Merge "Make enqueue-ref <new|old>rev optional" into feature/zuulv3
- Merge "More documentation for enqueue-ref" into feature/zuulv3
- Merge "Improve error handling in webapp /keys" into feature/zuulv3
- Merge "Remove zuul-migrate job" into feature/zuulv3
- Merge "Switch to threading model of socketserver" into feature/zuulv3
- Merge "web: add /{tenant}/status route" into feature/zuulv3
- Merge "web: add /tenants route" into feature/zuulv3
- Make enqueue-ref <new|old>rev optional
As described in the documentation in
Ibd7ee306bc461d1597c9c8febcaad89a48b9ff96 the new/old rev arguments
aren't required in all situations. For example when triggering a
periodic job and pretending to be the timer event, you just need the
refs/head/branch. For a release/tag replay, I believe you only need
the newref, so setting them both to zero by default triggers the check
incorrectly.
Thus we set the arguments to None intially and only if they are
explicitly set do we check they're not the same. If they're unset by
the user, we just set them to zero for the rpc calls.
Change-Id: I8125010efcf8d26dc3a99c87fe2c945314072b72
- More documentation for enqueue-ref
enqueue-ref didn't make it into the client documentation, add it with
some details about what it does, including some discussion of common
operations you might perform.
Change-Id: Ibd7ee306bc461d1597c9c8febcaad89a48b9ff96
- Remove tools/trigger-job.py
I don't believe this is submitting jobs correctly at all for v3
... while it may be fixable I think effort would be better spent
getting this into the client if we want the way to trigger a single
job (the enqueue / enqueue-ref will trigger *all* jobs ... which might
be enough anyway). My suggestion is to remove this to avoid
confusion.
Change-Id: I908657ca57471517a6f706980eaf5f04c15224ed
- web: add /{tenant}/status route
This change adds the 'status:get' job to the scheduler gearman worker
to expose the formatStatusJSON over gearman.
This change also copies the etc/status/public_html files to the zuul-web
static.
Change-Id: I46cec321f2e75b92ca02e430135a694216718948
- web: add /tenants route
This change adds the zuul:tenant_list to the scheduler gearman worker
to expose the list of tenants.
This change also adds the /tenants.json endpoint to the zuul-web as well as
a web interface to list tenants at / or /tenants.html.
Change-Id: Ia8edb52ec97ebe53205427c828944116eebe03b7
- Merge "Prime github app install map on connection load" into feature/zuulv3
- Merge "Make encrypt_secret.py work with OpenSSL 0.x" into feature/zuulv3
- Switch to threading model of socketserver
In an attempt to maybe get more information as to why the finger
daemon seems to be disappearing on us, try using threads instead
of forks. Alas, there is no spoon.
Change-Id: I473d874037fb81b940bfe71e5d6d1a17b131635f
- Make encrypt_secret.py work with OpenSSL 0.x
The format was changed in 1.0. This enables Mac OS's default openssl CLI
tool to work with encrypt_secret.py
Change-Id: Ib5d7a0c5cc6a729bed6fa4a64193444bb48022fb
- Prime github app install map on connection load
When scheduler comes online it will query every repo it knows about for
the branches of the repo. On GitHub this is done via the API. At this
point there wasn't a mapping of installation IDs to project names, so
the regular API was used as opposed to the app specific API, and that
chewed into the API limits rather quickly.
This change will preemptively query for every install known to the app,
and then every repo said install has access to, in order to build up the
map of installation IDs for the projects. It has a side effect of
priming the cache for app tokens as well. With this change when Zuul
asks for the branches of the projects it's far more likely that the
installation_id will be found within the cache.
Change-Id: I2173c089d816fa376ac9ca54addd3529d03e11d1
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
- Merge "Fix gerrit branch creation detection" into feature/zuulv3
- Fix gerrit branch creation detection
We detect when a new branch is created so that we know to perform
a reconfiguration which will include the new branch. However,
the detection for this case in the Gerrit driver still had the
pre-2.13 form of branch representation. Update it to support both
forms, and update the tests to use the new form.
Change-Id: I0480482a59a9f289ca4e449ce70fd1c0872e7999
- Improve error handling in webapp /keys
While investigating some problems with key loading, it was very
difficult to discern what exactly went wrong while requesting keys. One
important note is that if a project does not exist, the source drivers
just add it, which leads to 500 crashes later because those empty
projects don't have the public_key attribute.
Change-Id: Ic2eb0372620798dd7ca63e9e8205c39bd4629a92
- Merge "On reconfiguration, re-enqueue items at the same position" into feature/zuulv3
- Merge "Add support for override-checkout, deprecate override-branch" into feature/zuulv3
- On reconfiguration, re-enqueue items at the same position
Upon reconfiguration, we currently re-enqueue every item back
into its pipeline, in case a difference in the configuration would
change what should occur. In the simple case, this is fine, but
if a dependent pipeline has a branching dependency structure due
to already failing jobs, we would erroneously re-order the changes
back into a linear arrangement. The next pass through the pipeline
manager would move those items back to where they should be, but
in doing so, would reset their builds.
To correct this, after re-enqueueing a change, if the change that
was previously ahead of it in the pipeline was also successfully
re-enqueued, immediately move the change behind it (or if the
change ahead was "None" meaning it was its own head, move it behind
no change). This should have the effect of putting changes back
where they were in relation to other failing changes. If the change
ahead was not successfully re-enqueued, the current behavior of
simply putting it behind the nearest change in the queue is preserved.
If anything more complex happens, any errors will be corrected on the
next pass through the pipeline manager.
Change-Id: Ie3771d9bbbc1ca77425cf62751d8e5f70ba1f14c
- Merge "Check start time for wait_time key" into feature/zuulv3
- Merge "Use user home as work directory of executor" into feature/zuulv3
- Merge "Increase github delay to 10 seconds" into feature/zuulv3
- Use user home as work directory of executor
When we config non-root user as executor.user in zuul.conf,
we should switch to the user's home directory in order to avoid
"Permission deny" error, sometime we may launch zuul-executor
process in /root directory.
Change-Id: I20ddeace5822fa58235915d4629f3acd40e2a4b0
- Check start time for wait_time key
If the job didn't get started correctly, build.start_time may be None.
Check it before calculating the wait_time.
Change-Id: I61cc2d72cb56552ff56e517ef7bc87304d5d6eef
- Add support for override-checkout, deprecate override-branch
We want to support jobs specifying that they check out a tag
rather than merely a branch. This accidentally worked in Zuul v2
with zuul-cloner, and some jobs have come to rely on the behavior.
There's no reason not to support it, so let's do so.
However, for clarity, change the name of the option to
"override-checkout". This is intended to be intuitive (anything
you can tell git to 'checkout' you can put here), while avoiding
the suggestion that only branches or tags may be checked out.
The old form, "override-branch" is deprecated and will be removed.
Change-Id: Icc2907e72596626e96d2dc9f6ab1c3026f4085ab
- Remove zuul-migrate job
The job does not work anymore since it tests migration of files in
project-config that have been removed now.
Depends-On: Ic17bb17222068b1b5cf3ac57c4614af42a679961
Change-Id: Ide024a72d68112e76c114afe685ea7edf586e013
- Fix syntax with gear unRegisterFunction()
This is to fix the following exception:
2017-10-30 17:02:44,604 ERROR zuul.ExecutorServer: Exception in governor thread:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1883, in run_governor
self.manageLoad()
File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1897, in manageLoad
self.unregister_work()
File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1693, in unregister_work
self.executor_worker.unregisterFunction("executor:execute")
AttributeError: 'ExecutorExecuteWorker' object has no attribute 'unregisterFunction'
Change-Id: Ib898c4046b83829bf2a42b92897261956bc249bc
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Merge "Use cached branches on tenant reconfiguration" into feature/zuulv3
- Zuul: add file extension to playbook path
Zuul now supports including the file extension on the playbook path
and omitting the extension is now deprecrated. Update references
to include the extension.
Change-Id: I855444b69e037b1f6a01dac65e700a2a26e8be2b
- Merge "Do late decoding of log stream buffer" into feature/zuulv3
- Use cached branches on tenant reconfiguration
It takes about one minute for us to query gerrit for all the project
branches, so don't do that on tenant reconfiguration, except for
the project which prompted the reconfiguration (in case the event
was a branch creation or deletion).
Also, clean up some unused arguments to getProject methods.
Change-Id: I625dfc6af00d8d480fcb6bac5d0b86e4846844c0
- Merge "Add multi-branch support for project-templates" into feature/zuulv3
- Merge "Validate that a job has a run playbook on freeze" into feature/zuulv3
- Merge "Remove implied run" into feature/zuulv3
- Merge "Update test fixtures to use explicit run" into feature/zuulv3
- Add multi-branch support for project-templates
Currently, to avoid confusion, we simply ignore project-template
definitions after the first. However, a more intuitive approach
would be, if the template appears on multiple branches, to have
it reflect those branches.
To that end, add implied branch matchers to templates in the same
way that we do to jobs themselves. This should provide the same
intuitive behavior where a template defined in zuul-jobs will apply
to all branches, but one defined in a multi-branch repository will
add jobs with implied branch matchers.
When a template is defined more than once, combine them (in the same
way that multiple project definitions are defined) so the resulting
template contains all the jobs (likely with implied branch matchers).
Allow a template to be defined multiple times (e.g., branches) within
the same project, but do not allow it to be redefined in another
project.
Because the multiple project definitions in different branches may
end up applying the same template (which will have all of the
per-branch jobs) multiple times, detect duplicate job definitions
when applying templates and filter them out.
The test test_dynamic_template was originally written to verify that
a project could not redefine a template defined in another project.
Clarify that, and update it to support the newly reported error in
that condition.
Change-Id: I6613885a8c7ebf400e85041e0d68b2eb5ceb033f
- Increase github delay to 10 seconds
In testing with a GitHub Enterprise installation, 3s is sometimes not
enough for the API to update after an event webhook.
Change-Id: I36a7c465afa6e4232addf6fbc29b8a9c62f41d3f
- Merge "Support file extension in playbook path" into feature/zuulv3
- Merge "Add pragma directive" into feature/zuulv3
- Merge "Move test_job_auth_inheritance to test_v3" into feature/zuulv3
- Merge "Switch to late-binding inheritance" into feature/zuulv3
- Merge "Add implied branch matchers on 'master'" into feature/zuulv3
- Merge "Remove test_job_inheritance" into feature/zuulv3
- Merge "Move test_model.test_job_inheritance_configloader" into feature/zuulv3
- Do late decoding of log stream buffer
The log stream is read in chunked blocks. When having multi byte
unicode characters in the log stream it can happen that this character
is split into different buffers. This can break the decode step with
an exception [1]. This can be fixed by treating the buffer as binary
and decoding the final lines.
Further we must expect that the data also contains binary data. In
order to cope with this further harden the final decoding by adding
'backslashreplace'. This will replace every occurrence of an
undecodable character by an appropriate escape sequence. This way we
can retain all the information (even binary) without being unable to
decode the stream.
[1]: Log output
Ansible output: b'Exception in thread Thread-10:'
Ansible output: b'Traceback (most recent call last):'
Ansible output: b' File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner'
Ansible output: b' self.run()'
Ansible output: b' File "/usr/lib/python3.5/threading.py", line 862, in run'
Ansible output: b' self._target(*self._args, **self._kwargs)'
Ansible output: b' File "/var/lib/zuul/ansible/zuul/ansible/callback/zuul_stream.py", line 140, in _read_log'
Ansible output: b' more = s.recv(4096).decode("utf-8")'
Ansible output: b"UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data"
Ansible output: b''
Change-Id: I568ede2a2a4a64fd3a98480cebcbc2e86c54a2cf
- Validate that a job has a run playbook on freeze
We now know, before attempting to run a job, whether the inheritance
hierarchy has produced a main playbook for the job. If there is none,
error early.
Also, in the executor, assume that any specified playbooks are
required to exist, and use the more specific version of the error
message if they don't.
Change-Id: Id7dc5934c665cf939820b12b5ded53adeb60c0a8
- Remove implied run
With multiple levels of inheritance, the optional implied run
feature is making it difficult to figure out which jobs run
playbooks and which don't. Make the run attribute explicitly
required to aid in human understanding of the configuration.
Change-Id: Ia8f23bce9898cd4f387554e6787b091b63e75519
- Update test fixtures to use explicit run
Change-Id: I3060a2bf57cef10a5a7ec5299e3491f1f6751221
- Support file extension in playbook path
This allows us to specify the full playbook name, with file extension.
A documentation update is included in a following patch, as well as
a deprecation notice for the old form.
Change-Id: I19a3ec50e473f717a7e5e2824f702ad4e6acab02
- Add pragma directive
This allows the user to override the implied branch matcher behavior.
Change-Id: I3ef43fd868988666cb01e8a6bb28552cc42151b4
- Move test_job_auth_inheritance to test_v3
Move this into configuration files so that we can test the
functionality end-to-end rather than relying on internal APIs
which are frequently…
silverdaz
pushed a commit
to NBISweden/LocalEGA
that referenced
this issue
Jan 22, 2018
Jagrmi-C
added a commit
to Jagrmi-C/osm-validator
that referenced
this issue
Jan 22, 2018
move from requirements.txt to https://docs.pipenv.org/ keep locked dependencies on Pipfile.lock fix dependencies: aio-libs/aiohttp#2662 delete requirements.txt and requirements-dev.txt
DevAlone
added a commit
to DevAlone/proxy_py
that referenced
this issue
Jan 28, 2018
colin-nolan
pushed a commit
to wtsi-hgi/irobot
that referenced
this issue
Feb 6, 2018
This was referenced Mar 15, 2018
ibizaman
added a commit
to ibizaman/jsondispatch
that referenced
this issue
May 20, 2018
viklund
pushed a commit
to NBISweden/LocalEGA
that referenced
this issue
Nov 22, 2018
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Long story short
As of yarl 1.0.0 (available since one hour ago), yarl.unquote is no longer available, thus causing an import error at web_urldispatcher.py.
Expected behaviour
Been able to run:
from aiohttp import webActual behaviour
from aiohttp import webraises the following error:Steps to reproduce
Simply:
from aiohttp import webYour environment
aiohttp 2.3.7 (both)
Ubuntu Linux 16.04
Python 3.6.3
Workaround
Downgrade to yarl 0.18.0
The text was updated successfully, but these errors were encountered: