Update pip config

[MarkSymsCtx]

I'm currently running some tests on this to check it's not regressed anything

[ydirson]

I think we should keep the dev_requirements.txt, it is really much easier to start working on the code by running pip in a venv, than having to lookup in the github actions which packages to install and do so manually.

[MarkSymsCtx]

I think we should keep the dev_requirements.txt, it is really much easier to start working on the code by running pip in a venv, than having to lookup in the github actions which packages to install and do so manually.

We have a corporate security directive to avoid use of pip in general and source repositories are regularly scanned to nag about the presence of pip configuration information. These files got missed in earlier clean up activities.

[ydirson]

We have a corporate security directive to avoid use of pip in general and source repositories are regularly scanned to nag about the presence of pip configuration information.

Would it more acceptable in a setup.cfg ?

[MarkSymsCtx]

We have a corporate security directive to avoid use of pip in general and source repositories are regularly scanned to nag about the presence of pip configuration information.

Would it more acceptable in a setup.cfg ?

No, they get complained about as well. Basically anything with pip is deemed to be insecure and a problem with supply chain verification.

[ydirson]

We have a corporate security directive to avoid use of pip in general and source repositories are regularly scanned to nag about the presence of pip configuration information.

Would it more acceptable in a setup.cfg ?

No, they get complained about as well. Basically anything with pip is deemed to be insecure and a problem with supply chain verification.

I'm sorry corporate directives impair ability for devs inside Xenserver to work efficiently (even more so as I've endured such rules in past jobs), but surely we can find a way to provide dependency information in a standard machine-parsable way a public project?

[MarkSymsCtx]

We have a corporate security directive to avoid use of pip in general and source repositories are regularly scanned to nag about the presence of pip configuration information.

Would it more acceptable in a setup.cfg ?

No, they get complained about as well. Basically anything with pip is deemed to be insecure and a problem with supply chain verification.

I'm sorry corporate directives impair ability for devs inside Xenserver to work efficiently (even more so as I've endured such rules in past jobs), but surely we can find a way to provide dependency information in a standard machine-parsable way a public project?

It doesn't actually impair us at all as we use the RPM build mock to run this stuff and that just derives everything it needs from the specfile.

[ydirson]

But then, not providing machine-parsable dependency info is impairing the community - providing them to the community does not force their internal use.

[gwd]

@MarkSymsCtx As an open-source project under the umbrella of the Xen Project, it's important that there be a clear, practical way for external contributors to build and contribute xapi. Is such a method documented anywhere?

Also, are there any other upstream projects that XenServer either consumes or contributes to, which have setup.cfg or dev_requirements.txt files? How are those dealt with WRT the corporate anti-pip policy?

@ydirson Would it be practical for you to use mock to do building as well? Or alternately, would a Docker file or a vagrant file with the relevant dependencies be suitable?

[ydirson]

Would it be practical for you to use mock to do building as well? Or alternately, would a Docker file or a vagrant file with the relevant dependencies be suitable?

Mock is useful for build reproducibility of RPMs, but before we're at rpm-building stage we should be able to easily run unit tests, which can only run when we have the proper dependencies installed. Sure, the SM build scripts today ensure that unit tests and even pylint are run in the mock environment on each build, but this is much too costly for everyday use, for example when working on a specific failed test.

Further more, allowing users to use pip makes it easy to test with different versions of dependencies and checker tools (in separate python virtual environments), which in turn allows to further improve the project quality.

[MarkSymsCtx]

Fine requirements file reinstated but we will not be maintaining it or keeping it updated (not least because we have no environment in which we could test that it is valid and/or useful) so it will most likely become stale quite quickly.

[ydirson]

Keeping it used by github actions would likely be the best option, that would ensure the info in there is still tested - and would avoid getting it duplicated in 2 places.

[ydirson]

I'm surprised when running pylint locally (fresh venv on CentOS7, populated with same commands as github actions):

• when including tests/mocks/ in PYTHONPATH as the github actions do, I get 7 occurrences of Module 'XenAPI' has no 'xapi_local' member. git grep XenAPI.xapi_local shows them all (and more, which show that the files not named *.py are apparently skipped by pylint).
• aside from those I also have drivers/on_slave.py:130:4: E0401: Unable to import 'XenAPIPlugin' (import-error)

Any idea why those errors don't appear in the CI ?

[MarkSymsCtx]

I'm surprised when running pylint locally (fresh venv on CentOS7, populated with same commands as github actions):

• when including tests/mocks/ in PYTHONPATH as the github actions do, I get 7 occurrences of Module 'XenAPI' has no 'xapi_local' member. git grep XenAPI.xapi_local shows them all (and more, which show that the files not named *.py are apparently skipped by pylint).
• aside from those I also have drivers/on_slave.py:130:4: E0401: Unable to import 'XenAPIPlugin' (import-error)

Any idea why those errors don't appear in the CI ?

Unfortunately not, if you do get to the bottom of please send an update PR to rectify the issues.

[MarkSymsCtx]

Actually, the xapi_local complaint is probably because https://github.com/xapi-project/sm/blob/master/tests/mocks/XenAPI/__init__.py does indeed not have the xapi_local member. That code should actually get moved to a tests/mocks/XenAPI.py file and have the xapi_local method declared as a no-op for testing purposes.

Although actually making this change makes the unit tests fail through a lack of coverage.

[ydirson]

XenAPI/init.py does indeed not have the xapi_local member.

Yes, that's why my question was why it indeed works on the CI, and I'm still in the dark there.