bpo-37085: Expose SocketCAN bcm_msg_head flags #13646

karlding · 2019-05-29T07:25:53Z

Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct
flags (provided by <linux/can/bcm.h>) under the socket library.

This adds the following constants with a CAN_BCM prefix:

SETTIMER
STARTTIMER
TX_COUNTEVT
TX_ANNOUNCE
TX_CP_CAN_ID
RX_FILTER_ID
RX_CHECK_DLC
RX_NO_AUTOTIMER
RX_ANNOUNCE_RESUME
TX_RESET_MULTI_IDX
RX_RTR_FRAME
CAN_FD_FRAME

https://bugs.python.org/issue37085

auvipy

tests?

karlding · 2019-05-29T17:42:34Z

Could you clarify what you mean, @auvipy?

These are just binding the C #defines, so do you want me to test that these values are some known value? If so, wouldn't that just be testing the Linux implementation?

Thanks!

Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct flags (provided by <linux/can/bcm.h>) under the socket library. This adds the following constants with a CAN_BCM prefix: * SETTIMER * STARTTIMER * TX_COUNTEVT * TX_ANNOUNCE * TX_CP_CAN_ID * RX_FILTER_ID * RX_CHECK_DLC * RX_NO_AUTOTIMER * RX_ANNOUNCE_RESUME * TX_RESET_MULTI_IDX * RX_RTR_FRAME * CAN_FD_FRAME

taleinat

This certainly needs tests, at least checking availability as done for the other socket.CAN_BCM_* constants in test_socket.BasicCANTest.testBCMConstants().

taleinat · 2019-07-23T09:27:10Z

Modules/socketmodule.c

@@ -7668,6 +7668,42 @@ PyInit__socket(void)
    PyModule_AddIntConstant(m, "CAN_BCM_RX_STATUS", RX_STATUS);
    PyModule_AddIntConstant(m, "CAN_BCM_RX_TIMEOUT", RX_TIMEOUT);
    PyModule_AddIntConstant(m, "CAN_BCM_RX_CHANGED", RX_CHANGED);
+#ifdef  SETTIMER


Why these separate #ifdef checks for each constant? Are they not either all available together, or not? More specifically, isn't the #ifdef HAVE_LINUX_CAN_BCM_H check enough to ensure their availability?

This is not necessarily true for all the constants, namely the backwards-compatible introduction of the CAN_FD_FRAME flag, which was added in 6f3b911d5f29b98752e5da86a295210c0c4f4e14 as part of the the 4.8.x kernel series.

The other constants were always present since the SocketCAN drivers were mainlined in 2.6.25.

Should I just have the #ifdef on CAN_FD_FRAME then? I had added the other ones for consistency.

Should I just have the #ifdef on CAN_FD_FRAME then? I had added the other ones for consistency.

IMO that would be better, yes. And perhaps add a note about the availability of CAN_FD_FRAME in the docs.

bedevere-bot · 2019-07-23T09:28:49Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other ones were present since SocketCAN drivers were mainlined in 2.6.25. As such, it is probably unnecessary to guard against these constants being missing.

The CAN_BCM_CAN_FD_FRAME constant mirrors the availability in the Linux kernel. This constant was introduced in Linux 4.8 kernels.

karlding · 2019-07-23T19:49:42Z

Lib/test/test_socket.py

+        socket.CAN_BCM_TX_RESET_MULTI_IDX
+        socket.CAN_BCM_RX_RTR_FRAME
+
+    @unittest.skipUnless(hasattr(socket, "CAN_BCM_CAN_FD_FRAME"),


Is it better to parse the kernel version from platform.release as a way of checking whether this exists?

Yes, using the test.support.requires_linux_version decorator.

And don't forget to also use the same decorator as above, i.e.:

@unittest.skipUnless(hasattr(socket, "CAN_BCM"), 'socket.CAN_BCM required for this test.') @requires_linux_version(4, 8) def testBCMConstantsSupportsFD(self): # ...

Also the method name for the separate test is unclear to me ("supports FD"). Either of testBCMConstantsLinux48 or testBCMConstantsFD seems better.

Use support.requires_linux_version to guard BCM flag CAN_FD_FRAME, which is only present on Linux >= 4.8.

karlding · 2019-07-23T22:43:46Z

I have made the requested changes; please review again

bedevere-bot · 2019-07-23T22:43:49Z

Thanks for making the requested changes!

@taleinat: please review the changes made to this pull request.

taleinat · 2019-07-24T07:01:43Z

Misc/NEWS.d/next/Library/2019-06-18-16-29-31.bpo-37085.GeYaD6.rst

@@ -0,0 +1 @@
+Added support for the Linux SocketCAN Broadcast Manager API constants.


Some BCM constants were already there; perhaps more precisely describe the nature of the added constants/flags.

taleinat · 2019-07-24T07:02:49Z

Thanks, @karlding! This is looking good, I have just a small comment on the NEWS entry.

taleinat · 2019-07-25T19:40:04Z

The CI tests are failing due to the CAN_BCM_CAN_FD_FRAME flag being missing, on Linux (specifically, Ubuntu 16.04 on both Travis and Azure Pipelines).

taleinat · 2019-07-25T20:01:57Z

The CI tests are failing due to the CAN_BCM_CAN_FD_FRAME flag being missing, on Linux (specifically, Ubuntu 16.04 on both Travis and Azure Pipelines).

I ran the tests with this PR on an Ubuntu 18.10 installation and they passed. Perhaps get access to a host running Ubuntu 16.04 to reproduce and diagnose the issue.

karlding · 2019-07-28T06:18:39Z

So correct me if I'm wrong, but I believe this is because the Xenial headers included in linux-libc-dev for userspace (ie. glibc) in the repos are for an older version (4.4.x) of the kernel.

Looking at /usr/include/linux/version.h for a clean 16.04 install, we see:

#define LINUX_VERSION_CODE 263335

which corresponds to the following (4.4.167):

(((4) << 16) + ((4) << 8) + (167))

And even after updating linux-libc-dev, we see

#define LINUX_VERSION_CODE 263349

which corresponds to 4.4.181:

(((4) << 16) + ((4) << 8) + (181))

Despite running a 4.15 kernel:

karl@karl-VirtualBox:/usr/include$ uname -a
Linux karl-VirtualBox 4.15.0-45-generic #48~16.04.1-Ubuntu SMP Tue Jan 29 18:03:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I'm not too familiar with packaging, but if I'm reading this Ubuntu kernel release schedule page correctly, this seems to indicate that they ship 4.4.x headers.

And according to this AskUbuntu post, in Ubuntu LTS releases, it appears that they are not upgraded with the HWE kernels. So while the kernel may support it, the headers provided by the repos don't provide the necessary userspace API.

taleinat · 2019-07-28T08:14:03Z

@karlding, great research and write-up!

I think the existing implementation approach, exposing the flags only when they are available at the C level, is good. It's just the tests that need to be fixed.

In this case even just not testing for this flag would be fine IMO, but here I'd definitely want another core dev's opinion. So please remove it from the test, make sure tests are passing, and we'll wait for another reviewer.

@serhiy-storchaka, any chance for a quick review of this, perhaps?

This constant doesn't exist on Ubuntu 16.04 LTS, since linux-libc-dev ships 4.4.x headers from the repos, while the feature was added in the 4.8.x kernel series. However, Ubuntu 16.04 supports the 4.15.x kernel series, but since linux-libc-dev isn't updated past 4.4.x headers, there is no guarantee that /usr/include/linux/can/bcm.h contains the proper constant for the userspace API, despite having a supported kernel.

taleinat · 2019-07-29T07:14:26Z

@karlding, which current OS-s do support the CAN_BCM_CAN_FD_FRAME and provide it in their headers? Perhaps we can recognize at least one or two such OS-s and limit the test for the flag to those. Even if they're not one of those triggered here on GitHub, they're likely found in our buildbot fleet.

serhiy-storchaka

LGTM, just fix some spacing.

serhiy-storchaka · 2019-07-29T07:51:41Z

Modules/socketmodule.c

+    PyModule_AddIntConstant(m, "CAN_BCM_RX_ANNOUNCE_RESUME", RX_ANNOUNCE_RESUME);
+    PyModule_AddIntConstant(m, "CAN_BCM_TX_RESET_MULTI_IDX", TX_RESET_MULTI_IDX);
+    PyModule_AddIntConstant(m, "CAN_BCM_RX_RTR_FRAME", RX_RTR_FRAME);
+#ifdef  CAN_FD_FRAME


Suggested change

#ifdef CAN_FD_FRAME

#ifdef CAN_FD_FRAME

serhiy-storchaka · 2019-07-29T07:53:11Z

Doc/library/socket.rst

@@ -374,6 +374,9 @@ Constants

   .. availability:: Linux >= 2.6.25.

+   .. note::
+    The :data:`CAN_BCM_CAN_FD_FRAME` flag is only available on Linux >= 4.8.


Suggested change

The :data:`CAN_BCM_CAN_FD_FRAME` flag is only available on Linux >= 4.8.

The :data:`CAN_BCM_CAN_FD_FRAME` flag is only available on Linux >= 4.8.

serhiy-storchaka · 2019-07-29T08:03:17Z

AFAIK, the only OS besides Linux which supports CAN_BCM is NetBSD starting from version 8. But our NetBSD buildbots were broken and removed a long time ago.

taleinat · 2019-07-29T08:21:35Z

AFAIK, the only OS besides Linux which supports CAN_BCM is NetBSD starting from version 8. But our NetBSD buildbots were broken and removed a long time ago.

I was thinking of other Linux distros. Actually, I myself checked that it is available on Ubuntu 18.10 (see previous comment). I suggest to check on 18.04 (since it's LTS), and if found there, enable the removed test just on Ubuntu >= 18.04.

karlding · 2019-07-30T05:48:57Z

I suggest to check on 18.04 (since it's LTS), and if found there, enable the removed test just on Ubuntu >= 18.04.

I checked on Ubuntu 18.04, and after installing linux-libc-dev I see that the constant is present.

karl@karl-VirtualBox:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic
karl@karl-VirtualBox:~$ tail -n5 /usr/include/linux/can/bcm.h
#define TX_RESET_MULTI_IDX  0x0200
#define RX_RTR_FRAME        0x0400
#define CAN_FD_FRAME        0x0800

#endif /* !_UAPI_CAN_BCM_H */

With that being said, is there a non-deprecated way to check for Ubuntu versions (ie. besides platform.linux_distribution)? Or could you point me to an example or any documentation where you're only running the test on a particular builder type?

Fix the sphinx warning message that is causing the CI failure: Content block expected for the "note" directive; none found. This is done by increasing the indentation to place the text in a block.

karlding · 2019-07-30T06:03:27Z

The constant should also be available on Debian Stretch and beyond, since linux-libc-dev seems to provide 4.9.x headers in their repos.

taleinat · 2019-07-30T06:16:10Z

With that being said, is there a non-deprecated way to check for Ubuntu versions (ie. besides platform.linux_distribution)?

That's a good question, I hadn't realized it had been deprecated without any replacement.

Or could you point me to an example or any documentation where you're only running the test on a particular builder type?

No, I can't find an example of this in our codebase.

serhiy-storchaka

I think it can be merged without test for CAN_BCM_CAN_FD_FRAME.

karlding · 2019-07-30T16:01:24Z

Also, should this be backported, since the docs (incorrectly) indicate that all the BCM constants are available on older versions?

serhiy-storchaka · 2019-07-31T08:06:39Z

It may be backported to 3.8 (ask @ambv for this), but not to 3.7.

bedevere-bot · 2019-07-31T08:47:19Z

@ambv: Please replace # with GH- in the commit message next time. Thanks!

miss-islington · 2019-07-31T08:47:19Z

Thanks @karlding for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

bedevere-bot · 2019-07-31T08:47:37Z

GH-15049 is a backport of this pull request to the 3.8 branch.

Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct flags (provided by <linux/can/bcm.h>) under the socket library. This adds the following constants with a CAN_BCM prefix: * SETTIMER * STARTTIMER * TX_COUNTEVT * TX_ANNOUNCE * TX_CP_CAN_ID * RX_FILTER_ID * RX_CHECK_DLC * RX_NO_AUTOTIMER * RX_ANNOUNCE_RESUME * TX_RESET_MULTI_IDX * RX_RTR_FRAME * CAN_FD_FRAME The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other ones were present since SocketCAN drivers were mainlined in 2.6.25. As such, it is probably unnecessary to guard against these constants being missing. (cherry picked from commit 31c4fd2) Co-authored-by: karl ding <karlding@users.noreply.github.com>

taleinat · 2019-07-31T09:08:08Z

Shouldn't we add a note about this in "What's New"?

Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct flags (provided by <linux/can/bcm.h>) under the socket library. This adds the following constants with a CAN_BCM prefix: * SETTIMER * STARTTIMER * TX_COUNTEVT * TX_ANNOUNCE * TX_CP_CAN_ID * RX_FILTER_ID * RX_CHECK_DLC * RX_NO_AUTOTIMER * RX_ANNOUNCE_RESUME * TX_RESET_MULTI_IDX * RX_RTR_FRAME * CAN_FD_FRAME The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other ones were present since SocketCAN drivers were mainlined in 2.6.25. As such, it is probably unnecessary to guard against these constants being missing. (cherry picked from commit 31c4fd2) Co-authored-by: karl ding <karlding@users.noreply.github.com>

Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct flags (provided by <linux/can/bcm.h>) under the socket library. This adds the following constants with a CAN_BCM prefix: * SETTIMER * STARTTIMER * TX_COUNTEVT * TX_ANNOUNCE * TX_CP_CAN_ID * RX_FILTER_ID * RX_CHECK_DLC * RX_NO_AUTOTIMER * RX_ANNOUNCE_RESUME * TX_RESET_MULTI_IDX * RX_RTR_FRAME * CAN_FD_FRAME The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other ones were present since SocketCAN drivers were mainlined in 2.6.25. As such, it is probably unnecessary to guard against these constants being missing.

the-knights-who-say-ni added the CLA signed label May 29, 2019

bedevere-bot added the awaiting review label May 29, 2019

auvipy suggested changes May 29, 2019

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels May 29, 2019

brettcannon added the type-feature A feature request or enhancement label Jun 3, 2019

karlding force-pushed the issue-37085_add_bcm_msg_head_flags_socket_constants branch from baab234 to 6d1d880 Compare June 18, 2019 23:34

taleinat requested changes Jul 23, 2019

View reviewed changes

bedevere-bot removed the awaiting core review label Jul 23, 2019

bedevere-bot added the awaiting changes label Jul 23, 2019

karlding added 3 commits July 23, 2019 11:44

bpo-37085: Only use #ifdef for CAN_FD_FRAME

604bea9

The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other ones were present since SocketCAN drivers were mainlined in 2.6.25. As such, it is probably unnecessary to guard against these constants being missing.

bpo-37085: Add test for CAN_BCM_* flag constants

9e0b9c7

Add note in docs about CAN_BCM_CAN_FD_FRAME

28662f1

The CAN_BCM_CAN_FD_FRAME constant mirrors the availability in the Linux kernel. This constant was introduced in Linux 4.8 kernels.

karlding commented Jul 23, 2019

View reviewed changes

bpo-37085: Use support.requires_linux_version

b969e22

Use support.requires_linux_version to guard BCM flag CAN_FD_FRAME, which is only present on Linux >= 4.8.

bedevere-bot added awaiting change review and removed awaiting changes labels Jul 23, 2019

taleinat reviewed Jul 24, 2019

View reviewed changes

bpo-37085: Update NEWS entry

128c1c1

serhiy-storchaka reviewed Jul 29, 2019

View reviewed changes

taleinat added 2 commits July 29, 2019 11:14

fix whitespace as per Serhiy's review

0800558

reword NEWS entry

f1d212d

Fix sphinx note directive warning in socket docs

b28857f

Fix the sphinx warning message that is causing the CI failure: Content block expected for the "note" directive; none found. This is done by increasing the indentation to place the text in a block.

serhiy-storchaka approved these changes Jul 30, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Jul 30, 2019

ambv added the needs backport to 3.8 label Jul 31, 2019

ambv merged commit 31c4fd2 into python:master Jul 31, 2019

bedevere-bot removed awaiting merge needs backport to 3.8 labels Jul 31, 2019

		@@ -0,0 +1 @@
		Added support for the Linux SocketCAN Broadcast Manager API constants.

	The :data:`CAN_BCM_CAN_FD_FRAME` flag is only available on Linux >= 4.8.
	The :data:`CAN_BCM_CAN_FD_FRAME` flag is only available on Linux >= 4.8.

bpo-37085: Expose SocketCAN bcm_msg_head flags #13646

bpo-37085: Expose SocketCAN bcm_msg_head flags #13646

Conversation

karlding commented May 29, 2019 • edited by bedevere-bot Loading

auvipy left a comment

Choose a reason for hiding this comment

karlding commented May 29, 2019

taleinat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taleinat Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

bedevere-bot commented Jul 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlding commented Jul 23, 2019

bedevere-bot commented Jul 23, 2019

Choose a reason for hiding this comment

taleinat commented Jul 24, 2019

taleinat commented Jul 25, 2019

taleinat commented Jul 25, 2019

karlding commented Jul 28, 2019

taleinat commented Jul 28, 2019

taleinat commented Jul 29, 2019

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka commented Jul 29, 2019

taleinat commented Jul 29, 2019 • edited Loading

karlding commented Jul 30, 2019

karlding commented Jul 30, 2019

taleinat commented Jul 30, 2019

serhiy-storchaka left a comment

Choose a reason for hiding this comment

karlding commented Jul 30, 2019

serhiy-storchaka commented Jul 31, 2019

bedevere-bot commented Jul 31, 2019

miss-islington commented Jul 31, 2019

bedevere-bot commented Jul 31, 2019

taleinat commented Jul 31, 2019

karlding commented May 29, 2019 •

edited by bedevere-bot

Loading

taleinat Jul 23, 2019 •

edited

Loading

taleinat commented Jul 29, 2019 •

edited

Loading