Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I2C regression, last working commit is 43998c82a7e88df284b7aa30221b2d0d21b2b86a #261

Closed
VigibotDev opened this issue Mar 16, 2021 · 27 comments

Comments

@VigibotDev
Copy link

Hello,

With last raspbian apt update I get an I2C kernel problem on every platform (all RPI version) with :

RTIMULib IMU reads become unreliable (random corruption and FIFO overflow) on all RPI robotics platform >= kernel 5.10.14
RTIMULib IMU reads work perfectly on all RPI robotics platform <= kernel 5.10.11

Last working commit is 43998c8 (kernel: spi: bcm2835: Workaround/fix for zero-length transfers) Now we use this firmware commit ID on our official system image.

I2C regression can be inside :

@pelwell
Copy link
Collaborator

pelwell commented Mar 16, 2021

Can you confirm that between working and non-working systems vcgencmd version returns the same result?

@pelwell
Copy link
Collaborator

pelwell commented Mar 16, 2021

Or to word it better, can you confirm that the same version of the firmware can be found in the last working and first non-working system?

@VigibotDev
Copy link
Author

Last Raspbian stable, I2C NOT working :
Our fresh continuous integration image (with apt update / upgrade, rpi-update never used)

(root|~) uname -a
Linux raspberrypi 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l GNU/Linux
(root|~) vcgencmd version
Feb 25 2021 12:13:12
Copyright (c) 2012 Broadcom
version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start_x)

OLDEST version with I2C NOT WORKING :
rpi-update 115e3a5

(root|~) uname -a
Linux raspberrypi 5.10.14-v7+ #1401 SMP Mon Feb 8 14:23:24 GMT 2021 armv7l GNU/Linux
(root|~) vcgencmd version
Feb  8 2021 14:35:41
Copyright (c) 2012 Broadcom
version 73b3cad64181954e67f6e9fe6d275378d3eefa10 (clean) (release) (start_x)

LAST version with I2C working :
rpi-update 43998c8

(root|~) uname -a
Linux raspberrypi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/Linux
(root|~) vcgencmd version
Jan 27 2021 22:20:57
Copyright (c) 2012 Broadcom
version 99d9a48302e4553cff3688692bb7e9ac760a03fa (clean) (release) (start_x)

@pelwell
Copy link
Collaborator

pelwell commented Mar 16, 2021

Thanks. I'd dearly love it to not be my "untimely DONE signal" patch...

If you don't mind a bit of experimentation, starting with the last working commit (43998c8), do a sudo SKIP_KERNEL=1 rpi-update 115e3a5 to just update the firmware.

@VigibotDev
Copy link
Author

VigibotDev commented Mar 17, 2021

Hi

The next firmware version with the old kernel work, it's a success for you but please check if I understand all (my english is aproximative) :

From the last-working & complete version on one robot;
I returned to the oldest non-working version + kernel-update skip as you want, please check the complete trace with some "vcgencmd version" proof of firmware changing without kernel change :

(root|~) vcgencmd version
Jan 27 2021 22:27:56
Copyright (c) 2012 Broadcom
version 99d9a48302e4553cff3688692bb7e9ac760a03fa (clean) (release) (start_x)
(root|~) SKIP_KERNEL=1 rpi-update 115e3a5f77488d9ee30a33bcb5ac31eb587f60a8
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
#############################################################
WARNING: This update bumps to rpi-5.10.y linux tree
See: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=288234
'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.
 
DO NOT use 'rpi-update' as part of a regular update process.
 
##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   168  100   168    0     0   1976      0 --:--:-- --:--:-- --:--:--  2000
100  119M    0  119M    0     0  3957k      0 --:--:--  0:00:31 --:--:-- 4602k
 *** Updating firmware
 *** As requested, not updating kernel
 *** As requested, not updating kernel modules
 *** Updating VideoCore libraries
 *** Using HardFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to 115e3a5f77488d9ee30a33bcb5ac31eb587f60a8
 *** A reboot is needed to activate the new firmware
(root|~) uname -a
Linux raspberrypi 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l GNU/Linux
(root|~) reboot
Connection to 192.168.0.28 closed by remote host.
Connection to 192.168.0.28 closed.
(root|~) ssh 192.168.0.28
root@192.168.0.28's password:
Linux raspberrypi 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Mar 17 08:49:42 2021 from 192.168.0.2
(root|~) uname -a
Linux raspberrypi 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l GNU/Linux
(root|~) vcgencmd version
Feb  8 2021 14:35:41
Copyright (c) 2012 Broadcom
version 73b3cad64181954e67f6e9fe6d275378d3eefa10 (clean) (release) (start_x)
(root|~)  

It's good ?
Now can we update to the last master or stable firmware + old kernel to get the last up to date and best working ?

@VigibotDev
Copy link
Author

Or just ask for more tests...
I have a small fleet of 8 RPI robots dedicated to stability testing.
All 24/7 run, with H.264 low latency realtime video/audio with OpenCV graph overlay + multiples I2C sensors + UART LIDAR on the PI main UART @ 230K bauds + HPWM local audio + DMA PWM generators (PIGPIO) to run motors with mosfets.

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

Thank you for doing that - you've narrowed the problem down to a kernel change in the 5.10.14 update. As you thought, sudo SKIP_KERNEL=1 rpi-update stable will install the latest stable firmware but preserve the 5.10.11 kernel.

I'm tempted to revert that one commit I'm suspicious about, but I want to discuss with colleagues if there is a better way to distribute trial releases rather than change it for everybody.

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

What is the IMU device(s) you are using? It would be useful to get some for testing. Also, do you a very simple test script that demonstrates the problem?

@VigibotDev
Copy link
Author

Now I try to make the bug visible out of our Vigibot.com software stack :

On Vigibot we read the I2C bus from two simultaneous process :

  • The main Node.js websocket client process (for battery gauge / voltage read) no I2C problem appear with it.
  • And the optional fast C++ process for LIDAR (SLAM) and IMU I2C read. Where the IMU corrupt I2C frame problem appear "I2C read error from 104, 116 - Failed to read fifo data" and some time "mpu-9250 fifo has overflowed".

On Vigibot we use RTIMULib official binaries from Raspbian repo.

(root|~/RTIMULib/Linux/RTIMULibDrive) make
g++ -c -o objects/RTIMULibDrive.o RTIMULibDrive.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTMath.o ../../RTIMULib/RTMath.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUHal.o ../../RTIMULib/RTIMUHal.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTFusion.o ../../RTIMULib/RTFusion.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTFusionKalman4.o ../../RTIMULib/RTFusionKalman4.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTFusionRTQF.o ../../RTIMULib/RTFusionRTQF.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUSettings.o ../../RTIMULib/RTIMUSettings.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUAccelCal.o ../../RTIMULib/RTIMUAccelCal.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUMagCal.o ../../RTIMULib/RTIMUMagCal.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMU.o ../../RTIMULib/IMUDrivers/RTIMU.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUNull.o ../../RTIMULib/IMUDrivers/RTIMUNull.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUMPU9150.o ../../RTIMULib/IMUDrivers/RTIMUMPU9150.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUMPU9250.o ../../RTIMULib/IMUDrivers/RTIMUMPU9250.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUGD20HM303D.o ../../RTIMULib/IMUDrivers/RTIMUGD20HM303D.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUGD20M303DLHC.o ../../RTIMULib/IMUDrivers/RTIMUGD20M303DLHC.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUGD20HM303DLHC.o ../../RTIMULib/IMUDrivers/RTIMUGD20HM303DLHC.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMULSM9DS0.o ../../RTIMULib/IMUDrivers/RTIMULSM9DS0.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMULSM9DS1.o ../../RTIMULib/IMUDrivers/RTIMULSM9DS1.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUBMX055.o ../../RTIMULib/IMUDrivers/RTIMUBMX055.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTIMUBNO055.o ../../RTIMULib/IMUDrivers/RTIMUBNO055.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTPressure.o ../../RTIMULib/IMUDrivers/RTPressure.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTPressureBMP180.o ../../RTIMULib/IMUDrivers/RTPressureBMP180.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTPressureLPS25H.o ../../RTIMULib/IMUDrivers/RTPressureLPS25H.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTPressureMS5611.o ../../RTIMULib/IMUDrivers/RTPressureMS5611.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -c -o objects/RTPressureMS5637.o ../../RTIMULib/IMUDrivers/RTPressureMS5637.cpp -pipe -O2 -Wall -W  -I. -I../../RTIMULib
g++ -Wl,-O1 -o Output/RTIMULibDrive objects/RTIMULibDrive.o objects/RTMath.o objects/RTIMUHal.o objects/RTFusion.o objects/RTFusionKalman4.o objects/RTFusionRTQF.o objects/RTIMUSettings.o objects/RTIMUAccelCal.o objects/RTIMUMa
gCal.o objects/RTIMU.o objects/RTIMUNull.o objects/RTIMUMPU9150.o objects/RTIMUMPU9250.o objects/RTIMUGD20HM303D.o objects/RTIMUGD20M303DLHC.o objects/RTIMUGD20HM303DLHC.o objects/RTIMULSM9DS0.o objects/RTIMULSM9DS1.o objects/R
TIMUBMX055.o objects/RTIMUBNO055.o objects/RTPressure.o objects/RTPressureBMP180.o objects/RTPressureLPS25H.o objects/RTPressureMS5611.o objects/RTPressureMS5637.o  -L/usr/lib/arm-linux-gnueabihf
(root|~/RTIMULib/Linux/RTIMULibDrive) cd Output/
(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ls
RTIMULibDrive

And easy to run

(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ./RTIMULibDrive
Settings file not found. Using defaults and creating settings file
Detected MPU9250 at standard address
Using fusion algorithm RTQF
min/max compass calibration not in use
Ellipsoid compass calibration not in use
Accel calibration not in use
MPU-9250 init complete
^Cmple rate 84: : roll:169.933659, pitch:-4.159935, yaw:140.757385

The process need CTRL + C to exit.

I found strings of the problem inside C++ code "I2C read error from 104, 116 - Failed to read fifo data"

(root|~/RTIMULib) git grep "I2C read error from" && git grep "Failed to read fifo data"
RTIMULib/RTIMUHal.cpp:                    HAL_ERROR3("I2C read error from %d, %d - %s\n", slaveAddr, regAddr, errorMsg);
RTIMULib/RTIMUHal.cpp:                    HAL_ERROR2("I2C read error from %d - %s\n", slaveAddr, errorMsg);
RTIMULib/IMUDrivers/RTIMUMPU9150.cpp:        if (!m_settings->HALRead(m_slaveAddr, MPU9150_FIFO_R_W, MPU9150_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9150.cpp:                                m_cache[m_cacheIn].data, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9150.cpp:            if (!m_settings->HALRead(m_slaveAddr, MPU9150_FIFO_R_W, MPU9150_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9150.cpp:    if (!m_settings->HALRead(m_slaveAddr, MPU9150_FIFO_R_W, MPU9150_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9250.cpp:        if (!m_settings->HALRead(m_slaveAddr, MPU9250_FIFO_R_W, MPU9250_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9250.cpp:                    m_cache[m_cacheIn].data, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9250.cpp:            if (!m_settings->HALRead(m_slaveAddr, MPU9250_FIFO_R_W, MPU9250_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))
RTIMULib/IMUDrivers/RTIMUMPU9250.cpp:    if (!m_settings->HALRead(m_slaveAddr, MPU9250_FIFO_R_W, MPU9250_FIFO_CHUNK_SIZE, fifoData, "Failed to read fifo data"))

Now I need to return to a bad Kernel and reproduce the problem with the RTIMULibDrive process.

@VigibotDev
Copy link
Author

You need a mpu-9250 breakout board :

https://www.amazon.fr/gp/product/B07HMQZ7N9

Don't buy the blue PCB version without pressure sensor :
A lot of Non-working fake are on the market !!! We can't find working version of the no-pressure / blue PCB (tested aliexpress and amazon are all fake !)

@VigibotDev
Copy link
Author

Win ! You can use the compiled binary to get the problem,
(I used rpi-update without parameter to jump on the last firmware/kernel)

The error is more frequent with the Node.js client I2C slow pooling for battery monitoring. + the binary for testing :

(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ls
RTIMULibDrive  RTIMULib.ini
(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ./RTIMULibDrive
Settings file RTIMULib.ini loaded
Using fusion algorithm RTQF
min/max compass calibration not in use
Ellipsoid compass calibration not in use
Accel calibration not in use
MPU-9250 init complete
I2C read error from 104, 116 - Failed to read fifo data:140.464862
I2C read error from 104, 116 - Failed to read fifo data:140.428143ch:-4.165171, yaw:140.463510
^Cmple rate 84: : roll:170.000417, pitch:-4.202972, yaw:140.246870ch:-4.229920, yaw:140.305159

I stop the vigibot client

(root|~/RTIMULib/Linux/RTIMULibDrive/Output) systemctl stop vigiclient

But error still here, but very rare need a long run :

(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ./RTIMULibDrive
Settings file RTIMULib.ini loaded
Using fusion algorithm RTQF
min/max compass calibration not in use
Ellipsoid compass calibration not in use
Accel calibration not in use
MPU-9250 init complete
I2C read error from 104, 116 - Failed to read fifo data:140.810606
^Cmple rate 84: : roll:169.980568, pitch:-4.166409, yaw:140.560061ch:-4.223783, yaw:140.817723

The raw trace from our C++ process received on the website is :
I2C error appear way faster because Node.js pool battery fast + the C++ process high bandwidth on IMU :

16/03 15:15:40:972 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:42:057 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:44:778 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:46:529 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:48:257 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:49:456 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:52:156 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:53:582 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:56:034 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:58:032 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:15:59:556 |              | Client | Diffusion | 7334 | stderr | I2C read error from 104, 116 - Failed to read fifo data

or

16/03 15:57:51:622 |              | Client | Diffusion | 8554 | stderr | I2C read error from 104, 116 - Failed to read fifo data
16/03 15:57:53:837 |              | Client | Diffusion | 8554 | stderr | MPU-9250 fifo has overflowedI2C read error from 104, 116 - Failed to read fifo data
16/03 15:57:57:473 |              | Client | Diffusion | 8554 | stderr | MPU-9250 fifo has overflowedI2C read error from 104, 116 - Failed to read fifo data
16/03 15:57:59:208 |              | Client | Diffusion | 8554 | stderr | MPU-9250 fifo has overflowedI2C read error from 104, 116 - Failed to read fifo data

@VigibotDev
Copy link
Author

I remembrer we use 400K I2C bus rate :

# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=on,i2c_arm_baudrate=400000

I just tested @ 100K and it's harder to get the problem ! we must run @ 400K !

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

Thanks - I have one on order, but it might take a month to arrive.

@VigibotDev
Copy link
Author

Important you need 400KHz I2C clock and 1KHz IMU sample rate to get multiples I2C errors every minutes.

  • I launched binary once to get the RTIMULib.ini file + set "MPU9250GyroAccelSampleRate=" to 1000 for 1 KHz sample rate
  • I also re-tested this minimal setup on the old Kernel + stable firmware without any error in 30 minutes.

Few minute testing on the last Master (kernel + firmware) :

(root|~/RTIMULib/Linux/RTIMULibDrive/Output) systemctl stop vigiclient
(root|~/RTIMULib/Linux/RTIMULibDrive/Output) grep MPU9250GyroAccelSampleRate RTIMULib.ini
MPU9250GyroAccelSampleRate=1000
(root|~/RTIMULib/Linux/RTIMULibDrive/Output) ./RTIMULibDrive
Settings file RTIMULib.ini loaded
Using fusion algorithm RTQF
min/max compass calibration not in use
Ellipsoid compass calibration not in use
Accel calibration not in use
MPU-9250 init complete
I2C read error from 104, 116 - Failed to read fifo dataaw:141.097747
I2C read error from 104, 116 - Failed to read fifo dataaw:140.320786ch:-4.169592, yaw:139.758907
I2C read error from 104, 116 - Failed to read fifo dataaw:141.043283h:-4.156291, yaw:140.894522
I2C read error from 104, 116 - Failed to read fifo dataaw:139.960753ch:-4.190503, yaw:140.338340
I2C read error from 104, 116 - Failed to read fifo dataw:139.932872tch:-4.166954, yaw:140.754107
I2C read error from 104, 116 - Failed to read fifo dataaw:141.235307h:-4.188396, yaw:140.192857
I2C read error from 104, 116 - Failed to read fifo dataw:140.302590tch:-4.195192, yaw:139.979973
^Cmple rate 1010: : roll:169.974927, pitch:-4.183485, yaw:141.555575h:-4.148631, yaw:140.939587
(root|~/RTIMULib/Linux/RTIMULibDrive/Output)  

Also I can give you a root SSH access to a fresh robot with FTTH internet access if you want to try things before the package to arrive (you can crash it no problem, they are from my automatic integration).

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

That's a good idea. Email me - phil@raspberrypi.com - with the details. My plan would be to push a trial kernel to it to confirm that it solves the problem before I make the change public.

@VigibotDev
Copy link
Author

Now you can ssh to my domain with the email port + password.

It run on the last working version, all is ready to work or fail depend on the Kernel version.
(400KHz I2C 1KHz IMU sample rate).

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

Thanks - expect a visit soon.

@VigibotDev
Copy link
Author

VigibotDev commented Mar 17, 2021 via email

@VigibotDev
Copy link
Author

You can look https://www.youtube.com/watch?v=NmrKH4Qx6io how Vigibot perform on a smartphone :

Everything @ 30FPS and 0.1/0.2 second end to end / 4G to 4G latency.
Vigibot.com is a non-profit-making organisation (1901 french law)

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

Your target Pi is now running the latest kernel and firmware but with the suspect commit reverted, and I've not seen any I2C errors in a few minutes - with the current rpi-update kernel it saw errors after about 10 seconds.

On the basis of that I'll push the reversion to our current branches, then revisit the patch once the MPU9250 arrives.

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

If you want to try the kernel on other Pi 3s, download install.tgz from here (https://drive.google.com/file/d/1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df/view?usp=sharing), or use:

$ wget -O install.tgz "https://drive.google.com/uc?export=download&id=1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df"

directly on the command line. Install with:

$ tar --no-same-owner --keep-directory-symlink -zxf install.tgz -C /

(You will need to use sudo unless you run as root).

@VigibotDev
Copy link
Author

I confirm you the complete system work on the kernel (vigibot.com robot name is Lidar)
(root|~) uname -a
Linux raspberrypi 5.10.23-v7+ #1083 SMP Wed Mar 17 15:30:57 GMT 2021 armv7l GNU/Linux

If you can give me a procedure I can deploy on the 10 testing robots

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

N.B. The test kernel is only for a Pi 3B, 3B+ or 3A+ or 2B.

pelwell added a commit to raspberrypi/linux that referenced this issue Mar 17, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
@VigibotDev
Copy link
Author

LOL I see the answer few ms. before I clicked the "Comment" button.
Test success on the PI 3 A+ and B+ (No 4 as your N.B. And no 2B because is too slow for the IMU+Lidar Autopilot, 2B OK only for low latency basic FPV piloting)

you can check my executing your procedure here :

(root|~) wget -O install.tgz "https://drive.google.com/uc?export=download&id=1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df"
--2021-03-17 15:52:30--  https://drive.google.com/uc?export=download&id=1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df
Resolving drive.google.com (drive.google.com)... 142.250.179.78, 2a00:1450:4007:813::200e
Connecting to drive.google.com (drive.google.com)|142.250.179.78|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://doc-0c-9o-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/k3bsbojtnacgq8d047hti204gme0mbp0/1615996350000/18441922891                              031522895/*/1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df?e=download [following]
Warning: wildcards not supported in HTTP.
--2021-03-17 15:52:36--  https://doc-0c-9o-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/k3bsbojtnacgq8d047hti204gme0mbp0/1615996350                              000/18441922891031522895/*/1ItRLCyvgba8cQeU6qgtloxu_xgDsG-df?e=download
Resolving doc-0c-9o-docs.googleusercontent.com (doc-0c-9o-docs.googleusercontent.com)... 216.58.206.225, 2a00:1450:4007:817::2001
Connecting to doc-0c-9o-docs.googleusercontent.com (doc-0c-9o-docs.googleusercontent.com)|216.58.206.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-compressed]
Saving to: ‘install.tgz’

install.tgz                                 [                                    <=>                                         ]  25.38M  3.52MB/s    in 7.1s

2021-03-17 15:52:43 (3.57 MB/s) - ‘install.tgz’ saved [26614529]

(root|~) ls
install.tgz
(root|~) tar --no-same-owner --keep-directory-symlink -zxf install.tgz -C /
(root|~) uname -a
Linux raspberrypi 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021 armv7l GNU/Linux
(root|~) reboot
Connection to 192.168.0.25 closed by remote host.
Connection to 192.168.0.25 closed.
(root|~) ssh 192.168.0.25
root@192.168.0.25's password:
Linux raspberrypi 5.10.23-v7+ #1083 SMP Wed Mar 17 15:30:57 GMT 2021 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Mar 17 15:51:31 2021 from 192.168.0.2
(root|~) uname -a
Linux raspberrypi 5.10.23-v7+ #1083 SMP Wed Mar 17 15:30:57 GMT 2021 armv7l GNU/Linux

@pelwell
Copy link
Collaborator

pelwell commented Mar 17, 2021

Yes - that looks correct.

pelwell added a commit to raspberrypi/linux that referenced this issue Mar 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
pelwell added a commit to raspberrypi/linux that referenced this issue Mar 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Mar 22, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Mar 22, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
limeng-linux pushed a commit to limeng-linux/linux-yocto-develop that referenced this issue Mar 23, 2021
commit  ffb6947602d01b4dd60cc537f48e550332242b30 from
https://github.com/raspberrypi/linux.git rpi-5.12.y

This reverts commit 431ac1d21b5e842c5a564bc0bab31d7dc11611f5.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Signed-off-by: Meng Li <Meng.Li@windriver.com>
popcornmix added a commit to raspberrypi/firmware that referenced this issue Mar 23, 2021
kernel: Make rpi poe fan less noisy in cool environments
See: raspberrypi/linux#4222

kernel: overlays: Add pcie-32bit-dma overlay
See: raspberrypi/linux#4216

kernel: ARM: dts: bcm2711: Add aliases for additional SPIs

kernel: Revert i2c: bcm2835: Handle untimely DONE signal
See: Hexxeh/rpi-firmware#261
popcornmix added a commit that referenced this issue Mar 23, 2021
kernel: Make rpi poe fan less noisy in cool environments
See: raspberrypi/linux#4222

kernel: overlays: Add pcie-32bit-dma overlay
See: raspberrypi/linux#4216

kernel: ARM: dts: bcm2711: Add aliases for additional SPIs

kernel: Revert i2c: bcm2835: Handle untimely DONE signal
See: #261
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Mar 29, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Mar 29, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
limeng-linux pushed a commit to limeng-linux/linux-yocto-5.10 that referenced this issue Apr 2, 2021
commit  13c43880f2ee6db7d26949cb6a3e8db1a4b76736 from
https://github.com/raspberrypi/linux.git rpi-5.10.y

This reverts commit 431ac1d21b5e842c5a564bc0bab31d7dc11611f5.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Signed-off-by: Meng Li <Meng.Li@windriver.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 6, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 6, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 27, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 27, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Apr 30, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 7, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 7, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
@VigibotDev
Copy link
Author

My today testing :

  • Fresh image + apt update / upgrade on nine robots with Raspberry PI 3 A+ / B+ and 4 4Go

  • 9 * robots with IMU do not work (as expected)

  • rpi-update on all (without parameters)

  • 9 * robots with IMU all working OK

The master branch firmware (and kernel) are OK !

@pelwell
Copy link
Collaborator

pelwell commented May 10, 2021

Please close the issue,

popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 13, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 13, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 19, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue May 25, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Jun 8, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Jun 14, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
limeng-linux pushed a commit to limeng-linux/linux-yocto-develop that referenced this issue Jun 27, 2021
commit  26efbc6bfcffdfa389d8870adc95bc210bc2954b from
https://github.com/raspberrypi/linux.git rpi-5.12.y

This reverts commit 431ac1d21b5e842c5a564bc0bab31d7dc11611f5.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Signed-off-by: Meng Li <Meng.Li@windriver.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Jul 9, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Jul 15, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
popcornmix pushed a commit to raspberrypi/linux that referenced this issue Jul 27, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Noltari pushed a commit to Noltari/rpi-linux that referenced this issue Aug 20, 2021
This reverts commit 431ac1d.

Reverting this patch because it appears to have caused a regression
when used with an MPU-9250 IMU.

See: Hexxeh/rpi-firmware#261

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants