Rootfs corruption after a few months - bootloop #312

AnthraX1 · 2023-11-10T09:06:01Z

Describe the bug
The rootfs on yi dome 1080p camera would get corrupted and stuck in a yellow light reboot loop. Can be reset but won't connect to wifi. Log says no wifi dev and wpa_supplicant crashed.

[./watch_process][1/19/6:42:1:235]: cloud crashed![./watch_process][1/19/6:42:1:236]: p2p_tnp crashed![./watch_process][1/19/6:42:1:237]: arp_test crashed![./dispatch][1/19/6:42:8:277]: wpa may crashed, now reset
[./dispatch][1/19/6:42:8:329]: no usb dev

The only way to fix is to flash with an older version of rootfs_h20 and home_h20 files then load the sdcard with yi-hack-v5 0.38 files again. Flashing home_h20 alone won't solve the problem.

To Reproduce
Steps to reproduce the behaviour:
All two of my long running yi dome 1080p cameras had the same problem after maybe 2 months.

Expected behaviour
Camera doesn't crash

Screenshots
None

Set Up Details (please complete the following information):

Firmware Version: 0.38
Device: YI Dome 1080p
Model: 45US

Technical Details
It's exactly like this issue except I didn't change any settings and it just happened on its own: #204

The text was updated successfully, but these errors were encountered:

alienatedsec · 2023-11-10T09:24:50Z

@AnthraX1 Please read this thread #291 and make sure the following:

encryption security on your WiFi - the camera wpa_supplicant modules are not handling WPA3/WPA2 encryptions very well. Mine works fine on the below-highlighted setting, but it fails to connect when changing to WPA2-PSK only. I guess the camera needs WPA-PSK on its own.

if you or anyone can contribute to cross-compiling the latest wpa_supplicant before I find any time, it would be great https://w1.fi/releases/wpa_supplicant-2.10.tar.gz - the current version on our cameras is v2.6
[Edit] Also, it's worth checking the preamble settings as those are important for older Wi-Fi setups https://routerguide.net/preamble-type-short-or-long/

AnthraX1 · 2023-11-10T15:30:53Z

After reading #291 it made much more sense. I too have two APs using the same SSID. However my tests indicate the camera can roam between APs with the same SSID totally fine.

I wonder if the issue could be the rekey interval in WPA or WPA2. The max rekey interval is 4194303 seconds which is 48.5 days, very close to the time my camera was connected for. I don't know the inner workings of wpa_supplicant but I wonder if it writes the generated key(not the password) to a file then hope to reconnect with the key but didn't have time to do the full auth handshake before getting killed by watchdog?

alienatedsec · 2023-11-20T15:23:14Z

@AnthraX1 I get closer to the latest wpa_supplicant release - I found that different cameras have different versions. Please can you share which version is on your camera? Ideally, run the wpa_supplicant command from the console and share the content here.

alienatedsec · 2023-11-21T10:04:16Z

@AnthraX1 New release with the latest wpa_supplicant

https://github.com/alienatedsec/yi-hack-v5/releases/tag/0.4.1c

K13PTo · 2023-11-30T22:43:13Z

I've been having the same issue but cannot go beyond version 0.4.1b to see if it resolves for me. If i try versions 0.4.1c and 0.4.1d, i go into boot loop. Currently at 4.0

Firmware Version 0.4.0
Base Version 1.9.2.0I_201812141405
Model Suffix yi_dome_1080p
Hardware ID 65US
wpa_supplicant v2.0-devel

AnthraX1 · 2023-12-05T16:48:21Z

@AnthraX1 I get closer to the latest wpa_supplicant release - I found that different cameras have different versions. Please can you share which version is on your camera? Ideally, run the wpa_supplicant command from the console and share the content here.

I was using 0.40, not sure what version was in it. Isn't it compiled in and have binary file in yi-hack-v5/wpa/?

alienatedsec · 2023-12-05T17:00:08Z

0.4.1c has the first wpa_supplicant release @AnthraX1

AnthraX1 · 2023-12-05T17:11:01Z

for the new firmware, I'm stuck at the Retrieving Pairing Status page on the app after the cam successfully connected to wifi. the last step in the app just times out while the camera is actually connected to the network.

AnthraX1 · 2023-12-08T13:06:50Z

It appears cloud client is crashing.

[./rmm][1/19/6:42:58:948]: got RMM_SPEAK_WIFI_CONNECTTED[./dispatch][1/19/6:42:58:949]: msg snd success[./watch_process][1/19/6:42:58:975]: cloud crashed![./watch_process][1/19/6:42:58:975]: p2p_tnp crashed!

alienatedsec · 2023-12-08T13:41:27Z

@AnthraX1 what happens is that the ./watch_process is restarting the camera whenever a component is not operational. I suggest commenting a line in system.sh which is to execute it and boot the camera.

mariosc · 2023-12-18T19:16:28Z

the issue is not fixed with 0.4.1d. I thought it was ok but the multi reboot issue came back on. The only way to get the camera to work again is to run the yi-hack-v4 first, then run the v5 scripts.

alienatedsec · 2023-12-19T10:40:15Z

the issue is not fixed with 0.4.1d. I thought it was ok but the multi reboot issue came back on. The only way to get the camera to work again is to run the yi-hack-v4 first, then run the v5 scripts.

I confirm the same - both of my cameras are in the loop after reboot - I am investigating now.

RidingTheSpiral · 2023-12-19T17:53:04Z

I had the same problem twice in a week. First time with the last beta version, to restore it I've flashed back the stock firmware. Second time I didn't update and it boot looped with 038 beta. Now I've restore it with 037 beta (like someone suggested in the other discussion), let's see how it goes! I've keept all there was on the sdcard, there some log that can be useful??

github-actions · 2024-01-19T06:24:29Z

This issue has been stale for 30 days - it will be closed within the next 7 days if not updated

g1za · 2024-01-22T12:06:27Z

today after a power outage my yi dome camera went into a boot loop.
I'm now trying to re-flash the latest base firmware and then update to the latest pre-release.

g1za · 2024-01-22T12:28:02Z

no luck. the files seem to be deployed correctly on the SD card but it still goes into boot loop.
I upload what seems to contain some log files and now I'm trying to go back to stock firmware.
log.txt

EDIT:
As reported in the loop investigation discussion I as well see an error in the log on ARP:
[./dispatch][1/22/9:41:36:899]: wpa may crashed, now reset [./dispatch][1/22/9:41:37:19]: no usb dev [./watch_process][1/22/9:41:37:642]: arp_test crashed!

alienatedsec · 2024-01-22T15:55:30Z

the files seem to be deployed correctly on the SD card but it still goes into boot loop.

@g1za please remember to use FAT32

g1za · 2024-01-22T15:57:07Z

@alienatedsec always

EDIT:
In the end even the stock firmware could not exit the camera from this loop but I could solve using the reset button on the back and starting again from scratch.

alienatedsec · 2024-02-04T09:34:06Z

Please follow and test if needed https://github.com/alienatedsec/yi-hack-v5/releases/tag/0.4.1e

alienatedsec · 2024-02-11T20:44:16Z

So I need to rework busybox, strip it of some tools, and move them to the /tmp/sd/yi-hack-v5/. folder.

Simply, it is too big and is unlikely to make cameras work with as little as 32kB space on the rootfs partition.

#291 is another discussion covering this subject in many directions - I'll start now and will make you aware of any prorgess

Changes to be committed: new file: src/busybox_tools/cleanup.busybox_tools new file: src/busybox_tools/compile.busybox_tools new file: src/busybox_tools/config.busybox_tools new file: src/busybox_tools/init.busybox_tools new file: src/busybox_tools/install.busybox_tools

Changes to be committed: modified: src/busybox/.config modified: src/busybox_tools/install.busybox_tools

alienatedsec · 2024-02-12T11:13:10Z

Some positive news. I found the rootfs partition needs around 80KB of free space to operate. I managed to increase the free space up to 124KB. The Use% increases up to around 95% over time and drops to 90% after a while. It means that some processes are still writing onto the /dev/root filesystem.

If that will work fine for the rest of the day, and I'll manage the other issue with the wget on the SD card, it will be released as the final baseline very soon.

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.3M      1.1M    124.0K  90% /
tmpfs                    15.0M      4.0K     15.0M   0% /dev
/dev/mtdblock5           12.7M      9.1M      3.6M  72% /home
tmpfs                    16.0M      1.9M     14.1M  12% /tmp
/dev/mmcblk0p1            7.4G     82.1M      7.3G   1% /tmp/sd
/dev/mmcblk0p1            7.4G     82.1M      7.3G   1% /tmp/sd/yi-hack-v5/www/record

Mentioning other issues for awareness #338

alienatedsec · 2024-02-13T09:39:02Z

The new release 0.4.1h should resolve the boot loop issue.

alienatedsec · 2024-02-19T07:36:54Z

This is fixed in the latest pre-release - I appreciate it could take weeks or months to reproduce, but then please raise another issue if needed

ad-on-is · 2024-10-12T11:11:12Z

I'm having exactly the same issue with 0.4.1. Camera works for a day or two, and then suddenly starts bootlooping.

Hack (SD) Firmware Version 0.4.1
Baseline Firmware Version 0.4.1
Base Version 1.8.7.0F_xxx
Model Suffix yi_home
Hardware ID 47US
Serial Number Y9FYxxxx

alienatedsec · 2024-10-12T20:29:02Z

I'm having exactly the same issue with 0.4.1. Camera works for a day or two, and then suddenly starts bootlooping.

Hack (SD) Firmware Version 0.4.1

Baseline Firmware Version 0.4.1

Base Version 1.8.7.0F_xxx

Model Suffix yi_home

Hardware ID 47US

Serial Number Y9FYxxxx

@ad-on-is this hasn't been an issue for several months now. I guess without a terminal output we cannot find what would be the problem. Furthermore, the issue with boot looping was partition-related. The affected partition has been modified, so it has enough space for camera operations.

ad-on-is · 2024-10-12T20:36:39Z

@alienatedsec Unfortunately I can't SSH into the cam when it happens, since it doesn't connect to the wifi.

Is there a way to get to the latest logs when this happens?

Also, I've increased the SD size to 30%, since I read in other threads that it helped for some people.

AnthraX1 added the bug Something isn't working label Nov 10, 2023

AnthraX1 assigned alienatedsec Nov 10, 2023

github-actions bot added the issue-stale label Jan 19, 2024

github-actions bot removed the issue-stale label Jan 23, 2024

alienatedsec added a commit that referenced this issue Feb 11, 2024

[+] Stripping and adding tools for rootfs space #312

48292b2

Changes to be committed: modified: src/busybox/.config modified: src/busybox_tools/install.busybox_tools

alienatedsec changed the title ~~Rootfs corruption after a few months~~ Rootfs corruption after a few months - bootloop Feb 12, 2024

alienatedsec closed this as completed Feb 19, 2024

jseabaugh mentioned this issue Jul 6, 2024

[REQ] WiFi Recovery Guide #350

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootfs corruption after a few months - bootloop #312

Rootfs corruption after a few months - bootloop #312

AnthraX1 commented Nov 10, 2023 •

edited

Loading

alienatedsec commented Nov 10, 2023 •

edited

Loading

AnthraX1 commented Nov 10, 2023 •

edited

Loading

alienatedsec commented Nov 20, 2023 •

edited

Loading

alienatedsec commented Nov 21, 2023

K13PTo commented Nov 30, 2023

AnthraX1 commented Dec 5, 2023

alienatedsec commented Dec 5, 2023

AnthraX1 commented Dec 5, 2023

AnthraX1 commented Dec 8, 2023

alienatedsec commented Dec 8, 2023

mariosc commented Dec 18, 2023

alienatedsec commented Dec 19, 2023

RidingTheSpiral commented Dec 19, 2023

github-actions bot commented Jan 19, 2024

g1za commented Jan 22, 2024

g1za commented Jan 22, 2024 •

edited

Loading

alienatedsec commented Jan 22, 2024

g1za commented Jan 22, 2024 •

edited

Loading

alienatedsec commented Feb 4, 2024

alienatedsec commented Feb 11, 2024

alienatedsec commented Feb 12, 2024

alienatedsec commented Feb 13, 2024

alienatedsec commented Feb 19, 2024

ad-on-is commented Oct 12, 2024

alienatedsec commented Oct 12, 2024

ad-on-is commented Oct 12, 2024

Rootfs corruption after a few months - bootloop #312

Rootfs corruption after a few months - bootloop #312

Comments

AnthraX1 commented Nov 10, 2023 • edited Loading

alienatedsec commented Nov 10, 2023 • edited Loading

AnthraX1 commented Nov 10, 2023 • edited Loading

alienatedsec commented Nov 20, 2023 • edited Loading

alienatedsec commented Nov 21, 2023

K13PTo commented Nov 30, 2023

AnthraX1 commented Dec 5, 2023

alienatedsec commented Dec 5, 2023

AnthraX1 commented Dec 5, 2023

AnthraX1 commented Dec 8, 2023

alienatedsec commented Dec 8, 2023

mariosc commented Dec 18, 2023

alienatedsec commented Dec 19, 2023

RidingTheSpiral commented Dec 19, 2023

github-actions bot commented Jan 19, 2024

g1za commented Jan 22, 2024

g1za commented Jan 22, 2024 • edited Loading

alienatedsec commented Jan 22, 2024

g1za commented Jan 22, 2024 • edited Loading

alienatedsec commented Feb 4, 2024

alienatedsec commented Feb 11, 2024

alienatedsec commented Feb 12, 2024

alienatedsec commented Feb 13, 2024

alienatedsec commented Feb 19, 2024

ad-on-is commented Oct 12, 2024

alienatedsec commented Oct 12, 2024

ad-on-is commented Oct 12, 2024

AnthraX1 commented Nov 10, 2023 •

edited

Loading

alienatedsec commented Nov 10, 2023 •

edited

Loading

AnthraX1 commented Nov 10, 2023 •

edited

Loading

alienatedsec commented Nov 20, 2023 •

edited

Loading

g1za commented Jan 22, 2024 •

edited

Loading

g1za commented Jan 22, 2024 •

edited

Loading