Audit Log "audit" function crashes FTL on v.5.3.2 (aarch64) #960
Labels
Comments
|
This issue has been mentioned on Pi-hole Userspace. There might be relevant details there: https://discourse.pi-hole.net/t/auditlog-ftl-crash-nach-update-auf-5-2-1/40999/6 |
|
Thank you for your report and sorry for the inconvenience you're experiencing. The source of the bug you're seeing is from deeply inside the (third-party) SQLite3 database engine we're not modifying ourselves. There is some other report (#949) that is seeing a crash in the SQLite3 engine as well and we're already trying to work hard to get down to the source of the problem. Having said that, I just tried auditing at least fifty random domains (sometimes faster, sometimes slower) and did not experience a crash myself. |
|
Hello, after updating Pi-Hole, I seem to have the same Issue of Auditing anything causing Pi-Hole to "crash" the DNS to the point I have to restart it in the Command Line (no Errors there). I run this on a Pi-4B-4GB with PoE HAT (Pi-Hole being installed on the Micro-SD Card), and in the Web Interface it seems like clicking fast through the Audits causes it to crash, but this is a pure feeling based thing, since whenever I try clicking through them slowly it seems to work fine, but I am not sure. Might be an issue with accessing the Micro-SD Card Data. Also for reference, I installed Pi-Hole a few weeks ago, so the Issue did not exist when I cloned it directly from GitHub at the 17th of November 2020 at 5:26AM German Time (unsure if we are UTC+1 or +2 due to daylight savings crap). I hope this helps with finding the Issue. ^^ |
|
I'm seeing this problem too. I've tried the tweak/interruptsafe_systemcalls branch for FTL from some of the other issues and it still happens. I'm running this on a RPI-3B booted off a USB flash drive. |
|
I can also try the |
|
Tried with the latest |
|
Could you retry with |
|
|
I just tried the fix/all_the_things branch and it makes the audit problem a little harder to reproduce for me, but I can still crash it clicking audit buttons as fast as I can. |
|
And how often do you find yourself clicking the audit button as fast as you can? |
|
Pretty much every time I go to the page.
There are usually blocks of 2-4 addresses that I know I want to mark off at a glance and click up the list from the bottom.
Without this branch, FTL crashes after the 3rd or 4th click. With this patch, it’s more like 5 to 7 and I have to click a little faster than I normally would before it crashes.
…--
Robert Loomans
robert@loomans.org
On 21 Dec 2020, at 08:00, Dan Schaper ***@***.***> wrote:
And how often do you find yourself clicking the audit button as fast as you can?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
What exactly are you using the audit log for? Not to be flippant but I don't think I've ever seen anyone use it as a regular function. We've even discussed the need for it to continue to exist because it's really not that useful. |
|
I use the Audit Function after visiting each new Website to see what Stuff I already know about and which Stuff I might need to block for several reasons. For example blocking most of those "Font"-CDNs, which dramatically slow down the Internet Connection (most easily observed with Youtube). I also use Pi-Hole in combination with uMatrix to see what parts of a Website REALLY are absolutely not needed or only used for tracking. And if you use Youtube on a regular basis you often have to clean up a TON of dynamically generated googlevideo Entries from the Audit Log which results in me Spam Clicking through them, which then results in aforementioned Issue of having to use the restart DNS Command. The Audit Function is FAR better than running through the Query Logs all the time, and if you happen to have anything blocked that you did not intend to block, the Audit List is perfect for seeing where the culprit is, so it can be whitelisted manually if required. So yeah, there is a reason why the /admin/auditlog.php Page is bookmarked as THE page that I use to open the Web Interface of Pi-Hole. |
|
Pretty much what Gregorius said.
…On Mon, 21 Dec 2020 at 10:29, Gregorius Techneticies < ***@***.***> wrote:
I use the Audit Function after visiting each new Website to see what Stuff
I already know about and which Stuff I might need to block for several
reasons.
For example blocking most of those "Font"-CDNs, which dramatically slow
down the Internet Connection (most easily observed with Youtube).
I also use Pi-Hole in combination with uMatrix to see what parts of a
Website REALLY are absolutely not needed or only used for tracking.
And if you use Youtube on a regular basis you often have to clean up a TON
of dynamically generated googlevideo Entries from the Audit Log which
results in me Spam Clicking through them, which then results in
aforementioned Issue of having to use the restart DNS Command.
The Audit Function is FAR better than running through the Query Logs all
the time, and if you happen to have anything blocked that you did not
intend to block, the Audit List is perfect for seeing where the culprit is,
so it can be whitelisted manually if required.
So yeah, there is a reason why the /admin/auditlog.php Page is bookmarked
as THE page that I use to open the Web Interface of Pi-Hole.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#960 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAWPZKRFZ2YGLXRQ3FRF7TSV2JGJANCNFSM4UMOPNIQ>
.
--
Robert Loomans
robert@loomans.org
|
|
@yubiuser @rloomans (and anyone else still seeing the crash, maybe @GregoriusT as well): to the file Next time the crash happens, please include again the crash report from |
|
Happy to help with that as I see the same issue. On which branch do you suggest do run the debug? |
|
Somehow on |
|
While deleting some items from the blacklist I added accidentally while clicking fast on the audit tool I produced a crash |
|
These Lines seem to be the only thing that could be useful after enabling debugging the Database. Do note I do not use any of the "fixed" Versions, but more Data is always better. ^^
The Log Lines above are likely from when i crashed FTL, the ones below could be it too, but I am not sure.
And the Lines below this definitely should not be from the crash. That was after I needed to restart Pi Hole
edit: Code block formatting |
|
Okay, so this quite a bit of text coming in, I'll reply to each of you individually below or we will completely loose overview @danielvonwi @yubiuser
This asks for
Oh, this is not good. The first comment runs before opening the database. There is no error, so it succeeded. The second comment is when setting the temp. object location. This is an ordinary SQL query ( It also matches your backtrace
Seriously, I don't really know what to think about this right now, but I'll keep looking.
Well, yes, and no. Currently, it is better to use
Yes, they are from separate processes. You can see the PID (process ID) right after the time. This is the important number here. The number after the Was there really no other message after
?
Same PID, so from the same process, without a crash in between. I need some more lines around the crash (if there are any). Also, I don't see any lines indicating a restart of FTL in your log snippet. Debugging a partial snippet is ... difficult ;-) |
|
Strange... since I turned on (Log really ended there) |
|
The thing ended there, considering the 1.5 minute gap inbetween the thing that i pasted and Pi-Hole being started. That Last Line in this Log is the first line of that other thing I pasted. Unrelated but did I mention that I hate it when Logs contain way too much useless info, making the whole File just really bloated, especially if the same Log File does contain privacy critical material (aka all websites ever visited), so I can't just gist/paste the whole 8MiB thing, and have to read it in advance before, just to find what some other person was vaguely looking for? |
|
The debug messages are just verbose in what they do. When you add It seems that FTL did never crash but "just" got unresponsive. Maybe you can add should be sufficient to see if some locking was going on. |
Closed
|
Uh darn, when i enabled DEBUG_LOCKS=true (along with DEBUG_DATABASE=true) I lost the ability to reproduce the Issue no matter how hard I spam click. Edit: This MIGHT be an Issue with slow Storage Volumes like the Micro SD Card, which if the Logs are spammed sufficiently enough could cause enough delay to not cause the locking Issue maybe? I don't know... P.S. I am starting to run out of "unknown relatively trustworthy websites" that I could audit. |
|
@GregoriusT same here... I tried literally an hour to freeze/crash pihole. Thanks for confirming. |
|
It may be either due to additional delay (which should even with slow cards be in the microseconds regime as we're only writing a few bytes each time) or - and I'm afraid it is more this - it is the well-known Locks are sequential, a dead-lock cannot be recovered from. Any delay should not change anything here. Can you re-trigger the crash when you remove the debug lines and run (this does not do a full restart, however, FTL should re-check which debugging options are enabled) ? |
|
Bed time for me now. However, I wrote a custom SQL extension meanwhile that does the audit wildcard matching without the need for dynamic memory allocation inside SQLite3. If you still have time (and endurance) try (contains only the modified/improved audit statement), or to get both, Ideally you should not see a crash on these branches. |
|
I could trigger it while being on With And |
|
And another location |
|
Could you try running FTL in valgrind? Like described here: https://deploy-preview-485--pihole-docs.netlify.app/ftldns/valgrind/#command |
|
It won't start even without |
|
Ah, yes. It complain about capabilities. Try |
|
Ok. It started, but I'm not sure if it is/was running correctly. Immediate output of valgrind.log. The web interface was showing "DNS service not running" and red circle FTL. It also did not answer any queries. I clicked on "Enable" in the web interface and DNS resolution was restored (but this started FTL outside of valgrind?). Is there a way to disable/restart FTL inside valgrind? |
|
Yes, it is expected that the web interface may not be able to determine the running status of FTL because FTL is wrapped inside That it was not answering queries is strange. Please note that FTL runs a lot slower in
No but this should not be necessary. Though, |
|
I should also say that your log snippet looks fine and there is no indication that FTL had exited. Note that the first exit comes from forking into daemon (background) mode so this always happens. |
Ok, this made me belief it exited. So far I was not able to crash it running in valgrind. I run out of domains to audit/black/whitelist and to restore my database I had to kill the process (database was locked otherwise). Killing FTL produced some errors in valgrind (guess this is expected - or does valgrind "collect" errors and report all of them only at the end of a run?). |
Yes. If nothing too serious happens, the complete report will be given only at the end. This to avoid mentioning the same error potentially hundreds of times. It will just report this once instead and say it happened Errors like are something we ignore, as they are difficult to solve and lie deeply in We found three interesting things==2315== Thread 8 telnet-17: ==2315== Invalid read of size 8 ==2315== at 0x22AE44: columnMem (sqlite3.c:84688) ==2315== by 0x22AE44: sqlite3_column_int (sqlite3.c:84762) ==2315== by 0x16F3EB: domain_in_list (gravity-db.c:1179) ==2315== by 0x16F3EB: in_auditlist (gravity-db.c:1332) ==2315== by 0x16600F: getTopDomains (api.c:314) ==2315== by 0x169503: process_request (request.c:53) ==2315== by 0x169B7F: telnet_connection_handler_thread (socket.c:331) ==2315== by 0x49407E3: start_thread (pthread_create.c:486) ==2315== by 0x4A37ADB: thread_start (clone.S:78) ==2315== Address 0x4e89698 is 33,608 bytes inside a block of size 48,008 free'd ==2315== at 0x4848FE0: free (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==2315== by 0x23DB37: sqlite3LeaveMutexAndCloseZombie.part.537 (sqlite3.c:165403) ==2315== by 0x23E213: sqlite3LeaveMutexAndCloseZombie (sqlite3.c:165184) ==2315== by 0x23E213: sqlite3Close.part.538 (sqlite3.c:165240) ==2315== by 0x16C44F: gravityDB_close.part.2 (gravity-db.c:950) ==2315== by 0x15325B: cleanup (daemon.c:207) ==2315== by 0x14ED7B: main (main.c:110) ==2315== Block was alloc'd at ==2315== at 0x4847E4C: malloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==2315== by 0x20E32B: sqlite3MemMalloc (sqlite3.c:23979) ==2315== by 0x1EA7EF: setupLookaside (sqlite3.c:164778) ==2315== by 0x2673FF: openDatabase (sqlite3.c:167386) ==2315== by 0x16C4EF: gravityDB_open.part.3 (gravity-db.c:119) ==2315== by 0x16DAFB: gravityDB_open (gravity-db.c:110) ==2315== by 0x16DAFB: gravityDB_reopen (gravity-db.c:210) ==2315== by 0x153FF3: FTL_reload_all_domainlists (datastructure.c:463) ==2315== by 0x16C177: DB_thread (database-thread.c:86) ==2315== by 0x49407E3: start_thread (pthread_create.c:486) ==2315== by 0x4A37ADB: thread_start (clone.S:78) ==2315== Invalid read of size 8 ==2315== at 0x22AE4C: columnMem (sqlite3.c:84688) ==2315== by 0x22AE4C: sqlite3_column_int (sqlite3.c:84762) ==2315== by 0x16F3EB: domain_in_list (gravity-db.c:1179) ==2315== by 0x16F3EB: in_auditlist (gravity-db.c:1332) ==2315== by 0x16600F: getTopDomains (api.c:314) ==2315== by 0x169503: process_request (request.c:53) ==2315== by 0x169B7F: telnet_connection_handler_thread (socket.c:331) ==2315== by 0x49407E3: start_thread (pthread_create.c:486) ==2315== by 0x4A37ADB: thread_start (clone.S:78) ==2315== Address 0x4e89b60 is 34,832 bytes inside a block of size 48,008 free'd ==2315== at 0x4848FE0: free (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==2315== by 0x23DB37: sqlite3LeaveMutexAndCloseZombie.part.537 (sqlite3.c:165403) ==2315== by 0x23E213: sqlite3LeaveMutexAndCloseZombie (sqlite3.c:165184) ==2315== by 0x23E213: sqlite3Close.part.538 (sqlite3.c:165240) ==2315== by 0x16C44F: gravityDB_close.part.2 (gravity-db.c:950) ==2315== by 0x15325B: cleanup (daemon.c:207) ==2315== by 0x14ED7B: main (main.c:110) ==2315== Block was alloc'd at ==2315== at 0x4847E4C: malloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==2315== by 0x20E32B: sqlite3MemMalloc (sqlite3.c:23979) ==2315== by 0x1EA7EF: setupLookaside (sqlite3.c:164778) ==2315== by 0x2673FF: openDatabase (sqlite3.c:167386) ==2315== by 0x16C4EF: gravityDB_open.part.3 (gravity-db.c:119) ==2315== by 0x16DAFB: gravityDB_open (gravity-db.c:110) ==2315== by 0x16DAFB: gravityDB_reopen (gravity-db.c:210) ==2315== by 0x153FF3: FTL_reload_all_domainlists (datastructure.c:463) ==2315== by 0x16C177: DB_thread (database-thread.c:86) ==2315== by 0x49407E3: start_thread (pthread_create.c:486) ==2315== by 0x4A37ADB: thread_start (clone.S:78) ==2315== Invalid read of size 4 ==2315== at 0x4942D34: pthread_mutex_lock (pthread_mutex_lock.c:67) ==2315== by 0x22AE5F: sqlite3_mutex_enter (sqlite3.c:26247) ==2315== by 0x22AE5F: columnMem (sqlite3.c:84688) ==2315== by 0x22AE5F: sqlite3_column_int (sqlite3.c:84762) ==2315== by 0x16F3EB: domain_in_list (gravity-db.c:1179) ==2315== by 0x16F3EB: in_auditlist (gravity-db.c:1332) ==2315== by 0x16600F: getTopDomains (api.c:314) ==2315== by 0x169503: process_request (request.c:53) ==2315== by 0x169B7F: telnet_connection_handler_thread (socket.c:331) ==2315== by 0x49407E3: start_thread (pthread_create.c:486) ==2315== by 0x4A37ADB: thread_start (clone.S:78) ==2315== Address 0x534143202c6e6971 is not stack'd, malloc'd or (recently) free'd |
|
|
Is it normal that the log output of at all when I try to black/whitelist from the audit page. |
No, good catch! It seems like FTL is not receiving the signal to reload. This may also be why the crash isn't triggered. I'll check what needs to be changed and come back to you. |
|
@yubiuser Please run This should allow the reload commands to work even when FTL is running inside |
8 tasks
|
Perfect, reloading works now. Although it did not crash, valgrind reported some errors. |
|
Crashing after receiving |
|
Can it be a race condition? The moment I let it run bare metal I can trigger the crash immediately. Maybe it is due to FTL's slowness inside valgrind? |
|
It shouldn't be possible to be a race-collision because all the API accessing works inside of global locks to avoid exactly this scenario. I checked the errors in your log. Thread 9 telnet-18: Invalid read of size 4==27642== at 0x4942D34: pthread_mutex_lock (pthread_mutex_lock.c:67) ==27642== by 0x22AE5F: sqlite3_mutex_enter (sqlite3.c:26247) ==27642== by 0x22AE5F: columnMem (sqlite3.c:84688) ==27642== by 0x22AE5F: sqlite3_column_int (sqlite3.c:84762) ==27642== by 0x16F3EB: domain_in_list (gravity-db.c:1179) ==27642== by 0x16F3EB: in_auditlist (gravity-db.c:1332) ==27642== by 0x16600F: getTopDomains (api.c:314) ==27642== by 0x169503: process_request (request.c:53) ==27642== by 0x169B7F: telnet_connection_handler_thread (socket.c:331) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== Address 0x534143202c6e6971 is not stack'd, malloc'd or (recently) free'd We don't really see where this is coming from. It really seems to be an SQLite3 engine error not caused by us, I'll try to figure out what is going on here. X bytes in 1 blocks are definitely lost==27642== Thread 1: ==27642== 10 bytes in 1 blocks are definitely lost in loss record 7 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x173C1B: getNameFromIP (network-table.c:1810) ==27642== by 0x16CEC3: get_client_groupids (gravity-db.c:483) ==27642== by 0x16F51F: gravityDB_get_regex_client_groups (gravity-db.c:1342) ==27642== by 0x15ED3B: reload_per_client_regex (regex.c:429) ==27642== by 0x15ED3B: read_regex_from_database (regex.c:560) ==27642== by 0x15400B: FTL_reload_all_domainlists (datastructure.c:470) ==27642== by 0x16C177: DB_thread (database-thread.c:86) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== ==27642== 16 bytes in 8 blocks are definitely lost in loss record 31 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x17157B: add_FTL_clients_to_network_table (network-table.c:704) ==27642== by 0x17157B: parse_neighbor_cache (network-table.c:1367) ==27642== by 0x16C157: DB_thread (database-thread.c:82) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== ==27642== 18 bytes in 1 blocks are definitely lost in loss record 34 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x173587: getMACfromIP (network-table.c:1667) ==27642== by 0x16CDCF: get_client_groupids (gravity-db.c:367) ==27642== by 0x16F51F: gravityDB_get_regex_client_groups (gravity-db.c:1342) ==27642== by 0x15ED3B: reload_per_client_regex (regex.c:429) ==27642== by 0x15ED3B: read_regex_from_database (regex.c:560) ==27642== by 0x15400B: FTL_reload_all_domainlists (datastructure.c:470) ==27642== by 0x16C177: DB_thread (database-thread.c:86) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== ==27642== 18 bytes in 1 blocks are definitely lost in loss record 35 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x173587: getMACfromIP (network-table.c:1667) ==27642== by 0x16CDCF: get_client_groupids (gravity-db.c:367) ==27642== by 0x16F51F: gravityDB_get_regex_client_groups (gravity-db.c:1342) ==27642== by 0x15E973: reload_per_client_regex (regex.c:429) ==27642== by 0x16EC23: gravityDB_reload_groups (gravity-db.c:1207) ==27642== by 0x16EC23: gravityDB_client_check_again (gravity-db.c:1223) ==27642== by 0x16EC23: in_whitelist (gravity-db.c:1235) ==27642== by 0x155797: _FTL_check_blocking.isra.3.part.4 (dnsmasq_interface.c:283) ==27642== by 0x156D83: _FTL_check_blocking (dnsmasq_interface.c:628) ==27642== by 0x156D83: _FTL_new_query (dnsmasq_interface.c:761) ==27642== by 0x190A77: receive_query (forward.c:1639) ==27642== by 0x18664B: check_dns_listeners (dnsmasq.c:1797) ==27642== ==27642== 18 bytes in 1 blocks are definitely lost in loss record 36 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x173587: getMACfromIP (network-table.c:1667) ==27642== by 0x16CDCF: get_client_groupids (gravity-db.c:367) ==27642== by 0x16E1D7: gravityDB_prepare_client_statements (gravity-db.c:839) ==27642== by 0x16ED2F: in_whitelist (gravity-db.c:1242) ==27642== by 0x155797: _FTL_check_blocking.isra.3.part.4 (dnsmasq_interface.c:283) ==27642== by 0x156D83: _FTL_check_blocking (dnsmasq_interface.c:628) ==27642== by 0x156D83: _FTL_new_query (dnsmasq_interface.c:761) ==27642== by 0x190A77: receive_query (forward.c:1639) ==27642== by 0x18664B: check_dns_listeners (dnsmasq.c:1797) ==27642== by 0x187E6F: main_dnsmasq (dnsmasq.c:1209) ==27642== ==27642== 27 bytes in 1 blocks are definitely lost in loss record 55 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x173C1B: getNameFromIP (network-table.c:1810) ==27642== by 0x16CEC3: get_client_groupids (gravity-db.c:483) ==27642== by 0x16F51F: gravityDB_get_regex_client_groups (gravity-db.c:1342) ==27642== by 0x15E973: reload_per_client_regex (regex.c:429) ==27642== by 0x16EC23: gravityDB_reload_groups (gravity-db.c:1207) ==27642== by 0x16EC23: gravityDB_client_check_again (gravity-db.c:1223) ==27642== by 0x16EC23: in_whitelist (gravity-db.c:1235) ==27642== by 0x155797: _FTL_check_blocking.isra.3.part.4 (dnsmasq_interface.c:283) ==27642== by 0x156D83: _FTL_check_blocking (dnsmasq_interface.c:628) ==27642== by 0x156D83: _FTL_new_query (dnsmasq_interface.c:761) ==27642== by 0x190A77: receive_query (forward.c:1639) ==27642== by 0x18664B: check_dns_listeners (dnsmasq.c:1797) ==27642== ==27642== 76 bytes in 8 blocks are definitely lost in loss record 110 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x171543: add_FTL_clients_to_network_table (network-table.c:702) ==27642== by 0x171543: parse_neighbor_cache (network-table.c:1367) ==27642== by 0x16C157: DB_thread (database-thread.c:82) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== ==27642== 132 bytes in 8 blocks are definitely lost in loss record 132 of 212 ==27642== at 0x484A164: calloc (in /usr/lib/aarch64-linux-gnu/valgrind/vgpreload_memcheck-arm64-linux.so) ==27642== by 0x2E0D13: FTLcalloc (calloc.c:27) ==27642== by 0x2E161F: FTLstrdup (strdup.c:26) ==27642== by 0x17155F: add_FTL_clients_to_network_table (network-table.c:703) ==27642== by 0x17155F: parse_neighbor_cache (network-table.c:1367) ==27642== by 0x16C157: DB_thread (database-thread.c:82) ==27642== by 0x49407E3: start_thread (pthread_create.c:486) ==27642== by 0x4A37ADB: thread_start (clone.S:78) ==27642== Looking at the code, this can only happen when there are (severe enough) errors with the Gravity database. When you to reproduce this (or in case you still have the rotated log files), is there anything inside How did you terminate FTL? It looks a bit like it was interrupted somewhere in between and haven't had a chance to properly clean up behind itself (hence all the |
|
I haven't been at home the last days, rotating log I terminate it with |
|
|
|
I also have no idea... fresh logs from today (some new sqlite errors in valgrind). Still "the crash" did not happen inside valgrind. pihole-FTL.logvalgrind.logTo give you an idea how easy I can trigger it |
|
Do the recorded movie and the logs match? I though you weren't able to trigger a crash with |
|
No, they don't match. The recording is with bare metal pihole-FTL. |
|
So with Please try again with It is an attempt to fix the crash you see at shutdown time. I'm not sure where the crash comes from exactly (I'm guessing it is a dangling SQLite3 lock), so we'll see if the fix works. Maybe the shutdown itself is hanging with this fix. |
|
I just updated the branch again. |
|
How many commits should be in the new branch? I only see one from today... Trying to stop FTL via Last lines are Nothing more happens |
|
Okay, this means that there was an API thread that could not be canceled/joined. Let's have another try... |
|
This worked, no crash, logs attached. |
|
The next version of FTL has been released. Please update and run to get back on-track. The fix/feature branch you switched to will not receive any further updates. Thanks for helping us to make Pi-hole better for us all! If you have any issues, please either reopen this ticket or (preferably) create a new ticket describing the issues in further detail and only reference this ticket. This will help us to help you best. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Versions
Current Pi-hole version is v5.2.1.
Current AdminLTE version is v5.2.1.
Current FTL version is v5.3.2.
Platform
Expected behavior
FTL should not crash using the audit log.
Actual behavior / bug
Might be the same bug as in #947 but this is marked as "closed" (in the sense of fixed) already.
Steps to reproduce
Steps to reproduce the behavior:
Use the audit log to "audit" domains. It doesn't crash always, but I could reliably crash FTL by randomly clicking the "audit" button fast in series
Debug Token
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
The text was updated successfully, but these errors were encountered: