First discussion

[technicalpyro]

based on the pihole script installing under the user "pihole" would it fit into the code that at the end of what im guessing will be the eventual bash/whiptail install script to have it run the following commands

sudo touch /etc/pihole/FTL.log
sudo touch /etc/pihole/FTL.pid
sudo touch /etc/pihole/FTL.lport
sudo chown pihole:pihole /etc/pihole/FTL.*

i would guess that is the eventual plan but dont want to assume

so far despite not running with a web interface as it is not agreeing at the moment every operation i have run has been successful and damn fast

[dschaper]

Yeah, the permissions and location of the files are most likely going to be updated as the engine takes shape. It's in preview release now for functionality, the details for other items will be tuned as we progress along. Web interface interaction is something that is a bit off, but the CLI directions should cover most of what you can do for now.

There is a log file and a script to generate a massive log file to see how performant the code is when accessing data that would crush an existing install, and the results are indeed fast!

Thanks for the information and comments and please post any other observations or ideas that you have and we'll handle them as we can.

[technicalpyro]

My current install is blocking just over 1.24 million domains and a command like >getallqueries which using the current web interface i really can't run due to the ajax error runs in seconds or less

couple features I would love to see:

an ability to view more than just the last blocked address ie >recentBlocked-x where x is the number of lines you want to view

the same could be said for getallquieries-x kind of an idea the time ability s nice but if you dont understand the timecode the database is using its difficult

knowing what the codes mean in the getallqueries results the numbers at the end of the lines would be nice as well or show it as a formatted table but that is definitely just a feature not a requirement

i understand right now this is alpha with me being guinea pig extraordinaire and everything that should be working is working
just me thinking out loud to be honest.

you guys rock and im happy to be allowed to play i really am :)

[DL6ER]

I like the idea >recentBlocked-x and >getallquieries-x and will add them.

in the getallqueries results the numbers at the end of the lines would be nice as well

I don't understand that?

The web interface implementation is finished but kept back for a while since we have to sort out how we do the licensing on that as it will integrate into the already published AdminLTE repo code

[technicalpyro]

in the getallqueries results the numbers at the end of the lines would be nice as well
I don't understand that?

1486179259 IPv4 api-v2launch.trakt.tv 172.16.1.107 2

the number two at the end of the line ( in this case) sometimes its a 1 or a 3 as well i just dont understand what it refers to

[PromoFaux]

just dont understand what it refers to

I have't actually read the FTL code properly yet, but I'm going to take a guess at the number being the pi-holed/OK status. I'm going to guess... 1=OK, 2=Pi-holed(exact), 3=Pi-holed(willdcard)

[technicalpyro]

That would make sense i had looked through but got lost in the code a few times

[DL6ER]

The statuses are assigned here:

• 0 = unknown, that should not happen (and has not happened to me so far), but could potentially happen - these lines will be written to the FTL log for possible subsequent investigation
• 1 = blocked by gravity.list (i.e. exact blacklisting)
• 2 = forwarded to upstream server
• 3 = answered by local cache
• 4 = wildcard blocking

[DL6ER]

or show it as a formatted table but that is definitely just a feature not a requirement

That is a difficult one since I don't know how long the domains and client host names can get, i.e. two unknown widths. If there would have been only one unknown width (like the domain name), I could have moved that to the end and could it have stay out of a preformatted table but I don't know how to do it as it is right now.

[technicalpyro]

yeah all the research I've tried to do tells me the limit is 255 bytes but that doesn't necessarily translate to x amount of characters

perhaps attempt local host names limit of 25 characters and see where it goes or just ignore it because it was a want not a need

[DL6ER]

yeah all the research ive tried to do tells e its 255 bytes but that doesnt necessarily translate to x amount of characters

Well, up to 255 characters

perhaps attempt local hostnames limit of 25 characters and see where it goes or just ignore it because it was a want not a need

I know about at least one user that has host names which are much longer than that because they reflect quite some hierarchy in the network, like:

device.room.building.field.company.tld

that can easily become > 25 characters

[technicalpyro]

Yeah for sure. i forgot about those who implement on the larger basis. My bad.

[technicalpyro]

installed the latest commits
change of filename structure to include pihole-FTL caused some issues all good now

[dschaper]

We've got to keep you on your toes! :)

[technicalpyro]

yeah and i learn from it too :)
Edit: tested >recentBlocked 10 works great

[DL6ER]

Oh, I wrote some text and somehow that seems to have gotten lost.

Yeah, you already discovered two changes:

• Filestructure (will change again to some more Linux-typical places like `/var/log/, etc. soon)
• >recentBlocked XY
• There is also >getallqueries (15). The ( ) have the nice effect that I was able to extend the scope to all already available filters, e.g.
  • >getallqueries (2) -> show only the recent two queries
  • >getallqueries-client 127.0.0.1 (10) show only the queries from localhost within the recent 10 queries (might return nothing)
  • >getallqueries-domain www.google.com (50)
  • etc.

Maybe I should put in the ( ) syntax everywhere for consistency?

[DL6ER]

The file locations have been changed to

/var/log/pihole-FTL.log
/var/run/pihole-FTL.pid
/var/run/pihole-FTL.port

You can use make && sudo make install to have the files created with proper permissions.

[technicalpyro]

Done this morning before work. will be back in approx three hours to reap the benefits. definitely like being able to control very easily the number of results returned very clean code:)

[DL6ER]

Yeah, but I'm not entirely satisfied by the new code, since e.g. >getallqueries-client 127.0.0.1 (10) will show only the queries from localhost within the recent 10 queries (might return nothing) instead of (what I'd expect to see) 10 queries from localhost.

The problem is that we print out the latest query as the end of the results, i.e. I don't know where to start printing.

[technicalpyro]

oh OK so it doent know to filter the last ten result containing that client ... you guys are t==he coding masters but will think about it while doing my shift today maybe i can think of something inelegant you guys can turn to gold

[technicalpyro]

might have to be a silent command kind of idea and i dont know how to code this but the workflow for me would look something like

• query database filtering for "domain" or "host" depending on what is specified
• print results to a temporary file
• query temporary file for number of results requested
• Print result to window
• delete temporary file

[dschaper]

Sounds like a good use case for a hashtable. (Coding a hash in C may not be trivial, I don't know that much yet...)

[DL6ER]

To be honest, I'm not sure if that is worth the effort. I'd have to do all the fancy filtering once, extract only what I need and loop over everything a second time, printing only what I need. This will not only slow down FTL but also (considerable) increase the memory needed for the request routines and make the code more complicated.

That is because I don't collect the data somewhere and print it only afterwards, but immediately push every matching data set out to the requester. This is one of the reasons why FTL is so fast.

[technicalpyro]

Will download and install release and report back when I get home ... Got called in for a stupidly long shift today

[technicalpyro]

release seem to install fine although despite running >kill to terminate before update when i went to start the service it echoed already running .. didnt realize it was started by the make && make install
not sure if it was implemented but >getallqueries 10 shows all not just the ten i thought it was limited to

[DL6ER]

didnt realize it was started by the make && make install

That does not happen. Either you had another instance of pihole-FTL running somewhere or we have a bug. Have you used sudo make install?

not sure if it was implemented but >getallqueries 10 shows all not just the ten i thought it was limited to

Use >getallqueries (10)

[technicalpyro]

ok did a clean install following same steps i did last night and i must not have killed the process like i thought everything is working well and installed as expected

also using the () with getallqueries worked perfectly

[technicalpyro]

god im slow some days got it sorted see other issue for info for some reason not letting me log in via CLI but got the right info where it needs to be

[technicalpyro]

thinking i might put osmc on a new card and see if we run into the same issues we are seeing on the master thoughts?
or is there a different OS i should be testing ?

[DL6ER]

Oh, I might have missed that - which issue?

[technicalpyro]

ps -aux has dnsmasq at 43% of memory. also probably due to something i missed when setting up the vps i dont show temp or FTL status in the dashboard and lastly and i will be submitting a bug report on this on the main repo the package dialog if it is missing breaks the install script the os provided by my host didnt include that so might be a good idea to include it in required packages that are loaded or looked for before installation

[DL6ER]

ps -aux has dnsmasq at 43% of memory

Okay, we can attribute that to the insane number of domains

dont show temp or FTL status

Temperature is not displayed if it cannot be detected - they does not work on most VMs (virtual machines have nothing like a "physical" temperature). Remember that there is actually no FTL status display except if FTL is offline.

[technicalpyro]

thats what i figured

[DL6ER]

FTL is now merged to Pi-hole development. Closing this very long issue now. Thanks for the very nice discussion. Feel free to open another discussion at any point!