Manage adlists #252

DL6ER · 2016-12-15T16:39:01Z

Changes proposed in this pull request:

Enable/disable all lists from /etc/pihole/adlists.default from the Settings page
Add new ad lists to user-defined file that will be preserved on updates

This section is loaded in collapsed form since it needs quite some space of the Settings page...

@pi-hole/dashboard

…ttings page

AzureMarker · 2016-12-15T19:09:21Z

Could we add a text field on the bottom for custom lists? Then they can add them there. Also, how will we handle if the lists are changed in the repo and their adlists.list is out of date, but has their changes?

DL6ER · 2016-12-15T22:36:22Z

Sure, I can add a field where a user can add lists. I'd store them in an extra file, so the original adlists.list can always be updated, i.e. overwritten, by us when this it is necessary.

AzureMarker · 2016-12-16T01:22:53Z

I think it's more of a deeper problem. Since adlists.list is copied from adlists.default, it isn't updated with the normal update. I like the idea of adding extra lists in a separate file. Then we can change up a bit how we handle list updates. Perhaps only have one adlists.list file, which users can modify. On an update, we check the user's list against the updated list and add in the lists that are in the updated version but the the user's version? Then that raises the question of what to do if a list changes from active to commented out or flat deleted in an update. Do we comment out the user's list, delete the list from the file, etc? Or leave it up to the user?

DL6ER · 2016-12-16T09:05:06Z

Currently I do the following: Edit adlists.default directly such that it will get stashed and changed how we want it during an update. The extra.list would be separate. I'd propose to comment out everything in this file on update. That would lead to the situation that an user always has the configuration we think is the best after an update - however, his custom lists are not deleted, but only disabled for the moment. He can simply go to the Settings page and customize as he likes.

This might be some level of inconvenience connected to each update, but we can be sure that all users have a working situation (that we can control) after each update. Re-enabling the extra lists is done in seconds.

Conflicts: php/savesettings.php

…e list maintained by us will be reset on every update

…fficiency + speed)

… time (one URL per line)

DL6ER · 2016-12-16T14:36:08Z

Updated screenshot in PR description

DL6ER · 2016-12-16T14:50:36Z

@Mcat12 I'd propose to go with adlists.list + new adlists.user. Not sure what to do with an possibly already existing adlists.list, because I'd like to overwrite on every Pi-hole update like:

cp /etc/.pihole/adlists.default /etc/pihole/adlists.lists

to enforce an up-to-date list.

See also here

AzureMarker · 2016-12-16T20:20:52Z

Gravity actually already overrides the default list on every run. ;) https://github.com/pi-hole/pi-hole/blob/master/gravity.sh#L335

AzureMarker · 2016-12-16T20:22:16Z

That sounds like it should work well. Just will have to try and re-enable/disable the lists that are in the default list based on the user's prior settings, if possible.

AzureMarker · 2016-12-28T03:02:58Z

Does this require any Core changes?

AzureMarker · 2016-12-28T03:05:55Z

php/savesettings.php

+	{
+		while (($line = fgets($handle)) !== false)
+		{
+			if(substr($line, 0, 2) === "#h")


Could this check for #http instead? There might be a comment that starts like #here's some list...

AzureMarker · 2016-12-28T03:07:32Z

php/savesettings.php

+function readAdlists(&$list, $listname)
+{
+	// Reset list
+	$list = [];


Why take in a list and just rewrite it? Could this return a list instead?
$adlistsuser = readAdlists("user");

DL6ER · 2016-12-28T13:57:02Z

@Mcat12 These are the core changes:
Allow enabling/disabling of adlists via webUI pi-hole/pi-hole#988

DL6ER · 2017-01-02T14:11:04Z

WIP since we might change the format of the adlist to CSV

technicalpyro · 2017-03-10T21:05:09Z

i oticed this thread a few days ago and am wondering if the formatting of my personal adlist could be useful in it

###A word of warning (i.e: Please DO NOT just install all these lists at once):
###Installing all of these lists will result in the Query Log tool of Pi-hole to return an ajax datatables error, when it loads
###If this doesn't deter you, consider skipping a couple of lists known for false positives (e.g: hosts.eladkarako.com)
###If you have hardware with 1GB of RAM or more, you may increase PHP's memory: memory_limit = 180M within /etc/php5/fpm/php.ini OR /etc/php5/cgi/php.ini (FPM if it exists, otherwise CGI, and then sudo service php5-fpm restart OR sudo service php5-cgi restart)
###Raising PHP's memory beyond this suggestion may result in your hardware becoming unresponsive (freezing), as multiple 'children' with this limit are spawned via /etc/php5/fpm/pool.d/www.conf

##Suspicious Lists:

#Bill Stearn
http://v.firebog.net/hosts/BillStearns.txt
#CAMELEON
http://sysctl.org/cameleon/hosts
#DSHEILD Suspicious
https://www.dshield.org/feeds/suspiciousdomains_Low.txt
#Elad Karako
https://raw.githubusercontent.com/eladkarako/hosts.eladkarako.com/master/_raw__hosts.txt
#Joe Wein
http://www.joewein.net/dl/bl/dom-bl-base.txt
#Mahakala
https://adblock.mahakala.is
#hpHosts Spam
https://hosts-file.net/grm.txt
#Andy Short
http://hostsfile.mine.nu/hosts0.txt
#Piwik Referrer Spam
https://raw.githubusercontent.com/piwik/referrer-spam-blacklist/master/spammers.txt
#ReddestDream
https://raw.githubusercontent.com/ReddestDream/reddestdream.github.io/master/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
#Steven Blacks KAD
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
#Steven Blacks Spammers
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
#SecureMecca
http://securemecca.com/Downloads/hosts.txt
#Someone Who Cares
http://someonewhocares.org/hosts/zero/hosts
#Spam404
https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
#Vokins yhosts
https://raw.githubusercontent.com/vokins/yhosts/master/hosts
#Winhelp2002 MVPS
http://winhelp2002.mvps.org/hosts.txt

##Advertising Lists:

#AdAway
https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
#Disconnect Ads
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
#hpHosts Ads/Trackers
https://hosts-file.net/ad_servers.txt
#Easylist
http://v.firebog.net/hosts/Easylist.txt
#Steven Blacks Spotify Ads
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/SpotifyAds/hosts
#Steven Blacks Unchecky Ads
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts

##Tracking & Telemetry Lists:

#Airelle Trackers
http://v.firebog.net/hosts/Airelle-trc.txt
#Easyprivacy
http://v.firebog.net/hosts/Easyprivacy.txt
#Quidsup Trackers
https://raw.githubusercontent.com/quidsup/notrack/master/trackers.txt
#Steven Blacks Trackers
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
#Steven Blacks Microsoft Telemetry
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/tyzbit/hosts

##Malicious Lists:

#Airelle High Risk
http://v.firebog.net/hosts/Airelle-hrsk.txt
#Disconnect Malvertising
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
#DNS-BH Malware
http://mirror1.malwaredomains.com/files/justdomains
#hpHosts Exploit
https://hosts-file.net/exp.txt
#hpHosts Malware
https://hosts-file.net/emd.txt
#hpHosts Phishing
https://hosts-file.net/psh.txt
#'Immortal' Malware Domains
https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
#Malware Domain List
: https://www.malwaredomainlist.com/hostslist/hosts.txt
#Mandiant APT1 Report
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
#Openphish
https://openphish.com/feed.txt
#Quidsup Malicious
https://raw.githubusercontent.com/quidsup/notrack/master/malicious-sites.txt
#Ransomware Tracker
https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
#Steven Blacks Risky Hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
#Shalla Malicious
http://v.firebog.net/hosts/Shalla-mal.txt
#Zeus Tracker
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist

AzureMarker · 2017-03-31T22:43:25Z

AzureMarker · 2017-04-01T00:19:24Z

DL6ER · 2017-04-01T04:22:31Z

Why should the user want to delete them when he can disable them?

technicalpyro · 2017-04-01T15:40:40Z

is there a way to label the lists using a more descriptive title if is is present in the adlists.list file ?

PromoFaux · 2017-04-02T15:30:14Z

@DL6ER some people might just want to have a clean slate, and not use any of the defaults. Why have them all in the list disabled if they're never going to use them?

@technicalpyro possibly, for now I think it's nice just to get the functionality out there and work on the polish separately!

Enable/disable all lists from /etc/pihole/adlists.default from the Se…

af0b405

…ttings page

DL6ER mentioned this pull request Dec 15, 2016

[WIP] Allow enabling/disabling of adlists via webUI pi-hole/pi-hole#988

Closed

6 tasks

DL6ER added the Enhancement label Dec 15, 2016

Merge branch 'devel' into customizeadlists

dee11cb

Conflicts: php/savesettings.php

DL6ER added this to the next release milestone Dec 16, 2016

DL6ER added 4 commits December 16, 2016 13:47

Generalized the functions to allow different ad files to be loaded

51f1bfe

Be able to add lists to the user-defined list + tell the user that th…

7b87301

…e list maintained by us will be reset on every update

Only enable/disable those lists where the status should be changed (e…

6e7020a

…fficiency + speed)

Be able to enable/disable user lists and add multiple user lists at a…

c73c10f

… time (one URL per line)

DL6ER changed the title ~~Enable/disable all lists from /etc/pihole/adlists.default from the Settings page~~ Manage adlists [Settings] Dec 16, 2016

DL6ER changed the title ~~Manage adlists [Settings]~~ [WIP] Manage adlists [Settings] Dec 16, 2016

DL6ER modified the milestones: v2.1, next release Dec 16, 2016

DL6ER changed the title ~~[WIP] Manage adlists [Settings]~~ Manage adlists [Settings] Dec 27, 2016

DL6ER added the Ready for Check and Merge label Dec 27, 2016

DL6ER added 3 commits December 27, 2016 19:55

Merge branch 'devel' into customizeadlists

b51fbb0

Add notice on whitelist page that list domains are automatically added

3e8e153

Add shortcut to Update lists (since it is now hidden under Tools)

c9f9d5f

DL6ER changed the title ~~Manage adlists [Settings]~~ Manage adlists Dec 27, 2016

AzureMarker reviewed Dec 28, 2016

View reviewed changes

PromoFaux modified the milestones: v2.4, v2.3 Jan 1, 2017

DL6ER added 2 commits January 2, 2017 14:43

Merge branch 'devel' into customizeadlists

4b3173b

Add "Save and Update" button

ffe4000

DL6ER changed the title ~~Manage adlists~~ [WIP] Manage adlists Jan 2, 2017

DL6ER mentioned this pull request Jan 16, 2017

Feature Requests pi-hole/pi-hole#347

Closed

10 tasks

AzureMarker modified the milestones: v2.5, v2.4 Jan 27, 2017

DL6ER removed the Ready for Check and Merge label Jan 29, 2017

DL6ER removed this from the v2.5 milestone Mar 5, 2017

PromoFaux and others added 5 commits March 31, 2017 21:05

modify to work with changes in pihole#1365

c11172d

move to a more prominent location

42f4c51

Lists are no longer reset on updates

1f6c95a

Add delete checkbox

87b92d4

Handle deletion of list in savesettings.php

86d8869

AzureMarker added 2 commits March 31, 2017 20:16

Use button to remove lists

7a5c8e8

Change wording of block list label

db54cda

PromoFaux changed the title ~~[WIP] Manage adlists~~ Manage adlists Apr 4, 2017

AzureMarker approved these changes Apr 5, 2017

View reviewed changes

:codacy:

892bc5f

AzureMarker approved these changes Apr 6, 2017

View reviewed changes

AzureMarker merged commit acc8334 into devel Apr 6, 2017

AzureMarker deleted the customizeadlists branch April 6, 2017 20:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manage adlists #252

Manage adlists #252

DL6ER commented Dec 15, 2016 •

edited

Loading

AzureMarker commented Dec 15, 2016

DL6ER commented Dec 15, 2016

AzureMarker commented Dec 16, 2016

DL6ER commented Dec 16, 2016

DL6ER commented Dec 16, 2016

DL6ER commented Dec 16, 2016 •

edited

Loading

AzureMarker commented Dec 16, 2016

AzureMarker commented Dec 16, 2016

AzureMarker commented Dec 28, 2016

AzureMarker Dec 28, 2016

AzureMarker Dec 28, 2016

DL6ER commented Dec 28, 2016

DL6ER commented Jan 2, 2017

technicalpyro commented Mar 10, 2017

AzureMarker commented Mar 31, 2017

AzureMarker commented Apr 1, 2017 •

edited

Loading

DL6ER commented Apr 1, 2017

technicalpyro commented Apr 1, 2017

PromoFaux commented Apr 2, 2017

Manage adlists #252

Manage adlists #252

Conversation

DL6ER commented Dec 15, 2016 • edited Loading

AzureMarker commented Dec 15, 2016

DL6ER commented Dec 15, 2016

AzureMarker commented Dec 16, 2016

DL6ER commented Dec 16, 2016

DL6ER commented Dec 16, 2016

DL6ER commented Dec 16, 2016 • edited Loading

AzureMarker commented Dec 16, 2016

AzureMarker commented Dec 16, 2016

AzureMarker commented Dec 28, 2016

AzureMarker Dec 28, 2016

Choose a reason for hiding this comment

AzureMarker Dec 28, 2016

Choose a reason for hiding this comment

DL6ER commented Dec 28, 2016

DL6ER commented Jan 2, 2017

technicalpyro commented Mar 10, 2017

AzureMarker commented Mar 31, 2017

AzureMarker commented Apr 1, 2017 • edited Loading

DL6ER commented Apr 1, 2017

technicalpyro commented Apr 1, 2017

PromoFaux commented Apr 2, 2017

DL6ER commented Dec 15, 2016 •

edited

Loading

DL6ER commented Dec 16, 2016 •

edited

Loading

AzureMarker commented Apr 1, 2017 •

edited

Loading