Custom filters block resource too strictly

[luixxiul]

Describe the issue you encountered:
#6850 (comment)

when I enable */RealMedia/ uploading a file on this page by selecting the file is blocked.

Also no images on a PR page:

• Platform (Win7, 8, 10? macOS? Linux distro?): macOS

• Brave Version (revision SHA): f682695

• Steps to reproduce:

  1. Open a github page
  2. Open about:adblock
  3. Add */RealMedia/ following Youslist (Korean adblock list) block CSS and images #6850 (comment)
  4. Disable custom filters if they are loaded
  5. Reload the github page

• Actual result: every external resource is blocked

• Expected result: nothing would be blocked

• Extra QA steps:

  1. Open https://github.com/brave/browser-laptop/issues/new
  2. Click "selecting them"
  3. Select a file
  4. Make sure it is uploaded

• Any related issues:

[lukemulks]

thanks @luixxiul will investigate

[lukemulks]

@luixxiul I took a long look at this today - and I think at least part of issue is caused from is the placement of wildcards in the filters. I think there may be some conflicts between our default lists and regional lists. I think @bbondy may have pinged me about this a while back, and I think I may have made some inaccurate assumptions about how the lists were handled regionally at the time.

When I start pre-pending wildcards to the custom filters play with character casing in the filter rules, I get unexpected results.

For example,

• If I use */realmedia/ads/ as a custom filter, it completely breaks Twitter.com for me.
• If I use */RealMedia/ads/ it also breaks Twitter.com.
• If I use */REalMedia/ads/ Twitter.com works as expected.
• If I use */realMedia/ads/ Twitter.com works as expected.

The filters are treated as if they have a wildcard before and after the filter, unless character separators or other syntax are applied.

It looks like, by adding the wildcard in advance, if the string that follows is all lower case, or possibly matches an existing rule (i.e. RealMedia or ads), that it throws the blocking off for externally hosted resources. I'm not exactly sure what specific conditions cause this, since there is some variance, but it appears to be initially thrown off by the wildcard pre-pend, and character casing has some degree of impact as well.

@bbondy @SergeyZhukovsky
I think that I need to get a better understanding of how:

• Conflicting exceptions and blocking rules are handled for regional lists (easylist exception, vs regional blocking filter for the same filter, and vice versa)
• How custom filters may/may not override default or regional lists
• How our default lists are handled in priority compared to our regional lists.
• I also might need to get a little more info on how our privacy protection list is prioritized with the adblock engine lists, and how the privacy protection engine is handled with optional regional lists and custom filters.

Mainly, how conflicts are handled across lists, and what list takes priority.

If a regional list has a match or conflict with the privacy protection list, and then something like a wildcard is pre-pended, how is that throwing off the blocking? Things like that.

@bbondy if some of the above are unknowns w/the addition of the regional lists, that's totally cool too. I just need to know status so I can know which different test scenarios I should throw in the mix.

[luixxiul]

Thanks for investigation, appreciate that 😆

[luixxiul]

@lukemulks I feel youtube ads are not blocked because of that issue with regex and character casing, wdyt? Is this totally different stuff?

[bbondy]

@lukemulks I think this bit you:
brave-experiments/ad-block#33

[bbondy]

let me know if I should prioritize that higher