Work around with HTTPSE breaking ruleset format changes #3

cschanaj · 2018-01-11T07:07:55Z

Close EFForg/https-everywhere/issues/12590, Close brave/browser-laptop/issues/10976
Needed for brave/browser-laptop/issues/11598

Hopefully I did not break anything...

Bisaloo · 2018-02-12T15:41:12Z

@cschanaj, could you use GitHub auto-closing feature please? It works across folders

https://help.github.com/articles/closing-issues-using-keywords/

Also @diracdeltas, not sure if you were automatically pinged on this.

diracdeltas · 2018-02-12T15:43:44Z

Thanks, I did not see this. Will look at it today. Does this address brave/browser-laptop#10976 (comment) ?

diracdeltas · 2018-02-12T20:25:43Z

scripts/preloadHTTPSE.js

 const levelup = require('level')
 const rmDir = require('./util').rmDir
 const exec = require('child_process').exec

-const xpiVersion = '5.2.21' // Manually update this to latest version
+const xpiVersion = '2017.12.6' // Manually update this to latest version


please also bump the version in package.json to 6.0.0. this creates a new ruleset download in S3 instead of overwriting the current one (safer in case the new one has breaking changes).

diracdeltas · 2018-02-12T20:31:59Z

scripts/preloadHTTPSE.js

-  }
-
-  console.log('Writing httpse.json')
-  fs.writeFileSync('./out/httpse.json', JSON.stringify(rulesets), 'utf8')


we still need to create httpse.json since it's used on the Brave desktop browser (only mobile uses leveldb)

diracdeltas · 2018-02-12T20:32:46Z

scripts/preloadHTTPSE.js

@@ -33,84 +32,40 @@ const downloadRulesets = (dir, cb) => {
 }

 const buildDataFiles = () => {
-  // Manually exclude sites that are broken until they are fixed in the next
-  // HTTPS Everywhere release.
-  const exclusions = {


we don't want to delete these exclusions unless all of these rulesets are verified to be fixed in HTTPS Everywhere upstream

cschanaj · 2018-02-13T02:04:37Z

@diracdeltas I've made corresponding changes in my PR. Do I need update the xpiVersion (to 2018.1.29 or even latest) as well? It has been 2 months after 2017.12.6 was released.

diracdeltas · 2018-02-13T22:13:31Z

@cschanaj would be great to update it to 2018.1.29. we avoid latest because we want to pin the version number.

cschanaj · 2018-02-13T22:45:20Z

@diracdeltas please check. thanks!

PS. The original JSON format is not that straight forward...

{
   "ruleset":{
      "$":{
         "name":"101domain.com (partial)",
         "default_off":"failed ruleset test", // gone
         "f":"101domain.com.xml" // gone
      },
      "exclusion":[
         {
            "$":{
               "pattern":"^http://(?:portuguese|spanish)\\.101domain\\.com/+(?!css/|favicon\\.ico|images/)"
            }
         }
      ],
      "rule":[
         {
            "$":{
               "from":"^http://russian\\.101domain\\.com/[^?]*",
               "to":"https://www.101domain.ru/"
            }
         },
         {
            "$":{
               "from":"^http:",
               "to":"https:"
            }
         }
      ]
   }
}

cschanaj · 2018-02-13T23:19:16Z

scripts/preloadHTTPSE.js

+  }
+
+  for (const ruleset of rulesets) {
+    if (!ruleset.default_off && !ruleset.platform) {


the original code include default_off and platform ruleset in httpse.json, which is gone now.

diracdeltas · 2018-02-13T23:57:50Z

scripts/preloadHTTPSE.js

+    'Cisco.com (partial)': 'breaks http://www.cisco.com/c/m/en_us/training-events/events-webinars/techwise-tv/listings.html',
+    'GQ.com (partial)': 'mixed content on gq.com',
+    'Where 2 Get It (partial)': 'maps missing on http://us.coopertire.com/Customer-Care/Dealer-Locator.aspx?form=locator_search&addressline=92346',
+    'Thompson Hotels.com (partial)': 'missing stylesheets on http://www.thompsonhotels.com/'


thanks for going through and re-mapping these!

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas · 2018-02-14T00:06:15Z

@cschanaj yup, there's no reason to have the HTTPSE.json be as complicated as it used to be. i opened brave/browser-laptop#13133 so you can simplify it somewhat by flattening out the $ properties.

so the new schema can be:

{
   "ruleset":{
      "name": "101domain.com (partial)",
      "exclusion":[
         {
               "pattern":"^http://(?:portuguese|spanish)\\.101domain\\.com/+(?!css/|favicon\\.ico|images/)"
         }
      ],
      "rule":[
         {
               "from":"^http://russian\\.101domain\\.com/[^?]*",
               "to":"https://www.101domain.ru/"
         },
         {
               "from":"^http:",
               "to":"https:"
         }
      ]
   }
}
`

diracdeltas · 2018-02-14T00:15:36Z

scripts/preloadHTTPSE.js

+        })
+      }
+
+      if (ruleset.securecookie) {


we don't have plans to use securecookie in browser-laptop so this can be omitted

thanks for submitting a PR upstream!! I have remove the securecookie properties and simplified the schema in 31d9b8f.

diracdeltas · 2018-02-15T07:39:59Z

This works great, thank you! Please run rm -r node_modules/ and npm install so that it updates package-lock.json. Then commit the new package-lock.json file.

cschanaj · 2018-02-15T11:38:09Z

@diracdeltas done. thanks!!

diracdeltas · 2018-02-15T17:25:11Z

@bsclifton this can be squashed and merged :). once that is done, please run npm run build and npm run upload to upload it to the S3 test bucket. I'll ping you again once it's ready for upload to prod. (https://github.com/brave/https-everywhere-builder/blob/master/scripts/uploadDataFiles.js#L44)

- Update preloadHTTPSE.js - Update package.json - Bump version in package.json to 6.0.0 - Update scripts/preloadHTTPSE.json - Update to HTTPS Everywhere 2018.1.29 - Changes as per npm run lint - Update scripts/preloadHTTPSE.js - Simplify HTTPSE.json schema - Update package-lock.json

bsclifton

Rebased / squashed per comments by @diracdeltas 😄

bsclifton · 2018-02-15T22:29:00Z

@diracdeltas built + uploaded 👍

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

bsclifton · 2018-02-16T06:18:01Z

pushed to prod with https://jenkins.brave.com/view/laptop%20updates/job/https-everywhere-builder/10/

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas assigned diracdeltas and unassigned diracdeltas Feb 12, 2018

diracdeltas self-requested a review February 12, 2018 15:45

diracdeltas reviewed Feb 12, 2018

View reviewed changes

cschanaj changed the title ~~Https everywhere 2017.12.6~~ Work around with HTTPSE breaking ruleset format changes Feb 13, 2018

cschanaj commented Feb 13, 2018

View reviewed changes

diracdeltas reviewed Feb 13, 2018

View reviewed changes

diracdeltas added a commit to brave/browser-laptop that referenced this pull request Feb 14, 2018

[HTTPS Everywhere] simplify ruleset pattern for HTTPSE 6.0

e15dcd5

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

diracdeltas mentioned this pull request Feb 14, 2018

[HTTPS Everywhere] simplify ruleset pattern for HTTPSE 6.0 brave/browser-laptop#13133

Merged

10 tasks

diracdeltas reviewed Feb 14, 2018

View reviewed changes

diracdeltas previously approved these changes Feb 15, 2018

View reviewed changes

bsclifton dismissed diracdeltas’s stale review via 648c764 February 15, 2018 22:17

bsclifton approved these changes Feb 15, 2018

View reviewed changes

bsclifton merged commit 272d8c4 into brave:master Feb 15, 2018

cschanaj deleted the https-everywhere-2017.12.6 branch February 15, 2018 23:51

diracdeltas added a commit to brave/browser-laptop that referenced this pull request Feb 16, 2018

[HTTPS Everywhere] update to latest HTTPS Everywhere

a5c9936

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

bsclifton pushed a commit to brave/browser-laptop that referenced this pull request Feb 16, 2018

[HTTPS Everywhere] update to latest HTTPS Everywhere

4df2075

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

Bisaloo mentioned this pull request Feb 22, 2018

Brave using 5.2 release (outdated) for rulesets EFForg/https-everywhere#12590

Closed

ryanml pushed a commit to ryanml/browser-laptop that referenced this pull request Feb 27, 2018

[HTTPS Everywhere] update to latest HTTPS Everywhere

eb1078a

Needed for brave/https-everywhere-builder#3 Note that this is a breaking change from HTTPS Everywhere 5.x's ruleset format.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work around with HTTPSE breaking ruleset format changes #3

Work around with HTTPSE breaking ruleset format changes #3

cschanaj commented Jan 11, 2018 •

edited by diracdeltas

Loading

Bisaloo commented Feb 12, 2018

diracdeltas commented Feb 12, 2018

diracdeltas Feb 12, 2018

diracdeltas Feb 12, 2018

diracdeltas Feb 12, 2018

cschanaj commented Feb 13, 2018 •

edited

Loading

diracdeltas commented Feb 13, 2018

cschanaj commented Feb 13, 2018 •

edited

Loading

cschanaj Feb 13, 2018

diracdeltas Feb 13, 2018

diracdeltas commented Feb 14, 2018 •

edited

Loading

diracdeltas Feb 14, 2018

cschanaj Feb 14, 2018

diracdeltas commented Feb 15, 2018 •

edited

Loading

cschanaj commented Feb 15, 2018

diracdeltas commented Feb 15, 2018

bsclifton left a comment •

edited

Loading

bsclifton commented Feb 15, 2018

bsclifton commented Feb 16, 2018

Work around with HTTPSE breaking ruleset format changes #3

Work around with HTTPSE breaking ruleset format changes #3

Conversation

cschanaj commented Jan 11, 2018 • edited by diracdeltas Loading

Bisaloo commented Feb 12, 2018

diracdeltas commented Feb 12, 2018

diracdeltas Feb 12, 2018

Choose a reason for hiding this comment

diracdeltas Feb 12, 2018

Choose a reason for hiding this comment

diracdeltas Feb 12, 2018

Choose a reason for hiding this comment

cschanaj commented Feb 13, 2018 • edited Loading

diracdeltas commented Feb 13, 2018

cschanaj commented Feb 13, 2018 • edited Loading

cschanaj Feb 13, 2018

Choose a reason for hiding this comment

diracdeltas Feb 13, 2018

Choose a reason for hiding this comment

diracdeltas commented Feb 14, 2018 • edited Loading

diracdeltas Feb 14, 2018

Choose a reason for hiding this comment

cschanaj Feb 14, 2018

Choose a reason for hiding this comment

diracdeltas commented Feb 15, 2018 • edited Loading

cschanaj commented Feb 15, 2018

diracdeltas commented Feb 15, 2018

bsclifton left a comment • edited Loading

Choose a reason for hiding this comment

bsclifton commented Feb 15, 2018

bsclifton commented Feb 16, 2018

cschanaj commented Jan 11, 2018 •

edited by diracdeltas

Loading

cschanaj commented Feb 13, 2018 •

edited

Loading

cschanaj commented Feb 13, 2018 •

edited

Loading

diracdeltas commented Feb 14, 2018 •

edited

Loading

diracdeltas commented Feb 15, 2018 •

edited

Loading

bsclifton left a comment •

edited

Loading