Download network dapps and load them from filesystem #144

axelchalon · 2018-06-21T17:46:05Z

Network dapps are now completely handled by Parity UI.

Fetch icon hash, manifest hash, content hash from registry
Resolve to a URL using GitHub Hint
Download file as ~/.config/parity-ui/hashfetch/partial/<hash>.part
Once the download is finished, move it directly to ~/.config/parity-ui/hashfetch/files/<hash> if it's a regular file ; if it's a zip archive, extract it temporarily to ~/.config/parity-ui/hashfetch/partial-extract/<hash> and then move it to ~/.config/parity-ui/hashfetch/files/<hash>

Manifests and icons are fetched on the dapps list page (Dapps); the dapps themselves (their contents) are fetched on demand, when the user wants to open the dapp. Added a loading screen if the dapp takes more than 1.5 second to load (we then assume it's a network dapp and that it's being downloaded).

Tomek advised me to set an upper limit for the downloads :

The upper limit is just to prevent your node from downloading some super huge zip file that may crash the node (or just generate a lot of load on the target server - so pretty much a ddos attack)

I set 20MB as max file size to download, regardless of whether it's a regular file (manifest or icon) or a dapp zip archive.

Tomek's reply on my suggestion to blacklist the URL locally if the download fails (to prevent DDoSing a target victim server)

We can't really blacklist the destination. Maybe we can do that temporarily cause the file on the server might change in the future, same if the hash does not match
actually, given what we have seen with the uploader we should probably ban it temporarily with exponential increase so like, ban for 30 seconds, then 60s, 120s, and so on

In case of hash mismatch or file too big (not in case of connection error though), we register the timestamp of the failed attempt, and the next time we request this hash with this url, we check if enough time has elapsed. Failed attempts history is stored in hashfetch/failed_history.json

Could only test on Kovan. Would be nice to compare master and this branch on the mainnet to see if there are differences (in the dapps displayed, rejected, working..)

Other notes:

Dapp contents whose GitHub Hint entry points to a single file are rejected (we expect a zip archive)
Dapp contents whose GitHub Hint entry pattern designates the URL to a regular file (commit 0x0000000000000000000000000000000000000000) rather than the URL to a zip archive (commit 0x0000000000000000000000000000000000000001) are rejected
@amaurymartiny any idea of stuff that is obsolete now that we handle dapps completely on the Parity UI side? maybe the proxies? maybe in there https://github.com/parity-js/shell/blob/master/.gitlab-ci.yml?

Let me know what you think!

(cc @tomusdrw)

axelchalon · 2018-06-21T17:49:06Z

webpack/app.js

-  node: {
-    fs: 'empty'
-  },
+  target: 'electron-renderer',


Otherwise in src/ I couldn't use modules that used require('fs') (the latter would return undefined), even though window.require('fs') worked fine in my src/ files.

This also fixes having to use window.require instead of require in src/

Not sure if there are consequences to this (some warnings and errors occur during bundling -- CI doesn't pass because of this)

No problem, the react app is exactly in the renderer process. I'll have a look at the CI error, building locally works fine.

axelchalon · 2018-06-21T17:56:04Z

src/util/hashFetch/expoRetry.js

+
+  // We store the URL to allow GH Hint URL updates for a given hash
+  // to take effect immediately, and to disregard past failed attempts at
+  // getting this file from other URLs.


Attempts are keyed on hash+url right now.

Should we? It's a DDoS vector if the target host is the same. If the attacker registers (some hash, "http://victim-website.com/") on GH Hint and then every few minutes changes the URL to http://victim-website.com/?x http://victim-website.com/?xx http://victim-website.com/?xxx then ExpoRetry isn't used at all..
Or is this a non-issue because of costs etc.?

I'm thinking in particular if somebody registers a dapp with the icon or manifest URL in GitHub Hint set to some big file like http://victim-website.com/bigfile.zip
Parity UI tries to download the icons and manifests of network daps each time the user goes to the dapps list (homepage). If the download failed (file too big), ExpoRetry blacklists {url,hash} exponentially; however this exponential increase will be reset each time the URL updates on GH Hint. If the attacker regularly updates the URL to http://victim-website.com/bigfile.zip?x etc. then victim website has a bad time.

This could be fixed if we key the attempts based on the hash only (although the attacker could regularly update the manifest/icon hash in the dapp registry..). What do you think?

cc @tomusdrw

Sorry just noticed this comment today, can we maybe use ExpoRetry on hash, not the destination?

That said, modifying the entry is not really something that we should consider - it will always be possible either by modifying the URL or the hash itself.

axelchalon · 2018-06-21T17:57:54Z

electron/index.js

@@ -45,18 +49,10 @@ function createWindow () {

  mainWindow = new BrowserWindow({
    height: 800,
-    width: 1200
+    width: 1200,
+    webPreferences: { nodeIntegrationInWorker: true }


Nodeintegration is enabled for the BrowserWindow but disabled by default for its WebWorkers; I had to add this option to remove the following error (wasn't blocking though):

Should be fine, we are not requiring any external files as web workers.

Is the error stack longer? Who is requiring @parity/shared/lib/webWorker.js? I'd be more enthusiast to remove web workers altoghether.

that's the whole error stack

we import some files of @parity/shared (for example import { initStore } from '@parity/shared/lib/redux' in index.parity.js) that require the worker of @parity/shared (in that case, initStore calls setupWorker). I imagine that's where it's coming from

axelchalon · 2018-06-24T19:29:30Z

Does this need to be updated?

shell/electron/index.js

Lines 92 to 104 in 5ad324f

    
           session.defaultSession 
        
             .setPermissionRequestHandler((webContents, permission, callback) => { 
        
               if (!webContents.getURL().startsWith('file:')) { 
        
                 // Denies the permissions request for all non-file://. Currently all 
        
                 // network dapps are loaded on http://127.0.0.1:8545, so they won't 
        
                 // have any permissions. 
        
                 return callback(false); 
        
               } 
        
               // All others loaded on file:// (shell, builtin, local) can have those 
        
               // permissions. 
        
               return callback(true); 
        
             });

Also, this becomes outdated:

shell/src/inject.js

Lines 91 to 93 in 5ad324f

    
           // For now we simply allow eval(). All builtin dapps are trusted and can use 
        
           // eval(), and all network dapps are served on 127.0.0.1:8545, which have CSP 
        
           // that disallow eval(). So security-wise it should be enough.

tomusdrw

Looks good!

tomusdrw · 2018-07-02T09:10:21Z

src/Dapps/store.js

+        if (app.type === 'network') {
+          return HashFetch.get().fetch(this._api, app.contentHash, 'dapp')
+            .then(appPath => {
+              app.localUrl = `file://${appPath}/index.html`;


Does it work on windows? I thought that /// is required

Good question! Need to check

tomusdrw · 2018-07-02T09:12:11Z

src/util/dapps.js

+      const manifestHash = bytesToHex(manifestId).substr(2);
+
+      return Promise.all([
+        HashFetch.get().fetch(api, imageHash, 'file').catch((e) => { console.warn(`Couldn't download icon for dapp ${appId}`, e); }),


this will resolve the promise to undefined in case of error, is that ok?

yes, this is expected. we want to still display the dapp if the icon fetch failed. the only consequence is that the icon url will be file://undefined

tomusdrw · 2018-07-02T09:15:09Z

src/util/hashFetch/expoRetry.js

+  load () {
+    const filePath = this._getFilePath();
+
+    return fsExists(filePath)


I'd skip fsExists, it's the same as just reading the file.

tomusdrw · 2018-07-02T09:18:40Z

src/util/hashFetch/expoRetry.js

+
+        return fsWriteJson(this._getFilePath(), this.failHistory);
+      }
+    });


Since we are not handling errors here it seems that if it fails once it will never be attempted again, right? Should we at least print a warning if that happens?

Nice catch! (or rather, missing catch :P) I don't think we need to never attempt writing again if it failed once. I'll add a catch

tomusdrw · 2018-07-02T09:19:24Z

src/util/hashFetch/index.js

+}
+
+function checkHashMatch (hash, path) {
+  return fsReadFile(path).then(content => {


This will read the whole file to memory, might be better to just hash without allocating. In general we should avoid reading the files that were fetched over the network, better to just stream them if it's possible.

tomusdrw · 2018-07-02T09:21:44Z

src/util/hashFetch/index.js

+  const finalPath = path.join(getHashFetchPath(), 'files', hash);
+
+  return download(url, tempPath)
+      // Don't register a failed attempt if the download failed; the user might be offline.


We should actually distinguish between:

User offline

Size limit reached

Server returned non-200 response

We can only skip registering failed attempt in the first case, but skipping it in the other two opens up a ddos vector

You're right. I think it's easier for now to register the failed attempt in any case, and assume the user is online.

tomusdrw · 2018-07-02T09:23:11Z

src/util/hashFetch/index.js

+    });
+}
+
+function download (url, destinationPath) {


What will happen if the url is file:///etc/passwd?

download is only called by hashDownload, which is called with a url that is guaranteed to start with http:

shell/src/util/hashFetch/index.js

Lines 161 to 172 in f76f89c

if (commit === '0x0000000000000000000000000000000000000000') { // The slug is the URL to a file

if (!slug.toLowerCase().startsWith('http')) { throw new Error(`GitHub Hint URL ${slug} isn't HTTP/HTTPS.`); }

url = slug;

zip = false;

} else if (commit === '0x0000000000000000000000000000000000000001') { // The slug is the URL to a dapp zip file

if (!slug.toLowerCase().startsWith('http')) { throw new Error(`GitHub Hint URL ${slug} isn't HTTP/HTTPS.`); }

url = slug;

zip = true;

} else { // The slug is the `owner/repo` of a dapp stored in GitHub

url = `https://codeload.github.com/${slug}/zip/${commit.substr(2)}`;

zip = true;

}

I'll add a condition in the download function for future-proofing

tomusdrw · 2018-07-02T09:23:55Z

src/util/hashFetch/index.js

+    httpx.get(url, response => {
+      var size = 0;
+
+      response.on('data', (data) => {


Should we reject the promise .on('error') (either read error or fs write error?)

amaury1093

Works like a charm on Kovan! I'm syncing mainnet now

amaury1093 · 2018-07-02T09:37:31Z

electron/index.js

@@ -45,18 +49,10 @@ function createWindow () {

  mainWindow = new BrowserWindow({
    height: 800,
-    width: 1200
+    width: 1200,
+    webPreferences: { nodeIntegrationInWorker: true }


Should be fine, we are not requiring any external files as web workers.

Is the error stack longer? Who is requiring @parity/shared/lib/webWorker.js? I'd be more enthusiast to remove web workers altoghether.

amaury1093 · 2018-07-02T09:41:58Z

src/util/hashFetch/index.js

+const fsReadFile = util.promisify(fs.readFile);
+const fsUnlink = util.promisify(fs.unlink);
+const fsReaddir = util.promisify(fs.readdir);
+const fsRmdir = util.promisify(fs.rmdir);


I saw you included fs-extra in package.json, great idea. In this case you can fs-extra directly, it includes fs and is promise-based if no callback is passed.

amaury1093 · 2018-07-02T10:02:47Z

webpack/app.js

-  node: {
-    fs: 'empty'
-  },
+  target: 'electron-renderer',


No problem, the react app is exactly in the renderer process. I'll have a look at the CI error, building locally works fine.

amaury1093 · 2018-07-02T11:43:03Z

Running npm run ci:build gives the following error.

ERROR in bundle.js from UglifyJs
Invalid assignment [../node_modules/universalify/index.js:7,0][bundle.js:34656,43]

It's fs-extra which requires universalify. See RyanZim/universalify#6. Or a quicker hack might be to exlude universalify in the Uglify plugin

Re

shell/src/inject.js

Lines 91 to 93 in 5ad324f

    
           // For now we simply allow eval(). All builtin dapps are trusted and can use 
        
           // eval(), and all network dapps are served on 127.0.0.1:8545, which have CSP 
        
           // that disallow eval(). So security-wise it should be enough.

If we merge this PR, all dapps will be able to use eval(), which should be avoided. Ideally, in this same file, we'd be able to get the manifest.json of the dapp, @axelchalon do you think that's doable? If yes, then disable eval unless there's an allowEval field.

Re

shell/electron/index.js

Lines 92 to 104 in 5ad324f

    
           session.defaultSession 
        
             .setPermissionRequestHandler((webContents, permission, callback) => { 
        
               if (!webContents.getURL().startsWith('file:')) { 
        
                 // Denies the permissions request for all non-file://. Currently all 
        
                 // network dapps are loaded on http://127.0.0.1:8545, so they won't 
        
                 // have any permissions. 
        
                 return callback(false); 
        
               } 
        
               // All others loaded on file:// (shell, builtin, local) can have those 
        
               // permissions. 
        
               return callback(true); 
        
             });

We can probably remove this piece of code, as it's less a security issue. The only permission we're using that I can think of is the signer requiring camera access for Parity Signer. Again, same point as above, if the manifest.json is easily fetchable here, it'd be easier to allow all permissions for builtin dapps, and disallow for network ones.

5chdn · 2018-07-02T14:10:27Z

How realistic is it we get this through this week?

* Restrict in-frame navigation for dapps * Lint

axelchalon · 2018-07-03T19:37:10Z

@amaurymartiny

re 1. excluding universalify from the uglify plugin doesn't seem to make a difference (or maybe I didn't exclude it correctly). I tried

new webpack.optimize.UglifyJsPlugin({
        sourceMap: true,
        screwIe8: true,
        compress: {
          warnings: false
        },
        output: {
          comments: false
        },
+       exclude: /universalify/
      })

re 2. off the top of my head: we don't really have easy access to the manifest fields here.

re 3. same here. According to https://electronjs.org/docs/tutorial/security#4-handle-session-permission-requests-from-remote-content, By default, Electron will automatically approve all permission requests unless the developer has manually configured a custom handler, which is very worrying (however, strangely enough, I couldn't get permissions to ever be approved as far as I tested -- tried geolocation and camera --, with custom approving handler or without). Anyway: I don't think the Signer is a webview? So I suggest we deny all permissions for webviews (dapps); this is what I did in the latest commit. (once again, not sure if this has any effects; all permissions seem to be rejected on my side)

axelchalon · 2018-07-03T19:39:38Z

@5chdn I addressed the grumbles; last few commits need reviewing. Branch needs some testing to make sure it works fine (also on windows, and on the mainnet). CI still doesn't pass. Apart from that I think it's good to go.

amaury1093 · 2018-07-04T09:11:40Z

Concerning 3- yes, disabling all permissions for webviews seems reasonable.

5chdn · 2018-07-05T20:10:09Z

CI passed :)

amaury1093 · 2018-07-09T13:39:09Z

Even works well with --light

axelchalon added 2 commits June 21, 2018 19:37

Download network (registry) dapps and load them from filesystem

0af589d

Remove obsolete Rust files

ecb4e93

axelchalon added the A0-pleasereview Pull request needs code review. label Jun 21, 2018

axelchalon requested a review from amaury1093 June 21, 2018 17:46

axelchalon commented Jun 21, 2018

View reviewed changes

package-lock.json

5ef03f3

axelchalon commented Jun 21, 2018

View reviewed changes

axelchalon added 5 commits June 21, 2018 20:01

Linting

cd1a5e9

Fix tests

8452e2f

Lint

a2cc6f0

Fix .includes() is not a function (:88)

a76cfef

Remove rust files from CI build

943765d

Update @parity/ui dependency

f76f89c

This was referenced Jun 30, 2018

refactor parity-dapps-glue crate openethereum/parity-ethereum#8405

Closed

Remove the dapps system openethereum/parity-ethereum#9017

Merged

tomusdrw reviewed Jul 2, 2018

View reviewed changes

amaury1093 reviewed Jul 2, 2018

View reviewed changes

axelchalon added 3 commits July 3, 2018 21:31

Grumbles

24f68a7

Restrict in-frame navigation for dapps (#146)

90ce81b

* Restrict in-frame navigation for dapps * Lint

Grumbles

51ffdac

Merge from master

2d96027

fs-extra pathExists doesn't throw if the path doesn't exist

3db6aa7

amaury1093 force-pushed the ac-registry-dapps branch from 60d2e7f to 3db6aa7 Compare July 4, 2018 12:53

Fix bug uglifyjs

532df40

amaury1093 added A8-looksgood Pull request is reviewed well. and removed A0-pleasereview Pull request needs code review. labels Jul 9, 2018

amaury1093 approved these changes Jul 9, 2018

View reviewed changes

tomusdrw approved these changes Jul 9, 2018

View reviewed changes

amaury1093 merged commit c5f3207 into master Jul 9, 2018

amaury1093 deleted the ac-registry-dapps branch July 9, 2018 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download network dapps and load them from filesystem #144

Download network dapps and load them from filesystem #144

axelchalon commented Jun 21, 2018 •

edited

Loading

axelchalon Jun 21, 2018 •

edited

Loading

amaury1093 Jul 2, 2018

axelchalon Jun 21, 2018 •

edited

Loading

axelchalon Jun 21, 2018

tomusdrw Jul 4, 2018 •

edited

Loading

axelchalon Jun 21, 2018

amaury1093 Jul 2, 2018

axelchalon Jul 3, 2018

axelchalon commented Jun 24, 2018 •

edited

Loading

tomusdrw left a comment

tomusdrw Jul 2, 2018

axelchalon Jul 2, 2018

tomusdrw Jul 2, 2018

axelchalon Jul 2, 2018

tomusdrw Jul 2, 2018

tomusdrw Jul 2, 2018

axelchalon Jul 2, 2018

tomusdrw Jul 2, 2018

tomusdrw Jul 2, 2018

axelchalon Jul 3, 2018

tomusdrw Jul 2, 2018

axelchalon Jul 2, 2018 •

edited

Loading

tomusdrw Jul 2, 2018

amaury1093 left a comment

amaury1093 Jul 2, 2018

amaury1093 Jul 2, 2018

amaury1093 Jul 2, 2018

amaury1093 commented Jul 2, 2018 •

edited

Loading

5chdn commented Jul 2, 2018

axelchalon commented Jul 3, 2018 •

edited

Loading

axelchalon commented Jul 3, 2018 •

edited

Loading

amaury1093 commented Jul 4, 2018

5chdn commented Jul 5, 2018

amaury1093 commented Jul 9, 2018

	if (commit === '0x0000000000000000000000000000000000000000') { // The slug is the URL to a file
	if (!slug.toLowerCase().startsWith('http')) { throw new Error(`GitHub Hint URL ${slug} isn't HTTP/HTTPS.`); }
	url = slug;
	zip = false;
	} else if (commit === '0x0000000000000000000000000000000000000001') { // The slug is the URL to a dapp zip file
	if (!slug.toLowerCase().startsWith('http')) { throw new Error(`GitHub Hint URL ${slug} isn't HTTP/HTTPS.`); }
	url = slug;
	zip = true;
	} else { // The slug is the `owner/repo` of a dapp stored in GitHub
	url = `https://codeload.github.com/${slug}/zip/${commit.substr(2)}`;
	zip = true;
	}

Download network dapps and load them from filesystem #144

Download network dapps and load them from filesystem #144

Conversation

axelchalon commented Jun 21, 2018 • edited Loading

axelchalon Jun 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axelchalon Jun 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomusdrw Jul 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axelchalon commented Jun 24, 2018 • edited Loading

tomusdrw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axelchalon Jul 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaury1093 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaury1093 commented Jul 2, 2018 • edited Loading

5chdn commented Jul 2, 2018

axelchalon commented Jul 3, 2018 • edited Loading

axelchalon commented Jul 3, 2018 • edited Loading

amaury1093 commented Jul 4, 2018

5chdn commented Jul 5, 2018

amaury1093 commented Jul 9, 2018

axelchalon commented Jun 21, 2018 •

edited

Loading

axelchalon Jun 21, 2018 •

edited

Loading

axelchalon Jun 21, 2018 •

edited

Loading

tomusdrw Jul 4, 2018 •

edited

Loading

axelchalon commented Jun 24, 2018 •

edited

Loading

axelchalon Jul 2, 2018 •

edited

Loading

amaury1093 commented Jul 2, 2018 •

edited

Loading

axelchalon commented Jul 3, 2018 •

edited

Loading

axelchalon commented Jul 3, 2018 •

edited

Loading