Fetch protocol.json from chrome?

[paulirish]

What do you think.. if we allowed a means for you to grab the current protocol directly from Chrome?

[cyrus-and]

It would be great!
Actually, I was thinking about asking if you knew a better way than fetching it from time to time from the Blink repo.

Random suggestion: Maybe provide another URL, e.g., http://localhost:9222/protocol/json?

[paulirish]

Random suggestion: Maybe provide another URL, e.g., http://localhost:9222/protocol/json

sgtm!

[paulirish]

Filed upstream: https://code.google.com/p/chromium/issues/detail?id=538300

[cyrus-and]

Thanks! Starred.

[paulirish]

Right now we're not going to package the protocol in Chrome. It'd be 20kb but across every chrome desktop installs in the world, it's a pretty big bump, and we don't have many customers for it besides this (very awesome project).

We've discussed that the best approach for now is pulling a protocol.json from source control based on the version provided in /json/version. I've written an implementation which works well.

fetch("http://localhost:9222/json/version")
  .then( resp => resp.json() )
  .then( function(obj){   
    // eg: "WebKit-Version": "537.36 (@31978e66f0c2ae00926292b37ce61bb19827836c)"
    var ver = obj["WebKit-Version"];
    var match = ver.match(/\s\(@(\b[0-9a-f]{5,40}\b)/);
    var hash = match[1];
    return hash;
  })
  .then( hash => {
    // the blink/chromium repo merge changed file locations. https://goo.gl/JztV1A
    var url = (hash <= 202666) ?
      `http://src.chromium.org/blink/trunk/Source/devtools/protocol.json?p=${ hash }` :
      `https://chromium.googlesource.com/chromium/src/+/${ hash }/third_party/WebKit/Source/devtools/protocol.json?format=TEXT`;
    return fetch(url);
  })
  .then( resp =>  { return resp.text() }) 
  .then( function(txt){
    try { 
      // googlesource response is base64 encoded
      var protocol = atob(txt);  
    } catch (e) {
      var protocol = txt;
    }
    return Promise.resolve(JSON.parse(protocol));
  })
  .then( protocolObj => console.log(protocolObj) )

hope this helps.

[kdzwinel]

I thought protocol.json is already bundled with Chrome, just not exposed. Check out this snippet from the DT code:

It suggests that protocol.json is located in chrome-devtools://devtools/protocol.json, unfortunately it's not a valid path :(

[cyrus-and]

@paulirish OK, I'm doing that way. But I'll leave the hardcoded version too, as the scenario in which the instrumentation happens offline is not so uncommon.

@kdzwinel Neither via JavaScript? One could try to open a new tab, inject some code and fetch the protocol. Out of curiosity though; this would be too ugly to implement...

If it's already bundled how can it require 20KB more?

[kdzwinel]

@cyrus-and I tried messing with that code in the runtime to make DT download protocol.json, no luck though. Maybe it's departed or meant for some other environment (e.g. https://frontend.chrome-dev.tools ?).

[EDIT] I think I found an explanation. protocol.json is only requested when working on devtools:

When remotely debugging and developing the front-end of Blink, the content of the InspectorBackendCommands.js file is generated based on the content of protocol.json instead as a fallback of the Chromium build system.

https://developer.chrome.com/devtools/docs/contributing

[cyrus-and]

@kdzwinel It's still unclear to me whether the protocol descriptor is already bundled (even in such a way that it can be generated) in Chome or not?

@paulirish I implemented what you suggested but I don't know if it wise to enable it by default as it will introduce the non-negligible latency of fetching the protocol every time a connection with Chrome is established. This may or may not be a problem but surely it is a change in the expected behavior: what was almost immediate now takes about 1-2s.

[paulirish]

@kdzwinel good find. just asked @repenaxa about this and he confirmed mostly what you said. this would be loaded only when you're doing --remote-debugging-port=xxx and supplying a custom frontend via http://localhost:xxx#http://localhost:zz/custom_frontend/inspector.html. It's this case where we'd pull http://localhost:zz/custom_frontend/protocol.json. But that's being supplied by you and your custom frontend folder.. It's never bundled.

@cyrus-and

But I'll leave the hardcoded version too, as the scenario in which the instrumentation happens offline is not so uncommon.

Yeah I think that's wise.

what was almost immediate now takes about 1-2s.
Hmm. I don't have a great solution here.

Possibilities:

• caching on your side?
• create a new devtools-protocol.json NPM module. (It could also do this lookup work plus caching... but maybe...) It ships with all protocol.json possibilities baked in and isn't network dependent.
• something smarter?

[paulirish]

cc @repenaxa @dgozman

[cyrus-and]

@paulirish

caching on your side?

In-memory caching would solve the latency due to several connections (create tab, connect to tab, instrument, repeat...) to the same Chrome instance within a single Node.js process. But across multiple executions, the network is hit anyway. Since this is basically a library I don't know if caching on disk could be advisable.

It ships with all protocol.json possibilities baked in and isn't network dependent.

Do you mean fetching the whole version history of package.json from the repository? That module would require some manual intervention anyway any time a new version ships...

Manual caching is already possible with the current version: users may fetch and download the protocol.json once using the API, then they may pass that file to Chrome() via the protocol option during further executions.

I will think about it, but options seem quite limited.

[cyrus-and]

@paulirish Since the split, the snippet is no longer valid (see #40). Does this mean that starting from 53.0.2759.0 we need to fetch both and merge?

I think the information here needs a fix too:

The tip-of-tree protocol changes frequently. If you need the protocol version for a specific Chrome client, this snippet will provide the protocol JSON based on the client's /json/version.

[cyrus-and]

Well it seems you've already faced the issue (ChromeDevTools/debugger-protocol-viewer#54).

[kdzwinel]

yeah, we just grab both js_protocol and browser_protocol and merge them like so: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/gh-pages/merge-protocol-files.js

[cyrus-and]

@kdzwinel I'm implementing that. But from the chrome-remote-interface perspective this now means 2 GETs every time a new connection with Chrome is established, this is quite bad... I'm considering to remove the "fetch from Chrome" feature and rely on the hardcoded version only.

[c0b]

I'm considering to remove the "fetch from Chrome" feature and rely on the hardcoded version only.

oh, no, please don't do that; because that's already the behavior of set fallback: true if people want that, please can set that, or you may set fallback true as default; but for other people who works with latest nightly version of Chromium, to retrieve protocol on the fly is still the best way to try with latest features

[cyrus-and]

@c0b You're right, I'm not going to drop that feature, but it may be convenient to use the fallback version by default.

[paulirish]

@cyrus-and yup this is the state of things. Just spoke with @pavelfeldman and while we aim to inevitably expose the actual protocol to you, we're not there yet.

For now we'll need do to this sort of solution.

This file from @willchen90 has similar implementation: https://github.com/ChromeDevTools/devtools-frontend/blob/5c6f9d1a1a45af50c0179e61c73df85fa84feeb0/scripts/hosted_mode/server.js (it's since changed, slightly). It handles getting the right two files, but doesn't merge them. That part is easier. :)

---

also fwiw the protocol split happened at chromium git hash b34642153f00033ae8cdeea5bdb351f569c7ad7a which is at the revision 397883. (If you want to support both ways in the meantime)

[cyrus-and]

Just spoke with pavelfeldman and while we aim to inevitably expose the actual protocol to you, we're not there yet.

@paulirish thanks for that.

I pushed the fix (c61237f) a couple of days ago on a temporary branch, I'm waiting for some feedback from @c0b (issue opener) before merging, since I don't have a Chrome version new enough to try.

also fwiw the protocol split happened at chromium git hash b34642153f00033ae8cdeea5bdb351f569c7ad7a which is at the revision 397883. (If you want to support both ways in the meantime)

I found by bisecting that the first release to require the merge should be 53.0.2759.0, I needed a version to compare to the one returned by /json/version.

[paulirish]

ah true. Good call.
We can probably expose the chromium rev in /json/version to make that a bit easier.

---

edit: i looked and the chromium rev is not known to a chrome build, so that won't be easy.. :/

[c0b]

I've tested with Chrome 54 it seems working.

BTW, I'm packing chrome in a docker container to be running on server, and have multiple different chrome versions at same time, can be controlled in a VNC session by human or completely headless by chrome-remote-interface only; that way you don't have to upgrade your main working chrome to an unstable version, maybe I should share that Dockerfile in a separate repo...

[cyrus-and]

@c0b thanks, that would be useful indeed, even a Gist will do.

[c0b]

@cyrus-and now I have shared https://github.com/c0b/chrome-in-docker

[cyrus-and]

@paulirish

ChromeDevTools/debugger-protocol-viewer@633424b

Does this mean that js_protocol.json must be fetched from the V8 repo now? There is no v8 folder in https://chromium.googlesource.com/chromium/src/+/master/.

---

Edit (self-reply): Yes: ChromeDevTools/debugger-protocol-viewer@bbf6a71.

[cyrus-and]

PSA: this is now possible with Chrome >= 60.0.3097.0