tools: build all.json by combining generated JSON #21637

rubys · 2018-07-03T16:55:48Z

Notes:

Removed a number of root properties that did not seem relevant: source,
desc, and introduced_in. There no longer is a source, and the other two are
from the first include and do not reflect the entire API.
As with doc: all.html is seriously broken link-wise #20100, the current "desc"
properties sometimes contained in-page links, other times referenced another
page, and often did not match the links in the original HTML or JSON file.
I chose to standardize on external links as "desc" values are isolated
snippets as opposed to all.html which can be viewed as a standalone and self
contained document.
Eliminated preprocessing for @include entirely, including the test case
for this function.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are updated
commit message follows commit guidelines

rubys · 2018-07-03T17:06:08Z

See #21568 (comment) for the original request.

Trott · 2018-07-03T17:18:05Z

Not sure if @tolmasky or any of their RunKit colleagues will be able to test this ahead of time, but pinging anyway as their the one significant consumer of our docs-in-JSON-format that I'm aware of. (I don't know if this would help or not, but maybe it would help them if the resulting JSON could be stashed somewhere, maybe in a gist or something.)

vsemozhetbyt · 2018-07-03T17:22:52Z

What will happen with https://github.com/nodejs/node/blob/master/doc/api/index.md? It also contains one include directive.

rubys · 2018-07-03T17:30:17Z

@Trott @tolmasky gist containing both what all.json will look like after this change, and a diff between what it looks like after vs what it looked like before.

@vsemozhetbyt good catch! Initial thoughts: _toc.md should be renamed to _index.md (overwriting this file). allhtml.js and alljson.js would need to be updated to read this data from its new location.

If you agree, I'll make this change and squash the commit.

vsemozhetbyt · 2018-07-03T17:43:07Z

Seems good, but we should also check mentions like this one:

node/tools/doc/html.js

Line 39 in 6e16ad7

const gtocPath = path.join(docPath, 'api', '_toc.md');

vsemozhetbyt · 2018-07-03T19:43:12Z

I hope I will be able to review at the weekend. If anybody feels confident to review and land sooner please do)

Me1000 · 2018-07-03T19:58:01Z

Hey all! I work on the RunKit team, thanks for the ping @Trott.

The new output looks like it contains all the existing data. If that is indeed the case (the output is quite large, so maybe I missed something) then it should be fairly easy for us consume the new format. :)

vsemozhetbyt · 2018-07-07T11:21:31Z

One breaking change in the index.html: previously, preprocess.js not only processed @include directives but also stripped the special comments (see commentExpr mentions in preprocess.js). Maybe we should make these comments formatted as HTML comments — these are just two lines in the index.md. Otherwise, they are rendered in the index.html:

vsemozhetbyt · 2018-07-07T10:55:36Z

test/doctool/test-make-doc.js


 const actualDocs = allDocs.filter(
  (name) => {
    const extension = path.extname(name);
-    return (extension === '.html' || extension === '.json') &&
-           name !== '_toc.html';
+    return (extension === '.html' || extension === '.json');


It seems the parentheses are redundant now.

vsemozhetbyt · 2018-07-07T11:49:08Z

tools/doc/alljson.js

@@ -0,0 +1,56 @@
+'use strict';
+
+// Build all.json by combining the miscs, modules, classes, global, and methods


global -> globals?

vsemozhetbyt · 2018-07-07T11:49:42Z

tools/doc/alljson.js

+// Read the table of contents.
+const toc = fs.readFileSync(source + '/index.html', 'utf8');
+
+// initialize results.  Only these four data values will be collected.


initialize -> Initialize.

Extra space between the sentences.

vsemozhetbyt · 2018-07-07T13:29:36Z

tools/doc/alljson.js

+// Extract (and concatenate) the selected data from each document.
+// Expand hrefs found in json to include source HTML file.
+for (const link of toc.match(/<a.*?>/g)) {
+  const href = /href="(.*?)"/.exec(link)[1].replace('.html', '.json');


.replace('.html', '.json') seems erroneous here?

vsemozhetbyt · 2018-07-07T13:31:11Z

tools/doc/alljson.js

+  if (!jsonFiles.includes(json) || seen[json]) continue;
+  const data = JSON.parse(
+    fs.readFileSync(source + '/' + json, 'utf8')
+      .replace(/<a href="#/g, `<a href="${href}#`)


It seems this should be /<a href=\\"#/g, `<a href=\\"${href}#`, as double quotes are escaped in .json sources.

vsemozhetbyt · 2018-07-07T13:34:17Z

tools/doc/alljson.js

+  }
+
+  // Mark source as seen.
+  seen[href] = true;


seen[href] -> seen[json]?

vsemozhetbyt · 2018-07-07T16:56:33Z

tools/doc/html.js

@@ -36,7 +36,7 @@ marked.setOptions({ renderer });

 const docPath = path.resolve(__dirname, '..', '..', 'doc');

-const gtocPath = path.join(docPath, 'api', '_toc.md');
+const gtocPath = path.join(docPath, 'api', 'index.md');
 const gtocMD = fs.readFileSync(gtocPath, 'utf8').replace(/^@\/\/.*$/gm, '');


Sorry, it seems this line also need to be updated so that the new HTML comment from index.md could be excluded from all the docs except the index.html.

I mean the .replace(/^@\/\/.*$/gm, '') part.

vsemozhetbyt · 2018-07-07T19:13:04Z

tools/doc/html.js

-const gtocPath = path.join(docPath, 'api', '_toc.md');
-const gtocMD = fs.readFileSync(gtocPath, 'utf8').replace(/^@\/\/.*$/gm, '');
+const gtocPath = path.join(docPath, 'api', 'index.md');
+const gtocMD = fs.readFileSync(gtocPath, 'utf8').replace(/^<!--.*?-->/gms, '');


(Just to prevent doubts: dotAll flag is supported since V8 6.2, so this line is backportable for Node.js 8 LTS.)

vsemozhetbyt · 2018-07-07T19:45:58Z

tools/doc/alljson.js

+
+// Write results.
+fs.writeFileSync(source + '/all.json',
+                 JSON.stringify(results, null, 2), 'utf8');


Just one more significant nit I've found while diffing the old and new results: new all.json lacks the \n at the end of the file, just after the last }. So maybe it is worth to replace JSON.stringify(results, null, 2) with `${JSON.stringify(results, null, 2)}\n` to be on the safe side)

vsemozhetbyt · 2018-07-07T19:50:06Z

A note: amended commits are not reported to the people that are subscribed to a PR, so maybe it is worth to let everybody know about a new commit version by a comment)

@include

Notes: 1) Removed a number of root properties that did not seem relevant: source, desc, and introduced_in. There no longer is a source, and the other two are from the first include and do not reflect the entire API. 2) As with nodejs#20100, the current "desc" properties sometimes contained in-page links, other times referenced another page, and often did not match the links in the original HTML or JSON file. I chose to standardize on external links as "desc" values are isolated snippets as opposed to all.html which can be viewed as a standalone and self contained document. 3) Eliminated preprocessing for @include entirely, including the test case for this function. 4) _toc.md was renamed to index.md. 5) index comments no longer appear in embedded TOCs (left hand side column in the generated documentation.

rubys · 2018-07-07T20:08:32Z

Perhaps I should just stop squashing commits.

vsemozhetbyt · 2018-07-07T20:23:15Z

CI-lite: https://ci.nodejs.org/job/node-test-pull-request-lite/904/

vsemozhetbyt · 2018-07-07T22:13:10Z

Let's land this on Monday, maybe somebody from @nodejs/documentation will want to chime in.

@include

Notes: 1) Removed a number of root properties that did not seem relevant: source, desc, and introduced_in. There no longer is a source, and the other two are from the first include and do not reflect the entire API. 2) As with #20100, the current "desc" properties sometimes contained in-page links, other times referenced another page, and often did not match the links in the original HTML or JSON file. I chose to standardize on external links as "desc" values are isolated snippets as opposed to all.html which can be viewed as a standalone and self contained document. 3) Eliminated preprocessing for @include entirely, including the test case for this function. 4) _toc.md was renamed to index.md. 5) index comments no longer appear in embedded TOCs (left hand side column in the generated documentation. PR-URL: #21637 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

vsemozhetbyt · 2018-07-09T20:02:11Z

Landed in 0c743b5
Thank you!

@include

Notes: 1) Removed a number of root properties that did not seem relevant: source, desc, and introduced_in. There no longer is a source, and the other two are from the first include and do not reflect the entire API. 2) As with #20100, the current "desc" properties sometimes contained in-page links, other times referenced another page, and often did not match the links in the original HTML or JSON file. I chose to standardize on external links as "desc" values are isolated snippets as opposed to all.html which can be viewed as a standalone and self contained document. 3) Eliminated preprocessing for @include entirely, including the test case for this function. 4) _toc.md was renamed to index.md. 5) index comments no longer appear in embedded TOCs (left hand side column in the generated documentation. PR-URL: #21637 Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

nodejs-github-bot added build Issues and PRs related to build files or the CI. doc Issues and PRs related to the documentations. tools Issues and PRs related to the tools directory. labels Jul 3, 2018

rubys force-pushed the alljson branch 2 times, most recently from 95ad110 to 511f403 Compare July 3, 2018 19:04

vsemozhetbyt reviewed Jul 7, 2018

View reviewed changes

rubys force-pushed the alljson branch from 511f403 to f2308dc Compare July 7, 2018 15:03

vsemozhetbyt reviewed Jul 7, 2018

View reviewed changes

rubys force-pushed the alljson branch from f2308dc to f54c875 Compare July 7, 2018 17:26

vsemozhetbyt reviewed Jul 7, 2018

View reviewed changes

rubys force-pushed the alljson branch from f54c875 to 7ac03da Compare July 7, 2018 19:58

vsemozhetbyt approved these changes Jul 7, 2018

View reviewed changes

vsemozhetbyt added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 7, 2018

Trott approved these changes Jul 7, 2018

View reviewed changes

vsemozhetbyt closed this Jul 9, 2018

vsemozhetbyt mentioned this pull request Jul 9, 2018

Creating a branch of the Docs using RunKit #21723

Open

targos removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 12, 2018

vsemozhetbyt mentioned this pull request Jul 14, 2018

test/doctool/test-make-doc.js fails if .DS_Store exists #21519

Closed

targos mentioned this pull request Jul 17, 2018

v10.7.0 proposal #21851

Merged

mcollina mentioned this pull request Jul 20, 2018

test: unclear error with _toc.html #21912

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: build all.json by combining generated JSON #21637

tools: build all.json by combining generated JSON #21637

rubys commented Jul 3, 2018

rubys commented Jul 3, 2018

Trott commented Jul 3, 2018

vsemozhetbyt commented Jul 3, 2018

rubys commented Jul 3, 2018

vsemozhetbyt commented Jul 3, 2018 •

edited

Loading

vsemozhetbyt commented Jul 3, 2018

Me1000 commented Jul 3, 2018

vsemozhetbyt commented Jul 7, 2018 •

edited

Loading

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

vsemozhetbyt Jul 7, 2018

rubys Jul 7, 2018

vsemozhetbyt commented Jul 7, 2018

rubys commented Jul 7, 2018

vsemozhetbyt commented Jul 7, 2018

vsemozhetbyt commented Jul 7, 2018

vsemozhetbyt commented Jul 9, 2018

		@@ -0,0 +1,56 @@
		'use strict';

		// Build all.json by combining the miscs, modules, classes, global, and methods

tools: build all.json by combining generated JSON #21637

tools: build all.json by combining generated JSON #21637

Conversation

rubys commented Jul 3, 2018

Checklist

rubys commented Jul 3, 2018

Trott commented Jul 3, 2018

vsemozhetbyt commented Jul 3, 2018

rubys commented Jul 3, 2018

vsemozhetbyt commented Jul 3, 2018 • edited Loading

vsemozhetbyt commented Jul 3, 2018

Me1000 commented Jul 3, 2018

vsemozhetbyt commented Jul 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsemozhetbyt commented Jul 7, 2018

rubys commented Jul 7, 2018

vsemozhetbyt commented Jul 7, 2018

vsemozhetbyt commented Jul 7, 2018

vsemozhetbyt commented Jul 9, 2018

vsemozhetbyt commented Jul 3, 2018 •

edited

Loading

vsemozhetbyt commented Jul 7, 2018 •

edited

Loading