Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Auto)EmbedLiveSample #3973

Merged
merged 48 commits into from
Aug 16, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
28f6c16
first stab at auto-embed-live-sample
Gregoor Jun 4, 2021
8d2ee63
add token information to EmbedLiveSamples to enrich flaws
Gregoor Jun 9, 2021
89f3f0f
remove some logs
Gregoor Jun 9, 2021
b1bc8da
Update kumascript/src/live-sample.js
escattone Jun 16, 2021
8376fb1
Merge branch 'main' into auto-embed-live-sample
escattone Jun 16, 2021
3d6ed08
test increased timeout
Gregoor Jun 17, 2021
e1d9cc0
globally increased timeout
Gregoor Jun 17, 2021
da2a91f
more timeout
Gregoor Jun 17, 2021
4cfd16c
one last comical attempt at increasing test timoeut
Gregoor Jun 17, 2021
263f5ca
enough with the shenanigans
Gregoor Jun 17, 2021
0f8712d
use generator for collecting live-sample code
Gregoor Jun 22, 2021
3e884fe
increase page default timeout
Gregoor Jun 22, 2021
15ac70a
increase page default timeout
Gregoor Jun 22, 2021
4908219
more timeouting testing
Gregoor Jun 22, 2021
47c62f1
more timeouting testing
Gregoor Jun 22, 2021
838a2ac
more timeouting testing
Gregoor Jun 22, 2021
93983e8
remove timeout stuff
Gregoor Jun 22, 2021
e32798e
wait for networkIdle
Gregoor Jun 22, 2021
e9f9f16
simplify live-sample extraction conditional control flow
Gregoor Jun 22, 2021
6406608
set iframe title for AutoEmbedLiveSamples
Gregoor Jul 5, 2021
0f55205
waitUntil page nav
Gregoor Jul 5, 2021
4ab0272
change networkIdle value
Gregoor Jul 5, 2021
d884294
try networkIdle0
Gregoor Jul 5, 2021
ded604b
add a bit of arbitrary timeout
Gregoor Jul 5, 2021
d903e24
add a bit of arbitrary timeout
Gregoor Jul 5, 2021
f427ae8
use playwright for headless tests
Gregoor Jul 5, 2021
7fce379
Merge remote-tracking branch 'origin/main' into auto-embed-live-sample
Gregoor Jul 5, 2021
5ff49d0
Merge remote-tracking branch 'origin/main' into auto-embed-live-sample
Gregoor Jul 5, 2021
41d9afe
start server from playwright tests
Gregoor Jul 5, 2021
de9b7bb
revert adding playwright
Gregoor Jul 6, 2021
ac9d916
upload built docs
Gregoor Jul 6, 2021
23f75fa
upload built docs & screenshot
Gregoor Jul 6, 2021
c551cc6
upload screenshot
Gregoor Jul 6, 2021
1238cfc
log grid files
Gregoor Jul 6, 2021
5bcc145
remove log
Gregoor Jul 6, 2021
6f03ad3
lowercase live sample IDs
Gregoor Jul 6, 2021
2f7637c
remove upload artifacts action
Gregoor Jul 6, 2021
29323b5
explicitly test for live_sample names
Gregoor Jul 6, 2021
79fcae7
fix lint
Gregoor Jul 6, 2021
c8c53fb
use rawBody to construct KS livesample error
Gregoor Jul 8, 2021
5c77038
remove logs
Gregoor Jul 8, 2021
ba86e46
stop collecting flawed liveSamples
Gregoor Jul 8, 2021
57b1803
Merge branch 'main' into auto-embed-live-sample
escattone Aug 13, 2021
3893d4d
change "let" to "const"
escattone Aug 13, 2021
972564c
fix caching of EmbedLiveSample macros
escattone Aug 13, 2021
c1db1d2
make local live-samples behave like deployed
escattone Aug 13, 2021
f63697d
prettier fix
escattone Aug 13, 2021
fe1de10
improve live-sample error messages
escattone Aug 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 34 additions & 39 deletions build/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ const { gather: gatherGitHistory } = require("./git-history");
const { buildSPAs } = require("./spas");
const { renderCache: renderKumascriptCache } = require("../kumascript");
const LANGUAGES_RAW = require("../content/languages.json");
const { safeDecodeURIComponent } = require("../kumascript/src/api/util");

const LANGUAGES = new Map(
Object.entries(LANGUAGES_RAW).map(([locale, data]) => {
Expand Down Expand Up @@ -298,8 +299,8 @@ async function buildDocument(document, documentOptions = {}) {

doc.flaws = {};

let renderedHtml = "";
let flaws = [];
let renderedHtml = "";
const liveSamples = [];

if (options.clearKumascriptRenderCache) {
Expand All @@ -319,26 +320,21 @@ async function buildDocument(document, documentOptions = {}) {
`MacroInvocationError trying to parse ${error.filepath}, line ${error.line} column ${error.column} (${error.error.message})`
);
}

// Any other unexpected error re-thrown.
throw error;
}

const sampleIds = kumascript.getLiveSampleIDs(
document.metadata.slug,
const $ = cheerio.load(`<div id="_body">${renderedHtml}</div>`);

const liveSamplePages = kumascript.buildLiveSamplePages(
document.url,
document.metadata.title,
$,
document.rawBody
);
for (const sampleIdObject of sampleIds) {
const liveSamplePage = kumascript.buildLiveSamplePage(
document.url,
document.metadata.title,
renderedHtml,
sampleIdObject
);

let liveSampleHTML = liveSamplePage.html;
if (liveSamplePage.flaw) {
const flaw = liveSamplePage.flaw.updateFileInfo(fileInfo);
for (let { id, html, flaw } of liveSamplePages) {
if (flaw) {
flaw.updateFileInfo(fileInfo);
if (flaw.name === "MacroLiveSampleError") {
// As of April 2021 there are 0 pages in mdn/content that trigger
// a MacroLiveSampleError. So we can be a lot more strict with en-US
Expand All @@ -356,7 +352,7 @@ async function buildDocument(document, documentOptions = {}) {
}
}
flaws.push(flaw);
liveSampleHTML = `<!doctype html>
html = `<!doctype html>
<html>
<head>
<meta charset="utf-8">
Expand All @@ -382,10 +378,7 @@ async function buildDocument(document, documentOptions = {}) {
</html>
`;
}
liveSamples.push({
id: sampleIdObject.id.toLowerCase(),
html: liveSampleHTML,
});
liveSamples.push({ id: id.toLowerCase(), html });
}

if (flaws.length) {
Expand Down Expand Up @@ -438,7 +431,9 @@ async function buildDocument(document, documentOptions = {}) {
// its output with the `folder`.
validateSlug(metadata.slug);

const $ = cheerio.load(`<div id="_body">${renderedHtml}</div>`);
// EmbedLiveSamples carry their token information to enrich flaw error
// messages, these should not be in the final output
$("[data-token]").removeAttr("data-token");

// Kumascript rendering can't know about FLAW_LEVELS when it's building,
// because injecting it there would cause a circular dependency.
Expand Down Expand Up @@ -620,30 +615,30 @@ async function buildLiveSamplePageFromURL(url) {
throw new Error(`Unexpected URL format to extract live sample ('${url}')`);
}
const [documentURL, sampleID] = url.split(/\.html$/)[0].split("/_sample_.");
const decodedSampleID = safeDecodeURIComponent(sampleID).toLowerCase();
const document = Document.findByURL(documentURL);
if (!document) {
throw new Error(`No document found for ${documentURL}`);
}
// Convert the lower-case sampleID we extract from the incoming URL into
// the actual sampleID object with the properly-cased live-sample ID.
for (const sampleIDObject of kumascript.getLiveSampleIDs(
document.metadata.slug,
document.rawBody
)) {
if (sampleIDObject.id.toLowerCase() === sampleID) {
const liveSamplePage = kumascript.buildLiveSamplePage(
document.url,
document.metadata.title,
(await kumascript.render(document.url))[0],
sampleIDObject
);
if (liveSamplePage.flaw) {
throw new Error(liveSamplePage.flaw.toString());
}
return liveSamplePage.html;
const liveSamplePage = kumascript
.buildLiveSamplePages(
document.url,
document.metadata.title,
(await kumascript.render(document.url))[0],
document.rawBody
)
.find((page) => page.id.toLowerCase() == decodedSampleID);

if (liveSamplePage) {
if (liveSamplePage.flaw) {
throw new Error(liveSamplePage.flaw.toString());
}
return liveSamplePage.html;
}
throw new Error(`No live-sample "${sampleID}" found within ${documentURL}`);

throw new Error(
`No live-sample "${decodedSampleID}" found within ${documentURL}`
);
}

// This is used by the builder (yarn build) and by the server (JIT).
Expand Down
1 change: 1 addition & 0 deletions client/src/document/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ export function Document(props /* TODO: define a TS interface for this */) {
? props.doc
: null,
revalidateOnFocus: CRUD_MODE,
refreshInterval: CRUD_MODE ? 500 : 0,
}
);

Expand Down
4 changes: 2 additions & 2 deletions kumascript/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ const info = require("./src/info.js");
const { render: renderMacros } = require("./src/render.js");
const {
getLiveSampleIDs,
buildLiveSamplePage,
buildLiveSamplePages,
LiveSampleError,
} = require("./src/live-sample.js");
const { HTMLTool } = require("./src/api/util.js");
Expand Down Expand Up @@ -114,7 +114,7 @@ const renderFromURL = async (
};

module.exports = {
buildLiveSamplePage,
buildLiveSamplePages,
getLiveSampleIDs,
LiveSampleError,
render: renderFromURL,
Expand Down
3 changes: 2 additions & 1 deletion kumascript/macros/EmbedLiveSample.ejs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
//
// See also : LiveSampleLink

var sampleId = $0;
var sampleId = $0 || $token.location.start.offset;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ I like how you decided to pass the token as part of the macro's execution context.

var width = $1;
var height = $2;
var screenshotUrl = $3;
Expand Down Expand Up @@ -80,5 +80,6 @@ if (width) { %> width="<%= width %>"<% }
if (height) { %> height="<%= height %>"<% }
%> src="<%- url %>"<%
if (allowedFeatures) { %> allow="<%= allowedFeatures %>"<% }
if ($token) { %> data-token="<%= JSON.stringify($token) %>"<% }
%>></iframe><%
if (hasScreenshot) { %></td></tr></tbody></table><% } %>
124 changes: 97 additions & 27 deletions kumascript/src/api/util.js
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,68 @@ function safeDecodeURIComponent(text) {
}
}

const findSectionStart = ($, sectionID) =>
$(`#${cssesc(sectionID, { isIdentifier: true })}`);

const hasHeading = ($, sampleID) =>
!sampleID ? false : findSectionStart($, sampleID).length > 0;

function findTopLevelParent($el) {
while ($el.siblings(":header").length == 0 && $el.parent().length > 0) {
$el = $el.parent();
}
return $el;
}

const getLevel = ($header) => parseInt($header[0].name[1], 10);

const getHigherHeaderSelectors = (upTo) =>
Array.from({ length: upTo }, (_, i) => "h" + (i + 1)).join(", ");

function* collectLevels($el) {
// Initialized to 7 so that we pick up the lowest heading level which is <h6>
let level = 7;
let $prev = $el;
while (level !== 1) {
const nextHigherLevel = getHigherHeaderSelectors(level - 1);
const $header = $prev.prevAll(nextHigherLevel).first();
if ($header.length == 0) {
return;
}
level = getLevel($header);
$prev = $header;
yield $header.add($header.nextUntil(nextHigherLevel));
}
}

function collectClosestCode($start) {
const $el = findTopLevelParent($start);
for (const $level of collectLevels($el)) {
const pairs = LIVE_SAMPLE_PARTS.map((part) => {
const selector = `.${part}, pre[class*="brush:${part}"], pre[class*="${part};"]`;
const $filtered = $level.find(selector).add($level.filter(selector));
return [
part,
$filtered
.map((i, element) => cheerio(element).text())
.get()
.join("\n"),
];
});
if (pairs.some(([, code]) => !!code)) {
$start.prop("title", $level.first(":header").text());
return Object.fromEntries(pairs);
}
}
return null;
}

class HTMLTool {
constructor(html, pathDescription) {
this.$ = cheerio.load(html, { decodeEntities: true });
this.$ =
typeof html == "string"
? cheerio.load(html, { decodeEntities: true })
: html;
this.pathDescription = pathDescription;
}

Expand Down Expand Up @@ -139,7 +198,7 @@ class HTMLTool {
// Kuma looks for the first HTML tag of a limited set of section tags with ANY
// attribute equal to the "sectionID", but in practice it's always an "id" attribute,
// so let's simplify this as well as make it much faster.
const sectionStart = $(`#${cssesc(sectionID, { isIdentifier: true })}`);
const sectionStart = findSectionStart($, sectionID);
if (!sectionStart.length) {
let errorMessage = `unable to find an HTML element with an "id" of "${sectionID}" within ${this.pathDescription}`;
const hasMoreThanAscii = [...sectionID].some(
Expand Down Expand Up @@ -182,33 +241,44 @@ class HTMLTool {
}

extractLiveSampleObject(sampleID) {
const result = Object.create(null);
const sample = this.getSection(sampleID);
// We have to wrap the collection of elements from the section
// we've just acquired because we're going to search among all
// descendants and we want to include the elements themselves
// as well as their descendants.
const $ = cheerio.load(`<div>${cheerio.html(sample)}</div>`);
for (const part of LIVE_SAMPLE_PARTS) {
const src = $(
`.${part},pre[class*="brush:${part}"],pre[class*="${part};"]`
)
.map((i, element) => {
return $(element).text();
})
.get()
.join("\n");
// The string replacements below have been carried forward from Kuma:
// * Bugzilla 819999: &nbsp; gets decoded to \xa0, which trips up CSS.
// * Bugzilla 1284781: &nbsp; is incorrectly parsed on embed sample.
result[part] = src ? src.replace(/\u00a0/g, " ") : null;
}
if (!LIVE_SAMPLE_PARTS.some((part) => result[part])) {
throw new KumascriptError(
`unable to find any live code samples for "${sampleID}" within ${this.pathDescription}`
const sectionID = sampleID.substr("frame_".length);
if (hasHeading(this.$, sectionID)) {
const result = Object.create(null);
const sample = this.getSection(sectionID);
// We have to wrap the collection of elements from the section
// we've just acquired because we're going to search among all
// descendants and we want to include the elements themselves
// as well as their descendants.
const $ = cheerio.load(`<div>${cheerio.html(sample)}</div>`);
for (const part of LIVE_SAMPLE_PARTS) {
const src = $(
`.${part}, pre[class*="brush:${part}"], pre[class*="${part};"]`
)
.map((i, element) => $(element).text())
.get()
.join("\n");
// The string replacements below have been carried forward from Kuma:
// * Bugzilla 819999: &nbsp; gets decoded to \xa0, which trips up CSS.
// * Bugzilla 1284781: &nbsp; is incorrectly parsed on embed sample.
result[part] = src ? src.replace(/\u00a0/g, " ") : null;
}
if (!LIVE_SAMPLE_PARTS.some((part) => result[part])) {
throw new KumascriptError(
`unable to find any live code samples for "${sectionID}" within ${this.pathDescription}`
);
}
return result;
} else {
const result = collectClosestCode(
this.$("#" + cssesc(sampleID, { isIdentifier: true }))
);
if (!result) {
throw new KumascriptError(
`unable to find any live code samples for "${sectionID}" within ${this.pathDescription}`
);
}
return result;
}
return result;
}

html() {
Expand Down
6 changes: 5 additions & 1 deletion kumascript/src/environment.js
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ class Environment {

// Get a customized environment object that is specific to a single
// macro on a page by including the arguments to be passed to that macro.
getExecutionContext(args) {
getExecutionContext(args, token = null) {
const context = Object.create(this.prototypeEnvironment);

// Make a defensive copy of the arguments so that macros can't
Expand Down Expand Up @@ -188,6 +188,10 @@ class Environment {
context[`$${i}`] = "";
}

// Position of the macro inside of the source, which we need
// to identify EmbedLiveSample macros which do not receive an ID
context["$token"] = token;

return context;
}
}
Expand Down
Loading