Major gulp refactor: blueprint.defineTaskGroup() #617

giladgray · 2017-02-03T23:03:57Z

What is a task group?

A task group is a set of gulp tasks that performs the same operation on every relevant package, and a parent task that runs each of the package tasks respecting dependencies between packages. Relevant packages are those that have a given "block" present in their definition in Gulpfile.js.

For example, the "sass" group compiles .scss files to css. gulp sass will run sass-core, sass-datetime, sass-table, and sass-docs. sass-core will be run first, as the other three depend on it, then sass-datetime and sass-table in parallel, followed by sass-docs which depends on all of them.

How was it before?

The previous iteration of our gulp build (before this PR) used blueprint.task(block, name, deps, fn) to define a task group. This function was expressive but did not offer control of the task function itself. Instead, the fn arg received (package, isDevMode) and returned a stream or a promise. It was not able to use the simpler done callback syntax for gulp.task(), so a handful of our task groups were defined using blueprint.projectsWithBlock() directly, leading to some unsightly code.

Changes proposed in this pull request:

blueprint.defineTaskGroup(options, fn) defines a task group based on a block query with some cool options. The gulp task itself is defined in the callback so it can be configured precisely as needed.

This is a big improvement over the previous iteration because it provides full control over the task config: you can use gulp.task() exactly as you need (deps, no deps, done arg, etc). It also means that the internals of defineTaskGroup are straightforward and transparent—there's very little magic here (compare task to dTG).

Many of the compile tasks have been renamed. They are now shorter and more accurately describe what they do.

copy-files → copy (also removed the no-ops due to copy: false in config)
sass-compile → sass
sass-lint → stylelint
typescript-compile → tsc
typescript-lint → tslint
webpack-compile-docs → webpack-docs
webpack-compile-w-docs → webpack-docs-watch

The -w- segment for incremental tasks run by watchers has been renamed to :only suffix. For example, sass-compile-w-table is now sass-table:only. This better expresses the purpose of these tasks: to run only the one task without dependencies for the purpose of rebuilding.

Reviewers should focus on:

Understanding how it all fits together, so we can all support this going forward.

⭐️ Reading this PR commit-by-commit is encouraged. I worked hard to keep the commits separate and logical.

defines a task group based on a block query with some cool options. gulp task itself is defined in callback so it can be configured precisely as needed. this removes the need to specify deps in config. function accepts options object to support more complex usage and multiple args.

for tasks that care about project dependencies 😄

sass-compile => sass sass-lint => stylelint typescript-compile => tsc typescript-lint => tslint webpack-compile-docs => webpack-docs webpack-compile-w-docs => webpack-docs-watch -w- => :only for incremental build tasks so: sass-compile-table => sass-table:only dependencies are much easier to trace now because they're all listed in task definition.

blueprint-bot · 2017-02-03T23:17:26Z

fix isotest usage

Preview: docs | landing | table
Coverage: core | datetime

blueprint-bot · 2017-02-06T22:19:59Z

log files written by sass task

Preview: docs | landing | table
Coverage: core | datetime

adidahiya · 2017-02-06T22:37:55Z

gulp/index.js

+         * The task name is of the format `[name]-[project.id]`.
+         * Finally, a "group task" is defined with the base name that runs all the project tasks.
+         * This group task can be configured to run in parallel or in sequence.
+         * @param {{block: string, name?: string, parallel?: boolean}} options


instead of "block", I propose another name: "scope"

actually i'm less into scope now. config block sounds better than config scope, as does projectsWithBlock vs projectsWithScope.

i am open to a new name though, just don't think we've found it yet.

block really doesn't mean anything. what do you mean by "config block sounds better than config scope"? also projectsWithScope sounds fine to me

what is a scope?

adidahiya · 2017-02-06T22:38:29Z

gulp/typescript.js

+            gulp.src([
+                path.join(project.cwd, "!(dist|node_modules|typings)", "**", "*.{js,jsx,ts,tsx}"),
+                // exclude nested dist directories (ex: table/preview/dist)
+                "!" + path.join(project.cwd, "*", "dist", "**", "*.{js,jsx,ts,tsx}"),


this looks pretty awkward, is it a new consideration we have to take with this build system refactor?

no, it's always been there. it's actually due to table's webpack-based build which doesn't rely on these changes even a little bit.

adidahiya · 2017-02-06T22:38:41Z

overally looks good, just minor comments ^

cmslewis · 2017-02-07T21:09:12Z

@giladgray FYI I'm reading through all the existing Gulp code now, then I'll circle back to read through this afterward.

cmslewis

Phew, what a marathon. Finally perused all the changes, @giladgray. Left a few clarifying questions, but otherwise this looks good.

cmslewis · 2017-02-07T21:51:24Z

gulp/isotest.js

+        parallel: false,
+    }, (taskName, project) => {
+        gulp.task(taskName, [`typescript-compile-${project.id}`], () => {
+            return gulp.src(project.cwd + "test.iso/**/*")


@giladgray just gonna comment on every little thing until I build up a more solid understanding of this whole build system. There's a minor change in behavior here, yes (${project.cwd}/test/isotest.js => ${project.cwd}/test.io/**/*)? Is the reasoning that we expect to add more isomorphic test files later, so might as well support anything in that directory moving forward?

cmslewis · 2017-02-07T21:57:41Z

gulp/copy.js

-        // allow for no-op on project dependencies
-        if (project.copy === false) {
-            return;
-        }


So this check is gone now. In the new code, seems like omitting a key/blockname (e.g. copy) from a project config would cause projectsWithBlock to return an empty array in blueprint.taskGroup, which would then define a gulp.task that simply executes an empty set of sub-tasks—effectively a no-op. Is that behavior drastically different from this special no-op case above that used to be here? Was it ever particularly helpful or necessary?

blueprint.task always generated a root task that depended on corresponding tasks in other projects. so copy-files-docs depended on (the existence of) copy-files-core, which is why we had to add the no-op shortcut. this was awkward because there wasn't actually a dependency between the copy tasks (like, they're completely separate operations), whereas there is a real dependency for the typescript tasks (core must be compiled before datetime).

in this new world, the copy task simply ignores the depTaskNames argument and voila, packages can be copied without any dependencies, as they should be!

cmslewis · 2017-02-07T22:38:02Z

gulp/index.js

-                const taskName = [name, project.id].join("-");
-                taskFn(taskName, project);
+                const taskName = [name, id].join("-");
+                const depNames = dependencies.map((dep) => [name, dep].join("-"));


It appears that dep and id both project IDs in practice (e.g. "core", "datetime", "table"). It'd be helpful to name dep to depId IMO, just so it's clear that the names we're building on this line and the line above both follow the same scheme.

cmslewis · 2017-02-07T22:46:07Z

gulp/webpack.js

        webpack(configuration, webpackConfig.webpackDone(callback));
    });

-    gulp.task("webpack-compile-w-docs", (callback) => { // eslint-disable-line no-unused-vars


Do we not need the eslint-disable line anymore? Looks like we're still not using the callback argument anywhere. Did we need that comment in the first place?

we are not using eslint anymore! it's all TSLint

cmslewis · 2017-02-07T22:48:15Z

gulp/hygiene.js

@@ -17,8 +17,8 @@ module.exports = (blueprint, gulp, plugins) => {
        return del(cleanDirs, { force: true });
    });

-    gulp.task("tslint", () => (
-        gulp.src(["*.js", "gulp/**/*.js", "packages/*/*.js"])
+    gulp.task("tslint-gulp", () => (


What's the motivation behind this -gulp-suffixed tslint task as distinct from the non-suffixed tslint task? Looks like we're tslint-ing all JS files...and then also JS files within gulp/? Is that necessary on non-typescript files?

ok so: the previous iteration of this task ran tslint on all .js files (hence the package/**/*.js glob).

this new iteration runs tslint on the gulp code only because JS files in packages are now linted by tslint-<package>.

Gotcha 👍

cmslewis · 2017-02-07T23:00:56Z

gulp/typescript.js

+        // create a TypeScript project for each project to improve re-compile speed.
+        // this must be done outside of a task function so it can be reused across runs.
+        const tsconfigPath = path.join(project.cwd, "tsconfig.json");
+        project.typescriptProject = typescript.createProject(tsconfigPath, {


Where does typescript come from? Doesn't appear to be defined in this file.

Gilad Gray added 8 commits February 3, 2017 14:39

copy-files => copy + remove copy: false

e52a0f1

swap taskFn args order, add dependencies arg

bce87b7

for tasks that care about project dependencies 😄

update aliases and watch

fad2c05

clean up typescript project creation

b3c09e3

rename to defineTaskGroup()

183c315

fix linting of some newly captured files

0044b9b

giladgray added Domain: tooling Type: refactor labels Feb 3, 2017

Gilad Gray added 2 commits February 3, 2017 15:06

remove now unused blueprint.task()

b13f70c

fix isotest usage

6952e15

giladgray requested a review from cmslewis February 6, 2017 20:14

Gilad Gray added 4 commits February 6, 2017 13:34

refactor test-dist with helper assert function

9126e3d

refactor icons: helper functions at the bottom

aba0117

fix tslint glob to exclude nested dist dirs

aeb0fdd

log files written by sass task

2629f96

adidahiya reviewed Feb 6, 2017

View reviewed changes

cmslewis approved these changes Feb 7, 2017

View reviewed changes

giladgray merged commit 5d1997a into master Feb 8, 2017

giladgray deleted the gg/gulp-taskgroup branch February 8, 2017 20:58

giladgray mentioned this pull request Feb 11, 2017

Prepare Release v1.9.0 #671

Merged

gscshoyru mentioned this pull request Feb 14, 2017

Fixed scripts in table package's package.json #684

Merged

snyk-bot mentioned this pull request Apr 24, 2023

[Snyk] Fix for 1 vulnerabilities majacQ/blueprint#204

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Major gulp refactor: blueprint.defineTaskGroup() #617

Major gulp refactor: blueprint.defineTaskGroup() #617

giladgray commented Feb 3, 2017 •

edited

Loading

blueprint-bot commented Feb 3, 2017

blueprint-bot commented Feb 6, 2017

adidahiya Feb 6, 2017

giladgray Feb 8, 2017

giladgray Feb 8, 2017

adidahiya Feb 8, 2017

giladgray Feb 8, 2017

adidahiya Feb 6, 2017

giladgray Feb 8, 2017

adidahiya commented Feb 6, 2017

cmslewis commented Feb 7, 2017

cmslewis left a comment

cmslewis Feb 7, 2017

cmslewis Feb 7, 2017

giladgray Feb 8, 2017

cmslewis Feb 7, 2017

cmslewis Feb 7, 2017

giladgray Feb 8, 2017

cmslewis Feb 7, 2017

giladgray Feb 8, 2017

cmslewis Feb 8, 2017

cmslewis Feb 7, 2017

Major gulp refactor: blueprint.defineTaskGroup() #617

Major gulp refactor: blueprint.defineTaskGroup() #617

Conversation

giladgray commented Feb 3, 2017 • edited Loading

What is a task group?

How was it before?

Changes proposed in this pull request:

Reviewers should focus on:

blueprint-bot commented Feb 3, 2017

fix isotest usage

blueprint-bot commented Feb 6, 2017

log files written by sass task

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adidahiya commented Feb 6, 2017

cmslewis commented Feb 7, 2017

cmslewis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giladgray commented Feb 3, 2017 •

edited

Loading