Improve (speed up) build.dev.watch task

[mpetkov]

First off I wanted to say how much I love this seeder. It is very easy to use and follow. One possible improvement that I could suggest would speed up the build.dev.watch task. Currently, no matter what file changes, the entire dev destination gets rebuilt from scratch. So that means that, on average, each change takes about 10-15 seconds to complete. It would be faster if the watch was separated out so that it only updates files that were changes, instead of the whole directory. So if I change something on a css/sass file, only that file should compile and the others (such as build.js.dev) should not. This will make previewing in dev environment faster and less painful.

[joshwiens]

The change after the initial watch takes ~2.5 seconds, there was another discussion around this #496

The typescript build is incremental, an incremental build for things like css probably wouldn't be viable if you think about preprocessors ( which are supported ).

Rebuilding everything is the simple solution, rebuilding what changed is certainly possible but I'd have to tinker with it to see if it could be done in a way which can be generic enough for the seed tools.

Perhaps a possible feature after the cli project is finished.

//cc @mgechev @ludohenin

[mpetkov]

Thank you for the answer. Looks like from your comment in #496 node V5 is way faster than V4 (which I am using). In addition, when I have time I will try to separate the watchers so that they only happen per type so that if a css/html/whatever change happens, the typescript rebuild (which is the most time consuming) doesn't trigger.

[mgechev]

@mpetkov it will be great if you can help with this! Let me know what you find for node 5 vs 4, for your specific case.

[joshwiens]

@mgechev - Node 5 is definitely faster than 4.

• Mid 2014 ( 16gb + SSD )
• Fresh checkout of the angular2-seed

Node 4.4.0

Average of 5 runs dropping highest and lowest
Initial build.js.dev - 6.5s
Watch build.js.dev - 4.1s

Node 5.9.0

Average of 5 runs dropping highest and lowest
Initial build.js.dev - 4.1s
Watch build.js.dev - 1.8s

That said, on larger projects, those numbers obviously jump up. Rebuilding only what's needed in dev would be a solid change imo

[mpetkov]

@mgechev - Unfortunately my work hasn't approved Node 5 so I am unable to install it on my work machine. Here are my numbers on the latest version of node I am able to install:

Node 4.3.2:
Machine Specs: Windows 7; 8GB Ram; 2.6ghz i7-5600U
Current project size of src folder: 59 folders (110 Files) (will grow by about 2x or 3x by time its finished)

Average of 5 runs:
Initial build.js.dev - 12.67s
Watch build.js.dev - 10.47s

As noted by @d3viant0ne it would be a great addition/update to rebuild in dev only what has been changed.

[mgechev]

@mpetkov yes, sounds like terrible benchmarks.

I have a few other things in my schedule so I can schedule the optimization next weekend/the end of the week. If you have any ideas you're welcome to open a PR before that!

[joshwiens]

@mpetkov - Curious, does that system have a SSD or some sort of hybrid traditional mobile hard disk?

[mpetkov]

@d3viant0ne - It is a 256GB SSD (L8T-256L9G-HH).

[brian428]

I think part of the issue is that the watch triggers build.dev, which includes clean.dev. So I think the output folder is getting blown away after each change. I'm not sure if it's possible, but ideally the clean would only delete the files (or folders) affected by the change? Or, it might be worth considering whether a clean is actually needed during watch changes? I can see it being desired during startup, but I'm wondering if doing it for each change has more drawbacks than benefits?

[brian428]

Hmm no...just testing it by removing clean from the sequence didn't seem to affect the watch duration.

[joshwiens]

@brian428 - As an aside, rxjs5 beta.3 speeds up the watch task a bunch. I'll have it updated in the seed shortly.

[brian428]

I see you updated the RxJS dep to beta.3, but this seems to fail since Angular 2 still forces beta.2 as a peer dependency. Is there some trick I'm missing that will bypass that error?

[StefanKoenen]

Any progress on this? It would be really nice if single files are linted / compiled, a big performance hit atm compared to other seeds.

[ludohenin]

If performance is really bad with TS it's still possible to compile the tools, npm run gulp -- build.tools and npm run gulp -- clean.tools when developing tools.

[joshwiens]

@StefanKoenen - We can't realistically bump the seed until Angular2 bumps their rxjs version past beta2 which is on their radar but is blocked by a few issues.

[red010b37]

Could we do and approach similar to with this - https://www.npmjs.com/package/jspm-watch One of the devs I worked with built it for our previous project we had more than 800 files it cut our refresh build from 30s down to 5s.

[totev]

I don't know how you guys are using it, but I get constantly high numbers:
[17:19:44] Finished 'build.js.dev' after 6.65 s
[17:19:44] Finished 'build.dev' after 13 s
Every change I make, even in CSS causes the whole thing to build. This takes up to 1Minute until the browser refreshes.

[joshwiens]

@totev - A lot of that pain is due to rxjs beta.2

Compiling the tools as @ludohenin suggests can speed up the process but in the end it's going to come down to hardware & os.

OSX/Linux is faster than Windows
If you don't have a SSD, it's always going to be slower
Once angular2 updates beyond rxjs-beta.2 life will get a bit better

On a 2014 MBP running a heavily extended project, my initial builds are a third of what you posted and on an Ubuntu 14.04 VM on the same hardware via Fusion, it's actually slightly faster.

[joshwiens]

Also worth noting, Node 5.x is faster than 4.x and is quite stable.

[mgechev]

@totev TypeScript's type checking and transpilation takes a lot, given all the ambient type definitions. On Unix-like environment a big project gets transpiled in less than 8 seconds but on Windows everything happens slower even with incremental transpilation enabled.

Since recently, I'm experimenting with the angular2-hot-loader. Using it any change on template and/or style is instantly visualized in the browser. Hopefully in a few weeks I'll publish a stable version which has limited feature set but works well with templates and styles.

[joshwiens]

@totev - It may be worth experimenting with running a linux vm or developing inside docker ( there are filesystem change detection issues here depending on os )

In essence share a directory with an Ubuntu VM, modify locally but have node and the browser running in Ubuntu.

[totev]

Thanks for the tips @d3viant0ne! I am on osx and not lacking raw power. I will update the node version and test how it compares. I am however currently stuck with those times ~20sec on a cold start and ~15sec on every change regardless of the file type.

@mgechev I am looking forward to your angular2-hot-loader - your work and engagement are much appreciated!

@mgechev it is an enterprise project, however not yet a huge one - 200+ ts files with 60000+ locs. I just can't believe that an incremental compilation is taking place. I am surprised, that I am the only one suffering from high compile times and high latencies from code change to transpilation and browser reflection.

[joshwiens]

@totev - You can try bumping node to 5.x and rxjs to beta.4 ( 5 is broken, no clue about 6 ).

This deviates away from the Angular2 dependencies. I'm doing it on a similar size project so it "works" but it does purposefully deviate away from what's supported and may have side effects I have not run into yet.

Assuming it doesn't completely brick the app, it will most definitely increase the speed of the watch cycle.

[chl03ks]

Sorry guys i was wondering where i can put some questions about the seed l was about to create a issue but i see that i can't ask for support so where can i do that?

[joshwiens]

@gvsdan - The best spot for questions is actually the gitter chat room, click the badge in the readme

[StefanKoenen]

I've made a pullrequest to gulp-typescript, fixed a small bug there that will solve the compilation of TS while only changing the HTML.
Please upgrade when there is a new release :).

[joshwiens]

@StefanKoenen - I've been watching that repo and a few others that have incoming performance fixes. If I happen to miss it, just @ me or PR the update and we will get it in one way or the other.

[mgechev]

Using the seed for development during the last month, I noticed that css linting only slows us (and the build) down. What are your thoughts on commenting this task by default?

[totev]

We also have it currently disabled. Our custom seed project used to have css streaming to the browser, which also sped up the roundtrips. What currently helps is using gulp-cached in the middle of the the ts files pipes. This currently shaves off about 6s and is pretty reliable.

[joshwiens]

@mgechev - IMO css linting should be an explicit task separate from the watch cycle. It still needs to happen, it just doesn't need to happen on every change.

[dmittendorf]

Today I tried adding "isolatedModules: true" to my tsconfig.json file, which dramatically improved the build speed (build.js.dev step goes from 15s to 0.5s) and didn't seem to break anything. Can anyone explain whether it is safe to run with it set to true and/or whether this would be a good option to at least include on dev builds?

[brian428]

@totev: how are you using gulp-cached? I tried sticking it into the JS build dev task, but got errors about not finding classes. I assume that's because the pared-down set of files passed through gulp-cached doesn't give the compiler all the files it needs.

[totev]

@brian428 in your build.js.dev.ts after the declaration of your project related sources:

let typings = gulp.src([
        'typings/browser.d.ts',
        join(config.TOOLS_DIR + '/manual_typings/**/*.d.ts')]);
let projectFiles = gulp.src(src).pipe(plugins.cached());
let result = merge(typings,projectFiles)
        .pipe(plugins.debug())
        .pipe(plugins.plumber())
        .pipe(plugins.sourcemaps.init())
        .pipe(plugins.typescript(tsProject));

Please note, that in order for this to work, you should disable the cleaning step upon every change:
//'clean.dev',

[mgechev]

@dmittendorf by enabling isolatedModules you prevent compile-time type checking from running. At first this seems like you're giving up the selling point of TypeScript, however, on the other hand your text editor/IDE points you out the type errors anyway. Same for linting.

Although it doesn't seem like a good idea to disable type checking in dev build by default, I think it'll be great to point this compiler option out in the readme. Also, to me it seems that we can comment css-lint inside of gulpfile.ts for build.dev.

Thoughts?

[mgechev]

Here's a wiki page which recaps everything https://github.com/mgechev/angular2-seed/wiki/Speeding-the-build-up.

[mgechev]

FYI, applying everything from the wiki page above, dropped the build duration of my project from 11-12s to 500ms.

I think we can close this issue.

[mpetkov]

Thank you everybody! The updates that were implemented and described in the wiki page works amazing and cut down the build.js.dev step by about 10 times!

@mgechev - I was able to implement all of the changes except the gulp-cached. The command merge doesn't allow typings to be passed to it and gives following error:

Argument type NodeJS.ReadWriteStream is not assignable to parameter type Observable

Also I believe that the last thing that can be done to improve the dev watch even more is to seprate it from 1 watch to 3 watches: assets, js/ts, css. Basically that way when you do a change to css it shouldn't fire the assets and ts watch. The reason is because currently after all of these updates, the build.assets.dev takes the most time (which is awesome) because there are a lot of files in there. But since I just made a change to a ts file, it shouldn't even happen.

Thank you all again for the great work.

[mgechev]

@mpetkov there's one more optimization pending, which is introduced by a bugfix by @StefanKoenen.

The optimization you suggested definitely makes sense. In the next 2 weeks I will not be able to work on it but it is definitely in the roadmap.

[Paladinium]

@mgechev : has there been any further steps in the direction that @mpetkov suggests (separating assets/js/css)?

[mgechev]

@Paladinium good point! I'll open an issue for this.

[Paladinium]

Thanks, I really appreciate this! And many others using your great project, too ;-)

[mgechev]

This improvement is coming to the seed #1443.