Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-platform run of an arbitrary script #548

Closed
vsemozhetbyt opened this issue Mar 24, 2017 · 10 comments
Closed

Cross-platform run of an arbitrary script #548

vsemozhetbyt opened this issue Mar 24, 2017 · 10 comments

Comments

@vsemozhetbyt
Copy link

Is there a simple canonical way to run an arbitrary script with Node.js on different platforms? Something like CI, but just for one file.

Say, I want to know if this benchmark is broken just for my OS and environment or it is broken in general. Can I do this?

@bnoordhuis
Copy link
Member

https://ci.nodejs.org/job/node-stress-single-test/ ?

@gibfahn
Copy link
Member

gibfahn commented Mar 24, 2017

We should probably document this in the COLLABORATOR_GUIDE.md.

@vsemozhetbyt
Copy link
Author

vsemozhetbyt commented Mar 24, 2017

@bnoordhuis @gibfahn Maybe I am doing something wrong, I've made 2 attempts, both have failed without the needed output.

I was using ../benchmark/net/tcp-raw-pipe path for the RUN_TESTS field and 1 for the RUN_TIMES field. Is it right?

For the first run, I've used already selected RUN_LABEL, for the second one I've selected something else.

https://ci.nodejs.org/job/node-stress-single-test/1137/

https://ci.nodejs.org/job/node-stress-single-test/1138/

@gibfahn
Copy link
Member

gibfahn commented Mar 24, 2017

@vsemozhetbyt oh right, that won't work for benchmarks. the node-stress-single-test job runs tools/test.py with the arguments you give it, as I understand it the benchmarks are run with node benchmark/run.js.

I guess the answer would be to add a new job to run benchmarks (assuming they don't take too long).

cc/ @nodejs/benchmarking @nodejs/build @mscdex as to whether this is a good idea.

@mscdex
Copy link

mscdex commented Mar 24, 2017

I believe there was some discussion some time ago by the benchmarking group about setting up dedicated hardware for running benchmarks and I was thinking it was about node's benchmarks and not the acme air benchmarks.

@vsemozhetbyt
Copy link
Author

@gibfahn I am trying to do this hard way :)

@gibfahn
Copy link
Member

gibfahn commented Mar 25, 2017

According to the benchmarking WG README, there is currently one dedicated benchmarking machine, which is used to generate the number for https://benchmarking.nodejs.org/.

However, this doesn't really help with the cross-platform testing @vsemozhetbyt is talking about.

If it's just a single (short) script, and it can run from the tarballs on nodejs.org, then it should be pretty straighforward to set up a job that runs the script with a tarball from nodejs.org. If we have to build node then it's another 10-25 minutes per run (depending on the platform).

@mscdex
Copy link

mscdex commented Mar 25, 2017

@gibfahn Right, the keyword there is currently. I do not know what the status is for acquiring hardware for running node benchmarks or even if that is something the benchmarking group is looking into anymore.

Depending on the scope of this request, I think using the pre-compiled tarballs will only provide so much benefit, since I think in most cases we will want to be benchmarking changes made in PRs, which would require compilation.

@mscdex
Copy link

mscdex commented Mar 25, 2017

Additionally, I would think we would want actual dedicated hardware when it comes to running benchmarks, to avoid any interference/noise created by other processes (e.g. Jenkins) running at the same and such.

@gibfahn
Copy link
Member

gibfahn commented Mar 25, 2017

I'd be interested to know how much variability Jenkins adds, given that the current benchmarking jobs are run through ci.nodejs.org.

@mscdex I think both are probably useful, you need to build from source to test a PR, but you often want to test an existing build, that's certainly the case with the node-test-commit jobs, CitGM, and the other nodejs/ CI jobs.

Presumably if we were to have a complete suite of benchmarking machines that would double our hardware usage, I'm not sure whether that would be feasible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants