Use Release API for binary downloads #712

nschonni · 2015-02-27T01:29:44Z

While a tag shouldn't be altered once it's been made, the GitHub release API allows files to be attached to the release https://developer.github.com/v3/repos/releases/#upload-a-release-asset
This would mean that platform specific binaries could be built and attached after the code had been tagged. This would be a solution to the recent issue of frequent io.js releases in between official tagged releases. People could just check out the latest release tag and build for the new platform to be supported.

am11 · 2015-02-27T01:50:26Z

At some point, we were gauging the options to either use gh-API or amazon S3 bucket with node-pre-gyp (#56), until this CentOS incompatible binary issue showed up.

BTW, shouldn't we use this API for uploading the binaries only? I mean the download URL will remain the same: raw.githubusercontent.com.., no?

nschonni · 2015-02-27T02:20:21Z

I think the new download URL would be something like https://github.com/sass/node-sass/releases/download/v3.0.0/darwin-x64-4.1.0.21.zip (https://developer.github.com/v3/repos/releases/#get-a-single-release-asset) for OSX io.js 1.4 given what you found in #710 (comment) plus the switch back to using the V8 version as the identifier.

am11 · 2015-02-27T03:47:21Z

Nice one! So basically we can upload assets to node-sass repo, instead of having a separate node-sass-binaries repo.

I think something like this will do the trick: https://gist.github.com/am11/a4a708120f16934f5d81 (credit goes to this guy: http://stackoverflow.com/a/26680032/863980).

We can then place upload.js in scripts/ directory and call it like:

node scripts/build -f
# then
node scripts/upload --user am11 --auth myGitHubOAuthToken

myGitHubToken was build from public access token section: https://github.com/settings/applications#personal-access-tokens with public-repo check marked. (on personal device, we can probably store the token in env var and echo it on publishing the binary.

am11 · 2015-02-27T11:59:25Z

Incidentally, will these upload assets appear in the repository?

nschonni · 2015-02-28T23:37:03Z

I think the binaries should still stay in the separate repo, and then we just copy them to the release when the release goes live.

am11 · 2015-03-01T00:57:55Z

I agree. This way, we can take binaries contribution even after the node-sass build has released.

We can think about knitting #720, #389, #689 and this one to form a versatile build strategy, which will allow users to specify:

which binary they want to fetch at install time (Changing binary #689; supply an custom argument to npm install node-sass).
which commit they want to build the binary against: Expose config param to allow specific libsass commit for build #389 (perhaps we don't need to provision it in package.json, we can simply take libsass-commit-hash as argument like node scripts/node --libsass-hash <hash>). This can happen on the already existing installation as well as the install time (npm install node-sass --libsass-hash <hash>).
the location of binary by setting the environment var SASS_BINARY_PATH=/my/favorite/path/blah.node (not exactly but inspired by Download sass binary from environment variable url #720).
the URL to download binary from by setting the environment var SASS_BINARY_URL=http://my-favorite-cdn-url/downloads (Download sass binary from environment variable url #720).

Then for the release, we will:

bootstrap by tagging the node-sass-binary with same tag as node-sass (like we do today). This is just for our accounting and has nothing to do with npm install step.
Make a draft release on node-sass, throw in the collected binaries from node-sass-binaries repo:

Save release and publish npm.

On npm install node-sass, it will download from https://github.com/sass/node-sass/releases/download/---/--.node/

Later if we realize that some new binary needs to be uploaded for this (published) release, we can use the GH API upload script (node scripts/upload --argument-able) to add it to the release.
If we realize that the existing binary is broken and need replacing, we cannot delete it but we can rectify it by adding another binary with same_name + "v1" and ask the user to either:

Download the binary and set the SASS_BINARY_PATH=/my/downloads/blah.node or replace the existing vendor/blah/blah.node.
Reinstall node-sass like:
- SASS_BINARY_URL=https://download.github.com/sass/node-sass/blah/blah-v1.node && npm install node-sass
- or npm install node-sass --sass-binary linux-x64-<my-v8-version>.

If they need to download binaries for multiple OSes (for distilled deployment use-cases: as #689), they can npm install node-sass --save-binaries linux-x64-<my-v8-verson> linux-x86-<my-v8-verson> darwin-x64-<my-v8-verson> <-- this will essentially skip the binary testing on install.

Even the post-release-binaries will also be pushed to node-sass-binaries repo first and then a tagged release will be made for instance: v3.0.0-post-v1, v3.0.0-post-v2 etc. This way we will keep record of all binaries ever happen to node-sass from Sass Org and by whom (but we cannot assert this policy in code I suppose, since this is matter of workflow which can be violated by just not following the workflow)..

Thoughts?

xzyfer · 2015-03-04T11:07:19Z

👍 x 💯

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as envionment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload upload URL. * Note: this will superesede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as envionment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will superesede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: #.743.

@nschonni

from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. //cc @nschonni, @xzyfer

@nschonni

from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743. //cc @nschonni, @xzyfer

@nschonni

from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743. //cc @nschonni, @xzyfer

@nschonni

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743. //cc @nschonni, @xzyfer

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743.

am11 · 2015-03-08T17:16:31Z

Note that if the binary name is other than binding.node, it causes error on Linux only with node.js v0.10.x: https://travis-ci.org/am11/node-sass/jobs/53492321#L567.

Module._extensions[extension](this, filename);
                             ^
Error: Symbol linux_x64_3.14.5_module not found.

However, with njs v0.12 and io.js vEdge, it works on Linux. On Windows it works for all njs 0.10, 0.12 and io.js vEdge.

This happened when I was testing with linux-x64-3.14.5.node. So I have changed it back to binding.node and the file we will upload will be named linux-x64-3.14.5_binding.node. After downloading, it will be renamed to linux-x64-3.14.5/binding.node passed to path.join(vendor_directory, __) to form the abs. process.sass.binaryPath.

The only hiccup is now we would manually rename the binary files before uploading as release asset.

@koppa, it seems like there is a strong name bond in 0.10.x, which was removed in latter versions on node.js and io.js. Is there a work around for it?

nschonni · 2015-03-08T22:20:54Z

@am11 wouldn't the upload to the release API be a zip? The zip itself would have the correct folder nesting when you unzip it. I don't know what is the preferred npm package for dealing with cross-platform unzipping anymore.

nschonni · 2015-03-08T22:40:13Z

There is also the option of using Appveyor for the Windows binaries http://www.appveyor.com/docs/deployment/github

am11 · 2015-03-08T23:01:45Z

The AppVeyor technique is the one which node-pre-gyp uses. Our script is just a baby version of it. 👶

I think its better that we could avoid the unzipping step (with extra dependency).

If we somehow workaround that node 0.10 issue, we will save this extra folder nesting for good and a .replace(/_/, '/') statement from our code. 😃

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede default name. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * This name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Note: this will supersede both default name as well as the `SASS_BINARY_NAME` environment variable. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede default path. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * This name will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Note: this will supersede both default path as well as the `SASS_BINARY_PATH` environment variable. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743.

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * Note: this will supersede default name. * Allows user to set binary name as in package file `"nodeSassConfig": { "binary-name": "" }`. * Note: this will supersede default name and `SASS_BINARY_NAME` environment variable. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * Note: this will supersede default name, name defined in package `nodeSassConfig` as well as the `SASS_BINARY_NAME` environment variable. * This binary name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * Note: this will supersede default path. * Allows user to set binary name as in package file `"nodeSassConfig": { "binary-path": "" }`. * Note: this will supersede default path and `SASS_BINARY_PATH` environment variable. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * Note: this will supersede default path, path as well as the `SASS_BINARY_PATH` environment variable. * This binary path will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743.

am11 · 2015-03-11T00:24:07Z

Merged! 😃

* Delivers binary name, paths and download URL from lib/extensions.js. * Allows user to set binary name as environment variable with `SASS_BINARY_NAME`. * Note: this will supersede default name. * Allows user to set binary name as in package file `"nodeSassConfig": { "binary-name": "" }`. * Note: this will supersede default name and `SASS_BINARY_NAME` environment variable. * Allows user to set binary name as parameter to invoke any node-sass script with `--binary-name` flag. * Note: this will supersede default name, name defined in package `nodeSassConfig` as well as the `SASS_BINARY_NAME` environment variable. * This binary name will be used to construct the: * Binary path. * Binary download URL. * Upload URL. * Allows user to set binary path as environment variable with `SASS_BINARY_PATH`. * Note: this will supersede default path. * Allows user to set binary name as in package file `"nodeSassConfig": { "binary-path": "" }`. * Note: this will supersede default path and `SASS_BINARY_PATH` environment variable. * Allows user to set binary path as parameter to invoke any node-sass script with `--binary-path` flag. * Note: this will supersede default path, path as well as the `SASS_BINARY_PATH` environment variable. * This binary path will be used when: * Requiring node-sass package. * Downloading binary. * Uploading binary. * Wraps all extensions in `process.sass` namespace. Issue URL: sass#712. PR URL: sass#743.

nschonni · 2015-03-12T23:23:55Z

👍 I came across this today https://www.npmjs.com/package/semantic-release and it might be worth looking at later

am11 added the npm build label Feb 27, 2015

This was referenced Feb 27, 2015

Linux ARM: Add binaries for 2.0.1 sass/node-sass-binaries#42

Closed

npm ERR! node-sass@2.0.1 postinstall: node scripts/build.js npm #693

Closed

am11 mentioned this issue Mar 4, 2015

Binary for FreeBSD 10.x(amd64) node 0.12 sass/node-sass-binaries#46

Closed

This was referenced Mar 4, 2015

Support clang C++ library #733

Closed

Time to ship 3.0.0? #710

Closed

Build: Implements GH API to upload binaries #737

Merged

am11 mentioned this issue Mar 7, 2015

Build: Centralises binary naming conventions (build changes part 2) #743

Merged

am11 mentioned this issue Mar 9, 2015

node-sass + libsass source coupling #744

Closed

am11 closed this as completed Mar 11, 2015

am11 added this to the 3.0 milestone Mar 11, 2015

am11 mentioned this issue Mar 11, 2015

Download sass binary from environment variable url #720

Closed

xzyfer mentioned this issue Mar 26, 2016

Better error messages for missing binaries #1428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Release API for binary downloads #712

Use Release API for binary downloads #712

nschonni commented Feb 27, 2015

am11 commented Feb 27, 2015

nschonni commented Feb 27, 2015

am11 commented Feb 27, 2015

am11 commented Feb 27, 2015

nschonni commented Feb 28, 2015

am11 commented Mar 1, 2015

xzyfer commented Mar 4, 2015

am11 commented Mar 8, 2015

nschonni commented Mar 8, 2015

nschonni commented Mar 8, 2015

am11 commented Mar 8, 2015

am11 commented Mar 11, 2015

nschonni commented Mar 12, 2015

Use Release API for binary downloads #712

Use Release API for binary downloads #712

Comments

nschonni commented Feb 27, 2015

am11 commented Feb 27, 2015

nschonni commented Feb 27, 2015

am11 commented Feb 27, 2015

am11 commented Feb 27, 2015

nschonni commented Feb 28, 2015

am11 commented Mar 1, 2015

xzyfer commented Mar 4, 2015

am11 commented Mar 8, 2015

nschonni commented Mar 8, 2015

nschonni commented Mar 8, 2015

am11 commented Mar 8, 2015

am11 commented Mar 11, 2015

nschonni commented Mar 12, 2015