buffer: hard-deprecate calling Buffer without new #8169

seishun · 2016-08-18T20:52:01Z

Checklist

make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Description of change

Light version of #7152.

We want to make Buffer a class so that it can be subclassed. However, instantiating a class requires new. This hard-deprecates calling Buffer without new.

addaleax · 2016-08-18T20:54:52Z

LGTM, and really, thanks for doing this.

addaleax · 2016-08-18T20:54:58Z

CI: https://ci.nodejs.org/job/node-test-commit/4653/

Qard · 2016-08-18T20:57:29Z

LGTM too.

jasnell · 2016-08-18T21:57:24Z

lib/buffer.js

@@ -65,6 +71,9 @@ function alignPool() {
 * would ever actually be removed.
 **/
 function Buffer(arg, encodingOrOffset, length) {
+  if (!(this instanceof Buffer)) {


Perhaps new.target instead? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target

if (!new.target) { // print deprecation warning }

ChALkeR · 2016-08-18T22:01:30Z

LGTM once the comments above are fixed. One question, though — are there no tests that are affected by this?

jasnell · 2016-08-18T22:04:29Z

This should also include a test case that ensures that the deprecation warning is emitted when expected

ChALkeR · 2016-08-18T22:10:44Z

Btw, should this also hard-deprecate SlowBuffer() without a new?

jasnell · 2016-08-18T22:11:24Z

Good point @ChALkeR ... it probably should. /cc @trevnorris

ChALkeR · 2016-08-18T22:14:24Z

We could start filing some issues in advance.

Edited.

Qard · 2016-08-18T23:07:25Z

With that many occurrences, I feel we might need to do a longer deprecation cycle on this. Even with PRs, it could take awhile for patches to trickle down from deps of deps.

ChALkeR · 2016-08-18T23:14:10Z

@Quard, that's only 1193 modules (approx), and I expect this list to look much less scary once they are ordered by the actual downloads/month stats. I will begin filing issues to the most important ones, see sidorares/node-mysql2#380 for an example.

seishun · 2016-08-19T09:30:25Z

@ChALkeR

Btw, should this also hard-deprecate SlowBuffer() without a new?

I don't see the point. As far as I'm aware, we're not planning to make SlowBuffer subclassable.

@jasnell

This should also include a test case that ensures that the deprecation warning is emitted when expected

Could you or someone else link to an existing test that does a similar thing?

ChALkeR · 2016-08-19T09:34:10Z

@seishun Yes, I totally missed that, thanks.

Perhaps we should hard-deprecate SlowBuffer whatsoever in another PR, then. Not sure yet about the target on that one, though, but I guess 7.0.0 could be fine, as there are not so many SlowBuffer users and those of them who wish to support 0.12 and older v4.x versions could just use the shim. That should be discussed, though.

jasnell · 2016-08-19T13:20:06Z

@seishun ... you can look at test/parallel/test-crypto-deprecated.js for an example of validating that the deprecation warning is emitted. It's pretty straightforward.

process.on('warning', (warning) => {
  assert.strictEqual(warning.name, 'DeprecationWarning');
  // etc...
});

Fishrock123 · 2016-08-19T13:58:57Z

lib/buffer.js

+const bufferDeprecationWarning =
+    deprecate(() => {}, 'Using Buffer without `new` will soon stop working. ' +
+      'Use `new Buffer`, or preferably ' +
+      'Buffer.from, Buffer.allocUnsafe or Buffer.alloc instead.');


probably do not mention allocUnsafe?

if people need that they will find it I think :)

Fishrock123 · 2016-08-19T14:00:04Z

lib/buffer.js

+    newBufferWarned = true;
+    process.emitWarning(
+      'Using Buffer without `new` will soon stop working. ' +
+      'Use `new Buffer`, or preferably ' +


Actually since this will be in v7... maybe we should only recommend Buffer.from() and Buffer.alloc() (pref with the parenthesis?)

~~That works for me~~ misread the comment... see #8169 (comment) instead.

ChALkeR · 2016-08-19T14:32:03Z

SlowBuffer deprecation would affect only 444 packages.

addaleax · 2016-11-04T19:51:17Z

Yeah, I am in favor of reverting. The reactions we are hearing from outside of Node core make it pretty clear that a runtime deprecation for functionality like this is just too disruptive, and we’re not going to get away with it, possibly ever.

addaleax · 2016-11-04T19:52:19Z

Oh and since @Trott put this on the agenda (thanks!), ping @nodejs/ctc

seishun · 2016-11-04T19:55:48Z

The reactions we are hearing from outside of Node core make it pretty clear that a runtime deprecation for functionality like this is just too disruptive

Could you link to some of these reactions, because I'm only seeing negative comments from other Node.js collaborators?

MylesBorins · 2016-11-04T20:02:07Z

@mafintosh @yoshuawuyts @substack have contributed to various wg's but afaik are not part of core

seishun · 2016-11-04T20:23:53Z

To put that in perspective, there were 1193 packages that would affected by this in August (#8169 (comment)). Evidently most of their developers don't mind fixing their code, or their users don't mind the warning, or there are no users.

MylesBorins · 2016-11-04T20:28:40Z

@seisun how many are still a problem?

also 7 has only been out for a couple weeks, and does not have mass adoption yet... we can expect way more problems with 8, especially when it goes Lts

alex7kom · 2016-11-04T21:24:01Z

Maintained modules can be easily fixed. Unmaintained modules may have security issues or bugs, or they may simply be obsolete. So keeping old and insecure APIs just to keep unmaintained modules running will hurt the ecosystem long-term. Unmaintained things usually begin to decay, so building an ecosystem on them is not a great idea. Made by an unpaid volunteer or not, old code is old code.

feross · 2016-11-04T23:35:53Z

If we want to hard-deprecate Buffer() and then eventually new Buffer(), then IMO we should do them both at the same time, and we should be clear about the reasoning: this is for security reasons. Buffer.alloc() is preferred over Buffer() since it zeroes out the memory.

It would be really unfortunate if folks started updating their code to use new just to have that get deprecated shortly thereafter.

billiegoose · 2016-11-07T20:05:44Z

So I've been following this thread, and what isn't clear to me is from the security point of view, why you can't you just zero-fill buffers by default? Specifically, make Buffer() and new Buffer() aliases for Buffer.from(). At worst that would cause old code that hasn't been updated to Buffer.allocUnsafe()to run slower.

jasnell · 2016-11-07T21:35:25Z

I've been trying to think of a way forward on this that would allow us to maintain the existing uses of Buffer() with or without the use of new while still allowing us to progress with improved internals. There's one possible approach that I've come up with that's going to need a bit more investigation, but at first blush it's doable.

If we followed a pattern similar to:

class BufferArray extends Uint8Array {
  constructor(/*.. args ..*/) {
    /* ... */
  }
}

function Buffer(/* .. args ..*/) {
  if (!(this instanceof Buffer))
    return new Buffer(/* .. args ..*/);
  return Reflect.construct(BufferArray, [/** args **/], new.target);
}
util.inherits(Buffer, BufferArray);

Then calling Buffer() and new Buffer() as currently done would still continue to work and the resulting object would properly inherit from the new BufferArray ES6 class where the improved internal implementation would live. It would mean introducing a new object into the prototype chain, but it allows us a path forward that would enable existing code to keep functioning. Buffer itself essentially just becomes a backwards compatible shim for the new implementation.

There are obviously some details that would need to be worked out in this but, at the very least, it provides on possible option.

addaleax · 2016-11-07T21:39:00Z

@jasnell have you seen addaleax/node@ec09d43? It goes pretty much into the direction you’re describing, with tests passing and everything, it would just take a bit of work to match the current performance (because it currently creates unnecessary intermediate objects).

(I’ve linked this before – in the other thread, I think – but I do see that these things can get a bit lost 😄)

evanlucas · 2016-11-07T21:39:23Z

I think I am +1 on reverting as well

jasnell · 2016-11-07T21:41:34Z

@addaleax ... I think I saw the link at some point but I don't believe I'd actually seen the code. Will dig in more! Thank you!

seishun · 2016-11-08T08:05:36Z

@jasnell I am -1 on introducing an extra class. Things are already complicated enough with Array, Buffer, ArrayBuffer and UintXArray. Please try to imagine yourself in the shoes of someone who is just getting familiar with JavaScript and Node.js. Backward compatibility is important, but it would be unfortunate to permanently increase complexity in order to avoid a temporary disruption.

mafintosh · 2016-11-08T08:32:41Z

@jasnell I don't fully understand the Reflect thing but that seems like something that could work and not break things.

addaleax · 2016-11-08T12:26:44Z

@seishun If we do this right, the additional complexity will be limited to something that can be explained in a single sentence; namely, the only difference between Buffer and BufferArray would be that BufferArray is that the latter is an ES6 class, meaning that it can’t be instantiated without new and that it can be inherited from.

And yeah, I’m not too keen on the name BufferArray (or my BufferClass) either, but I get that it’s hard to come up with a name for something that basically already exists under a different name. If nothing else works, I’d be for SubclassableBuffer, because that’s what it is.

This reverts commit f2fe558 (nodejs#8169) as the original justification for the runtime-deprecation does not appear to justify the disruption to Node’s existing ecosystem. Futhermore, the possibility of deprecating the Buffer constructor entirely in v8.0 might lead to people having to change their code twice.

addaleax · 2016-11-09T21:05:07Z

I would like to ask everyone who has commented here to move the discussion to #9531, if only so the public discussion can take in a central place and not over multiple GH threads at once.

This reverts commit f2fe558 (#8169) as the original justification for the runtime-deprecation does not appear to justify the disruption to Node’s existing ecosystem. Futhermore, the possibility of deprecating the Buffer constructor entirely in v8.0 might lead to people having to change their code twice. PR-URL: #9529 Reviewed-By: Roman Reiss <me@silverwind.io> Reviewed-By: Myles Borins <myles.borins@gmail.com> Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com> Reviewed-By: Evan Lucas <evanlucas@me.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

buffer: hard-deprecate calling Buffer without new

996389c

nodejs-github-bot added the buffer Issues and PRs related to the buffer subsystem. label Aug 18, 2016

addaleax added the semver-major PRs that contain breaking changes and should be released in the next major version. label Aug 18, 2016

seishun mentioned this pull request Aug 18, 2016

buffer: runtime-deprecate Buffer constructor #7152

Closed

2 tasks

jasnell reviewed Aug 18, 2016
View reviewed changes

ChALkeR mentioned this pull request Aug 18, 2016

Buffer() without a new keyword is going to be hard-deprecated soon sidorares/node-mysql2#380

Closed

use shiny new syntax

63172e4

Fishrock123 reviewed Aug 19, 2016
View reviewed changes

use emitWarning directly

54ca2d8

Fishrock123 reviewed Aug 19, 2016
View reviewed changes

add backticks and parentheses

fccd038

addaleax mentioned this pull request Nov 7, 2016

Node.js Foundation Core Technical Committee (CTC) Meeting 2016-11-09 #9507

Closed

addaleax mentioned this pull request Nov 9, 2016

Revert "buffer: runtime deprecation of calling Buffer without new" #9529

Closed

2 tasks

addaleax mentioned this pull request Nov 9, 2016

buffer: discuss future direction of Buffer constructor API #9531

Closed

Trott removed the ctc-agenda label Nov 16, 2016

gibfahn mentioned this pull request Jun 15, 2017

Auditing for 6.11.1 nodejs/Release#230

Closed

3 tasks

seishun deleted the new branch August 12, 2017 11:48

nektro mentioned this pull request Sep 24, 2018

change new Buffer to Buffer.from pgilad/gulp-sitemap#27

Closed

sam-github mentioned this pull request Jul 23, 2019

What to do about Buffer? nodejs/security-wg#564

Closed

buffer: hard-deprecate calling Buffer without new #8169

buffer: hard-deprecate calling Buffer without new #8169

Conversation

seishun commented Aug 18, 2016

Checklist

Affected core subsystem(s)

Description of change

addaleax commented Aug 18, 2016

addaleax commented Aug 18, 2016

Qard commented Aug 18, 2016

jasnell Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

ChALkeR commented Aug 18, 2016

jasnell commented Aug 18, 2016

ChALkeR commented Aug 18, 2016 • edited Loading

jasnell commented Aug 18, 2016

ChALkeR commented Aug 18, 2016 • edited Loading

Qard commented Aug 18, 2016

ChALkeR commented Aug 18, 2016 • edited Loading

seishun commented Aug 19, 2016 • edited Loading

ChALkeR commented Aug 19, 2016 • edited Loading

jasnell commented Aug 19, 2016

Fishrock123 Aug 19, 2016

Choose a reason for hiding this comment

Fishrock123 Aug 19, 2016

Choose a reason for hiding this comment

jasnell Aug 19, 2016 • edited Loading

Choose a reason for hiding this comment

ChALkeR commented Aug 19, 2016

addaleax commented Nov 4, 2016

addaleax commented Nov 4, 2016

seishun commented Nov 4, 2016

MylesBorins commented Nov 4, 2016

seishun commented Nov 4, 2016

MylesBorins commented Nov 4, 2016

alex7kom commented Nov 4, 2016

feross commented Nov 4, 2016

billiegoose commented Nov 7, 2016

jasnell commented Nov 7, 2016

addaleax commented Nov 7, 2016 • edited Loading

evanlucas commented Nov 7, 2016

jasnell commented Nov 7, 2016

seishun commented Nov 8, 2016

mafintosh commented Nov 8, 2016

addaleax commented Nov 8, 2016

addaleax commented Nov 9, 2016

jasnell Aug 18, 2016 •

edited

Loading

ChALkeR commented Aug 18, 2016 •

edited

Loading

ChALkeR commented Aug 18, 2016 •

edited

Loading

ChALkeR commented Aug 18, 2016 •

edited

Loading

seishun commented Aug 19, 2016 •

edited

Loading

ChALkeR commented Aug 19, 2016 •

edited

Loading

jasnell Aug 19, 2016 •

edited

Loading

addaleax commented Nov 7, 2016 •

edited

Loading