Cleanup _writableState and _readableState access across codebase

[vkurchatkin]

This is a meta-issue to keep to track of all usages of _writableState and _readableState outside of streams source. These properties are considered private and should not be used unless absolutely necessary. Usage of them can indicate a few things:

• the code can be rewritten using existing documented API to achieve the same result;
• streams lack some consumer functionality and new public API should be introduced;
• streams lack some implementor functionality and new protected API should be introduced;
• documentation needs to be added for some parts of private state for implementors;
• it is an optimization that is and always be possible only in core.

The list of all _writableState and _readableState usages:

src/node.js

• L564 stdin._readableState.reading = false. Added in commit: bb56dcc by @isaacs; (src: nix stdin _readableState.reading manipulation #454)
• L573 stdin._readableState.reading = false. Added in commit: bb56dcc by @isaacs; (src: nix stdin _readableState.reading manipulation #454)

lib/_debug_agent.js

• L87 this._readableState.objectMode = true (_debug_agent: use readableObjectMode option for client stream #270);

lib/_http_server.js

• L348 socket._readableState.flowing = null (TODO(isaacs): Need a way to reset a stream to fresh state IE, not flowing, and not explicitly paused.). Added in 967b5db by @isaacs;
• L408 var needPause = socket._writableState.needDrain. Added in 085dd30 by @isaacs ;
• L445 req._readableState.resumeScheduled. Not sure where this originated, but in 2efe4ab @indutny added oldMode check here;

lib/_tls_legacy.js

• L421 this._writableState.finished;
• L508 self._readableState.length > 0;

lib/_tls_wrap.js

• L311 self._writableState.errorEmitted (Cleanup stream state in net #465);
• L313 self._writableState.errorEmitted = true (Cleanup stream state in net #465);
• L350 socket._readableState.length;

lib/child_process.js

• L1061 stream._readableState.flowing (child_process: remove redundant condition #511);

lib/crypto.js

• whole LazyTransform thing. Is it really necessary? Maybe it should go to stream? Maybe it should be public? Maybe transforms should be lazy by default?;
• L56 this._writableState.decodeStrings = false;
• L57 this._writableState.defaultEncoding = 'binary';
• L90 var encoding = this._readableState.encoding || 'buffer' (crypto: remove use of this._readableState #610);

lib/fs.js

• L1624 allocNewPool(this._readableState.highWaterMark);

lib/net.js

• L162 this._writableState.decodeStrings = false (Cleanup stream state in net #465);
• L174 this._readableState.flowing = false (Cleanup stream state in net #465);
• L196 this._readableState.ended (Cleanup stream state in net #465);
• L226 self._readableState.ended;
• L242 this._readableState.ended = true (comment: ended should already be true, since this is called after the EOF errno and onread has eof'ed) (Cleanup stream state in net #465);
• L243 this._readableState.endEmitted;
• L362 this._writableState.length;
• L392 this._readableState.endEmitted;
• L405 socket._writableState.length;
• L415 if (this._writableState.finished);
• L429 self._writableState.errorEmitted (Cleanup stream state in net #465);
• L433 self._writableState.errorEmitted = true (Cleanup stream state in net #465);
• L535 self._readableState.length === 0;
• L715 state.getBuffer();
• L842 this._readableState.reading = false;
• L843 this._readableState.ended = false;
• L844 this._readableState.endEmitted = false;
• L845 this._writableState.ended = false;
• L846 this._writableState.ending = false;
• L847 this._writableState.finished = false;
• L848 this._writableState.errorEmitted = false (Cleanup stream state in net #465);

lib/zlib.js

• L423 ws.ended;
• L426 ws.ending;
• L429 ws.needDrain;
• L460 ws.ending || ws.ended;
• L461 ws.length;
• L479 ws.length.

List of used properties:

_readableState

• reading;
• objectMode;
• flowing (boolean?) is used to determine which mode readable stream is in; can be true, false or null; `null is the initial state which means that is implicitly paused;
• resumeScheduled;
• length;
• encoding;
• highWaterMark;
• ended;
• endEmitted;

_writableState

• needDrain;
• ended;
• ending;
• finished;
• errorEmitted;
• decodeStrings;
• defaultEncoding;
• length;
• getBuffer();

/cc @chrisdickinson

[chrisdickinson]

This is great! Thanks for compiling a comprehensive list. The most egregious overreaches, I think, are any of the places we reach in and directly modify state. Informational access is less bad.

I'd like to encourage folks to pick these off one at a time. For changes that might require a new internal API, it'd be great to briefly discuss plans here first before cutting a PR.

On Jan 15, 2015, at 8:18 AM, Vladimir Kurchatkin notifications@github.com wrote:

This is a meta-issue to keep to track of all usages of _writableState and _readableState outside of streams source. These properties are considered private and should not be used unless absolutely necessary. Usage of them can indicate a few things:

the code can be rewritten using existing documented API to achieve the same result;
streams lack some consumer functionality and new public API should be introduced;
streams lack some implementor functionality and new protected API should be introduced;
documentation needs to be added for some parts of private state for implementors;
it is an optimization that is and always be possible only in core.
The list of all _writableState and _readableState usages:

src/node.js

L564 stdin._readableState.reading = false;
L573 stdin._readableState.reading = false;
lib/_debug_agent.js

L87 this._readableState.objectMode = true (#270);
lib/_http_server.js

L348 socket._readableState.flowing = null (TODO(isaacs): Need a way to reset a stream to fresh state IE, not flowing, and not explicitly paused.);
L408 var needPause = socket._writableState.needDrain;
L445 req._readableState.resumeScheduled;
lib/_tls_legacy.js

L421 his._writableState.finished;
L508 self._readableState.length > 0;
lib/_tls_wrap.js

L311 self._writableState.errorEmitted;
L313 self._writableState.errorEmitted = true;
L350 socket._readableState.length;
lib/child_process.js

L1061 stream._readableState.flowing;
lib/crypto.js

whole LazyTransform thing. Is it really necessary? Maybe it should go to stream? Maybe it should be public? Maybe transforms should be lazy by default?;
L56 this._writableState.decodeStrings = false;
L57 this._writableState.defaultEncoding = 'binary';
L90 var encoding = this._readableState.encoding || 'buffer';
lib/fs.js

L1624 allocNewPool(this._readableState.highWaterMark);
lib/net.js

L162 this._writableState.decodeStrings = false;
L174 this._readableState.flowing = false;
L196 this._readableState.ended;
L226 self._readableState.ended;
L242 this._readableState.ended = true (comment: ended should already be true, since this is called after the EOF errno and onread has eof'ed);
L243 this._readableState.endEmitted;
L362 this._writableState.length;
L392 this._readableState.endEmitted;
L405 socket._writableState.length;
L415 if (this._writableState.finished);
L429 self._writableState.errorEmitted;
L433 self._writableState.errorEmitted = true;
L535 self._readableState.length === 0;
L715 state.getBuffer();
L842 this._readableState.reading = false;
L843 this._readableState.ended = false;
L844 this._readableState.endEmitted = false;
L845 this._writableState.ended = false;
L846 this._writableState.ending = false;
L847 this._writableState.finished = false;
L848 this._writableState.errorEmitted = false;
/cc @chrisdickinson

—
Reply to this email directly or view it on GitHub.

[chrisdickinson]

Of note: I'm not opposed to all non-streams access to _readableState, just accesses that are directly inspecting a property or manipulating state. APIs private to io.js itself are okay to hang off of ReadableState, but I'd like to make sure that there's a clear API there, and not just random attribute access/manipulation.

[vkurchatkin]

We should also probably inspect userland modules to find out what people are doing with internals.

[chrisdickinson]

As a heads up, I edited your issue comment to link #454 for the node.js _readableState use (sorry sorry). In the future, I can ping you instead if you'd like to keep the list maintained!

[vkurchatkin]

@chrisdickinson fine by me. But think it's a good idea to reference this issue so we can see the timeline of all relevant stuff here

[chrisdickinson]

We might add lib/zlib.js:

• L423: ws.ended
• L426: ws.ending
• L429: ws.needDrain
• L460: ws.ending || ws.ended
• L461: ws.length
• L479: ws.length

[chrisdickinson]

Cross posting this from nodejs/readable-stream#109:

To amend the intent of this issue a bit – the goal is not to absolutely remove all use of _{{writ,read}able,transform}State, but to remove all places where:

• subclasses directly manipulate private state, or...
• state is being preserved in the base classes for the exclusive use of a subclass.

Ultimately, the places where the subclasses access those state objects should be catalogued and we should look into the best way to deal with that access – whether that's promotion into a public or protected "reflection" API for subclasses.

[Trott]

This issue hasn't seen any activity in almost a year. Is it still active and useful? Or not so much?

[indutny]

I think it is very informative, and perhaps needs some work to be done.

[Fishrock123]

Marking as stalled, is this still viable after the problems we encountered last time?

[RedYetiDev]

(The stalled bot doesn't appear to be working, so I've manually closed. Feel free to reopen if this is no longer stalled)