Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat!: throw error with out of bounds integer values instead of truncating, add custom 'integerValue' type cast option #488

Closed
wants to merge 22 commits into from
Closed

feat!: throw error with out of bounds integer values instead of truncating, add custom 'integerValue' type cast option #488

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
60c0900
feat: handle outside of bounds numeric values, add custom 'integerVal…
AVaksman Sep 5, 2019
6d1c7cc
chore: add unit tests
AVaksman Sep 5, 2019
edeec3e
Merge branch 'master' into numeric_typeCast
AVaksman Sep 5, 2019
e882977
feat: add validation agains MIN_SAFE_INTEGER
AVaksman Sep 5, 2019
5863365
refactor: remove extra condition
AVaksman Sep 6, 2019
9358b68
Merge branch 'master' into numeric_typeCast
AVaksman Sep 6, 2019
a7856d8
Merge branch 'master' into numeric_typeCast
AVaksman Sep 6, 2019
5f5be01
Merge branch 'master' into numeric_typeCast
AVaksman Sep 9, 2019
8668bd5
refactor: more specific name
AVaksman Sep 9, 2019
e5b8635
Merge branch 'master' into numeric_typeCast
AVaksman Sep 11, 2019
d66115b
Merge branch 'master' into numeric_typeCast
JustinBeckwith Sep 11, 2019
40f2f90
Merge branch 'master' into numeric_typeCast
AVaksman Sep 12, 2019
4e6f530
refactor: use Number.isSafeInteger
AVaksman Sep 15, 2019
b856dc8
fix: docs
AVaksman Sep 16, 2019
e201f6e
fix: docs2
AVaksman Sep 17, 2019
4c2ba71
Merge branch 'master' into numeric_typeCast
AVaksman Sep 19, 2019
1366a7a
Merge branch 'master' into numeric_typeCast
AVaksman Sep 26, 2019
4fb2306
Merge branch 'master' into numeric_typeCast
AVaksman Oct 2, 2019
6770518
Merge branch 'master' into numeric_typeCast
AVaksman Oct 2, 2019
68ce578
Merge branch 'master' into numeric_typeCast
jkwlui Oct 3, 2019
7aa3617
Merge branch 'master' into numeric_typeCast
AVaksman Oct 7, 2019
4b7cb3e
Merge branch 'master' into numeric_typeCast
JustinBeckwith Oct 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 95 additions & 8 deletions src/entity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import arrify = require('arrify');
import * as extend from 'extend';
import * as is from 'is';
import {Query, QueryProto} from './query';
import {Query, QueryProto, IntegerTypeCastOptions} from './query';
import {PathType} from '.';
import {Entities} from './request';
import * as Protobuf from 'protobufjs';
Expand Down Expand Up @@ -327,11 +327,68 @@ export namespace entity {
return value instanceof entity.Key;
}

/**
* Convert a protobuf `integerValue`.
*
* @private
* @param {object} valueProto The protobuf `integerValue` to convert.
* @param {object} integerTypeCastOptions Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
jkwlui marked this conversation as resolved.
Show resolved Hide resolved
* provided function to convert `integerValue`.
* @property {sting|string[]} {typeCastConig.names} `Entity` property
jkwlui marked this conversation as resolved.
Show resolved Hide resolved
* names to be converted using `integerTypeCastFunction`.
*/
function decodeIntegerValue(
valueProto: ValueProto,
integerTypeCastOptions?: IntegerTypeCastOptions
) {
let customCast = false;
if (integerTypeCastOptions) {
if (
typeof integerTypeCastOptions.integerTypeCastFunction !== 'function'
) {
throw new Error(
`integerTypeCastFunction is not a function or was not provided.`
);
}
customCast = true;
if (integerTypeCastOptions.names) {
integerTypeCastOptions.names = arrify(integerTypeCastOptions.names);
if (!integerTypeCastOptions.names.includes(valueProto.name!)) {
customCast = false;
}
}
}

if (customCast) {
try {
return integerTypeCastOptions!.integerTypeCastFunction(
valueProto.integerValue
);
} catch (error) {
throw new Error(`integerTypeCastFunction threw an error:\n${error}`);
}
} else {
const num = Number(valueProto[`integerValue`]);
if (num > Number.MAX_SAFE_INTEGER || num < Number.MIN_SAFE_INTEGER) {
throw new Error(
`Integer value ${valueProto[`integerValue`]} is out of bounds.`
);
}
return num;
}
}

/**
* Convert a protobuf Value message to its native value.
*
* @private
* @param {object} valueProto The protobuf Value message to convert.
* @param {object} integerTypeCastOptions Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
jkwlui marked this conversation as resolved.
Show resolved Hide resolved
* provided function to convert `integerValue`.
* @property {sting|string[]} [integerTypeCastOptions.names] `Entity` property
* names to be converted using `integerTypeCastFunction`.
* @returns {*}
*
* @example
Expand All @@ -350,13 +407,19 @@ export namespace entity {
* });
* // <Buffer 68 65 6c 6c 6f>
*/
export function decodeValueProto(valueProto: ValueProto) {
export function decodeValueProto(
valueProto: ValueProto,
integerTypeCastOptions?: IntegerTypeCastOptions
) {
const valueType = valueProto.valueType!;
const value = valueProto[valueType];

switch (valueType) {
case 'arrayValue': {
return value.values.map(entity.decodeValueProto);
// tslint:disable-next-line no-any
return value.values.map((val: any) =>
entity.decodeValueProto(val, integerTypeCastOptions)
);
}

case 'blobValue': {
Expand All @@ -372,7 +435,7 @@ export namespace entity {
}

case 'integerValue': {
return Number(value);
return decodeIntegerValue(valueProto, integerTypeCastOptions);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely a more robust option, but was just returning the DS.Int type considered?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I mixed up Datastore's "Int" with Spanner's: https://github.com/googleapis/nodejs-spanner/blob/f8ad0ece28e99dd9c6d7e985de56bd1b376f4712/src/codec.ts#L139 -- for simplicity's sake, would it be worth considering using that format instead? When a value is out of bounds, it will throw. The value is accessible via the "value" property in string form.

Copy link
Contributor Author

@AVaksman AVaksman Sep 12, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I do like the ability to still access the value in string form without extra network call in case of value is out of balance.
  • it will definitely be a breaking change.
  • IMO it will add complexity to the user's code.
    will need to try=>catch for valueOf and if error still pass the value to custom typeCastFunction.
  • Is it overkill to combine these two approaches?
    • return Spanner's style object for integerValue.
    • for valueOf on default throw in case of out of bounds.
    • able to pass integerTypeCastFunction and valueOf will return the custom function result.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I do like the ability to still access the value in string form without extra network call in case of value is out of balance.

Sorry if I misunderstand. Neither of the solutions make an extra network call, do they?

  • it will definitely be a breaking change.

I think the throwing behavior is identical to what is in this PR. It will first try to just use it as a number, but if it's out of bounds when called upon, it throws. Or, is it because the new "Int" wrapper isn't technically a Number? Lots of gray area in determining a breaking change 😕

  • Is it overkill to combine these two approaches?

I quite like that approach! Maybe @callmehiphop has a thought on this, too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clearing that up for me! For the Number vs customer "Int" type issue, they can at least be used interchangeably, the only difference would be typeof checks for Number wouldn't pass... unless Int extended Number? Curious to hear other thoughts on the best way to go forward.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we ever make progress on this, maybe in another chat?

Copy link
Contributor Author

@AVaksman AVaksman Sep 19, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick summary
Would like to hear the thoughts on the approach

  1. Throw on default, let user pass integerTypeCastFunction (currently implemented in this PR)
  2. Return an object Integer similar to Spanner, where
    • #valueOf would try to convert to Number and throw if out of bounds
    • #value returns string representation of the value
  3. Combine tow approaches above
    • do return an Integer object
    • let user provide integerTypeCast function to be used by #valueOf
    • still return string via #value

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My vote would go for #­3. Ping to @callmehiphop and @bcoe for more votes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My vote is starting simple with option 1, as a breaking change, and we can move towards an object representation in the future, if there's good reason.

}

case 'entityValue': {
Expand Down Expand Up @@ -505,6 +568,11 @@ export namespace entity {
*
* @private
* @param {object} entityProto The protocol entity object to convert.
* @param {object} integerTypeCastOptions Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
* provided function to convert `integerValue`.
* @property {sting|string[]} [integerTypeCastOptions.names] `Entity` property
* names to be converted using `integerTypeCastFunction`.
* @returns {object}
*
* @example
Expand All @@ -525,15 +593,22 @@ export namespace entity {
* // }
*/
// tslint:disable-next-line no-any
export function entityFromEntityProto(entityProto: EntityProto): any {
export function entityFromEntityProto(
entityProto: EntityProto,
integerTypeCastOptions?: IntegerTypeCastOptions
) {
// tslint:disable-next-line no-any
const entityObject: any = {};
const properties = entityProto.properties || {};

// tslint:disable-next-line forin
for (const property in properties) {
const value = properties[property];
entityObject[property] = entity.decodeValueProto(value);
value.name = property;
entityObject[property] = entity.decodeValueProto(
value,
integerTypeCastOptions
);
}

return entityObject;
Expand Down Expand Up @@ -719,6 +794,11 @@ export namespace entity {
* @param {object[]} results The response array.
* @param {object} results.entity An entity object.
* @param {object} results.entity.key The entity's key.
* @param {object} integerTypeCastOptions Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
jkwlui marked this conversation as resolved.
Show resolved Hide resolved
* provided function to convert `integerValue`.
* @property {sting|string[]} [integerTypeCastOptions.names] `Entity` property
* names to be converted using `integerTypeCastFunction`.
* @returns {object[]}
*
* @example
Expand All @@ -733,9 +813,15 @@ export namespace entity {
* //
* });
*/
export function formatArray(results: ResponseResult[]) {
export function formatArray(
results: ResponseResult[],
integerTypeCastOptions?: IntegerTypeCastOptions
) {
return results.map(result => {
const ent = entity.entityFromEntityProto(result.entity!);
const ent = entity.entityFromEntityProto(
result.entity!,
integerTypeCastOptions
);
ent[entity.KEY_SYMBOL] = entity.keyFromKeyProto(result.entity!.key!);
return ent;
});
Expand Down Expand Up @@ -1225,6 +1311,7 @@ export interface ValueProto {
values?: ValueProto[];
// tslint:disable-next-line no-any
value?: any;
name?: string;
}

export interface EntityProto {
Expand Down
6 changes: 6 additions & 0 deletions src/query.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -517,9 +517,15 @@ export interface QueryProto {
*/
export {Query};

export interface IntegerTypeCastOptions {
integerTypeCastFunction: Function;
names?: string | string[];
}

export interface RunQueryOptions {
consistency?: 'strong' | 'eventual';
gaxOptions?: CallOptions;
integerTypeCastOptions?: IntegerTypeCastOptions;
}

export interface RunQueryCallback {
Expand Down
25 changes: 19 additions & 6 deletions src/request.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,10 @@ class DatastoreRequest {
return;
}

const entities = entity.formatArray(resp!.found! as ResponseResult[]);
const entities = entity.formatArray(
resp!.found! as ResponseResult[],
options.integerTypeCastOptions
);
const nextKeys = (resp!.deferred || [])
.map(entity.keyFromKeyProto)
.map(entity.keyToKeyProto);
Expand Down Expand Up @@ -430,6 +433,11 @@ class DatastoreRequest {
* [here](https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore).
* @param {object} [options.gaxOptions] Request configuration options, outlined
* here: https://googleapis.github.io/gax-nodejs/global.html#CallOptions.
* @param {object} [options.integerTypeCastOptions] Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
* provided function to convert `integerValue`.
* @property {sting|string[]} [integerTypeCastOptions.names] `Entity` property
* names to be converted using `integerTypeCastFunction`.
* @param {function} callback The callback function.
* @param {?error} callback.err An error returned while making this request
* @param {object|object[]} callback.entity The entity object(s) which match
Expand Down Expand Up @@ -684,6 +692,11 @@ class DatastoreRequest {
* @param {object} [options] Optional configuration.
* @param {object} [options.gaxOptions] Request configuration options, outlined
* here: https://googleapis.github.io/gax-nodejs/global.html#CallOptions.
* @param {object} [options.integerTypeCastOptions] Config for custom `integerValue` cast.
* @property {function} {integerTypeCastOptions.integerTypeCastFunction} A custom user
* provided function to convert `integerValue`.
* @property {sting|string[]} [integerTypeCastOptions.names] `Entity` property
* names to be converted using `integerTypeCastFunction`.
*
* @example
* datastore.runQueryStream(query)
Expand Down Expand Up @@ -762,7 +775,10 @@ class DatastoreRequest {
let entities: Entity[] = [];

if (resp.batch.entityResults) {
entities = entity.formatArray(resp.batch.entityResults);
entities = entity.formatArray(
resp.batch.entityResults,
options.integerTypeCastOptions
);
}

// Emit each result right away, then get the rest if necessary.
Expand Down Expand Up @@ -1340,10 +1356,7 @@ export interface AllocateIdsOptions {
allocations?: number;
gaxOptions?: CallOptions;
}
export interface CreateReadStreamOptions {
consistency?: string;
gaxOptions?: CallOptions;
}
export interface CreateReadStreamOptions extends RunQueryOptions {}
export interface GetCallback {
(err?: Error | null, entity?: Entities): void;
}
Expand Down
Loading