✨ Generate dangerous strings by default #3043

dubzzz · 2022-06-24T18:30:48Z

Related to #484

Category:

Potential impacts:

Related to #484

codesandbox-ci · 2022-06-24T18:32:39Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 6052f5d:

Sandbox	Source
Vanilla	Configuration
@fast-check/examples	Configuration

packages/fast-check/src/arbitrary/_internals/ArrayArbitrary.ts

codecov · 2022-07-04T21:15:32Z

Codecov Report

Merging #3043 (6052f5d) into main (e815295) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #3043      +/-   ##
==========================================
+ Coverage   95.92%   95.97%   +0.05%     
==========================================
  Files         209      210       +1     
  Lines        5297     5348      +51     
  Branches     1025     1027       +2     
==========================================
+ Hits         5081     5133      +52     
+ Misses        216      215       -1

Flag	Coverage Δ
unit-tests	`95.97% <100.00%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...trary/_internals/helpers/SlicesForStringBuilder.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/asciiString.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/base64String.ts	`91.66% <100.00%> (+1.66%)`	⬆️
...ages/fast-check/src/arbitrary/fullUnicodeString.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/hexaString.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/string.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/string16bits.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/stringOf.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/unicodeString.ts	`100.00% <100.00%> (ø)`
packages/fast-check/src/arbitrary/double.ts	`97.14% <0.00%> (-2.86%)`	⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e815295...6052f5d. Read the comment docs.

packages/fast-check/src/arbitrary/_internals/helpers/SlicesForStringBuilder.ts

…StringBuilder.ts

dubzzz · 2022-07-06T07:05:00Z

Current iteration fails with:

FAIL test/unit/arbitrary/base64String.spec.ts
  ● base64String (integration) › should produce the same values given the same seed
    Property failed after 90 tests
    { seed: -637164509, path: "89:1:2", endOnFailure: true }
    Counterexample: [-13,2,Stream(0,18,6,2,9…),{"minLength":9,"maxLength":12}]
    Shrunk 2 time(s)
    Got error: Error: Cannot unmap non-string
    Stack trace: Error: Cannot unmap non-string
      at indexToCharStringUnmapper (src/arbitrary/_internals/mappers/IndexToCharString.ts:479:11)
      at MapArbitrary.unmapper (src/arbitrary/_internals/builders/CharacterArbitraryBuilder.ts:236:76)
      at MapArbitrary.shrink (src/check/arbitrary/definition/Arbitrary.ts:2918:39)
      at LazyIterableIterator.producer (src/arbitrary/_internals/ArrayArbitrary.ts:3645:25)
      at LazyIterableIterator.next (src/stream/LazyIterableIterator.ts:379:22)
      at joinHelper (src/stream/StreamHelpers.ts:976:33)
          at joinHelper.next (<anonymous>)
      at joinHelper (src/stream/StreamHelpers.ts:961:59)
          at joinHelper.next (<anonymous>)
      at joinHelper (src/stream/StreamHelpers.ts:961:59)
      Hint: Enable verbose mode in order to have the list of all failing values encountered during the run
      at throwIfFailed (lib/check/runner/utils/RunDetailsFormatter.js:128:11)
      at reportRunDetails (lib/check/runner/utils/RunDetailsFormatter.js:141:16)
      at Object.assert (lib/check/runner/Runner.js:71:52)
      at assertProduceSameValueGivenSameSeed (test/unit/arbitrary/__test-helpers__/ArbitraryAssertions.ts:15:8)
      at Object.<anonymous> (test/unit/arbitrary/base64String.spec.ts:113:71)

dubzzz · 2022-07-08T07:02:37Z

Here are some stats (based on a pimped versions of statistics making the call below equivalent to 10,000 executions with default 100 runs):

Legacy code snippet

declare const ds: string; // list of all the dangerous strings hardcoded within fast-check
fc.statistics(fc.string(), v => {
  return [
    ...(ds.includes(v) ? ['exact:' + v] : ['nothing']),
    ...ds.filter(d => v.includes(d)).map(d => 'partial:' + d),
  ]
}, { numRuns: 1_000_000 })

// WARNING: flags are not exclusive! One string can be both 'nothing' and 'partial:toString' at the same time.

Code snippet

declare const ds: string; // list of all the dangerous strings hardcoded within fast-check

fc.statistics(fc.string(), v => {
  return ds.includes(v) ? ['exact:' + v] : ['nothing'];
}, { numRuns: 1_000_000 })

fc.statistics(fc.string(), v => {
  const seenPartials = ds.filter(d => v.includes(d));
  return seenPartials.length !== 0 ? ['partial:' + seenPartials.join(',')] : ['nothing'];
}, { numRuns: 1_000_000 })

// alternative for large ones
fc.statistics(fc.string(), v => {
  const seenPartials = ds.filter(d => v.includes(d));
  return seenPartials.length !== 0 ? seenPartials.map(p => 'partial:' + p) : ['nothing'];
}, { numRuns: 1_000_000 })

Results

Default size

nothing..........97.70%
exact:ref.........0.35%
exact:key.........0.35%
exact:apply.......0.27%
exact:toString....0.26%
exact:valueOf.....0.26%
exact:caller......0.14%
exact:length......0.13%
exact:bind........0.09%
exact:__proto__...0.09%
exact:call........0.09%
exact:name........0.09%
exact:prototype...0.09%
exact:arguments...0.09%

---

nothing...................96.94%
partial:key................0.46%
partial:ref................0.45%
partial:apply..............0.33%
partial:valueOf............0.29%
partial:toString...........0.28%
partial:call...............0.21%
partial:length.............0.19%
partial:call,caller........0.18%
partial:name...............0.17%
partial:bind...............0.16%
partial:prototype..........0.11%
partial:__proto__..........0.10%
partial:arguments..........0.10%
partial:key,ref............0.00%
partial:call,ref...........0.00%
partial:bind,ref...........0.00%
partial:name,ref...........0.00%
partial:call,key...........0.00%
partial:bind,call..........0.00%
partial:bind,key...........0.00%
partial:apply,key..........0.00%
partial:call,name..........0.00%
partial:name,key...........0.00%
partial:bind,name..........0.00%
partial:apply,call.........0.00%
partial:apply,ref..........0.00%
partial:length,ref.........0.00%
partial:apply,bind.........0.00%
partial:call,caller,ref....0.00%
partial:length,key.........0.00%
partial:valueOf,key........0.00%
partial:call,caller,name...0.00%
partial:valueOf,ref........0.00%
partial:apply,name.........0.00%
partial:bind,length........0.00%
partial:call,length........0.00%
partial:bind,call,caller...0.00%
partial:call,caller,key....0.00%

---

nothing............96.93%
partial:key.........0.47%
partial:ref.........0.46%
partial:call........0.40%
partial:apply.......0.33%
partial:valueOf.....0.29%
partial:toString....0.29%
partial:length......0.18%
partial:name........0.18%
partial:caller......0.18%
partial:bind........0.18%
partial:arguments...0.10%
partial:prototype...0.10%
partial:__proto__...0.10%

large size

nothing.....................98.75%
exact:apply..................0.18%
exact:toString...............0.17%
exact:valueOf................0.17%
exact:length.................0.09%
exact:key....................0.09%
exact:caller.................0.09%
exact:ref....................0.09%
exact:call...................0.06%
exact:bind...................0.06%
exact:__proto__..............0.06%
exact:arguments..............0.06%
exact:name...................0.06%
exact:prototype..............0.06%
exact:propertyIsEnumerable...0.00%
exact:constructor............0.00%
exact:isPrototypeOf..........0.00%
exact:hasOwnProperty.........0.00%
exact:toLocaleString.........0.00%
exact:__defineSetter__.......0.00%
exact:__defineGetter__.......0.00%
exact:__lookupGetter__.......0.00%
exact:__lookupSetter__.......0.00%

---

nothing.......................90.67%
partial:call...................6.87%
partial:key....................5.82%
partial:ref....................5.80%
partial:apply..................5.64%
partial:name...................5.62%
partial:valueOf................5.60%
partial:toString...............5.60%
partial:bind...................5.60%
partial:caller.................5.55%
partial:length.................5.54%
partial:arguments..............5.47%
partial:__proto__..............5.46%
partial:prototype..............5.46%
partial:__lookupSetter__.......5.38%
partial:constructor............5.38%
partial:propertyIsEnumerable...5.37%
partial:isPrototypeOf..........5.37%
partial:__lookupGetter__.......5.37%
partial:toLocaleString.........5.36%
partial:__defineGetter__.......5.36%
partial:__defineSetter__.......5.36%
partial:hasOwnProperty.........5.36%

Same measurements with normal statistics ie. not emulating many runs but emulating a single long run:

Results

Default size

nothing..........99.81%
exact:apply.......0.03%
exact:toString....0.03%
exact:valueOf.....0.03%
exact:ref.........0.02%
exact:key.........0.02%
exact:caller......0.01%
exact:length......0.01%
exact:call........0.01%
exact:prototype...0.01%
exact:name........0.01%
exact:bind........0.01%
exact:arguments...0.01%
exact:__proto__...0.01%

---

nothing............99.65%
partial:call........0.06%
partial:ref.........0.05%
partial:key.........0.04%
partial:apply.......0.04%
partial:valueOf.....0.03%
partial:toString....0.03%
partial:bind........0.03%
partial:name........0.03%
partial:length......0.02%
partial:caller......0.02%
partial:arguments...0.01%
partial:__proto__...0.01%
partial:prototype...0.01%

large size

nothing.....................99.96%
exact:apply..................0.01%
exact:valueOf................0.00%
exact:toString...............0.00%
exact:bind...................0.00%
exact:key....................0.00%
exact:__proto__..............0.00%
exact:ref....................0.00%
exact:length.................0.00%
exact:caller.................0.00%
exact:name...................0.00%
exact:prototype..............0.00%
exact:arguments..............0.00%
exact:call...................0.00%
exact:propertyIsEnumerable...0.00%
exact:constructor............0.00%
exact:toLocaleString.........0.00%
exact:__lookupSetter__.......0.00%

---

nothing.......................98.05%
partial:call...................1.50%
partial:key....................1.26%
partial:ref....................1.26%
partial:name...................1.20%
partial:constructor............1.20%
partial:apply..................1.20%
partial:bind...................1.19%
partial:valueOf................1.19%
partial:length.................1.19%
partial:__lookupGetter__.......1.19%
partial:caller.................1.19%
partial:prototype..............1.19%
partial:arguments..............1.19%
partial:__proto__..............1.18%
partial:toString...............1.18%
partial:hasOwnProperty.........1.18%
partial:toLocaleString.........1.18%
partial:propertyIsEnumerable...1.18%
partial:__defineSetter__.......1.18%
partial:__defineGetter__.......1.17%
partial:__lookupSetter__.......1.17%
partial:isPrototypeOf..........1.17%

dubzzz · 2022-07-11T21:31:32Z

please deploy

✨ Generate dangerous strings by default

b39e5e9

Related to #484

Update snapshots accordingly

5b1ce28

dubzzz commented Jun 24, 2022

View reviewed changes

packages/fast-check/src/arbitrary/_internals/ArrayArbitrary.ts Outdated Show resolved Hide resolved

dubzzz added 8 commits June 24, 2022 20:57

Update packages/fast-check/src/arbitrary/_internals/ArrayArbitrary.ts

90e1ba2

Less keywords

4694e43

Do not always bias with dangerous

d0b9d00

Update snaps

3c31b71

Merge remote-tracking branch 'origin/main' into dangerous-strings

8683d41

Drop dead code

688e523

Rename prop usages to experimentalCustomSlices

643e61d

Bump versions

8339906

Rework createSlicesForString

4497388

dubzzz commented Jul 5, 2022

View reviewed changes

packages/fast-check/src/arbitrary/_internals/helpers/SlicesForStringBuilder.ts Outdated Show resolved Hide resolved

Update packages/fast-check/src/arbitrary/_internals/helpers/SlicesFor…

1162470

…StringBuilder.ts

dubzzz added 2 commits July 6, 2022 20:06

Merge branch 'main' into dangerous-strings

02f3093

Update snaps

231a692

dubzzz added 3 commits July 9, 2022 08:47

Merge branch 'main' into dangerous-strings

60fd0f9

Update examples

3d8f6d5

Update dictionary.spec.ts

6052f5d

dubzzz merged commit a6fd30c into main Jul 11, 2022

dubzzz deleted the dangerous-strings branch July 11, 2022 20:37

thewilkybarkid mentioned this pull request Jul 26, 2022

Don't throw a TypeError when trying to decode a sum value that contains a dangerous string gcanti/io-ts#650

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Generate dangerous strings by default #3043

✨ Generate dangerous strings by default #3043

dubzzz commented Jun 24, 2022

codesandbox-ci bot commented Jun 24, 2022 •

edited

Loading

codecov bot commented Jul 4, 2022 •

edited

Loading

dubzzz commented Jul 6, 2022

dubzzz commented Jul 8, 2022 •

edited

Loading

dubzzz commented Jul 11, 2022

✨ Generate dangerous strings by default #3043

✨ Generate dangerous strings by default #3043

Conversation

dubzzz commented Jun 24, 2022

codesandbox-ci bot commented Jun 24, 2022 • edited Loading

codecov bot commented Jul 4, 2022 • edited Loading

Codecov Report

dubzzz commented Jul 6, 2022

dubzzz commented Jul 8, 2022 • edited Loading

dubzzz commented Jul 11, 2022

codesandbox-ci bot commented Jun 24, 2022 •

edited

Loading

codecov bot commented Jul 4, 2022 •

edited

Loading

dubzzz commented Jul 8, 2022 •

edited

Loading