Skip to content

Commit

Permalink
Docs: update docs; doc --skip-render (#178)
Browse files Browse the repository at this point in the history 
* Update glsl-fuzz-walkthrough.md

* Remove unreferenced glsl-fuzz-explore.md

* Update glsl-fuzz-generate (manual)

* Document skip-render in more detail.

* Document --seed for glsl-generate.

* Document the fact that adb must be on your PATH.

* vkrun: use bundled binaries.

* Command line args: fix default seed value.

* Update glsl-fuzz-reduce.md
  • Loading branch information
@paulthomson
paulthomson committed Dec 19, 2018
1 parent df28465 commit 699e819
Show file tree
Hide file tree
Showing 17 changed files with 349 additions and 186 deletions.
3 changes: 2 additions & 1 deletion docs/android-notes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ you may need to open IntelliJ IDEA from the terminal depending on how
you set the environment variables.

You should also add the `android-sdk/platform-tools` directory
to your path so that you can use tools like `adb`.
to your path so that you can use `adb`.
The Vulkan worker requires that `adb` is on your path.

## Android networking: connecting Android apps to the server

Expand Down
29 changes: 0 additions & 29 deletions docs/glsl-fuzz-explore.md
View file

This file was deleted.

118 changes: 72 additions & 46 deletions docs/glsl-fuzz-generate.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,53 +33,79 @@ Each shader family will be output to a directory starting with `OUTPUT_FOLDER/PR
## Options

```shell
usage: glsl-generate.py [-h] [--webgl] [--keep_bad_variants] [--stop_on_fail]
[--verbose] [--small] [--avoid_long_loops]
[--max_bytes MAX_BYTES] [--max_factor MAX_FACTOR]
[--replace_float_literals] [--multi_pass]
[--require_license] [--generate_uniform_bindings]
[--max_uniforms MAX_UNIFORMS]
donors references num_variants glsl_version prefix
output_folder

Generate a set of shader families
usage: glsl-generate [-h] [--seed SEED] [--webgl] [--small]
[--allow-long-loops] [--disable DISABLE]
[--enable-only ENABLE_ONLY]
[--aggressively-complicate-control-flow]
[--single-pass] [--replace-float-literals]
[--generate-uniform-bindings]
[--max-uniforms MAX_UNIFORMS] [--no-injection-switch]
[--disable-validator] [--keep-bad-variants]
[--stop-on-fail] [--verbose] [--max-bytes MAX_BYTES]
[--max-factor MAX_FACTOR] [--require-license]
references donors num_variants glsl_version prefix
output_dir

Generate a set of shader families.

positional arguments:
donors Path to directory of donor shaders.
references Path to directory of reference shaders.
num_variants Number of variants to be produced for each shader in
the donor set.
glsl_version Version of GLSL to be used.
prefix String with which to prefix shader family name.
output_folder Output directory for shader families.
references Path to directory of reference shaders.
donors Path to directory of donor shaders.
num_variants Number of variants to be produced for each
generated shader family.
glsl_version Version of GLSL to target.
prefix String with which to prefix shader family names.
output_dir Output directory for shader families.

optional arguments:
-h, --help show this help message and exit
--webgl Restrict transformations to be WebGL-compatible.
--keep_bad_variants Do not remove invalid variants upon generation.
--stop_on_fail Quit if an invalid variant is generated.
--verbose Emit detailed information regarding the progress of
the generation.
--small Try to generate smaller variants.
--avoid_long_loops Avoid long-running loops during live code injection.
--max_bytes MAX_BYTES
Restrict maximum size of generated variants.
--max_factor MAX_FACTOR
Maximum blowup allowed, compared with size of
reference shader (default: no limit).
--replace_float_literals
Replace float literals with uniforms.
--multi_pass Apply multiple transformation passes.
--require_license Require that each shader has an accompanying license,
and use this during generation.
--generate_uniform_bindings
Put all uniforms in uniform blocks and generate
associated bindings. Necessary for Vulkan
compatibility.
--max_uniforms MAX_UNIFORMS
Ensure that no more than the given number of uniforms
are included in generated shaders. Necessary for
Vulkan compatibility.
-h, --help show this help message and exit
--seed SEED Seed to initialize random number generator with.
--webgl Restrict to WebGL-compatible features. (default:
false)
--small Try to generate small shaders. (default: false)
--allow-long-loops During live code injection, care is taken by
default to avoid loops with very long or infinite
iteration counts. This option disables this care
so that loops may end up being very long running.
(default: false)
--disable DISABLE Disable a given series of transformations.
--enable-only ENABLE_ONLY
Disable all but the given series of
transformations.
--aggressively-complicate-control-flow
Make control flow very complicated. (default:
false)
--single-pass Do not apply any individual transformation pass
more than once. (default: false)
--replace-float-literals
Replace float literals with uniforms. (default:
false)
--generate-uniform-bindings
Put all uniforms in uniform blocks and generate
bindings; required for Vulkan compatibility.
(default: false)
--max-uniforms MAX_UNIFORMS
Ensure that generated shaders have no more than
the given number of uniforms; required for Vulkan
compatibility. (default: 0)
--no-injection-switch Do not generate the injectionSwitch uniform.
(default: false)
--disable-validator Disable calling validation tools on generated
variants. (default: false)
--keep-bad-variants Do not remove invalid variants upon generation.
(default: false)
--stop-on-fail Quit if an invalid variant is generated.
(default: false)
--verbose Emit detailed information regarding the progress
of the generation. (default: false)
--max-bytes MAX_BYTES Maximum allowed size, in bytes, for variant
shader. (default: 500000)
--max-factor MAX_FACTOR
Maximum blowup allowed, compared with size of
reference shader (default: no limit).
--require-license Require a license file to be provided alongside
the reference and pass details through to
generated shaders. (default: false)

```
Expand All @@ -101,13 +127,13 @@ mkdir -p work/shaderfamilies
# glsl-generate [options] donors references num_variants glsl_version prefix output_folder

# Generate some GLSL version 300 es shaders.
glsl-generate --max_bytes 500000 --multi_pass samples/donors samples/300es 10 "300 es" family_300es work/shaderfamilies
glsl-generate --seed 0 samples/donors samples/300es 10 "300 es" family_300es work/shaderfamilies

# Generate some GLSL version 100 shaders.
glsl-generate --max_bytes 500000 --multi_pass samples/donors samples/100 10 "100" family_100 work/shaderfamilies
glsl-generate --seed 0 samples/donors samples/100 10 "100" family_100 work/shaderfamilies

# Generate some "Vulkan-compatible" GLSL version 300 es shaders that can be translated to SPIR-V for Vulkan testing.
glsl-generate --max_bytes 500000 --multi_pass --generate_uniform_bindings --max_uniforms 10 samples/donors samples/310es 10 "310 es" family_vulkan work/shaderfamilies
glsl-generate --seed 0 --generate_uniform_bindings --max_uniforms 10 samples/donors samples/310es 10 "310 es" family_vulkan work/shaderfamilies

# The lines above will take approx. 1-2 minutes each, and will generate a shader family for every
# shader in samples/300es or samples/100:
Expand Down
108 changes: 74 additions & 34 deletions docs/glsl-fuzz-reduce.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ See below for common examples.

glsl-reduce takes a *shader job* `SHADER_JOB` (a .json file) as an argument
as well as further arguments or options to specify the interestingness test.
glsl-reduce will try to simplify the given shader job to a smaller,
simpler shader job that is still deemed "interesting".
glsl-reduce will try to simplify the given shader to a smaller,
simpler shader that is still deemed "interesting".

`SHADER_JOB` should be a `.json` shader job file that represents all shaders and metadata needed to
render an image. If the shader job is named `foo.json`,
Expand All @@ -39,11 +39,24 @@ and the reference image will no longer be valid.
a shader job is interesting if it does *not* produce an
image and, optionally, the run log from running the shader job includes the
regular expression `ERROR`.
This reduction kind is typically used **without** `--preserve-semantics`,
* This reduction kind is typically used **without** `--preserve-semantics`,
so the reducer can change the semantics of the shader;
we are usually reducing a crash or incorrect shader compilation error,
so changing the semantics of the shaders is fine, as they should still compile and run.
Note that the reducer always produces valid shaders that should compile.
* This reduction kind is often used with `--skip-render`,
which causes remote workers to skip rendering the shader.
This can be much faster.
In cases
where the interesting error occurs during shader compilation,
there is no need to render the shader to check if the error still occurs
so `--skip-render` should be used.
However, if the error occurs while rendering the shader,
or as a result of rendering the shader,
`--skip-render` should **not** be used.

Note that unless `--preserve-semantics` is passed, changes made by the reducer can cause a shader to exhibit very high or infinite runtime. For example, the reducer might remove statements from a loop that prevent the loop from terminating.


Given a shader job `shader-job.json`,
the following are common reduction commands:
Expand All @@ -52,7 +65,15 @@ the following are common reduction commands:
# Wrong image reduction.
glsl-reduce shader-job.json --preserve-semantics --reduction-kind ABOVE_THRESHOLD --reference reference.info.json

# No image reduction:
# A "no image" reduction for a crash that occurs during shader compilation.
# --skip-render should be used to speed up the interestingness test,
# since the error occurs during shader compilation; successful shader
# compilation will be regarded as uninteresting.
glsl-reduce shader-job.json --reduction-kind NO_IMAGE --error-string "Fatal signal 11" --skip-render

# A "no image" reduction for a crash that occurs during rendering.
# --skip-render must NOT be used since the error occurs during rendering;
# we need to render to reproduce the crash.
glsl-reduce shader-job.json --reduction-kind NO_IMAGE --error-string "Fatal signal 11"

# Custom interestingness test:
Expand All @@ -70,8 +91,21 @@ by glsl-generate.
Options:

```
Reduce GLSL shaders, driven by a criterion of interest.
usage: glsl-reduce [-h] [--reduction-kind REDUCTION_KIND] [--output OUTPUT]
[--preserve-semantics] [--max-steps MAX_STEPS]
[--verbose] [--seed SEED] [--timeout TIMEOUT]
[--metric METRIC] [--reference REFERENCE]
[--threshold THRESHOLD] [--retry-limit RETRY_LIMIT]
[--skip-render] [--error-string ERROR_STRING]
[--server SERVER] [--worker WORKER] [--stop-on-error]
[--swiftshader] [--continue-previous-reduction]
shader_job [interestingness_test]
Reduce GLSL shaders, driven by a criterion of interest. The tool takes a
"shader job" as input, which is a json file (e.g. NAME.json) and a set of
files with the samename in the same directory. NAME.json is a metadata
file that can just contain "{}". There should also be some graphics
shaders (NAME.frag and/or NAME.vert) or a compute shader (NAME.comp).
positional arguments:
shader_job Path of shader job to be reduced. E.g.
Expand Down Expand Up @@ -104,48 +138,54 @@ optional arguments:
ALWAYS_REDUCE Always reduces (useful for
testing)
(default: CUSTOM)
--metric METRIC Metric to be used for image comparison. Options
are:
--output OUTPUT Directory to which reduction intermediate and
final results will be written. (default: .)
--preserve-semantics Only perform semantics-preserving reductions.
(default: false)
--max-steps MAX_STEPS The maximum number of reduction steps to take
before giving up and outputting the final reduced
file. (default: 2000)
--verbose Emit detailed information related to the
reduction process. (default: false)
--seed SEED Seed with which to initialize the random number
generator that is used to control reduction
decisions.
--timeout TIMEOUT Time in seconds after which checking
interestingness of a shader job is aborted.
(default: 30)
--metric METRIC Metric used for image comparison. Options are:
HISTOGRAM_CHISQR
PSNR
(default: HISTOGRAM_CHISQR)
--reference REFERENCE Path to reference .info.json result (with image
result) for comparison.
--threshold THRESHOLD Threshold used for histogram differencing.
(default: 100.0)
--timeout TIMEOUT Time in seconds after which execution of an
individual variant is terminated during
reduction. (default: 30)
--max-steps MAX_STEPS The maximum number of reduction steps to take
before giving up and outputting the final reduced
file. (default: 250)
--threshold THRESHOLD Threshold used for image comparison. (default:
100.0)
--retry-limit RETRY_LIMIT
When getting an image via the server, the number
of times the server should allow the client to
of times the server should allow the worker to
retry a shader before assuming the shader crashes
the client and marking it as SKIPPED. (default: 2)
--verbose Emit detailed information related to the
reduction process. (default: false)
--skip-render Don't render the shader on remote clients. Useful
when reducing compile or link errors. (default:
false)
--seed SEED Seed to initialize random number generator with.
(default: 1354327545)
the worker and marking it as SKIPPED. (default: 2)
--skip-render Don't render (just compile) the shader on remote
workers. Useful when reducing compile or link
errors. (default: false)
--error-string ERROR_STRING
String checked for containment in validation or
compilation tool error message.
--server SERVER Server URL to which image jobs are sent.
--token TOKEN Client token to which image jobs are sent. Used
with --server.
--output OUTPUT Output directory. (default: .)
--preserve-semantics Only perform semantics-preserving reductions.
(default: false)
For NO_IMAGE reductions, a shader is deemed
interesting if this string is found in the run
log. E.g. "Signal 11".
--server SERVER For non-CUSTOM reductions, shaders will be run
via a worker connected to this server URL.
--worker WORKER For non-CUSTOM reductions, shaders will be run on
this worker. Use with --server.
--stop-on-error Quit if something goes wrong during reduction;
useful for testing. (default: false)
--swiftshader Use swiftshader for rendering. (default: false)
--continue-previous-reduction
Carry on from where a previous reduction attempt
left off. (default: false)
left off. Requires the temporary files written
by the previous reduction to be intact, including
the presence of a REDUCTION_INCOMPLETE file.
(default: false)
```

Loading

0 comments on commit 699e819

Please sign in to comment.