chore(deps): update dependency open-policy-agent/opa to v0.38.0

[renovate]

This PR contains the following updates:

Package	Update	Change
open-policy-agent/opa	minor	v0.24.0 -> v0.38.0

---

Release Notes

open-policy-agent/opa

v0.38.0

Compare Source

This release contains a number of fixes and enhancements.

It contains one backwards-incompatible change to the JSON representation
of metrics in Status API payloads, please see the section below.

Rich Metadata

It is now possible to annotate Rego policies in a way that can be
processed programmatically, using Rich Metadata.

METADATA

title: My rule

description: A rule that determines if x is allowed.

authors:

- Jane Austin jane@example.com

allow {
  ...
}

The available keys are:

• title
• description
• authors
• organizations
• related_resources
• schemas
• scope
• custom

Custom annotations can be used to annotate rules, packages, and
documents with whatever you specifically need, beyond the generic
keywords.

Annotations can be retrieved using the Golang library
or via the CLI, opa inspect -a.

All the details can be found in the documentation on Annotations.

Every Keyword

A new keyword for explicit iteration is added to Rego: every.

It comes in two forms, iterating values, or keys and values, of a
collection, and asserting that the body evaluates successfully for
each binding of key and value to the collection's elements:

every k, v in {"foo": "FOO", "bar": "BAR" } {
  upper(k) == v
}

To use it, import future.keywords.every or future.keywords.

For further information, please refer to the Every Keyword docs
and the new section on FOR SOME and FOR ALL in the Intro docs.

Tooling, SDK, and Runtime

• Compile API: add disableInlining option (#​4357) reported and fixed by @​srlk
• Status API: add http_code to response (#​4259) reported and fixed by @​jkbschmid
• Status plugin: publish experimental bundle-related metrics via prometheus endpoint (authored by @​rafaelreinert) -- See Status Metrics for details.
• SDK: don't panic without config (#​4303) authored by @​damienjburks
• Storage: Support index for array appends (for JSON Patch compatibility)
• opa deps: Fix pretty printed output to show virtual documents (#​4342)

Rego and Topdown

• Parser: parse 'with' on 'some x in xs' expression (#​4226)
• AST: hash containers on insert/update (#​4345), fixing a data race reported by @​skillcoder
• Planner: Fix bug related to undefined results in dynamic lookups

Documentation and Website

• Policy Reference: update EBNF to include "every" and "some x in ..." (#​4216)
• REST API: Update docs on 400 response
• README: Include Google Analytic Instructions
• Envoy primer: use variables instead of objects
• Istio tutorial: expose application to outside traffic
• New "Community" Webpage (authored by @​msorens)

WebAssembly

• OPA now uses Wasmtime 0.34.0 to evaluate its Wasm modules.

Miscellaneous

• Build: make build now builds without errors (by disabling Wasm) on darwin/arm64 (M1)
• Various dependency bumps.
  • OpenTelemetry SDK: 1.4.1
  • github.com/prometheus/client_golang: 1.12.1

Backwards incompatible changes

The JSON representation of the Status API's payloads -- both for GET /v1/status
responses and the metrics sent to a remote Status API endpoint -- have changed:

Previously, they had been serialized into JSON using the standard library "encoding/json"
methods. However, the metrics coming from the Prometheus integration are only available
in Golang structs generated from Protobuf definitions. For serializing these into JSON,
the standard library functions are unsuited:

• enums would be converted into numbers,
• field names would be snake_case, not camelCase,
• and NaNs would cause the encoder to panic.

Now, we're using the protobuf ecosystem's jsonpb package, to serialize the Prometheus
metrics into JSON in a way that is compliant with the Protobuf specification.

Concretely, what would before be

  "metrics": {
    "prometheus": {
      "go_gc_duration_seconds": {
        "help": "A summary of the GC invocation durations.",
        "metric": [
          {
            "summary": {
              "quantile": [
                {
                  "quantile": 0,
                  "value": 0.000011799
                },
                {
                  "quantile": 0.25,
                  "value": 0.000011905
                },
                {
                  "quantile": 0.5,
                  "value": 0.000040002
                },
                {
                  "quantile": 0.75,
                  "value": 0.000065238
                },
                {
                  "quantile": 1,
                  "value": 0.000104897
                }
              ],
              "sample_count": 7,
              "sample_sum": 0.000309117
            }
          }
        ],
        "name": "go_gc_duration_seconds",
        "type": 2
      },

is now:

  "metrics": {
    "prometheus": {
      "go_gc_duration_seconds": {
        "name": "go_gc_duration_seconds",
        "help": "A summary of the pause duration of garbage collection cycles.",
        "type": "SUMMARY",
        "metric": [
          {
            "summary": {
              "sampleCount": "1",
              "sampleSum": 4.1765e-05,
              "quantile": [
                {
                  "quantile": 0,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.25,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.5,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.75,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 1,
                  "value": 4.1765e-05
                }
              ]
            }
          }
        ]
      },

Note that sample_count is now sampleCount, and the type is using the enum's
string representation, "SUMMARY", not 2.

Note: For compatibility reasons (the Prometheus golang client doesn't use the V2
protobuf API), this change uses jsonpb and not protojson.

v0.37.2

Compare Source

This is a bugfix release addressing two bugs:

1. A regression introduced in the formatter fix for CVE-2022-23628.
2. Support indices for appending to an array, conforming to JSON Patch (RFC6902)
   for patch bundles.

Miscellaneous

• format: generated vars may have a proper location
• storage: Support index for array appends

v0.37.1

Compare Source

This is a bug fix release that reverts the github.com/prometheus/client_golang
upgrade in v0.37.0. The upgrade exposed an issue in the serialization of Go
runtime metrics in the Status API
(#​4319).

Miscellaneous

• Revert "build(deps): bump github.com/prometheus/client_golang (#​4307)"

v0.37.0

Compare Source

This release contains a number of fixes and enhancements.

This is the first release that includes a binary and a docker image for
linux/arm64, opa_linux_arm64_static and openpolicyagent/opa:0.37.0-static.
Thanks to @​ngraef for contributing the build changes necessary.

Strict Mode

There have been numerous possible checks in the compiler that fall into this category:

1. They would help avoid common mistakes; but
2. Introducing them would potentially break some uncommon, but legitimate use.

We've thus far refrained from introducing them. Now, a new "strict mode"
allows you to opt-in to these checks, and we encourage you to do so!

With OPA 1.0, they will become the new default behaviour.

For more details, see the docs on Compiler Strict Mode.

Delta Bundles

Delta bundles provide a more efficient way to make data changes by containing
patches to data instead of snapshots.
Using them together with HTTP Long Polling,
you can propagate small changes to bundles without waiting for polling delays.

See the documentation
for more details.

Tooling and Runtime

• Bundles bug fix: Roundtrip manifest before hashing to allow changing the manifest
  and still using signature verification of bundles (#​4233),
  reported by @​CristianJena

• The test runner now also supports custom builtins, when invoked through the Golang
  interface (authored by @​MIA-Deltat1995)

• The compile package and the opa build command support a new output format: "plan".
  It represents a query plan, steps needed to take to evaluate a query (with policies).
  The plan format is a JSON encoding of the intermediate representation (IR) used for
  compiling queries and policies into Wasm.

  When calling opa build -t plan ..., the plan can be found in plan.json at the top-
  level directory of the resulting bundle.tar.gz.
  See the documentation for details..

• Compiler+Bundles: Metadata to be added to a bundle's manifest can now be provided via WithMetadata
  (#​4289), authored by @​marensws, reported by @​johanneslarsson

• Plugins: failures in auth plugin resolution are now output, previously panicked, authored by @​jcchavezs

• Plugins: Fix error when initializing empty decision logging or status plugin (#​4291)

• Bundles: Persisted bundle activation failures are treated like failures with
  non-persisted bundles (#​3840), reported by @​dsoguet

• Server: http.send caching now works in system policy system.authz (#​3946),
  reported by @​amrap030.

• Runtime: Apply credentials masking on opa.runtime().config (#​4159)

• opa test: removing deprecated code for --show-failure-line (-l), authored by @​damienjburks

• opa eval: add description to all output formats

• opa inspect: unhide command for bundle inspection

Rego and Topdown

Built-in function enhancements and fixes:

• object.union_n: New built-in for creating the union of more than two objects (#​4012),
  reported by @​eliw00d
• graph.reachable_paths: New built-in to calculate the set of reachable paths in a graph (authored by @​justinlindh-wf)
• indexof_n: New built-in function to get all the indexes of a specific substring (or character) from a string (authored by @​shuheiktgw)
• indexof: Improved performance (authored by @​shuheiktgw)
• object.get: Support nested key array for deeper lookups with default (authored by @​charlieegan3)
• json.is_valid: Use Golang's json.Valid to avoid unnecessary allocations (authored by @​kristiansvalland)

Strict-mode features:

• Add duplicate imports check (#​2698) reported by @​mikol
• Deprecate any() and all() built-in functions (#​2437)
• Make input and data reserved keywords (#​2600) reported by @​jpeach
• Add unused local assignment check (#​2514)

Miscellaneous fixes and enhancements:

• format: don't group iterable when one has defaulted location
• topdown: ability to retrieve input and plug bindings in the Event, authored by @​istalker2
• print() built-in: fix bug when used with with modifier and a function call value (#​4227)
• ast: don't error when future keyword import is redundant during parsing

Documentation

• A new "CLI" docs section describes the various
  OPA CLI commands and their arguments (#​3915)
• Policy Testing: Add reference to rule indexing in the context of test code coverage
  (#​4170), reported by @​ekcs
• Management: Add hint that S3 regional endpoint should be used with bundles (authored by @​danoliver1)
• Many broken links were fixed, thanks to @​phelewski
• Fix rendering of details: add detail-tab for collapsable markdown (authored by @​bugg123)

WebAssembly

• Add native support for json.is_valid built-in function
  (#​4140), authored by @​kristiansvalland
• Dependencies: bump wasmtime-go from 0.32.0 to 0.33.1

Miscellaneous

• Publish multi-arch image manifest lists including linux/arm64 (#​2233),
  authored by @​ngraef, reported by @​povilasv
• logging: Remove logger GetFields function (#​4114),
  authored by @​viovanov
• Website: add versioned docs for latest version, so when 0.37.0 is released, both
  https://www.openpolicyagent.org/docs/v0.37.0/ and https://www.openpolicyagent.org/docs/latest
  contain docs, and 0.37.0 can already be used for stable links to versioned docs pages.
• Community: Initial draft of the community badges program
• make test: fix "too many open files" issue on Mac OS
• Various dependency bumps

v0.36.1

Compare Source

This release includes a number of documentation fixes.
It also includes the experimental binary for darwin/arm64.

There are no code changes.

Documentation

• OpenTelemetry: fix configuration example, authored by @​rvalkenaers
• Configuration: fix typo for tls-cert-refresh-period, authored by @​mattmahn
• SSH and Sudo authorization: Add missing filename
• Integration: fix example policy

Release

• Build darwin/arm64 in post tag workflow

v0.36.0

Compare Source

This release contains a number of fixes and enhancements.

OpenTelemetry and opa exec

This release adds OpenTelemetry support to OPA. This makes it possible to emit spans to an OpenTelemetry collector via gRPC on both incoming and outgoing (i.e. http.send) calls in the server. See the updated docs on monitoring for more information and configuration options (#​1469 authored by @​rvalkenaers)

This release also adds a new opa exec command for doing one-off evaluations of policy against input similar to opa eval, but using the full capabilities of the server (config file, plugins, etc). This is particularly useful in contexts such as CI/CD or when enforcing policy for infrastructure as code, where one might want to run OPA with remote bundles and decision logs but without having a running server. See the updated docs on Terraform for an example use case.
(#​3525)

Built-in Functions

• Four new functions for working with HMAC (crypto.hmac.md5, crypto.hmac.sha1, crypto.hmac.sha256, and crypto.hmac.sha512) was added (#​1740 reported by @​jshaw86)
• array.reverse(array) and strings.reverse(string) was added for reversing arrays and strings (#​3736 authored by @​kristiansvalland and @​olamiko)
• The http.send built-in function now uses a metric for counting inter-query cache hits (#​4023 authored by @​mirayadav)
• An overflow issue with dates very far in the future has been fixed in the time.* built-in functions (#​4098 reported by @​morgante)

Tooling

• A problem with future keyword import of in was fixed for opa fmt (#​4111, reported by @​keshavprasadms)
• An issue with opa fmt when refs contained operators was fixed (authored by @​jaspervdj-luminal)
• Fix file renaming check in optimization using opa build (authored by @​davidmarne-wf)
• The allow_net capability was added, allowing setting limits on what hosts can be reached in built-ins like http.send and net.lookup_ip_addr (#​3665)

Server

• A new credential provider for AWS credential files was added (#​2786 reported by @​rgueldem)
• The new --tls-cert-refresh-period flag can now be provided to opa run. If used with a positive duration, such as "5m" (5 minutes),
  "24h", etc, the server will track the certificate and key files' contents. When their content changes, the certificates will be
  reloaded (#​2500 reported by @​patoarvizu)
• A new v1/status endpoint was added, providing the same data as the status plugin would send to a remote endpoint (#​4089)
• The HTTP router of OPA is now exposed to the plugin manager (#​2777 authored by @​bhoriuchi reported by @​mneil)
• Calling print now works in decision masking policies
• An unintended switch between long/regular polling on 304 HTTP status was fixed (#​3923 authored by @​floriangasc)
• The error message about prohibited config in the discovery plugin has been improved
• The discovery plugin no longer panics in Trigger() if downloader is nil
• The bundle plugin now ignores service errors for file:// resources
• The bundle plugin file loader was updated to support directories
• A timer to HTTP request was added to the downloader
• The requested_by field in the logging plugin is now optional

Rego

• The error message raised when using - with a number and a set is now more specific (as opposed to the correct usage with two sets, or two numbers) (#​1643)

• Fixed an edge case when using print and arrays in unification (#​4078)

• Improved performance of some array operations by caching an array's groundness bit (#​3679)

• ⚠️ Stricter check of arity in undefined function stage (#​4054).
  This change will fail evaluation in some unusual cases where it previously would succeed, but these policies should be very uncommon.

  An example policy that previously would succeed but no longer will (wrong arity):

package policy

default p = false
p {
    x := is_blue()
    input.bar[x]
}

is_blue(fruit) = y { # doesn't use fruit
    y := input.foo
}

SDK

• The opa.runtime() built-in is now made available to the SDK (#​4050 authored by @​oren-zohar and @​cmschuetz)
• Plugins are now exposed on the SDK object
• The SDK now supports graceful shutdown (#​3980 reported by @​brianchhun-chime)
• print output is now sent to the configured logger

Website and Documentation

• All pages in the docs now have a feedback button (#​3664 authored by @​alan-ma)
• The Kafka docs have been updated to use the new Kafka plugin, and to use the OPA management APIs
• The Terraform tutorial was updated to use opa exec (#​3965)
• The docs on Contributing as well as the Vendor Guidelines have been updated
• The term "whitelist" has been replaced by "allowlist" across the docs
• A simple destructuring assignment example was added to the docs
• The docs have been reviewed on the use of assignment, equality and comparison operators, to make sure they follow best practice

CI

• SHA256 checksums of CI builds now published to release directory (#​3448 authored by @​johanneslarsson reported by @​raesene)
• golangci-lint upgraded to v1.43.0 (authored by @​shuheiktgw)
• The build now creates an executable for darwin/arm64. This should work as expected, but is currently tested in the CI pipeline like the other binaries
• PRs targeting the ecosystem page are now checked for mistakes using Rego policies

v0.35.0

Compare Source

This release contains a number of fixes and enhancements.

Early Exit Optimization

This release adds an early exit optimization to the evaluator. With this optimization, the evaluator stops evaluating rules when an answer has been found and subsequent evaluation would not yield any new answers. The optimization is automatically applied to complete rules and functions that meet specific requirements. For more information see the Early Exit in Rule Evaluation section in the docs. #​2092

Built-in Functions

• The net.lookup_ip_addr function was added to allow policies to resolve hostnames to IPv4/IPv6 addresses (#​3993)
• The http.send function has been improved to close TCP connections quickly after receiving the HTTP response and avoid creating HTTP clients unnecessarily when a cached response exists (#​4015). This change reduces the number of open file descriptors required in high-throughput environments and prevents OPA from encountering ulimit errors.

Rego

• print() calls in the head of rules no longer cause runtime errors (#​3967)
• Type errors for calls to undefined functions no longer contain rewritten variable names (#​4031)
• The rego.SkipPartialNamespace option now correctly sets the flag on the partial evaluation queries (previously it would always set the value to true) (#​3996) authored by @​thomascoquet
• The internal set implementation has been updated to insert elements in sorted order rather than lazily sorting during comparisons.
• Fixed import alias parsing bug identified by fuzzer (#​3988)

WebAssembly

• The Golang SDK will now issue a grow() call if the input document exceeds the available memory space.
• The malloc() implementation will now call opa_abort if the grow() call fails.

Server

• The decision logger adapts upload chunk sizes based on previous outputs. This allows the decision loggger to encode significantly more decisions into each upload chunk, thereby reducing heap usage for buffered decisions. For more information on the adapative chunking behaviour, see the Decision Logs page in the docs.
• The decision logger can be configured to send records to a custom plugin as well as an HTTP endpoint at the same time (#​4013)
• print() calls from the system.authz policy are now included in the logs (#​4048)
• OPA can use an Azure Managed Identities Token to authenticate with control plane services (#​3916) authored by @​Scowluga.
• The logging configuration will be correctly applied to service clients so that DEBUG logs are surfaced (#​4071)

Tooling

• The opa fmt command will not generate a line-break when there are generated variables in a function call (#​4018) reported by @​torsrex
• The opa inspect command no longer prints a blank namespace when a data.json file is included at the root (#​4022)
• The opa build command will output debug messages if an optimized entrypoint is discarded.

Website and Documentation

• The website has been updated to build with Hugo 0.88.1 (#​3787)
• The version picker in the documentation is now scrollable (#​3955) authored by @​orweis
• The description of the urlquery built-in functions have been clarified (#​1592) reported by @​klarose
• The decision logger documentation has been improved to cover controls for large-scale environments (#​3976)
• The "strict built-in errors" mode is now covered in the docs along with built-in function error behaviour (#​3686)
• The OAuth2 and OIDC examples around key rotation and caching have been improved

CI

• Issues and PRs that have not seen activity in 30 days will be automatically marked as "inactive"
• The Makefile can now produce Docker images for other architectures. We do not yet publish binaries or images for non-amd64 architectures however if you want to build OPA yourself, the Makefile does not prohibit it.

Backwards Compatibility

• The diagnostics buffer in the OPA server has been completely removed as part of the deprecation and removal of the diagnostic feature (#​1052)

v0.34.2

Compare Source

Fixes

• ast: Fix print call rewriting for calls in head (#​3967)

v0.34.1

Compare Source

Fixes

• runtime: Fix logging configuration (#​3959) (#​3958)

v0.34.0

Compare Source

This release includes a number of enhancements and fixes. In particular, this
release adds a new keyword for membership and iteration (in) and a specialized
built-in function (print) for debugging.

The in operator

This release adds a new in operator that provides syntactic sugar for
references that perform membership tests or iteration on collections (i.e.,
arrays, sets, and objects.) The following table shows common patterns for arrays
with the old and new syntax:

Pattern | Existing Syntax | New Syntax
--- | --- | ---
Check if 7 exists in array | 7 == arr[_] | 7 in arr
Check if 7 does not exist in array | n/a (requires helper rule) | not 7 in arr
Iterate over the elements of array | x := arr[_] | some x in arr

For more information on the in operator see Membership and iteration:
in
in the docs.

The print function

This release adds a new print function for debugging purposes. The print
function can be used to output any value inside of the policy. The print
function has special handling for undefined values so that execution does not
stop if any of the operands are undefined. Instead, a special marker is emitted
in the output. For example:

package example

default allow = false

allow {
  print("the subject's username is:", input.subject.username)
  input.subject.username == "admin"
}

Given the policy above, we can see the output of the print function via STDERR when using opa eval:

echo '{"subject": {"username": "admin"}}' | opa eval -d policy.rego -I -f pretty 'data.example.allow'

Output:

the subject's username is: admin
true

If the username, subject, or entire input document was undefined, the print function will still execute:

echo '{}' | opa eval -d policy.rego -I -f pretty 'data.example.allow'

Output:

the subject's username is: <undefined>
false

The print function is integrated into the opa subcommands, REPL, server, VS
Code extension, and the playground. Library users must opt-in to print
statements. For more information see the
Debugging
section in the docs.

Enhancements

• SDK: Allow map of plugins to be passed to SDK (#​3826) authored by @​edpaget
• opa test: Change exit status when tests are skipped (#​3773) authored by @​kirk-patton
• Bundles: Improve loading performance (#​3860) authored by @​0xAP
• opa fmt: Keep new lines in between function arguments (#​3836) reported by @​anbrsap
• opa inspect: Add experimental subcommand for bundle inspection (#​3754)

Fixes

• Bundles/API: When deleting a policy, the check determining if it's bundle-owned was using the path prefix, which would yield false positives under certain circumstances.
  It now checks the path properly, piece-by-piece. (#​3863 authored by @​edpaget

• CLI: Using --set with null value again translates to empty object (#​3846)

• Rego: Forbid dynamic recursion with hidden (system.*) document (#​3876

• Rego: Raise conflict errors in functions when output not captured (#​3912)

  This change has the potential to break policies that previously evaluated successfully!
  See Backwards Compatibility notes below for details.

• Experimental disk storage: React to "txn too big" errors (#​3879), reported and authored by @​floriangasc

Documentation

• Kubernetes and Istio: Update tutorials for recent Kubernetes versions (#​3910) authored by @​olamiko
• Deployment: Add section about Capabilities (#​3769)
• Built-in functions: Add warning to http.send and extension docs about side-effects in other systems (#​3922) (#​3893)
• Docker Authorization: The tutorial now uses a Bundles API server.
• SDK: An example of SDK use is provided.

Miscellaneous

• Runtime: Refactor logger usage -- see below for Backwards Compatibility notes.
• Wasm: fix an issue with undefined, plain input references (#​3891)
• test/e2e: Extend TestRuntime to avoid global fixture
• types: Fix Arity function to return zero when type is known (#​3932)
• Wasm/builder: bump LLVM to 13.0.0, latest versions of wabt and binaryen (#​3908)
• Wasm: deal with importing memory in the compiler (#​3763)

Backwards Compatibility

• Function return values need to be well-defined: for a single input x, the function's
  output f(x) can only be one value. When evaluating policies, this condition had not
  been ensured for function calls that don't make use of their values, like

  package p
  r {
      f(1)
  }
  f(_) = true
  f(_) = false

  Before, data.p.r evaluated to true. Now, it will (correctly) return an error:

  eval_conflict_error: functions must not produce multiple outputs for same inputs
  

  In more realistic settings, this can be encountered when true/false return values
  are captured and returned where they don't need to be:

  package p
  r {
      f("any", "baz")
  }
  f(path, _) = r {
      r := path == "any"
  }
  f(path, x) = r {
      r := glob.match(path, ["/"], x)
  }

  In this example, any function input containing "any" would make the function yield
  two different results:

  1. The first function body returns true, matching the "any" argument.
  2. The second function body returns the result of the glob.match call -- false.

  The fix here would be to not capture the return value in the function bodies:

  f(path, _) {
      path == "any"
  }
  f(path, x) {
      glob.match(path, ["/"], x)
  }

• The github.com/open-policy-agent/opa/runtime#NewLoggingHandler function now
  requires a logger instance. Requiring the logger avoids the need for the
  logging handler to depend on the global logrus logger (which is useful for
  test purposes.) This change is unlikely to affect users.

v0.33.1

Compare Source

This is a bugfix release addressing an issue in the formatting of rego code that contains
object literals. With the last release, those objects would under some conditions have their
keys re-ordered, with some of them put into a single line.

Thanks to @​iainmcgin for reporting.

Fixes

• format: make groupIterable sort by row (#​3849)

v0.33.0

Compare Source

This release includes a number of improvements and fixes.

Built-in Functions

This release introduces crypto.x509.parse_rsa_private_key so that policy authors can decode RSA private keys and structure them as JWKs (#​3765). Authored by @​cris-he.

Fixes

• Fix object comparison to avoid sorting keys in-place. This prevents the interpreter from generating non-deterministic results when values are inserted into the partial set memoization cache. (#​3819)
• Fix data races in ast package caused by sorting types.Any instances in-place and shallow-copying module comments when a deep-copy should be performed (#​3793). Reported by @​markushinz.
• Fix "file name too long" error caused by bundle loader treating PEM encoded private keys as file paths (#​3766)
• Fix plugins to support manual triggering mode when discovery is disabled (#​3797)

Server & Tooling

• The server now supports policy-based health checks that can inspect the state of plugins and other internal components (#​3759) authored by @​gshively11
• The bundle reader now loads files lazily to avoid hitting file descriptor limits (#​3777). Authored by @​bhoriuchi
• The opa eval sub-command supports a --timeout option for limiting how long evaluation can run.

Rego

• The type checker now supports variadic arguments on void functions. This change paves the way for print() support as well as variadic arguments on all functions.
• The parser now memoizes term parsing. This prevents non-linear runtime for large nested objects and sets.

CI & Dependencies

• Fix spurious build errors in wasm library.
• Update wasmtime dependency to v0.30.0.
• Run PR checks on macOS in addition to Linux (#​3176).

Documentation

• Update the Kubernetes and Envoy (standalone) tutorials to show how the OPA management APIs can be used to distribute policies.

Backwards Compatibility

• The github.com/open-policy-agent/opa/ast#ArgErrDetail struct has been
  modified to use the new types.FuncArgs struct to represent the required
  arguments. Callers that depend on the exact structure of the error details
  must update to use the types.FuncArgs struct.

v0.32.1

Compare Source

This is a bugfix release to address a problem related to mismatching checksums in the official go mod proxy.
As a consequence, users with code depending on the OPA Go module that bypassed the proxy would see an error like

go get github.com/google/flatbuffers/go: github.com/google/flatbuffers@v1.12.0: verifying module: checksum mismatch
    downloaded: h1:N8EguYFm2wwdpoNcpchQY0tPs85vOJkboFb2dPxmixo=
    sum.golang.org: h1:/PtAHvnBY4Kqnx/xCQ3OIV9uYcSFGScBsWI3Oogeh6w=

Be aware that Github's Dependabot feature makes use of that check, and will start to fail for projects using the OPA Go module version 0.32.0.

There workaround applied to OPA is to replace to flatbuffers dependency's version manually.

For more information, see

• https://github.com/google/flatbuffers/issues/6466: The issue has been discussed upstream, and a 1.12.1 release has been published to address it.
• https://github.com/dgraph-io/badger/pull/1746: OPA transitively depends on the flatbuffer package because of badger.

There are no functional changes in this bugfix release.
If you use the container images, or the published binaries, of OPA 0.32.0, you are not affected by this.

Many thanks to James Alseth for triaging this, and engaging with upstream to fix this.

v0.32.0

Compare Source

This release includes a number of improvements and fixes.

💾 Disk-based Storage (Experimental)

This release adds a disk-based storage implementation to OPA. The implementation can be found in github.com/open-policy-agent/storage/disk. There is also an example in the rego package that shows how policies can be evaluated with the disk-based store. The disk-based store is currently only available as a library (i.e., it is not integrated into the rest of OPA yet.) In the next few releases, we are planning to integrate the implementation into the OPA server and provide tooling to help leverage the disk-based store.

Built-in Functions

This release includes a few improvements to existing built-in functions:

• The http.send function now supports UNIX domain sockets (#​3661) authored by @​kirk-patton
• The units.parse_bytes function now supports E* and P* units (#​2911)
• The io.jwt.encode_sign function uses the built-in context randomization source (which is helpful for replay purposes)

Server

This release includes multiple improvements for OPA server deployments in serverless environments:

• Plugins can now be triggered manually within OPA. This feature allows users extending and customizing OPA to control exactly when operations like bundle downloads and decision log uploads occur. The built-in plugins now include a trigger configuration that can be set to manual or periodic (which is the default). When manual triggering is enabled, the plugins WILL NOT perform any periodic/background operations. Instead, the plugins will only execute when the Trigger API is invoked.
• Plugins can now wait for server initialization. When runtime initialization is finished, plugins can be notified. This allows plugins to synchronize their behaviour with server startup. #​3701 authored by @​gshively11.
• The Health API now supports an exclude-plugin parameter to control which plugins are checked. #​3713 authored by @​gshively11.

Tooling

• The compiler no longer fetches remote schemas by default when used as as library. Capabilities have been updated to include an allow_net field to control whether network operations can be performed (#​3746). This field is only used to control schema fetching today. In future versions of OPA, the allow_net parameter will be used to control other behaviour like http.send.
• The WebAssembly runtime not supported error message has been improved #​3739.

Rego

• Added support for anyOf and allOf keywords in JSON schema support in the type checker (#​3592) authored by @​jchen10500 and @​juliafriedman8.
• Added support for custom JSON result marshalling in the rego package.
• Added a new convenience function (Allowed() bool) to the rego.ResultSet API.
• Improved string-representation construction performance for arrays, sets, and objects.
• Improved the topdown evaluator to support ast.Value results from the store so that unnecessary conversions can be avoided.
• Improved the rego package to make the wasmtime-go dependency optional at build-time (#​3545).
• Fixed a bug in the comprehension indexer whereby index keys were not constructed correctly leading to incorrect outputs (#​3579).
• Fixed a stack overflow during partial evaluation due to incorrect term rewriting in the copy propagation implementation (#​3071).
• Fixed a bug in partial evaluation when shallow inlinign is enabled that resulted in built-in functions being invoked instead of saved (#​3681).

WebAssembly

• The internal Wasm SDK now supports the inter-query built-in cache.
• The pre-compiled runtime is now built with llvm 12.0.1 and the builder image includes clang-format.
• The internal Wasm SDK has been updated to use wasmtime-go v0.29.0.

Documentation

This release includes a number of documentation improvements:

• The wasm opa_eval arguments have been clarified #​3699
• The contributing and development guide have been moved into a dedicated Contributing section on the website #​3751
• The Envoy standalone tutorial includes cleanup steps now (thanks @​princespaghetti)
• Various typos have been fixed by multiple folks (thanks @​Tej-Singh-Rana @​gujun4990)
• The Kubernetes ingress validation tutorial has been updated to include new mandatory attributes and newer API versions (thanks @​ereslibre)
• The recommendations around using OPA Gatekeeper have been improved.

Infrastructure

• OPA is now built with Go v1.17 and CI jobs have been added to ensure OPA builds with older versions of Go.

Backwards Compatibility

The rego package no longer relies on build constraints to enable the Wasm runtime. Instead, library users must opt-in to Wasm runtime support by adding an import statement in the Go code:

import _ "github.com/open-policy-agent/opa/features/wasm"

This change ensures that (by default) the wasmtime-go blobs are not vendored in projects that embed OPA as a library. If you are currently relying on the Wasm runtime support in the rego package (via the rego.Target("wasm") option), please update you code to include the import above. See #​3545 for more details.

v0.31.0

Compare Source

This release contains performance improvements for evaluating partial sets and objects,
and introduces a new ABI call to OPA's Wasm modules to speed up Wasm evaluations.

It also comes with an improvement for checking policies -- unsafe declared variables are now caught at compile time.
This means that some policies that have been working fine with previous versions, because their unsafe variables
had not ever been queried, will fail to compile with OPA 0.31.0.
See below for details and what to do about that.

Spotlights

Partial Sets and Objects Performance

Resolving an issue (#​822) created on July 4th 2018,
OPA can now cache the results of partial sets and partial objects.

A benchmark that accesses a partial set of increasing size twice shows a saving of more than 50%:

name                             old time/op    new time/op    delta
PartialRuleSetIteration/10-16       230µs ±10%     101µs ± 3%  -56.10%  (p=0.000 n=10+10)
PartialRuleSetIteration/100-16     13.4ms ± 9%     5.5ms ± 9%  -58.74%  (p=0.000 n=10+9)
PartialRuleSetIteration/1000-16     1.31s ±10%     0.51s ± 8%  -61.12%  (p=0.000 n=10+9)

name                             old alloc/op   new alloc/op   delta
PartialRuleSetIteration/10-16      77.7kB ± 0%    35.8kB ± 0%  -53.94%  (p=0.000 n=10+10)
PartialRuleSetIteration/100-16     3.72MB ± 0%    1.29MB ± 0%  -65.26%  (p=0.000 n=10+10)
PartialRuleSetIteration/1000-16     365MB ± 0%     114MB ± 0%  -68.86%  (p=0.000 n=10+10)

name                             old allocs/op  new allocs/op  delta
PartialRuleSetIteration/10-16       1.84k ± 0%     0.69k ± 0%  -62.42%  (p=0.000 n=10+10)
PartialRuleSetIteration/100-16      99.3k ± 0%     14.5k ± 0%  -85.43%  (p=0.000 n=10+9)
PartialRuleSetIteration/1000-16     10.0M ± 0%      1.0M ± 0%  -89.58%  (p=0.000 n=10+9)

These numbers were gathered querying fixture[i]; fixture[j] with a policy of

fixture[x] {
	x := numbers.range(1, n)[_]
}

where n is 10, 100, or 1000.

There are multiple access patterns that are accounted for: if a ground scalar is used to
access a previously not-cached partial rule,

allow {
	managers[input.user] # here
}

managers[x] {
### some logic here
}

the evaluation algorithm will calculate the set membership of input.user only, and cache the result.

If there is a query that requires evaluating the entire partial, however, the algorithm will also cache the entire partial:

allow {
	some person
	managers[person]
### more expressions
}

managers[x] {
### some logic here
}

thus avoiding extra evaluations later on.
The same is true if managers was used as a fully materialized set in an execution.

This also means that the question about whether to write

q = { x | ... } # set comprehension

or

q[x] { ... } # partial set rule

becomes much less important for policy evaluation performance.

WebAssembly Performance

OPA-generated Wasm modules have gotten a fast-path evaluation method:
By calling the one-off function

opa_eval(reserved, entrypoint, data_addr, input_addr, input_len, format)

which returns a pointer to the serialized result set (in JSON if format is 0, "value" format if 1),
the number of VM calls needed for evaluating a policy via Wasm is drastically reduced.

The performance benefit is huge:

name         old time/op    new time/op    delta
WasmRego-16    84.3µs ± 6%    15.1µs ± 0%  -82.07%  (p=0.008 n=5+5)

The added opa_eval export comes with an ABI bump to version 1.2.
See #​3627 for all details.

Along the same line, we've examined the processing of query evaluations that are Wasm-backed through the rego package.
This allowed us to avoid unneccessary work (#​3666).

Unsafe declared variables now cause a compile-time error

Before this release, local variables that had been declared, i.e. introduced via the some keyword, had been able
to slip through the safety checks unnoticed.

For example, a policy like

package demo

q { 
	input == "open sesame"
}

p[x] {
	some x
}

would have not caused any error if data.demo.p wasn't queried.
Querying data.demo.p would return an "var requires evaluation" error.

With this release, the erroneous rule no longer goes unnoticed, but is caught at compile time: "var x is unsafe".

The most likely fix is to remove the rule with the unsafe variable, since it cannot have contributed to a successful
evaluation in previous OPA versions.

See #​3580 for details.

Topdown and Rego

• New built-in function: crypto.x509.parse_and_verify_certificates (#​3601), authored by @​jalseth

  This function enables you to verify that there is a chain from a leaf certificate back to the trusted root.

• New built-in function: rand.intn generates a random number between 0 and n (#​3615), authored by @​base698

  The function takes a string argument to ensure that the same call, within one policy evaluation, returns the same random number.

• http.send enhancement: New caching_mode parameter to configure if deserialized or serialized response bodies should be cached (#​3599)

• Custom built-in function enhancement: let custom builtins halt evaluation (#​3534)

• Partial evaluation: Fix stack overflow on certain expressions (#​3559)

Tooling

• Query Profiling: opa eval --profile now supports a --count=# flag to gather metrics and profiling data over multiple runs, and displays aggregate statistics for the results (#​3651).

  This allows you to gather more robust numbers to assess policy performance.

• Docker images: Publish static image (#​3633)

  As of this release, you can use the staticly-built Linux binary from a docker image: openpolicyagent/opa:0.31.0-static.
  It contains the same binary that has been pu

---

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by WhiteSource Renovate. View repository job log here.