-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Allow named layers and some Misc Fixes #44
Conversation
Codecov Report
@@ Coverage Diff @@
## main #44 +/- ##
==========================================
+ Coverage 45.07% 45.13% +0.05%
==========================================
Files 13 13
Lines 945 955 +10
==========================================
+ Hits 426 431 +5
- Misses 519 524 +5
Continue to review full report at Codecov.
|
Closing this since, most features have been pulled into different PRs (already in main). We still need to work on named layers for container types. |
…ns/create-pull-request-6 Bump peter-evans/create-pull-request from 5 to 6
Change the Environment Variables
Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.2 to 1.23.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.2...v1.23.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* perf: cleanup the benchmarking script * perf: add benchmarks for Zygote * perf: try reclaiming memory * fix: incorrect system parameters * perf: temporarily disable non-dense benchmarks [skip tests] * ci(benchmark): allow proceed on failure [skip tests] * perf: update polyalg selection for matmul and matmuladd * test: ensure no additional allocations for matmul * fix: typo in AMDGPU batched matmul * perf: restore running all benchmarks * docs: add link to benchmarks * ci: fix benchmarks config * test: run allocs test only on CPU * fix: mixed-precision use Octavian if possible * feat: add traits to fuse activation functions [skip ci] * perf: selective vectorization of operations bias_add/activation * perf: fused bias activation for certain operations * perf: optimize batchnorm implementation * perf: don't fuse tanh * perf: run specific benchmarks * perf: be conservative while fusing activation functions * refactor: qualify CPU functions with `_cpu` * perf: restore running all benchmarks * fix(tracker): expand custom Tracker AD for wrapper types * fix: subtyping correction * test: ignore tests for batched_vec (not our code) * perf: faster version of groupnorm * ci: run downstream testing only on pull requests * refactor: remove unnecessary forced inlining * refactor: move PartialFunctions into a module * refactor: move utilities into Utils * refactor: move device agnostic functions to `DeviceAgnostic` * test: separate out the testing project file * refactor: move internal functions into separate modules * test: separate out the testing project file * fix: incorrect internal calls * refactor: remove unnecessary turbo loop * perf: don't rely on compile time branch removal for KA * perf: static ndrange kernel launches * perf: let it autotune * refactor: use multiple dispatch for cleaner kernels * refactor: disable cpu codegen for kernels * fix: nicer information for fallback mixed-precision matmul * fix: allow zero-sized arrays in bias_activation * fix: don't restrict bias_act to number * fix: don't restrict traits/ext/utils to number * fix: more aggressive type specialization * chore: update version * fix: broken qa tests * fix: use `fmap_with_path` to correctly identify all internal states * chore: apply formatting suggestion Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: don't error on detecting arrays with undefined entries * refactor: move ChainRulesCore into an extension * fix: skip enzyme tests if it is a pre-release * chore: bump version for release * fix: decide internal operation based on unwrapped arrays * fix: avoid wrappers for SVector using `insert_batch_dim` * fix: enzyme forward mode with octavian * feat: swap Enzyme forward rules along with reverse * test: simple enzyme forward test to check no crash * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add `unsafe_free!` * feat: add DeviceIterator (and support parallel Device DataLoader) * test: basic tests for free-ing data * refactor: simplify parallel dataloader * test: DataLoader aggressive freeing * docs: add docstrings for `DeviceIterator` * refactor: deprecate "Explicit" in favor of "Lux" * chore: add deprecation for the single arg outputsize * fix: remove old uses of Explicit * fix!: remove deprecations * chore: add exports for abstract layers * refactor: move Functors and Setfield into ext * fix!: remove hacky version of outputsize * feat: add `AbstractLuxWrapperLayer` * refactor: cleanup extension usage * test: update test to new API * test: extension loading errors * feat: support functors for WrappedLayer * test: LuxWrappedLayer tested * test: don't qualify unnecessarily * refactor: cleanup internal functions * fix!: remove default slow handling of outputsize * fix: update removed API * test: update old tests * fix!: remove unused `inputsize` * fix: add fmap_with_path support * chore: fix formatting * feat: default call for wrapper layers * fix: remove hacky usage of module getproperty rrules * fix: accidental dual usage of `ofeltype_array` * feat: auto-training mode and strict checks * chore: bump compat for LuxCore to 1, (keep existing compat) (#147) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * feat: extend the layernorm API * test: more detailed layernorm testing * chore: bump version for release * fix!: remove deprecations for 1.0 release * chore!: remove Reexport of NNlib (will be done via Lux) * perf: add NNlib to benchmarks deps * fix: remove unused explicit imports * chore: update to using LuxCore@1.0 * fix!: remove dropout branching based on size * fix!: change the default layernorm dims * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add enzyme reverse rules for `fused_dense!` * test: add tests for the enzyme fused_dense rules * fix: typo in reverse rule * test: run tests with more activations * feat: instancenorm with running statistics * fix: fixes for testing * fix: modify the dropout testing * fix: windows testing for dropout * chore(deps): bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 (#19) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * test: add tests comparing the fused op with unfused op * fix: improve load times by moving CRC to ext * fix: remove UnrolledUtilities dep * fix: remove UnrolledUtilities dep * chore: bump minimum MLDataDevices version * fix: dropout tests are no longer broken * chore: accidentally left deprecations file * fix: missing enzyme rules for matmuladd! (CUDA support) * test: incorrect condition * test: incorrect function name * fix: zero out shadows * fix: enzyme reverse bias needs a check on Const * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * feat: better test integration in test_gradients * feat: add test_gradients macro * chore: apply formatting suggestion Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: update to use test_gradients macro * fix: bias needs to add accum gradients * chore: bump `EnzymeCore` version * CompatHelper: bump compat for EnzymeCore in [weakdeps] to 0.8, (keep existing compat) * chore: bump version for release --------- Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Avik Pal <avikpal@mit.edu> * chore: install latest enzyme version * chore: update Enzyme version * chore: bump minimum versions * ci: update buildkite settings * feat: wider support for batched_matmul * perf: benchmark fallback batched_matmul * feat: slow fallback conv impl * feat: parallel fallback batchedmm * ci(buildkite): add GPU testing for Metal and oneAPI * test: check for FP64 support * fix: convert element type before broadcasting * fix: dispatch for NNlib conv * ci(buildkite): disable testing for Metal and oneAPI * chore: bump version * feat: update minimum version of Enzyme to 0.13 * feat: support within_gradient for Enzyme * refactor: rename within_gradient to within_autodiff * fix: update forward rules to new API * fix: use known on the return type * fix: forward enzyme rules * fix: broken enzyme tests * feat: support runtime activity for enzyme * fix: check was accidentally broken * chore(deps): bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix: rollback custom gelu implementation * feat: XLADevice via Reactant * chore: apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: bump version * feat: more extensive testing of XLA backend * fix: incorrect function call * test: rename * test: incorrect env var * fix: copy to XLA in main thread * fix: don't support pre-moving the data * fix: urgent patch for reactant breakage * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.24.6 to 1.25.0 (#41) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.26.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * ci: run on `1.10` and `1` (#57) * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` (#81) * ci: run on 1.10 and 1 * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` (#43) * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * test: mark truncated normal on Metal as unbroken * ci: run buildkite on `1.10` and `1` * chore: bump peter-evans/create-pull-request from 6 to 7 (#40) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: run tests only on `1.10` for now (#172) * fix: relax cublaslt types (#173) * docs: add Flux.jl to the README (#83) After FluxML/Flux.jl#2492 also Flux relies on MLDataDevices. * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#58) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#44) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#174) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump compat for GPUArrays in [weakdeps] to 11, (keep existing compat) (#86) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * chore: bump version for release * chore: bump compat for GPUArrays in [weakdeps] to 11, (keep existing compat) (#46) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * chore: bump compat for GPUArraysCore to 0.2, (keep existing compat) (#47) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Avik Pal <avikpal@mit.edu> * chore: bump version for release * feat: add fallbacks for unknown objects (#87) * feat: add fallbacks for unknown objects * feat: handle RNGs and undef arrays gracefully * test: RNG movement * test: functions and closures * refactor: move `JuliaSIMD` deps to extensions (#175) * fix: remove LV.vmap! usage * fix: remove LV handling for bias_activation * fix: remove LV usage in dropout * refactor: move LV and octavian behind an extension * docs: add docs for loading packages * refactor: move SLEEFPirates to an ext * fix: enzyme rules for batched matmul * fix: patch more enzyme issues * feat: add a preference to disable loop vectorization * fix: incorrect dispatch called * fix: enzyme segfault bypass * feat: define isleaf (#84) * isleaf * exclude * add tests and docs * more tests * import functors * fix test * chore: reduce min compat * chore: run formatter * chore: bump version for release * fix: handle bitstypes and wrapped arrays in isleaf (#88) * bitstype and wrapped arrays * fixes * fix import * bound * cleanup * chore: fix min version of LinearAlgebra * chore: run formatter --------- Co-authored-by: Avik Pal <avik.pal.2017@gmail.com> Co-authored-by: Avik Pal <avikpal@mit.edu> * fix: task switching in AMDGPU complex batched_matmul (#178) * ci(buildkite): add downstream testing for NeuralOperators * perf: restore old batched_mul * fix: disable threading for certain devices * revert: "perf: restore old batched_mul" This reverts commit a8c0f3b4615f96a8773577e16fac61ba310d8123. * fix: correctly handle adjoints of wrapped arrays (#90) * fix: correctly handle adjoints of wrapped arrays * fix: use fast paths for adapt * fix: adapt ranges to JuliaGPU/Adapt.jl#86 * chore(deps): bump crate-ci/typos from 1.25.0 to 1.26.8 (#44) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#49) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#60) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: missing import; fixes #179 (#180) * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#93) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: merge LuxCUDA testing scripts * ci: merge LuxCore testing scripts * ci: merge WeightInitializers testing scripts * ci: add WI to pipeline launch * ci: add MLDataDevices to pipeline launch * ci: change 1.10 to "lts" * test: LuxCore test fixes * ci: soft fail MLDataDevices * ci: add a central downstream testing * ci: partially migrate LuxLib CI * ci: remove name field * ci: minor fixes to build scripts * ci: move LuxTestUtils CI scripts * ci: update LuxLib workflow * ci: update LuxLib workflows * ci: split out downstream testing * ci: fix certain pipelines * ci: minor tweaks * fix: workflows * test: use local LuxCUDA for tests * fix: use develop * docs: update * fix: add dev packages * docs: dev required packages * perf: merge the benchmarks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Christopher Rackauckas <accounts@chrisrackauckas.com> Co-authored-by: Carlo Lucibello <carlo.lucibello@gmail.com>
* perf: cleanup the benchmarking script * perf: add benchmarks for Zygote * perf: try reclaiming memory * fix: incorrect system parameters * perf: temporarily disable non-dense benchmarks [skip tests] * ci(benchmark): allow proceed on failure [skip tests] * perf: update polyalg selection for matmul and matmuladd * test: ensure no additional allocations for matmul * fix: typo in AMDGPU batched matmul * perf: restore running all benchmarks * docs: add link to benchmarks * ci: fix benchmarks config * test: run allocs test only on CPU * fix: mixed-precision use Octavian if possible * feat: add traits to fuse activation functions [skip ci] * perf: selective vectorization of operations bias_add/activation * perf: fused bias activation for certain operations * perf: optimize batchnorm implementation * perf: don't fuse tanh * perf: run specific benchmarks * perf: be conservative while fusing activation functions * refactor: qualify CPU functions with `_cpu` * perf: restore running all benchmarks * fix(tracker): expand custom Tracker AD for wrapper types * fix: subtyping correction * test: ignore tests for batched_vec (not our code) * perf: faster version of groupnorm * ci: run downstream testing only on pull requests * refactor: remove unnecessary forced inlining * refactor: move PartialFunctions into a module * refactor: move utilities into Utils * refactor: move device agnostic functions to `DeviceAgnostic` * test: separate out the testing project file * refactor: move internal functions into separate modules * test: separate out the testing project file * fix: incorrect internal calls * refactor: remove unnecessary turbo loop * perf: don't rely on compile time branch removal for KA * perf: static ndrange kernel launches * perf: let it autotune * refactor: use multiple dispatch for cleaner kernels * refactor: disable cpu codegen for kernels * fix: nicer information for fallback mixed-precision matmul * fix: allow zero-sized arrays in bias_activation * fix: don't restrict bias_act to number * fix: don't restrict traits/ext/utils to number * fix: more aggressive type specialization * chore: update version * fix: broken qa tests * fix: use `fmap_with_path` to correctly identify all internal states * chore: apply formatting suggestion Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: don't error on detecting arrays with undefined entries * refactor: move ChainRulesCore into an extension * fix: skip enzyme tests if it is a pre-release * chore: bump version for release * fix: decide internal operation based on unwrapped arrays * fix: avoid wrappers for SVector using `insert_batch_dim` * fix: enzyme forward mode with octavian * feat: swap Enzyme forward rules along with reverse * test: simple enzyme forward test to check no crash * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.23.6 to 1.24.1 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.23.6 to 1.24.1. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.23.6...v1.24.1) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add `unsafe_free!` * feat: add DeviceIterator (and support parallel Device DataLoader) * test: basic tests for free-ing data * refactor: simplify parallel dataloader * test: DataLoader aggressive freeing * docs: add docstrings for `DeviceIterator` * refactor: deprecate "Explicit" in favor of "Lux" * chore: add deprecation for the single arg outputsize * fix: remove old uses of Explicit * fix!: remove deprecations * chore: add exports for abstract layers * refactor: move Functors and Setfield into ext * fix!: remove hacky version of outputsize * feat: add `AbstractLuxWrapperLayer` * refactor: cleanup extension usage * test: update test to new API * test: extension loading errors * feat: support functors for WrappedLayer * test: LuxWrappedLayer tested * test: don't qualify unnecessarily * refactor: cleanup internal functions * fix!: remove default slow handling of outputsize * fix: update removed API * test: update old tests * fix!: remove unused `inputsize` * fix: add fmap_with_path support * chore: fix formatting * feat: default call for wrapper layers * fix: remove hacky usage of module getproperty rrules * fix: accidental dual usage of `ofeltype_array` * feat: auto-training mode and strict checks * chore: bump compat for LuxCore to 1, (keep existing compat) (#147) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * feat: extend the layernorm API * test: more detailed layernorm testing * chore: bump version for release * fix!: remove deprecations for 1.0 release * chore!: remove Reexport of NNlib (will be done via Lux) * perf: add NNlib to benchmarks deps * fix: remove unused explicit imports * chore: update to using LuxCore@1.0 * fix!: remove dropout branching based on size * fix!: change the default layernorm dims * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.1 to 1.24.3 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.1 to 1.24.3. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.1...v1.24.3) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add enzyme reverse rules for `fused_dense!` * test: add tests for the enzyme fused_dense rules * fix: typo in reverse rule * test: run tests with more activations * feat: instancenorm with running statistics * fix: fixes for testing * fix: modify the dropout testing * fix: windows testing for dropout * chore(deps): bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 (#19) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump peter-evans/create-pull-request from 6 to 7 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.5 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.5) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * test: add tests comparing the fused op with unfused op * fix: improve load times by moving CRC to ext * fix: remove UnrolledUtilities dep * fix: remove UnrolledUtilities dep * chore: bump minimum MLDataDevices version * fix: dropout tests are no longer broken * chore: accidentally left deprecations file * fix: missing enzyme rules for matmuladd! (CUDA support) * test: incorrect condition * test: incorrect function name * fix: zero out shadows * fix: enzyme reverse bias needs a check on Const * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * feat: better test integration in test_gradients * feat: add test_gradients macro * chore: apply formatting suggestion Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: update to use test_gradients macro * fix: bias needs to add accum gradients * chore: bump `EnzymeCore` version * CompatHelper: bump compat for EnzymeCore in [weakdeps] to 0.8, (keep existing compat) * chore: bump version for release --------- Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Avik Pal <avikpal@mit.edu> * chore: install latest enzyme version * chore: update Enzyme version * chore: bump minimum versions * ci: update buildkite settings * feat: wider support for batched_matmul * perf: benchmark fallback batched_matmul * feat: slow fallback conv impl * feat: parallel fallback batchedmm * ci(buildkite): add GPU testing for Metal and oneAPI * test: check for FP64 support * fix: convert element type before broadcasting * fix: dispatch for NNlib conv * ci(buildkite): disable testing for Metal and oneAPI * chore: bump version * feat: update minimum version of Enzyme to 0.13 * feat: support within_gradient for Enzyme * refactor: rename within_gradient to within_autodiff * fix: update forward rules to new API * fix: use known on the return type * fix: forward enzyme rules * fix: broken enzyme tests * feat: support runtime activity for enzyme * fix: check was accidentally broken * chore(deps): bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.3 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.3...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.5 to 1.24.6 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.5...v1.24.6) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix: rollback custom gelu implementation * feat: XLADevice via Reactant * chore: apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: bump version * feat: more extensive testing of XLA backend * fix: incorrect function call * test: rename * test: incorrect env var * fix: copy to XLA in main thread * fix: don't support pre-moving the data * fix: urgent patch for reactant breakage * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump crate-ci/typos from 1.24.6 to 1.25.0 (#41) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.25.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.25.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.25.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore: bump crate-ci/typos from 1.24.6 to 1.26.0 Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.6 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.24.6...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * ci: run on `1.10` and `1` (#57) * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` (#81) * ci: run on 1.10 and 1 * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` (#43) * ci: run on `1.10` and `1` * ci: run on `1.10` and `1` * test: mark truncated normal on Metal as unbroken * ci: run buildkite on `1.10` and `1` * chore: bump peter-evans/create-pull-request from 6 to 7 (#40) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 7. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v7) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: run tests only on `1.10` for now (#172) * fix: relax cublaslt types (#173) * docs: add Flux.jl to the README (#83) After FluxML/Flux.jl#2492 also Flux relies on MLDataDevices. * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#58) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#44) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.25.0 to 1.26.0 (#174) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.0. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.0) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump compat for GPUArrays in [weakdeps] to 11, (keep existing compat) (#86) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * chore: bump version for release * chore: bump compat for GPUArrays in [weakdeps] to 11, (keep existing compat) (#46) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> * chore: bump compat for GPUArraysCore to 0.2, (keep existing compat) (#47) Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Avik Pal <avikpal@mit.edu> * chore: bump version for release * feat: add fallbacks for unknown objects (#87) * feat: add fallbacks for unknown objects * feat: handle RNGs and undef arrays gracefully * test: RNG movement * test: functions and closures * refactor: move `JuliaSIMD` deps to extensions (#175) * fix: remove LV.vmap! usage * fix: remove LV handling for bias_activation * fix: remove LV usage in dropout * refactor: move LV and octavian behind an extension * docs: add docs for loading packages * refactor: move SLEEFPirates to an ext * fix: enzyme rules for batched matmul * fix: patch more enzyme issues * feat: add a preference to disable loop vectorization * fix: incorrect dispatch called * fix: enzyme segfault bypass * feat: define isleaf (#84) * isleaf * exclude * add tests and docs * more tests * import functors * fix test * chore: reduce min compat * chore: run formatter * chore: bump version for release * fix: handle bitstypes and wrapped arrays in isleaf (#88) * bitstype and wrapped arrays * fixes * fix import * bound * cleanup * chore: fix min version of LinearAlgebra * chore: run formatter --------- Co-authored-by: Avik Pal <avik.pal.2017@gmail.com> Co-authored-by: Avik Pal <avikpal@mit.edu> * fix: task switching in AMDGPU complex batched_matmul (#178) * ci(buildkite): add downstream testing for NeuralOperators * perf: restore old batched_mul * fix: disable threading for certain devices * revert: "perf: restore old batched_mul" This reverts commit a8c0f3b4615f96a8773577e16fac61ba310d8123. * fix: correctly handle adjoints of wrapped arrays (#90) * fix: correctly handle adjoints of wrapped arrays * fix: use fast paths for adapt * fix: adapt ranges to JuliaGPU/Adapt.jl#86 * chore(deps): bump crate-ci/typos from 1.25.0 to 1.26.8 (#44) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.25.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.25.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#49) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#60) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: missing import; fixes #179 (#180) * chore: bump crate-ci/typos from 1.26.0 to 1.26.8 (#93) Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.26.0 to 1.26.8. - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](crate-ci/typos@v1.26.0...v1.26.8) --- updated-dependencies: - dependency-name: crate-ci/typos dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: merge LuxCUDA testing scripts * ci: merge LuxCore testing scripts * ci: merge WeightInitializers testing scripts * ci: add WI to pipeline launch * ci: add MLDataDevices to pipeline launch * ci: change 1.10 to "lts" * test: LuxCore test fixes * ci: soft fail MLDataDevices * ci: add a central downstream testing * ci: partially migrate LuxLib CI * ci: remove name field * ci: minor fixes to build scripts * ci: move LuxTestUtils CI scripts * ci: update LuxLib workflow * ci: update LuxLib workflows * ci: split out downstream testing * ci: fix certain pipelines * ci: minor tweaks * fix: workflows * test: use local LuxCUDA for tests * fix: use develop * docs: update * fix: add dev packages * docs: dev required packages * perf: merge the benchmarks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: CompatHelper Julia <compathelper_noreply@julialang.org> Co-authored-by: Christopher Rackauckas <accounts@chrisrackauckas.com> Co-authored-by: Carlo Lucibello <carlo.lucibello@gmail.com>
Example