Code

Wip/codecs shards#135

Code

Draft

cmhulbert wants to merge 128 commits intomastersaalfeldlab/n5:masterfrom wip/codecsShardssaalfeldlab/n5:wip/codecsShardsCopy head branch name to clipboard

+5,846-175

Commits

Commits on May 21, 2024

• feat: wip toward codecs

  bogovicj

  committed
  c4f96f2

  Copy full SHA for c4f96f2

Commits on May 28, 2024

• feat(wip): reading and writing blocks uses codecs

  bogovicj

  committed
  14d3b69

  Copy full SHA for 14d3b69

Commits on Jul 17, 2024

• Merge remote-tracking branch 'origin/codecs' into dev/shards

  Show description for 76e1ec2
  bogovicj

  committed
  76e1ec2

  Copy full SHA for 76e1ec2

Commits on Jul 18, 2024

• feat: add ChecksumCodec

  Show description for adb84bb
  bogovicj

  committed
  adb84bb

  Copy full SHA for adb84bb

Commits on Jul 19, 2024

• wip: use CheckedInput/Output streams from java.util.zip

  Show description for c54bdb9
  bogovicj

  committed
  c54bdb9

  Copy full SHA for c54bdb9

• wip: add ChecksumException

  bogovicj

  committed
  e756f8e

  Copy full SHA for e756f8e

Commits on Jul 23, 2024

• refactor: codec getId to getName

  Show description for 5a82159
  bogovicj

  committed
  5a82159

  Copy full SHA for 5a82159

• feat: add BytesCodec

  bogovicj

  committed
  1cd307c

  Copy full SHA for 1cd307c

• feat: add ShardingCodec

  bogovicj

  committed
  72cd669

  Copy full SHA for 72cd669

• feat(wip): add prelim Shard classes

  bogovicj

  committed
  6fbb729

  Copy full SHA for 6fbb729

• feat(wip): update core n5 api with sharding

  bogovicj

  committed
  a213190

  Copy full SHA for a213190

• wip: more shard/codec work

  cmhulbert

  committed
  352427e

  Copy full SHA for 352427e

Commits on Jul 24, 2024

• feat/wip: add size partial read lockForReading methods

  bogovicj

  committed
  ce0b25d

  Copy full SHA for ce0b25d

• wip/feat: ShardIndex progress, VirtualShard progress

  bogovicj

  committed
  8aabcf4

  Copy full SHA for 8aabcf4

• feat/wip: add VirtualShard

  bogovicj

  committed
  77c096d

  Copy full SHA for 77c096d

• test: reading a zarr shard demo

  bogovicj

  committed
  6874c39

  Copy full SHA for 6874c39

• feat: make partial writes possible for key value access

  bogovicj

  committed
  be87013

  Copy full SHA for be87013

• fix: AbstractShard getBlockSize

  bogovicj

  committed
  c5ee84d

  Copy full SHA for c5ee84d

• wip: toward block writing through shard

  bogovicj

  committed
  0b4d73c

  Copy full SHA for 0b4d73c

Commits on Aug 1, 2024

• fix: partial write defaults with 0

  Show description for 083cb54
  cmhulbert

  committed
  083cb54

  Copy full SHA for 083cb54

Commits on Aug 2, 2024

• wip: zone serialization and sharding?

  cmhulbert

  committed
  bdadb58

  Copy full SHA for bdadb58

• wip/feat: add VirtualShard.createIndex

  Show description for 1990424
  bogovicj

  committed
  1990424

  Copy full SHA for 1990424

• fix: return KVA return for types for ranged lockForReading/Writing

  bogovicj

  committed
  1931530

  Copy full SHA for 1931530

Commits on Aug 5, 2024

• feat: add getAttributesKey

  Show description for 786ec1f
  bogovicj

  committed
  786ec1f

  Copy full SHA for 786ec1f

• wip: move getAttributesKey to GsonN5Reader

  bogovicj

  committed
  bf4f63e

  Copy full SHA for bf4f63e

Commits on Aug 6, 2024

• wip add constant N5_DATASET_ATTRIBUTES

  bogovicj

  committed
  b01b6be

  Copy full SHA for b01b6be

Commits on Aug 8, 2024

• refactor: Compression interface extends Codec

  Show description for 0cb5a0f
  bogovicj

  committed
  0cb5a0f

  Copy full SHA for 0cb5a0f

Commits on Aug 9, 2024

• wip: BytesCodec update

  bogovicj

  committed
  553c85e

  Copy full SHA for 553c85e

• test: remove outdated wip config parsing

  cmhulbert

  committed
  35a4d35

  Copy full SHA for 35a4d35

Commits on Aug 12, 2024

• feat: annotations for extensible serialization for codecs

  cmhulbert

  committed
  d636b83

  Copy full SHA for d636b83

• refactor: rename former 'chunkGrid' variables

  bogovicj

  committed
  1d9c0c2

  Copy full SHA for 1d9c0c2

• test: start CodecSerialization test

  Show description for c7fb316
  bogovicj

  committed
  c7fb316

  Copy full SHA for c7fb316

• test: codec array with a compressor

  bogovicj

  committed
  344aa46

  Copy full SHA for 344aa46

• test: deserialization behavior

  bogovicj

  committed
  6b1e9a1

  Copy full SHA for 6b1e9a1

Commits on Aug 14, 2024

• feat: more shard and codec work

  cmhulbert

  committed
  664007b

  Copy full SHA for 664007b

• test: minor updates

  bogovicj

  committed
  21ab72c

  Copy full SHA for 21ab72c

• perf: BlockReader have read call static method

  bogovicj

  committed
  314b199

  Copy full SHA for 314b199

Commits on Aug 16, 2024

• feat: WIP initial read/write blocks through composed codecs

  Show description for 5ad00fd
  cmhulbert

  committed
  5ad00fd

  Copy full SHA for 5ad00fd

Commits on Aug 23, 2024

• fix: isDataset caching

  cmhulbert

  committed
  f38fef7

  Copy full SHA for f38fef7

Commits on Aug 26, 2024

• test: fix FixedScaleOffsetTests

  bogovicj

  committed
  9155ec5

  Copy full SHA for 9155ec5

• fix: LockedFileChannel locking

  bogovicj

  committed
  e8cfefd

  Copy full SHA for e8cfefd

• fix: LockedFileChannel truncation logic

  bogovicj

  committed
  17ef0fe

  Copy full SHA for 17ef0fe

• fix: NameConfigAdapter avoid NPE

  Show description for 2e18288
  bogovicj

  committed
  2e18288

  Copy full SHA for 2e18288

• pref: normalGetDatasetAttributes should call createDatasetAttributes

  Show description for 7180290
  bogovicj

  committed
  7180290

  Copy full SHA for 7180290

Commits on Sep 4, 2024

• feat: DataBlock methods to read/write directly from DataInput/Output

  bogovicj

  committed
  764d05a

  Copy full SHA for 764d05a

• refactor: create N5BytesCodec, BytesCodec is simple zarr approach

  bogovicj

  committed
  dbd7b15

  Copy full SHA for dbd7b15

• chore(pom): depend on guava

  bogovicj

  committed
  786e4d5

  Copy full SHA for 786e4d5

• wip: undo ArrayToBytes codec changes

  bogovicj

  committed
  2a7327a

  Copy full SHA for 2a7327a

• fix(wip): DefaultBlockReader/Writer can get ArrayToBytesCodec directly

  bogovicj

  committed
  c24605d

  Copy full SHA for c24605d

• test/fix: BytesTests now works with refactor

  bogovicj

  committed
  5f8fa62

  Copy full SHA for 5f8fa62

• wip: toward supporting endianness

  bogovicj

  committed
  784a4a2

  Copy full SHA for 784a4a2

• wip: DatasetAttributes allow empty codecs

  bogovicj

  committed
  87456a4

  Copy full SHA for 87456a4

• style: import order

  bogovicj

  committed
  7ef4297

  Copy full SHA for 7ef4297

• fix: BytesTest N5BytesCodec has name "n5bytes"

  bogovicj

  committed
  b80fef0

  Copy full SHA for b80fef0

Commits on Sep 16, 2024

• feat: wip shard/codec support

  Show description for ae1f347
  cmhulbert

  committed
  ae1f347

  Copy full SHA for ae1f347

Commits on Sep 17, 2024

• feat: rethrow NoSuchFile as NoSuchKey

  cmhulbert

  committed
  7a1bbcb

  Copy full SHA for 7a1bbcb

• feat: wip support for ShardIndex location and bytesorder

  Show description for 9fac328
  cmhulbert

  committed
  9fac328

  Copy full SHA for 9fac328

• Merge pull request #128 from cmhulbert/wip/codecsShards

  Show description for 48695aa
  bogovicj

  authored
  48695aa

  Copy full SHA for 48695aa

Commits on Sep 19, 2024

• fix: ShardingCodec indexLocation should default to END

  bogovicj

  committed
  f667bb5

  Copy full SHA for f667bb5

• style: ShardingCodec

  bogovicj

  committed
  6635107

  Copy full SHA for 6635107

Commits on Sep 20, 2024

• fix: getBlockPositionInShard

  bogovicj

  committed
  4489116

  Copy full SHA for 4489116

• fix/wip: sharding codec block sizes needs reversing in zarr

  bogovicj

  committed
  eb6de85

  Copy full SHA for eb6de85

• feat: add getShardAttributes method to DatasetAttributes

  Show description for a8df678
  bogovicj

  committed
  a8df678

  Copy full SHA for a8df678

Commits on Nov 19, 2024

• wip: n5 exception and InMemoryShard

  bogovicj

  committed
  865c861

  Copy full SHA for 865c861

• wip: dummy impl of writeShard

  bogovicj

  committed
  fccdb9a

  Copy full SHA for fccdb9a

Commits on Dec 16, 2024

• doc: getShardSize

  bogovicj

  committed
  19b8618

  Copy full SHA for 19b8618

• fix: ShardedDatasetAttributes.getBlockPosition

  bogovicj

  committed
  3229fdd

  Copy full SHA for 3229fdd

Commits on Dec 18, 2024

• perf: override writeData

  Show description for b270ece
  bogovicj

  committed
  b270ece

  Copy full SHA for b270ece

• chore: bump pom-scijva to 40.0.0

  bogovicj

  committed
  f46aa52

  Copy full SHA for f46aa52

Commits on Dec 20, 2024

• perf: initialize cache only if using it

  bogovicj

  committed
  362c74d

  Copy full SHA for 362c74d

• Merge branch 'wip/codecsShards' of github.com:saalfeldlab/n5 into wip/codecsShards

  bogovicj

  committed
  4f6b49b

  Copy full SHA for 4f6b49b

Commits on Dec 23, 2024

• fix: make removeAttribute methods' behavior more consistent

  Show description for edbdef6
  bogovicj

  committed
  edbdef6

  Copy full SHA for edbdef6

• fix: gzip make uzeZlib parameter optional

  Show description for 1b672de
  bogovicj

  committed
  1b672de

  Copy full SHA for 1b672de

• fix: BlockWriter should not close stream

  Show description for bec2996
  bogovicj

  committed
  bec2996

  Copy full SHA for bec2996

Commits on Jan 2, 2025

• Revert "fix: BlockWriter should not close stream"

  Show description for 2c753a1
  cmhulbert

  committed
  2c753a1

  Copy full SHA for 2c753a1

• feat(test): parameterize ShardDemo read/write test; add new test

  cmhulbert

  committed
  275aaa6

  Copy full SHA for 275aaa6

• feat(wip): toward an implementation of writeShard

  bogovicj

  committed
  9183d11

  Copy full SHA for 9183d11

• chore: stop using deprecated BoundedInputStream constructor

  bogovicj

  committed
  d4142c5

  Copy full SHA for d4142c5

• fix: be quiet

  bogovicj

  committed
  bc5d103

  Copy full SHA for bc5d103

• Merge branch 'master' into wip/codecsShards

  cmhulbert

  committed
  f294898

  Copy full SHA for f294898

• fix(test): failing on github actions

  cmhulbert

  committed
  1c9e601

  Copy full SHA for 1c9e601

• fix/test: add writeShardTest

  Show description for b62dc00
  bogovicj

  committed
  b62dc00

  Copy full SHA for b62dc00

• feat: writeShardEndStream

  cmhulbert

  committed
  1ced570

  Copy full SHA for 1ced570

Commits on Jan 3, 2025

• feat: serialize shardSize in DatasetAttributes for n5

  bogovicj

  committed
  0e045cd

  Copy full SHA for 0e045cd

• test: BytesTest operates on n5 container

  bogovicj

  committed
  f98b11a

  Copy full SHA for f98b11a

• feat: ShardedDatasetAttributes validate shard/block size on construction

  bogovicj

  committed
  cc5b8c5

  Copy full SHA for cc5b8c5

Commits on Jan 6, 2025

• refactor: remove DatasetAttributes#getShardedAttributes()

  Show description for 367b987
  cmhulbert

  committed
  367b987

  Copy full SHA for 367b987

• feat: writeBlocks aggregate shard

  cmhulbert

  committed
  20a9677

  Copy full SHA for 20a9677

• fix: index offset calculation

  cmhulbert

  committed
  c3c3ceb

  Copy full SHA for c3c3ceb

• feat(test): wip shard writeBlocks

  Show description for 5badbb7
  cmhulbert

  committed
  5badbb7

  Copy full SHA for 5badbb7

• feat/refactor: add BlockParameters and ShardParameters interfaces

  Show description for eb9fbc1
  bogovicj

  committed
  eb9fbc1

  Copy full SHA for eb9fbc1

Commits on Jan 7, 2025

• fix: null compression should result in empty byteCodecs list

  bogovicj

  committed
  b0092d3

  Copy full SHA for b0092d3

• refactor: createIndex now a default method in ShardParameters

  bogovicj

  committed
  458295c

  Copy full SHA for 458295c

• feat: add getByteOrder method for ArrayCodecs

  bogovicj

  committed
  da9b9ed

  Copy full SHA for da9b9ed

Commits on Jan 8, 2025

• feat: writeBlocks respects existing blocks in a given shard if not overwriting those blocks explicitly

  cmhulbert

  committed
  52f762e

  Copy full SHA for 52f762e

• refactor: remove generic from ShardParameter

  cmhulbert

  committed
  d4dcbe8

  Copy full SHA for d4dcbe8

• refactor: some signatures

  cmhulbert

  committed
  b2b3d2b

  Copy full SHA for b2b3d2b

• fix: Shard as Iterator<DataBlock<>>

  cmhulbert

  committed
  52751f5

  Copy full SHA for 52751f5

• feat: merge wip ShardParameters interface

  cmhulbert

  committed
  1161859

  Copy full SHA for 1161859

• test: ShardIndexTest

  bogovicj

  committed
  6e3cbe5

  Copy full SHA for 6e3cbe5

• feat: toward direct reading of InMemoryShard

  bogovicj

  committed
  0647755

  Copy full SHA for 0647755

Commits on Jan 9, 2025

• feat/test: add block position iterator for shard

  Show description for b17cb1f
  bogovicj

  committed
  b17cb1f

  Copy full SHA for b17cb1f

• feat: ShardIndex get properties by block index

  bogovicj

  committed
  334e46c

  Copy full SHA for 334e46c

• feat: add Shard.getNumBlocks

  bogovicj

  committed
  8449382

  Copy full SHA for 8449382

• perf: VirtualShard smart override of getBlocks

  bogovicj

  committed
  b6a5b4f

  Copy full SHA for b6a5b4f

• fix: GridIterator iteration order

  bogovicj

  committed
  55682ee

  Copy full SHA for 55682ee

Commits on Jan 10, 2025

• feat: DataBlockIterator skips missing blocks in Shard

  cmhulbert

  committed
  707addf

  Copy full SHA for 707addf

• chore: rm unused ShardReader/Writer classes

  bogovicj

  committed
  9a59de2

  Copy full SHA for 9a59de2

• fix/test: clone gridPosition

  bogovicj

  committed
  edb61a5

  Copy full SHA for edb61a5

• refactor: EMPTY_INDEX_NBYTES to ShardIndex

  bogovicj

  committed
  539959f

  Copy full SHA for 539959f

• refactor: EMPTY_INDEX_NBYTES to ShardIndex

  bogovicj

  committed
  0affc94

  Copy full SHA for 0affc94

• Merge branch 'wip/codecsShards' of github.com:saalfeldlab/n5 into wip/codecsShards

  bogovicj

  committed
  dd962e0

  Copy full SHA for dd962e0

• chore: rm unused method

  bogovicj

  committed
  7dde5bb

  Copy full SHA for 7dde5bb

Commits on Jan 13, 2025

• feat: Codec add composition helpers

  bogovicj

  committed
  a791244

  Copy full SHA for a791244

• fix: EMPTY_INDEX_NBYTES now in ShardIndex

  bogovicj

  committed
  cd54053

  Copy full SHA for cd54053

• feat: add getBlocks(int[] blockIndexes)

  bogovicj

  committed
  1c92d14

  Copy full SHA for 1c92d14

• feat: InMemoryShard add new write methods

  bogovicj

  committed
  52aaa2a

  Copy full SHA for 52aaa2a

• wip: minor change to writeBlocks, implement readBlocks

  Show description for 6fac5f2
  bogovicj

  committed
  6fac5f2

  Copy full SHA for 6fac5f2

Commits on Jan 14, 2025

• demo: BlockIterators

  bogovicj

  committed
  aca03d4

  Copy full SHA for aca03d4

Commits on Jan 15, 2025

• fix: ShardIndex.getOffsetIndex

  bogovicj

  committed
  367cadb

  Copy full SHA for 367cadb

Commits on Jan 17, 2025

• feat: add Position

  Show description for 1a44168
  bogovicj

  committed
  1a44168

  Copy full SHA for 1a44168

• feat: add positionToIndex static methods in GridIterator

  bogovicj

  committed
  43bc1e0

  Copy full SHA for 43bc1e0

• feat: ShardParameters methods

  Show description for 0e353e4
  bogovicj

  committed
  0e353e4

  Copy full SHA for 0e353e4

• wip: rm unused flatIndex in Shard

  Show description for 7097308
  bogovicj

  committed
  7097308

  Copy full SHA for 7097308

• refactor: InMemoryShard, read/writeBlocks

  Show description for f67943d
  bogovicj

  committed
  f67943d

  Copy full SHA for f67943d

Commits on Jan 21, 2025

• wip/feat: N5Reader.readShard

  bogovicj

  committed
  9224215

  Copy full SHA for 9224215

• feat: add ShardIndex.isEmpty

  Show description for a74d343
  bogovicj

  committed
  a74d343

  Copy full SHA for a74d343

Commits on Jan 24, 2025

• test: add a test for nested sharding codecs

  Show description for ce50fa9
  bogovicj

  committed
  ce50fa9

  Copy full SHA for ce50fa9