Refactoring Storage #12119

jdunkerley · 2025-01-23T10:33:36Z

Pull Request Description

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

Replace getItemAsObject with getBoxed.

Still in progress.

Left `AddGroupNumber`, `AddRowNumber` and `AddRunning`.

Remove last usage of `getItem`. Fix so can build.

Working on size still.

Still to rename back getBoxed and getPrimitive. Failing 10 tests. Rebase next.

std-bits/table/src/main/java/org/enso/table/parsing/TypeInferringParser.java

getItemAsLong and getItemAsBoolean.

std-bits/table/src/main/java/org/enso/table/data/column/builder/Builder.java

radeusgd · 2025-01-27T12:44:03Z

std-bits/table/src/main/java/org/enso/table/data/column/builder/DoubleBuilder.java

+      } else if (storage instanceof ColumnDoubleStorage doubleStorage) {
+        long n = doubleStorage.getSize();
+        for (long i = 0; i < n; i++) {
+          if (storage.isNothing(i)) {
+            appendNulls(1);
+          } else {
+            appendDouble(doubleStorage.getItemAsDouble(i));
+          }
+        }


About these: I'm a little bit slightly worried that because currently all (most?) storages are the 'base' e.g. DoubleStorage, this more 'abstract' variant will rarely run and so is not covered that much by tests.

This is a recurring problem that there are often more edge cases in the implementation of these low-level operations than seems feasible to have high-level tests (especially also ones shared with DB which are often slower).

Not for this PR but perhaps we need to add some 'unit' tests to std-table that test the internals more, to reinforce our correctness here?

No DoubleStorage is a ColumnDoubleStorage so this will run.
All the typed storages are their specific ColumnStorage.

TypedStorage<T> => ColumnStorage<T>

BooleanStorage => ColumnStorage<Boolean> and ColumnBooleanStorage.

Agree on some tests on the builders to check we get the correct interfaces out makes sense.

No DoubleStorage is a ColumnDoubleStorage so this will run. All the typed storages are their specific ColumnStorage.

No, because above it is a more specific branch that has a specialization for DoubleStorage specifically. Which is good. But that means that specialization above runs, not this generic code. That was my point above.

Ah ok - sorry yes.

It felt sensible to add these in at this point but will revisit as pull operations out one by one.

These will be used pretty quickly as ConstantColumnDouble would be a ColumnDoubleStorage for example.

I agree that we should have some java unit tests for these type of internals. I had a similar problem when I was looking to change the expression lanaguage.

radeusgd · 2025-01-27T12:51:00Z

std-bits/table/src/main/java/org/enso/table/data/column/storage/StringStorage.java

  @Override
-  protected SpecializedStorage<String> newInstance(String[] data, int size) {
-    return new StringStorage(data, size, type);
+  public TextType getType() {
+    // As the type is fixed, we can safely cast it.
+    return (TextType) super.getType();
  }


I'm tempted to make the Type also part of the generic arguments of SpecializedStorage to avoid these casts.

Please treat as a very optional suggestion

...bits/table/src/main/java/org/enso/table/data/column/storage/numeric/AbstractLongStorage.java

jdunkerley force-pushed the wip/jd/operations-1 branch from df39a40 to b2d2195 Compare January 23, 2025 17:59

jdunkerley added 23 commits January 25, 2025 14:46

Update Interface.

0c34336

Update base Storage and SpecializedStorage.

e6d714c

Update BigDecimalStorage.

c95a526

Update BigIntegerStorage.

e635538

Replace getItemAsObject with getBoxed.

Fix as ColumnStorage is now generic.

4991572

Update the datetime storages.

50328f3

Update StringStorage.

8add472

Update ObjectStorage and MixedStorage.

e3e69f6

Removing StringStorage constructors in favour of builders.

8bf5b45

Work on Long and Double storage.

56989a0

Still in progress.

Remove a fair amount of new LongStorage.

49f36ae

Left `AddGroupNumber`, `AddRowNumber` and `AddRunning`.

Java formatting.

1f18276

Java formatting.

6158438

More WIP.

e0c6d6c

Rename get to getPrimitive so can find easily.

156fd34

Remove last usage of `getItem`. Fix so can build.

Working through size.

2aa0865

Working through size (2).

8e3a2fc

Working through size (3).

dbe4b3a

Fix build issue.

1572a4e

Some tests fixed.

a0c90cc

Working on size still.

Use long indexes.

598efaa

Most of size gone and most of constructor use replaces.

b265313

Basically complete.

8ad8adc

Still to rename back getBoxed and getPrimitive. Failing 10 tests. Rebase next.

jdunkerley added the CI: No changelog needed Do not require a changelog entry for this PR. label Jan 25, 2025

Rebase fixes.

f95c03a

jdunkerley force-pushed the wip/jd/operations-1 branch from 66e54f1 to f95c03a Compare January 25, 2025 21:21

jdunkerley added 3 commits January 26, 2025 18:00

Java formatting.

d8783ef

Revert getBoxed to getItemBoxed.

1af40f2

Revert getPrimitive renames.

27bcd77

Java format.

5d69693

jdunkerley marked this pull request as ready for review January 26, 2025 21:58

jdunkerley requested review from radeusgd, GregoryTravis, AdRiley and marthasharkey as code owners January 26, 2025 21:58

AdRiley reviewed Jan 27, 2025

View reviewed changes

std-bits/table/src/main/java/org/enso/table/parsing/TypeInferringParser.java Outdated Show resolved Hide resolved

AdRiley reviewed Jan 27, 2025

View reviewed changes

std-bits/table/src/main/java/org/enso/table/parsing/TypeInferringParser.java Outdated Show resolved Hide resolved

jdunkerley added 4 commits January 27, 2025 12:03

PR comments.

415c37b

Change order of arguments.

65231cd

Rename getItemDouble back to getItemAsDouble.

40e3527

Rename getItemLong and getItemBoolean to

199e997

getItemAsLong and getItemAsBoolean.

radeusgd reviewed Jan 27, 2025

View reviewed changes

std-bits/table/src/main/java/org/enso/table/data/column/builder/Builder.java Show resolved Hide resolved

radeusgd reviewed Jan 27, 2025

View reviewed changes

jdunkerley added 2 commits January 27, 2025 13:02

Self PR fixes to Builders.

45f3827

Self PR fixes to Cast.

d2a98ce

radeusgd reviewed Jan 27, 2025

View reviewed changes

...bits/table/src/main/java/org/enso/table/data/column/storage/numeric/AbstractLongStorage.java Outdated Show resolved Hide resolved

radeusgd reviewed Jan 27, 2025

View reviewed changes

...bits/table/src/main/java/org/enso/table/data/column/storage/numeric/AbstractLongStorage.java Outdated Show resolved Hide resolved

Use same integer type when slicing, masking, filtering.

2917ea1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring Storage #12119

Refactoring Storage #12119

jdunkerley commented Jan 23, 2025

radeusgd Jan 27, 2025

jdunkerley Jan 27, 2025 •

edited

Loading

jdunkerley Jan 27, 2025

radeusgd Jan 27, 2025

jdunkerley Jan 27, 2025

jdunkerley Jan 27, 2025

jdunkerley Jan 27, 2025

AdRiley Jan 27, 2025

radeusgd Jan 27, 2025

Refactoring Storage #12119

Are you sure you want to change the base?

Refactoring Storage #12119

Conversation

jdunkerley commented Jan 23, 2025

Pull Request Description

Important Notes

Checklist

Choose a reason for hiding this comment

jdunkerley Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdunkerley Jan 27, 2025 •

edited

Loading