v0.6.0

What's Changed

• ArrowFlight: ignore messages that have no app_metadata set instead of NPE by @nbauernfeind in #1441
• Fix up importjava and importstatic methods by @mofojed in #1449
• Known fixes for cpp client build, plus an initial version of an automated build for deps. by @jcferretti in #1437
• Bump for next release by @devinrsmith in #1454
• Updated build instructions for C++ client by @kosak in #1337
• java-client add barrage support by @nbauernfeind in #1345
• Update web version to v0.5.1 by @mofojed in #1456
• More friendly utility methods for arrow::Status and arrow::Result by @kosak in #1457
• Remove garbage code that was left in by accident by @kosak in #1459
• Add API methods for conditional formatting by @mofojed in #1375
• layout hints api by @mofojed in #1452
• CSV reader example for the cpp client by @jcferretti in #1455
• C++ comprehensive build project and cleanups by @kosak in #1462
• Switching build-dependencies.sh default build type to Debug. by @jcferretti in #1464
• Fix an issue with the previous generator changes. by @jcferretti in #1465
• Upgrade compiler to JDK17, fixes #1367. by @devinrsmith in #1397
• Initial (untested) Kafka publishing support by @rcaudy in #1463
• Json catchup. by @jcferretti in #1470
• Upgrade to JDK17 test runtime, fixes #1368 by @devinrsmith in #1443
• Enable liveness scope release to destroy table created in repl by @nbauernfeind in #1448
• [jsapi] Set rollback to null to enable orphan cleanup by @nbauernfeind in #1476
• [barrage] use Timestamp column for DBDateTime by @nbauernfeind in #1481
• Rework QueryConstants to add missing values and to remove improper values by @chipkent in #1386
• FetchTable: make apply preview columns optional by @nbauernfeind in #1487
• Add resolve URI interface for local tables, remote barrage subscripti… by @devinrsmith in #1472
• Fix varargs call for python implicit calls of arrays by @nbauernfeind in #1479
• Added the csv module and the dependent ones, fixes #1391 by @jmao-denver in #1477
• Added the deephaven.ResolveTools module, fixes #1497 by @jmao-denver in #1499
• [parser] fix NPE when casting null to a type by @nbauernfeind in #1500
• User visible API for publishing to Kafka. by @jcferretti in #1377
• Upgrade runtime images to JDK17, fixes #1369 by @devinrsmith in #1494
• Upgrade application target from JDK8 -> JDK11 by @devinrsmith in #1495
• Upload JVM error logs if present on CI failure by @devinrsmith in #1505
• Web v0.6.0 by @mofojed in #1504

Full Changelog: v0.5.2...v0.6.0

v0.5.2

What's Changed

• Add a ProcessMemoryLog by @jcferretti in #1381
• Fix up formatting when multiple types of formats are set by @mofojed in #1409
• Fix KafkaTools wrt dictionary keys()/values() by @devinrsmith in #1410
• Fix *ChunkColumnSource.getChunk() with empty OrderedKeys by @rcaudy in #1412
• Improve the process memory view. by @jcferretti in #1413
• Remove unused data members in QueryPerformanceNugget. by @jcferretti in #1419
• method signatures cleaning up, Docstrings cleaning up, etc. fixes #1359 by @jmao-denver in #1390
• Add generated json files. by @jcferretti in #1418
• fixed a bug introduced in the commit for #1294 that led to the failur… by @jmao-denver in #1426
• Add gitter chat to the readme. by @chipkent in #1431
• C++ Client: documentation and sample code by @kosak in #1315
• Generate cpp client and example sphinx docs in ci build by @niloc132 in #1433
• Docker tasks should always tell gradle if they have changes by @niloc132 in #1427
• Time zone fetch optimizations by @mofojed in #1436
• Separate cppdoc job. Remove JDK15 where unneccessary. by @devinrsmith in #1435
• added tutorials.crypto_trades module, fixes #1442 by @jmao-denver in #1444
• Bump pre-release to version 0.5.2 by @devinrsmith in #1445

Full Changelog: v0.5.1...v0.5.2

v0.5.1

What's Changed

• Adding generated json. by @jcferretti in #1385
• Add support for java client doPut against NewTable. by @devinrsmith in #1374
• Fix handling for non-dictionary pages in dictionary chunks by @rcaudy in #1387
• Don't print out on our re-directed stdout/stderr, print out on the or… by @devinrsmith in #1383
• created the basic project structure, automated build, installation and testing, fixes #1294 by @jmao-denver in #1378
• Added CsvSpecs and readCsv for CsvSpecs. by @devinrsmith in #1361
• Update git clone instructions to use https instead of git (the latter… by @devinrsmith in #1198
• [envoy] route websockets to grpc-proxy and everything else to grpc_api by @nbauernfeind in #1365
• Format the client and server pydocs the same by @chipkent in #1392
• Add executeTimeout to session impl config builder, fixes #1395 by @devinrsmith in #1396
• Fix Kafka streaming referential integrity and shutdown processing by @rcaudy in #1393
• readCsv against String is now smarter, and can delegate to URL. by @devinrsmith in #1394
• Bump pre-release to version 0.5.1 by @devinrsmith in #1402

Full Changelog: v0.5.0...v0.5.1

[Release] 0.5.0

Fixed bugs:

• [Groovy] upperBin() (and lowerBin()) method signature doesn't match the javadocs #1341
• Nightly Check Transient Failure: QueryTableWhereTest#testInterFilterInterruption #1317
• Cannot add two series from the same table to a plot. #1292
• diffDay, diffNanos, and diffYear have incorrect docs for the situation that gives negative values #1288
• io.deephaven.db.v2.replay.Replayer tries to grab the LTM exclusive lock when it shouldn't #1283
• Syntax error found in the overview example of pydeephaven #1279
• StreamToTableAdapter.refresh error logging is substandard (because there isn't any) #1277
• deephaven.ParquetTools.readTable() doesn't support Parquet date32 or date64 data types #1253
• Python Issues with pre-defined constants #1247
• The sort direction enum in the Python client isn't correctly assigned values #1233
• Bug in ChunkColumnSources observed via streaming lastBy #1227
• Copy Selection* menu options are copying the entire table not just the selection. #1225
• The Python client releases tickets not generated by the client. #1219
• No need to be verbose about cancelled barrage subscriptions #1205
• "Running..." in IDE disappears before computation is complete #842
• Port fix for DH-11620 (WouldMatch fillChunk bug fix) #1342 (rcaudy)
• Move default nginx root outside of first location #1338 (nbauernfeind)
• Make QueryTableWhereTest#testInterFilterInterruption more robust for CI #1318 (nbauernfeind)
• Reduce noise on cancelled barrage subscriptions #1291 (nbauernfeind)
• Fix plot not allowing more than one series per table instance #1289 (nbauernfeind)
• Fix LTM lock usage in Replayer #1284 (rcaudy)
• Log errors when a failure is encountered in StreamToTableAdapter.refresh #1280 (rcaudy)
• Update web to v0.3.2 #1278 (mofojed)
• DBLanguageParser allow unboxing and widening when finding available methods #1248 (nbauernfeind)
• Fix bug in ChunkColumnSources #1243 (rcaudy)
• Reduce verbosity of UNAUTHENTICATED calls from alternative GrpcUtil.rpcWrapper #1238 (nbauernfeind)
• Fix case insensitive string filters #1229 (mofojed)
• Fix zulu-open-jdk-debian base image dependency #1213 (devinrsmith)
• TableService's gRPC lazyUpdate/update/select impls must acquire LTM shared lock #1211 (nbauernfeind)
• BarrageStreamGenerator#Drainable should be empty after drainTo #1209 (nbauernfeind)
• Send UI to Unknown Error if auth token cannot be refreshed after a few tries #1193 (nbauernfeind)
• Fix console type for containers/ #1192 (devinrsmith)
• Propagate query scope changes even if script throws a RuntimeError #1191 (nbauernfeind)
• Add no-cache headers to responses from the web #1185 (mofojed)
• Propagate Barrage propagation errors to the client table widget #1182 (nbauernfeind)
• Properly ignore non-refreshing dynamic nodes; fix jsapi getObject to wait for Ticket #1173 (nbauernfeind)
• ScriptSession#setVariable require subclasses to notify old vs new Value #1166 (nbauernfeind)

Closed issues:

• Consolidate slf4j versions and make sure they all match #1332
• Consider using standardized Java classes for date/time #1304
• Make it easy to use PyCharm to remotely debug Python code inside the Deephaven grpc-api container #1302
• Determine if an incremental-wrapping approach is possible or makes sense to make the server Python integration more Pythonic #1276
• Make pydeeephaven.Session usable in Python 'with' statement. #1244
• dummy issue #1197
• deephaven-client-session should not depend on flight-core or arrow-x #1186
• Code examples to demo how to use the major functions of the DH Python client. #1146
• java-client DoPut #1124
• Automate the code generation, building, and testing for the new Python client #1073
• Kafka: Handle DBDateTime from JSON #1046
• Kafka: KafkaTools: Make Avro/Json and Simple Raw Types Play Nicely Together as Key/Values #1025
• Kafka: Support nested records in avro schemas by linearizing them. #985
• Convert JSAPI examples from Groovy #904
• Reduce log noise from autocomplete #796
• Add convenience types for DynamicTableWriter #793
• Establish and advertise minimum required docker version #769
• CSV (and related table) uploads #696
• EPIC: Java API supporting connection to remote worker + simple table ops #657
• Consider open-telemetry #100

Merged pull requests:

• Bump version to 0.5.0 pre-release #1376 (devinrsmith)
• Add C++ to the readme #1371 (chipkent)
• Add java client DoPut support. Fixes #1124. #1366 (devinrsmith)
• GitHub Action for CLA compliance #1363 (chipkent)
• Update web to v0.4.0 #1357 (mofojed)
• Update container readme files. #1355 (chipkent)
• Consolidate slf4j versions, fixes #1332 #1340 (devinrsmith)
• Update CONTRIBUTING.md based on user feedback #1335 (chipkent)
• Add methods to get/set DBTimeZone#TZ_DEFAULT #1331 (AlSpinks)
• Avoid log4j dropping warnings in the console for dependent libraries. #1329 (jcferretti)
• Update redpanda from 21.4.13 ...

[Release] 0.4.1

New Feature

Kafka Ingestion Support

• Schema service integration for Confluent/Apicurio.
• Decode from Apache Avro, JSON, or primitives.
• Start from beginning, end, current, or an arbitrary offset.
• Stream tables to the Deephaven query engine for efficient event processing or aggregations.
• Record full history in-memory to allow richer Deephaven queries.
• Commingle all partitions, or segregate them into distinct sub-tables within Deephaven TableMaps.

Stream Table Optimizations

• Developed in support of Kafka ingestion.
• Allow for very efficient aggregations over the full data set, while only storing “new” rows in their entirety.
• Can be transformed into full-history in-memory tables.
• Suitable for any streaming ingestion use case.

Parquet Support Expansion

• Use Spark/Dask style _metadata and _common_metadata files when present for faster partition discovery.
• Support multiple row groups in a single .parquet file.
• Major scalability enhancements for files with many dictionary pages and/or row groups.
• Support dictionary size constraints in writing tools.
• Single entry point for “do the right thing” reading.
• This expands on existing support for Parquet, which allows for symmetrical writing/reading of Deephaven tables, in addition to ingestion support for single files, flat layouts, and hive-style partitioning with most standard compression options and data types.

Application Mode

• Loads all .app files in deephaven.application.dir (CLI flag to directory).
• Use ApplicationService to subscribe to field changes.
• Script types of application must match deephaven.console.type.
• Static (script-less, immutable) applications subclass StaticClassApplication.
• Dynamic (script-less, non-immutable) applications subclass DynamicApplication.
• Initialize REPL environment with a Script Application (Query Scope changes are available to REPL sessions).

Python Integration

• New deephaven.Types import, defines constants for Deephaven column data types and adds support for table creation from constant data with an API resembling Panda’s DataFrame. This builds on the QST support for building tables also mentioned in the Java client section.

Python client

• Session object connects to DH server, convenience methods for creating empty tables, time tables, importing table from Arrow data, merging tables, opening tables by names, and running scripts.
• Table object represents a DH table, supports most table operations including filtering, selection, snapshotting, aggregation and joins.
• Query object defines a set of table operations to be performed in a single batch on the server.
• Available for download from PYPI.

Java client

• Arrow Flight integration; doGet (doPut next release).
• Query syntax tree (QST) support.
• Serial mode.
• Batch mode.
• Console session support.
• Command line utilities and examples.
• Graphviz visualization for QST.
• Jars published to Maven Central; https://search.maven.org/search?q=g:io.deephaven

C++ Client

• Arrow flight integration (basically a passthrough to the C++ FlightClient provided by the Arrow libraries). Provides doGet, doPut, and others.
• Async mode.
• Unit tests and examples.

UI

• CSV upload and download are now supported via the web UI.

Enhancement

Improved interactions between two windows communicating with the same worker.

• The shared query-scope fields show up in the panels drop-down.
• The UI recognizes initial change to an existing query-scope field as a new field.

Improved detection of scenarios where the backend is not working as expected.

• If any of grpc-web, envoy, or the worker exit, the UI detects this and notifies the user.
• If a Barrage subscription fails to snapshot, all subscribers are notified and closed.

Improvements to Arrow Flight integration

• Migrated all Barrage metadata into FlightData metadata field.
• doGet works from official Java / Python and C++ Arrow Flight clients.

Improved error messages related to tickets (missing and invalid) to gRPC clients

gRPC API

• Added FetchTable to TableService; can be used in a batch.
• Added ExportFromTicket to SessionService; does not validate type of export.
• Added ExportNotification.RUNNING state to distinguish from QUEUED.
• Reduced log volume related to UNAUTHENTICATED requests.

Javascript Client (jsapi)

• Wired up selectDistinct (re-enables existing UI functionality for filter pop-ups).

Address bugs and inefficiencies in autocomplete.

Improved error handling in the JS API.

Bug fix

• Fixed several issues regarding Liveness tracking in SessionState et. al.
• Fix SessionState NPE caused by cancel / perform-export-work race.
• Fix barrage subscription update race due to acquiring wrong lock.
• Fix bugs in SortedRanges.SearchIterator for hasNext after exhaustion.

Internal

• Cleanup many of the rawtypes related to ColumnSources.
• Upgrade JavaParser from 2.0.0 to 3.23.0.
• AutoComplete is now performed over a bi-directional stream.
• Javascript Client
  • Prefers to use websockets in non-ssl for bi-directional streams.
  • Uses HTTP headers x-deephaven-stream-ticket, x-deephaven-stream-sequence, and x-deephaven-stream-halfclose to simulate client-streams (and bidirectional-streams) when using SSL.

System Administration

• Flag io.deephaven.console.type renamed to deephaven.console.type.
• New Flag deephaven.application.dir (see Application Mode in New Features).
• Added no-cache headers to nginx_default.conf.
• Added layouts directory adjacent to notebooks in nginx_default.conf.
• Disable websocket idle timeout in envoy via stream_idle_timeout: 0s.

Commits

Raw Git release notes:
3349c42 Cut for 0.4.0
f9fb90b Remove old docker/ contents (#1170)
b7edf2c Deephaven C++ client (#1176)
5681bad Propagate Barrage propagation errors to the client table widget (#1182)
d11aa6c Give table update listeners short version of propagation error (#1179)
7b23050 MVP release, version set to 0.4.0, fixes #1171 (#1175)
d79bed1 Improve Ticket/Descriptor error messages for all gRPC usages (#1174)
b9bbef2 Added python tests for every Kafka spec type. Closes #1123. (#1168)
8361935 Properly ignore non-refreshing dynamic nodes; fix jsapi getObject to wait for Ticket (#1173)
2ff6962 Add execute script support for java client (#1156)
2b11f45 Add publishing for java-client jars and dependencies, fixes #1126 (#1149)
0753e2d Remove files that should have been never added. (#1169)
34cd17a ScriptSession#setVariable require subclasses to notify old vs new Value (#1166)
b181861 Add python documentation for Kafka integration. (#1160)
36b4c2d add compose files for standalone docker containers (#1058)
ff335f4 Use the correct location (Classpaths.groovy) to declare which java parser version (#1165)
e7ca952 Update web to v0.3.1 (#1164)
5cfd357 Improved validations for comboagg grpc request (#1014)
cf19dfa Initial Kafka python tests. (#1155)
657a716 fixed a compatibilty issue in the Python client due to the latest chanage to session proto, fixes #1158 (#1159)
ad6ff1b Update all .env files (#1161)
c2bd84f Updating web UI to v0.3.0 (#1154)
e0d85c4 Application mode for long running workers (#1082)
37e5f32 Adding Javadoc to KafkaTools.java. Closes #1122 (#1148)
6ef97c8 Deephaven Python API MVP (#1094)
404d5fd Upgrade javaparser-core to 3.23.0 (#1106)
a791fd3 Massive RawTypes Cleanup (#1145)
cd8745d Add safety check for .gitignore, fixes #1114 (#1147)
2e36920 Learn library logic, no dependencies (#1140)
51f19a5 Support bidirectional streaming on websockets (#1111)
a45154c Use DBDateTime insteat of qst...instant from python in deephaven.Types for the time being. (#1144)
602dfd9 Disambiguate DynamicTableWriter constructors taking an array from the point of view of jpy. (#1143)
590a746 Disable envoy websocket timeout (#1139)
7b2eacf Parquet: Lazy dictionary loading, shared page caches, and better file channel cache behavior (#1130)
d3d6aa9 QST to graphviz DOT format; and SVG, PNG, and others (#935)
34552e7 Fix CODEOWNERS references (#1137)
0add3f8 Run ./gradlew :Generators:generateAllPython (#1132)
473a40a DynamicTableWriter ctor to support using qst...Types (#1125)
f39d8f2 java-client session and flight (#953)
86319e2 Update code style (#1121)
a7a0454 Downgrade Alpine to avoid a DNS issue on older mac docker installs (#1034)
10fc2f6 Update CONTRIBUTING.md for styleguide (#1118)
9b851e5 Add DynamicTableWriter to deephaven module (#1117)
3d46a74 Fix tests after styling applied
8a5099f Apply style guide
038f8ee Remove style guide ratchet
abcfa45 Add style guide for most projects
14a997c Fix .gitignore rules (#1115)
c646eb2 Rename KafkaIngester module to Kafka. (#1110)
5e135dd Some Avro converter unit tests. (#1108)
8e64658 Adding Types.py for wrapping DH column types and table operations around them. (#1088)
11bda2e Fix a bug for the implementation of simple in the python side of KafkaTools. (#1109)
c932216 Add support for seek to end to Kafka API. (#1103)
440a166 rerun ./gradlew :DB:replicate (#1099)
2074ed2 Force jvm to load class in desired package (#1095)
7fe69b9 CrossJoin cardinality calculation may overflow integer (#1098)
87aa98e Properly use liveness API for dynamic non-refreshing nodes in SessionState exports (#1092)
8addc73 Revert "First iteration of Learn library (#974)" (#1096)
216ca7d First iteration of Learn library (#974)
0c22475 Native arrays, Db Arrays, and types. (#1076)
152b2b8 Add support for converting stream tables to append from python. (#1090)
41cd9a3 JS API Error handling (#1086)
686e5f8 TableMap support in KafkaTools (#1079)
c80f7c8...

[Release] 0.4.0

v0.4.0

[Release] 0.4.0

[Release] 0.3.0

* fc05fcd4 - (HEAD -> main, upstream/main) Remove @JsMethod from JsTable and JsTreeTable features that do not currently work. (#931) (13 minutes ago) <Mike Bender>
* f37f20a5 - Update web UI to version v0.2.1 (#939) (2 hours ago) <Mike Bender>
* 7b268072 - StreamToTableAdapter implementation and simple unit test. (#930) (24 hours ago) <Charles P. Wright>
* 4e6c3254 - QST (#715) (25 hours ago) <Devin Smith>
* ad17d768 - Remove unused python requirements files (#933) (25 hours ago) <Devin Smith>
* c1123e4f - Upgrade to junit 4.13.2 (#932) (25 hours ago) <Devin Smith>
* 50d8eb72 - Split Join gRPC call into one per type of join (#905) (26 hours ago) <Nate Bauernfeind>
* e3dcc764 - Add dockerized redpanda; do an initial very simple spike end-to-end (#922) (29 hours ago) <Cristian Ferretti>
* 73a10d49 - Standardize on Arrow's Flight Schema representation (#926) (2 days ago) <Nate Bauernfeind>
* 78c42311 - Split docker-compose in preparation of kafka/redpanda. Reduce version requirement. (#928) (2 days ago) <Cristian Ferretti>
* aa431d08 - WIP: stream table interfaces for discussion (#917) (2 days ago) <Ryan Caudy>
* ec33284d - Chunk column source. (#921) (2 days ago) <Charles P. Wright>
* 2a198596 - fixes #880 listFlights is now compatible with proper FlightClient (#891) (3 days ago) <Nate Bauernfeind>
* 55d9a373 - Remove non-class level usages of out of band test annotation. (#915) (3 days ago) <Cristian Ferretti>
* ed9f3ec2 - Change the API of ParquetTools.writeTable* to be consistent with readTable. (#919) (3 days ago) <Cristian Ferretti>
* cccec7d9 - Adding missing doc dir to git. (#920) (3 days ago) <Cristian Ferretti>
* 12c7b55e - Expose IdeConnection and getConsoleTypes to the client (#863) (4 days ago) <Mike Bender>
* d63f17cf - Fix QueryTableTest.testSnapshot to follow new initial snapshot semant… (#913) (4 days ago) <Charles P. Wright>
* d0bf1047 - ExportTicketResolver: resolves ticket before computing flight info fixes #879 (4 days ago) <Nathaniel Bauernfeind>
* 3ea4212a - update javadoc from Ryan's feedback (4 days ago) <Nathaniel Bauernfeind>
* f5e9403e - Fix #824: make default behavior of QueryTable#snapshot take an initial snapshot (4 days ago) <Nathaniel Bauernfeind>
* a6b73efa - Move parquet compression tests to an out of band test class. (#899) (6 days ago) <Cristian Ferretti>
* a3eeb730 - Make ScriptSession a LivenessScope instead of LivenessNode (8 days ago) <Nathaniel Bauernfeind>
* 69e7eec3 - Ensure that bindTableToVariable manages live references (#887) (8 days ago) <Devin Smith>
* 0ce46a76 - Support arbitrary types in partitioning columns (#885) (8 days ago) <Ryan Caudy>
* 6ab40bd0 - Partitioned table abstraction/generification and hierarchical parquet support (#869) (9 days ago) <Ryan Caudy>
* 53d38165 - Update web-client-ui to v0.2.0 (#876) (9 days ago) <Mike Bender>
*   b5ae662e - Merge pull request #870 from deephaven/Add-DataStructures-toJDocs (10 days ago) <JJ Brosnan>
|\  
| * 1e82de57 - Update build.gradle (10 days ago) <JJ Brosnan>
|/  
* 42e273f6 - Arrow flight DoGet bugfixes (#825) (11 days ago) <Colin Alworth>
* 52b5c0c7 - Ensure replicate is deterministic, doesn't try writing to the wrong project (#864) (11 days ago) <Colin Alworth>
* 0a78efd8 - Use symbol tables for table definitions loaded from parquet files when appropriate.  Fixes #847 (#848) (11 days ago) <Cristian Ferretti>
* 491be6be - Fix writing empty tables; fix missing beginning magic number. Fixes #855 (#856) (13 days ago) <Cristian Ferretti>
* d3b8d7c2 - Name validator rename (#851) (2 weeks ago) <Devin Smith>
* 21e6d2b1 - Fix aj/raj match expression parsing (#852) (2 weeks ago) <Devin Smith>
* 2cf978fe - Introduce gRPC calls for plotting, update js api to use pb objects (#761) (2 weeks ago) <Colin Alworth>
* 3fb181b5 - Table operations (#762) (2 weeks ago) <Devin Smith>
* 5daca589 - Make autocomplete stop spamming logs with junk (#843) (2 weeks ago) <James Nelson>
* 4c6601ec - Fix loading of parquet non-dictionary encoded pages from a column chunk that has a dictionary.  Fixes #838 (#839) (2 weeks ago) <Cristian Ferretti>
* 241a0371 - Out TableManagementTools, In ParquetTools. Fixes #742. Fixes #321. Fixes #813. (#819) (2 weeks ago) <Cristian Ferretti>
* a51f6c98 -  Add static analysis to autocomplete from newTable calls issue: (#783) (3 weeks ago) <James Nelson>
* 52013217 - Update Web UI v0.1.7 (#823) (3 weeks ago) <Mike Bender>
* 167d0b9b - Address component type issues causing nightly test failures (#818) (4 weeks ago) <Ryan Caudy>
* ce330744 - Bump next version to 0.3.0 (#815) (4 weeks ago) <Devin Smith>

Initial release

Initial release v0.1.0.