v1.5.0-RC1

Changelog

[v1.5.0-RC1] - 2023-06-16

Bug Fixes

• build/Dockerfile.immuadmin to reduce vulnerabilities
• build/Dockerfile.immuclient to reduce vulnerabilities
• build/Dockerfile.immuadmin to reduce vulnerabilities
• build/Dockerfile.rndpass to reduce vulnerabilities
• build/Dockerfile.full to reduce vulnerabilities
• modify tests for new object db initialisation
• build/Dockerfile.full to reduce vulnerabilities
• build/Dockerfile.rndpass to reduce vulnerabilities
• table id generation
• build/Dockerfile.immuclient to reduce vulnerabilities
• docs: bump golang.org/x/net to 0.7.0 in docs and test pkg
• embedded/ahtree: correct calculation of payload offset
• embedded/appendable: proper closing of non-required chunks
• embedded/document: support nil docs
• embedded/document: proper column renaming
• embedded/document: assign correct revision number
• embedded/document: close readers before updating document
• embedded/document: id field conversion
• embedded/document: proper handling of deleted documents
• embedded/document: validate doc is properly initialized
• embedded/document: validate doc is properly initialized
• embedded/sql: do not force columns to have max key length when unspecified
• embedded/sql: include explicit close into sqlTx options
• embedded/sql: multi-row conflict handling
• embedded/sql: implicit conversion within expressions
• embedded/sql: consider 0 as no limit
• embedded/sql: parsing of exists stmt
• embedded/sql: like operator supporting null values
• embedded/sql: crash when RowReader.Read() returns error
• embedded/sql: proper handling of parameters in row readers
• embedded/store: force snapshot to include mandatory mvcc changes
• embedded/store: ensure snapshot is closed for read-only txs
• embedded/store: integrity checks covering empty values
• embedded/store: avoid dead-lock when exporting tx with external commit allowance mode
• embedded/tbtree: fix error comparison
• embedded/tbtree: rollback to the most recent snapshot when insertion fails
• embedded/tbtree: fix snapshot getKeyWithPrefix
• embedded/tbtree: proper kv validation
• embedded/tbtree: fix snapshot getKeyWithPrefix
• go.mod: bump go version to 1.17 in go.mod
• helm: set securityContext and podSecurityContext at correct location
• pkg/api: create collection endpoint with path parameter
• pkg/api: fix and implement LIKE and NOT_LIKE operator when querying documents
• pkg/client: ensure ticker is properly stopped
• pkg/client: return error when verifiedGet operation fails
• pkg/database: read-only document API for replicas
• pkg/database: skip eof error during scan
• pkg/database: read from err channel
• pkg/database: wrap propagated context
• pkg/database: fix truncation and contemplate entry-less txs
• pkg/replicator: check stream is properly initialized
• pkg/server: do not set trailer metadata when replication is done with bidirectional streamming
• pkg/server: ensure error propagation when sending headers
• pkg/server: use grpc interceptors with grpc proxy
• pkg/server: request explicit close when creating a rw sql tx
• pkg/server: ensure tx is closed upon error
• pkg/server: close document readers before cancelling txs
• pkg/server: thread-safe doc reader during session handling
• pkg/stream: handle the case when message fits in a single chunk
• pkg/truncator: adjust plan logic and contemplate empty txs
• pkg/verification: document comparison with proto equals
• push.yml: update min go version

Changes

• Add reformatting of protobuf file on build/codegen
• rename DocumentBulkInsert to DocumentInsertMany
• Update build/RELEASING.md file
• allow multiple order by clauses
• replace schemav2 with protomodel in truncator test
• handle no more doc error inside response in search
• use sys/unix package
• remove docker test provider
• update document with id if not nil
• use gosec action
• add updatecollection api
• copy document catalogue when truncating db
• add DocumentFindOneAndUpdate api
• wait for immudb to get initialized
• add test case for uncommitted tx not increasing table count
• pass transaction to upsert function
• check invalid search id in search request
• add monotically increasing number to doc id generation
• add document audit api
• change DocumentFindOneAndUpdate to DocumentUpdate
• return sql reader on document search
• add order by clause in search
• use sql statement for delete than raw query
• add ReadN method to document reader
• add documentReader iterator to read documents
• remove initial swagger support
• exclude generated code from coverage
• exclude generated code from coverage
• add unique search id for paginated readers
• fix TestFloatSupport test case
• simplified codegen
• fix tests
• change DeleteTableStmt to DropTableStmt
• generate proto requests for DocumentDelete api
• return ErrNoMoreDocuments instead of sql.ErrNoMoreRows
• add DocumentDelete api
• add default size for document reader lru cache
• add lru cache for paginated readers
• fix merge issues
• add pagination support when fetching documents
• fix failing verification test
• increase test coverage for document engine
• add bulk insert api
• TruncateDatabase endpoint should use the same ongoing Truncator if present
• address review comment
• add go-acc and goveralls to ext-tools folder
• delete columns on table deletion
• check for column before adding index on collection update
• add option for non unique indexes on collection
• ci: improve notifications
• cmd/immuadmin: flag to specify the usage of embedded values
• cmd/immuadmin: add truncate cmd to immuadmin
• cmd/immuadmin: modify truncation settings schema
• deps: bump golang.org/x/crypto
• deps: bump github.com/jaswdr/faker from 1.4.3 to 1.16.0
• deps: bump securego/gosec from 2.14.0 to 2.15.0
• deps: bump github.com/codenotary/immudb
• deps: bump github.com/influxdata/influxdb-client-go/v2
• deps: bump github.com/golang/protobuf from 1.5.2 to 1.5.3
• deps: bump github.com/rogpeppe/go-internal from 1.8.0 to 1.9.0
• deps: bump github.com/lib/pq from 1.10.7 to 1.10.9
• deps: bump github.com/lib/pq from 1.10.2 to 1.10.7
• deps: bump golang.org/x/net from 0.8.0 to 0.9.0
• deps: bump github.com/rs/xid from 1.3.0 to 1.5.0
• deps: bump github.com/stretchr/testify from 1.8.0 to 1.8.2
• deps: bump aws-actions/configure-aws-credentials from 1 to 2
• deps: bump github.com/spf13/cobra from 1.2.1 to 1.6.1
• deps: bump github.com/grpc-ecosystem/grpc-gateway/v2
• deps: bump google.golang.org/grpc from 1.46.2 to 1.54.0
• deps: bump github.com/spf13/viper from 1.12.0 to 1.15.0
• embedded/ahtree: add inline comments
• embedded/appendable: fsync freebsd
• embedded/appendable: automatic file creation only when appending
• embedded/appendable: use fdatasync when file is preallocated
• embedded/appendable: file syncing per os
• embedded/appendable: support file preallocation
• embedded/appendable: file syncing using fdatasync when available
• embedded/appendable: minor improvements reading files
• embedded/appendable: metadats with putBool
• embedded/document: remove dead-code
• embedded/document: add float support for doc engine
• embedded/document: raw document validation
• embedded/document: improve error handling
• embedded/document: transactional collection update
• embedded/document: support null values in indexed attributes
• embedded/document: retrieval of raw document
• embedded/document: move source code into dedicated files
• embedded/document: use onclose callback to close the tx
• embedded/document: return struct when auditing document history
• embedded/document: add test to ensure key ordering in document during serialization
• embedded/document: blob type not yet supported
• embedded/document: catch key alredy exists error
• embedded/document: catch tx read conflict error
• embedded/document: translate table already exists error
• embedded/document: minor var renaming
• embedded/document: fix query stmt generator and add tests
• embedded/document: leverage sqlengine lazy index contraint evaluation
• embedded/document: binary serialization of doc payload
• embedded/document: use query limit when searching
• embedded/document: wip continue with improvements
• embedded/document: wip continue with improvements
• embedded/document: wip continue with improvements
• embedded/document: wip improvements
• embedded/document: add collection deletion api support
• embedded/document: minor code simplification
• embedded/document: add variable length support for multiple types
• embedded/document: ensure order by clauses are used when deleting and updating
• embedded/document: improve error messages
• embedded/document: minor code adjustments
• embedded/document: transactional collection and document creation
• embedded/document: improve error messages
• embedded/document: possibility to specify desc order when querying document history
• embedded/document: add tests for blob type
• embedded/document: avoid public dependency on sql
• embedded/document: add test cases for collection on doc engine
• embedded/document: change querier from BinBoolExp to CmpBoolExp
• embedded/document: transactional document creation
• embedded/document: add document id generation
• embedded/document: typo in error message
• embedded/htree: allow creation of empty hash trees
• embedded/object: add document abstraction
• embedded/object: add collection/database statements
• embedded/sql: use read-only txs whenever possible
• embedded/sql: make sql engine generic for object store
• embedded/sql: limit and offset boundary validation
• embedded/sql: minor numeric type adjustments
• embedded/sql: implicit conversion support in limit and offset clauses
• embedded/sql: simplified sql tx
• embedded/sql: return closed sql txs
• embedded/sql: implicit conversion from varchar to int and float types
• embedded/sql: upgraded row reader
• embedded/sql: ddl stmts register catalog mutation
• embedded/sql: snapshot reuse improvements
• embedded/sql: lazy index contraint validation
• embedded/sql: WIP singledb sql engine
• embedded/sql: transient context
• embedded/sql: validate total key length at index creation time
• embedded/sql: extend max key length to 512
• embedded/sql: cancellable row reader
• embedded/store: add in-line documentation for store options
• embedded/store: validate Eh only when integrity checks are not disabled
• embedded/store: addition of a cache for values
• embedded/store: consume all tx content even if integrity checks are disabled
• embedded/store: optional integrity checking when reading values
• embedded/store: validate Eh only when integrity checks are not disabled
• embedded/store: inline comments
• embedded/store: fix typo in inline comment
• embedded/store: validate gets using filters
• embedded/store: MVCC read-set with boundaries
• embedded/store: context propagation
• embedded/store: mvcc validation only if another tx was processed
• embedded/store: minor changes after rebasing from master
• embedded/store: simplified dualproof implementation
• embedded/store: skipIntegrityCheck parameter when reading data
• embedded/store: minor code improvement
• embedded/store: add min limit for truncation frequency
• embedded/store: use syncSnapshot to validate ongoing txs
• embedded/store: unify Read and ReadBetween
• embedded/store: transient context
• embedded/store: tx header is returned when fully committed
• embedded/store: api upgrade
• embedded/store: set ctx as first argument
• embedded/store: readValueAt and exportTx improvements
• embedded/store: update ReadBetween
• embedded/store: contextualized transactions
• embedded/store: inline comments
• embedded/store: snapshot reuse improvements
• embedded/store: set smaller default value for indexing bulk size
• embedded/store: wip mvcc validations
• embedded/store: simplified indexer initialization
• embedded/store: multi-tx unsafe mvcc
• embedded/store: set tx as closed upon cancellation
• embedded/store: wip mvcc validations
• embedded/store: set ctx as first argument
• embedded/store: embedded meta attribute required if version is greater than 1
• embedded/store: set embedded values mode as default one
• embedded/store: backward compatible embedded value mode
• embedded/store: optional integrity checking when reading values
• embedded/store: preallocate tx header log files
• embedded/store: wip mvcc validations
• embedded/store: handle eof when reading last committed tx
• embedded/store: support preallocated files when reading tx data
• embedded/store: wip preallocated clog
• embedded/store: option to prealloc files
• embedded/store: optional integrity checking
• embedded/store: further in-line documentation
• embedded/store: make truncation validation tolerate entryless txs
• embedded/store: allow tx without entries as long as it contains metadata
• embedded/store: add hashValue as fixed 32 byte size
• embedded/store: add test for TxOptions
• embedded/store: add min limit for truncation frequency
• embedded/store: multi-tx bulk indexing
• embedded/store: fix default vlog cache size and add validation for hash when reading from cache
• embedded/store: safe key copy for mvcc validation
• embedded/store: improve log messages when discarding precommitted transactions
• embedded/store: mvcc validations
• embedded/store: fix rebase issue with readValueAt for vlogcache
• embedded/store: multi-timed bulk insertions
• embedded/store: clog file size adjustment only when preallocation is disabled
• embedded/store: added more in-line comments
• embedded/store: file preallocation not enabled by default
• embedded/store: propagate context usage
• embedded/tbtree: parametrize snapshot creation specs
• embedded/tbtree: in-line documentation
• embedded/tbtree: initialize tbtree with a non-mutated leaf
• embedded/tbtree: getWithPrefix
• embedded/tbtree: remove unnecessary kv sorting
• embedded/tbtree: add in-line documentation
• embedded/tbtree: minor code improvements
• embedded/tbtree: rollback not needed as updates are made in a copy
• embedded/tbtree: add in-line comments
• embedded/tbtree: wip reduce allocs while updating inner node
• embedded/tbtree: optimized bulk insertion
• embedded/tbtree: optimize snapshot renewal
• embedded/tbtree: wip optimized insertion
• embedded/tbtree: variable renaming after rebasing
• embedded/tools: upgrade embedded tools with transient context
• embedded/watchers: return context error upon cancellation
• embedded/watchers: set ctx as first arg
• embedded/watchers: use context instead of cancellation channel
• package/database: bunch of fixes and improvements in document engine
• pkg: add more tests admin truncate command
• pkg/api: expose db setting to enable file preallocation
• pkg/api: use ErrrIs/ErrorContains in error checks
• pkg/api: rename idFieldName to documentIdFieldName
• pkg/api: annotate required fields
• pkg/api: use of path parameters for document-related endpoints
• pkg/api: expose new store indexing options
• pkg/api: document update with path parameter
• pkg/api: document api improvements
• pkg/api: remove bool from tx metadata conversion
• pkg/api: document api improvements
• pkg/api: snapshot reuse attributes
• pkg/api: search api improvements
• pkg/api: return txID when inserting or updating documents
• pkg/api: expose MVCC read-set settings
• pkg/api: remove unsupported attribute from response messages
• pkg/api: revert changes in swagger spec
• pkg/api: re-generated httpclient with DeleteDocument endpoint
• pkg/api: singular document path for audit and proof endpoints
• pkg/api: manual adjustments post-code generation
• pkg/api: expose replication settings for skipping integrity checks and indexing
• pkg/api: swagger gen
• pkg/api: value cache settings exposed
• pkg/api: annotate primitive types as required
• pkg/api: endpoint renaming
• pkg/api: cleaner session id header
• pkg/api: minor proof request renaming
• pkg/api: authorization in swagger spec
• pkg/api: re-generated httpclient
• pkg/api: expose support for unsafe mvcc transactions
• pkg/api: expose embeddedValue database setting
• pkg/api: remove generated httpclient
• pkg/api: buch of implementation improvements
• pkg/api: revised document and authentication apis
• pkg/api: annotated required message fields
• pkg/api: add tx metadata conversion
• pkg/api: re-generated httpclient
• pkg/api: change retention period in TruncateDatabase message to int64
• pkg/auth: add document update permissions
• pkg/client: minor renaming in tx options
• pkg/client: move heartbeater.go to pkg/client
• pkg/client/cache: improve test coverage
• pkg/database: upgrade after rebasing
• pkg/database: remove object store db initialisation
• pkg/database: upgraded reader specs
• pkg/database: add document query struct to abstract request query
• pkg/database: change objectEngine to documentEngine
• pkg/database: add mvcc test for truncation, parse retention period using duration
• pkg/database: add more tests for truncation
• pkg/database: add document store db initialisation
• pkg/database: updated APIs with schema updates
• pkg/database: implement GetCollection API
• pkg/database: remove search through first query
• pkg/database: add query parser for object to generate sql expression
• pkg/database: fix truncation deletion point checks in test
• pkg/database: add search document api implementation for object store
• pkg/database: create document/collection from schemav2 requests
• pkg/database: context propagation
• pkg/database: minor document renaming
• pkg/database: add and implement object db interface
• pkg/database: add DocumentUpdate api
• pkg/database: minor code aligments
• pkg/database: add object store
• pkg/database: context propagation from server to embedded layer
• pkg/database: proper calculation of source tx
• pkg/database: create txs with default options
• pkg/database: hard limit on page size
• pkg/database: check encoded value is consistent with raw document
• pkg/database: document verfication
• pkg/database: use _obj to hold raw document payload
• pkg/database: snapshot reuse changes
• pkg/database: add document engine abstraction
• pkg/integration: exportTx benchmarking
• pkg/replication: replicator using bidirectional streaming
• pkg/replication: improve options validation
• pkg/replication: skip integrity check when exporting transactions
• pkg/replication: context propagation
• pkg/replication: wip stream replication - only async replication working
• pkg/server: multi-grpc request context propagation
• pkg/server: added inline comments
• pkg/server: set default replication settings
• pkg/server: context propagation from grpc api to embedded package
• pkg/server: upgrades after rebasing from master
• pkg/server: upgrade to new insecure credentials api
• pkg/server: log error when closing document reader
• pkg/server: support snapshot reuse
• pkg/server: close all paginated readers on close of session
• pkg/server: integrate document functions with server apis
• pkg/server: add pagination test for document search
• pkg/server: ensure document reader is closed when swithing pages
• pkg/server: minor code reuse
• pkg/server: add test successful load/unload of db with truncator
• pkg/store: skipIntegrityChecks parameter when reading data
• pkg/stream: handle eof when sending data
• pkg/truncator: return error if expiration time hasn't been met
• pkg/truncator: add context to Truncate method
• pkg/truncator: refactor truncator process
• pkg/verfication: document verification methods
• pkg/verification: strengthen proof validations
• pkg/verification: minor renaming
• pkg/verification: document verification using embedded identifier
• pkg/verification: use proto serialization
• test/objects: add tests to create collections
• test/objects: use httpexpect
• test/objects: add more tests to create collection
• test/perf: fix version value for flag
• test/perf: add immudb version to influxdb data
• test/perf: add runner to results for influxdb
• test/perf-tests: remove runner check
• test/perf-tests: use proxy on benchmark runner
• test/performance: call cleanup method
• test/performance-test-suite: fix replica directory path
• test/performance-test-suite: add sync benchmarks
• test/performance-test-suite: changed server concrete implementation
• test/performance-test-suite: replicas are able to communicate with primary
• test/performance-test-suite: send results to influxdb
• test/performance-test-suite: extract json from results
• test/performance-test-suite: add influxdb host and toke arguments
• test/performance-test-suite: use temp folders for primary, replicas and clients
• truncator: add more coverage for truncator

Features

• add vlog truncation functionality
• ci: change notification
• embedded/document: count documents
• embedded/object: add object store to embedded pkg
• embedded/sql: Initial float support
• embedded/sql: limit and offset as expressions
• embedded/sql: sql transaction creation with options
• embedded/sql: short casting syntax
• embedded/sql: wip unsafe and optimized mvcc
• embedded/sql: implicit type conversion of numeric types
• embedded/store: tx creation with options
• embedded/store: expose GetWithPrefixAndFilters
• embedded/store: embeddable values
• embedded/store: unsafe mvcc mode
• embedded/store: embedded values option
• embedded/store: read-only transactions
• embedded/store: GetWithPrefixAndFilters
• embedded/tbtree: multi-timed bulk insertions
• pkg/api: document proof endpoint
• pkg/api: improved replace documents endpoint
• pkg/api: keepOpen parameter to instruct server to maintain a document reader in memory
• pkg/api: count documents endpoint
• pkg/client: optional tx options are now available during the creation process

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

immudb Binaries

File	SHA256
immudb-v1.5.0-RC1-darwin-amd64	adc27c7af5cea449c6066968ec142c3e2de9c4ed1cd4507b023874e6ac86f07b
immudb-v1.5.0-RC1-darwin-arm64	3395f8921ebdf4a57a4b63a488cd27bcfb05b4f3d7d104eb298bcffb7d30ce21
immudb-v1.5.0-RC1-freebsd-amd64	ee96402296490af7de3a90f6b44e681ba69f6f90ef5f727a90241219f1d6272e
immudb-v1.5.0-RC1-linux-amd64	d925e86463366f68c952b050cafa5475ca4be2c1f61432d91301c537c7387b76
immudb-v1.5.0-RC1-linux-amd64-fips	774cd076faa29708ffafaf249e8e37aab126c474736f2b8a933fd04214190ae2
immudb-v1.5.0-RC1-linux-amd64-static	e3c63a8fd2ff32a1a97a96ed44b10e72e48901bf8ed7c5211e69e985b79bb2d2
immudb-v1.5.0-RC1-linux-arm64	0eed077d9a4492a968dd167c390ada2a2b1a299b5e0241bbccbe87ceb275a427
immudb-v1.5.0-RC1-linux-s390x	8dbfe322d21cef912b06040d0cf32a49106b9f6b7d486293fc787effe14d7b15
immudb-v1.5.0-RC1-windows-amd64.exe	3b5fb2e85039599524d72ef8750e5561c2227c706c994edfff3e9cda2bd1bca6

immuclient Binaries

File	SHA256
immuclient-v1.5.0-RC1-darwin-amd64	23be535a67846b2d3701edfce14eb47b90dc2bbf16420133d7abcaf3a31bfa95
immuclient-v1.5.0-RC1-darwin-arm64	acc9496f541c3c109e6b42b253c427617f057a8a36577ab84fd7509a0da3e08f
immuclient-v1.5.0-RC1-freebsd-amd64	d6f001fd361290787b9a81481c358c307df66e11ddf8b9925a5de1162df8db7b
immuclient-v1.5.0-RC1-linux-amd64	121bab9fe5fe7851011048e71007724594167695b12a606381c330ed5258c2f9
immuclient-v1.5.0-RC1-linux-amd64-fips	28432d3b86234767a731aed96302285e0d1b94edf6947582a67b611fbbc7ec05
immuclient-v1.5.0-RC1-linux-amd64-static	bdacddd65fb6e5c8052c8a0a11849c19de69ed52a5958085733c3dc68195e9f2
immuclient-v1.5.0-RC1-linux-arm64	f665821ec60244f6ce522e1c6fe3d18c6e43c06d90448765988cfc3877157e8a
immuclient-v1.5.0-RC1-linux-s390x	bcc0af4ad11071e1cf668ed9f042df20ad44bc83ee424595e7aa9b1c6da3ba4b
immuclient-v1.5.0-RC1-windows-amd64.exe	a61ce4e021b89b921975640dc519384b787c384c8560465d21387642f74ae121

immuadmin Binaries

File	SHA256
immuadmin-v1.5.0-RC1-darwin-amd64	d0d055a5fb43dcbd4afc663ad84a8b5c39a832c6c223d02687809ada2aa11965
immuadmin-v1.5.0-RC1-darwin-arm64	3c88a2c178d96d7ebf6cfb2437f42cda4e88f10924f51c7c0344dc46aae451f9
immuadmin-v1.5.0-RC1-freebsd-amd64	763e2c0d2763220ed5205717a30872d4ed5d3c7699a28028c63ebeda44aadf9d
immuadmin-v1.5.0-RC1-linux-amd64	3c6f277dbb286399c6a44c99ac94d44c42f1424e7173e67931adc0d2a821056d
immuadmin-v1.5.0-RC1-linux-amd64-fips	6ff96208437539c723afe61825c13b38021b963dc5e9ea6195996a0eeb1467ff
immuadmin-v1.5.0-RC1-linux-amd64-static	aba650e13e76f495999159f1ec4d18d5f6fab37b3c9652afbf2d2a0c5999b02d
immuadmin-v1.5.0-RC1-linux-arm64	649534fc458b1b3a7f06898aceedf111d37085a958f86fd45994d1bcdac738fa
immuadmin-v1.5.0-RC1-linux-s390x	0f947b5957f03514dc1761c81ec345f5135d43f926861c2c52e001ee4b11364f
immuadmin-v1.5.0-RC1-windows-amd64.exe	d81f4e90af6fd7226d833e254164b51b787f517a35d5c01b572ed26e9beca474