Releases: apache/shardingsphere
Releases · apache/shardingsphere
5.5.1
API Changes
- Authority: Mark privilege provider ALL_PERMITTED as deprecated and will be removed in future
- DistSQL: Remove optional param usageCount from show storage units
- Readwrite-splitting: Change dataSources to dataSourceGroups for YAML
New Features
- Kernel: Add new doris, hive and presto sql parser module and database type
Enhancements
- DistSQL: Check privilege when registering or altering storage unit
- DistSQL: Check duplicate actual data nodes when creating or altering sharding table rule
- DistSQL: Add like support for show storage units
- DistSQL: Rollback if import database configuration failed
- DistSQL: add table type to result set of show logical tables
- SQL Parser: Support MySQL change replication with no server ids
- SQL Parser: Support MySQL row and column aliases with ON DUPLICATE KEY UPDATE
- SQL Parser: Support MySQL CALL sql with mysql prefix
- SQL Parser: Support MySQL create loadable function returns int/dec
- SQL Parser: Support parsing MySQL CREATE TABLESPACE/TABLE in Mysql-8
- SQL Parser: Support parsing MySQL CREATE INDEX with ENGINE_ATTRIBUTE
- SQL Parser: Support parsing MySQL LOAD DATA with @ variable
- SQL Parser: Support parsing MySQL START REPLICA statement
- SQL Parser: Support parsing MySQL json table function
- SQL Parser: Add MySQL column not null SQL parsing
- SQL Parser: Add MySQL charset SQL parsing
- SQL Parser: Enhance mysql json function visit statement result
- SQL Parser: Support PostgreSQL COLLATION and CONSTRAINT keywords
- SQL Parser: Support PostgreSQL, openGauss function table and update from parse
- SQL Parser: Support CTE with PostgreSQL and openGauss
- SQL Parser: Optimize the timezone support of PostgreSQLDateValueParser.
- SQL Parser: Support PostgreSQL limit and offset as null
- SQL Parser: Enhance Oracle update statement as alias parse and add test case
- SQL Parser: Support Oracle column length unit type SQL parsing
- Metadata: Collect table type for PostgreSQL
- Kernel: Support sql hint extract when sql contains dbeaver hint comment
- Kernel: Add extract combine left select in extractFromSelectStatementWithoutProjection method
- Kernel: Add assisted node to create and delete databases
- Transaction: Use same transaction type in one transaction in JDBC adapter
- Transaction: Support for switching transaction types
- SQL Federation: Support sql federation bit_count function for mysql
- SQL Federation: Improve atan and anan2 sql function for mysql
- SQL Federation: Support federated query mysql bin function
- SQL Federation: Support for federated query NOT operator
- Proxy: Make the results of show tables in order
- Proxy: Trigger metadata collection when creating and deleting database and table
- Proxy: Optimize Agent to support collecting metrics data from multiple data sources when used with Driver
- Proxy Native: Add more graalvm reachability metadata for caffeine cache
- Sharding: Revise all local index for sharding table and add object uniqueness level spi to control index token generator
- Sharding: Add NullsOrderType.LOW and NullsOrderType.HIGH to handle NULL order by in sharding feature
- Sharding: Add inline sharding algorithms match actual data nodes check
- Encrypt: Add unsupported check for combine statement with encrypt columns
- Encrypt: Support select distinct(column) encrypt rewrite and refactor SubstitutableColumnNameToken build logic
- Encrypt: Support like concat nested concat statement rewrite with encrypt feature
- Pipeline: Add SHARDING_TOTAL_COUNT impl of JobExecutorServiceHandler to improve CPU core requirement
- Pipeline: Support page query for inventory dumper and data consistency streaming query
- Pipeline: Use case-insensitive identifiers to enhance the table metadata loader
- Pipeline: Support primary key columns ordering for standard pipeline table metadata loader
Bug Fixes
- DistSQL: Fix load single table error after creating logical data source
- DistSQL: Fix set default single table storage unit to random failed
- DistSQL: Fix set default single table storage unit to logical data source failed
- DistSQL: Fix NPE when import metadata
- SQL Parser: Support PostgreSQL do nothing conflict action
- SQL Parser: Fix oracle nvl function and interval hour parse error
- Metadata: Keep in-used storage node when unregister storage unit
- Metadata: Fixed the problem of missing storage unit when registering storage unit
- Kernel: Replace ThreadLocal with TransmittableThreadLocal in HintManager to support cross Thread usage
- Kernel: Use ConnectionPropertiesParser spi to parse jdbcUrl to solve oracle url parse error
- Kernel: Fix column count assert exception caused by postgresql system table judge
- SQL Binder: Add TableAvailable interface for CloseStatementContext, MoveStatementContext, FetchStatementContext and fix sql rewrite test case
- Transaction: Fix setSavepoint method invocation not cleaning up
- Transaction: Fix savepoint manager not cleaned up in distributed transactions
- Transaction: Fix transaction context not cleaned up when xa transaction is committed
- Transaction: Fix the issue that cursor is not rewritten
- SQL Federation: Fix federated query LocalDateTime conversion
- SQL Federation: Fix push down sql execute error when sql contains chinese character with sql federation
- SQL Federation: Fix federation query binary type data query
- SQL Federation: Fix null result in federated query for a single projected column
- SQL Federation: Fix sql federation unknown type exception caused by calcite wrong result type with bigint
- Proxy: Support binary type bool value in PostgreSQL
- Proxy: Support array type prepared param in PostgreSQL
- Proxy: Support binary protocol value for text array in PostgreSQL
- Proxy: Fix duplicate results when querying information_schema.SCHEMATA
- Proxy: Fix incorrect results for querying information_schema.SCHEMATA
- Proxy: Fix NPE when execute show table status
- Proxy: Fix no database selected exception occurs when show tables from db
- Proxy: Fix the error that the process does not exit after proxy startup fail
- Proxy: Fix the error that the persisted system metadata was not cleared after the database was deleted in the PostgreSQL scenario
- Proxy: Fix no database selected exception in the query SQL
- JDBC: Resolve the issue where "zip file closed" in versions prior to SpringBoot 2.3.
- JDBC: Fix the NPE when it does not contain a specified logic database in Driver.
- Sharding: Fix Sharding column not tracked through aliases
- Sharding: Fix alter view exception when config sharding rule and binding table rule
- Sharding: Fix is need accumulate logic
- Sharding: Fix the generated key column name case insensitivity error
- Encrypt: Fix show create table wrong result with encrypt when data type contains float
- Encrypt: Fix the issue where updating a non-encrypted table and using a subquery on an encrypted table.
- Encrypt: Add insert select rewrite for encrypt
- Pipeline: InventoryTaskSplitter compatible with
BigInteger
primary key - Pipeline: Fix default data source lost when create migration job
5.5.0
API Change
- Proxy: Remove the outdated schemaName configuration of the Proxy
- Kernel: Remove the sqlCommentParseEnabled configuration in SQL Parser Rule to reduce code complexity
- SQL Parser: Consider removing sqlCommentParseEnabled config in SQLParser rule
- Federation: Add allQueryUseSQLFederation config for sql federation
Bug Fix
- Pipeline: Fix commit/rollback migration job doesn't drop related consistency check job when check job is not completed
- Pipeline: Fix show consistency check status stop_time display
- Proxy: Restore original databaseName in connectionSession after unicast
- Proxy: Fix show tables can be executed without use database
- Proxy: Fix the incorrect current database after unicast routing
- Proxy: Fix the problem that show tables can be executed before use database
- Readwrite-splitting: Fix check exception when using shadow data source
- Governance: Fix thread blocking problem when create logic database for Etcd register center
- Governance:Fix register storage units and create feature rules failure when use Standalone mode
- MetaData: Fix database system schema is not built when create database
- Metadata: Fix NPE of system schema builder rule
- SQL Parser: Fix PostgreSQL NPE when parse columnRef
- SQL Parser: Fix npe cause by parse MySQL select window statement
- Sharding: Fix routing error when joining tables in uppercase
- Sharding: Fix drop sharding table exception when table name is uppercase
- Sharding: Fix generated key with upper case column name
- Federation: Fix Object 'DUAL' not found exception when execute select 1 from dual with sql federation
- Transaction: Fix xa auto commit in executeQuery
- Proxy: Fix multi statements with specified database name
Enhancement
- Pipeline: Improve CDC stability and performance
- Pipeline: MemoryPipelineChannel supports configurable zero queue size for less memory consumption
- Pipeline: Show dedicated error when mode type is not Cluster
- Authority: Add isSuper option for user
- Pipeline: Ignore error message in status DistSQL result on job cancelling
- DistSQL: Use JSON format to output props in RQL
- DistSQL: Optimize REFRESH DATABASE METADATA logic
- DistSQL: SHOW COMPUTE NODES supports JDBC nodes
- DistSQL: Optimize syntax of REFRESH DATABASE METADATA
- DistSQL: Optimize the output of SHOW TABLE METADATA
- Proxy: Add MySQL precompiled parameter verification to avoid turning on rewriteBatchedStatements=true when BenchmarkSQL connects to Proxy, causing an ArrayIndexOutOfBoundsException exception when the Proxy parameterCount exceeds 65535.
- SQL Parser: Add EOF to throw exception when parse distsql rollback migration statement
- SQL Parser: Support more plsql statement parse and add plsql parse assert logic
- SQL Parser: Support parse index hint
- SQL Parser: Support mysql intersect combine operation sql parse
- SQL Parser: Supoort parse chinese white space for oracle
- SQL Parser: Fix mysql TimeStampDiff function parse
- SQL Parser: Fix sqlServer unqualified shorthand parsing
- SQL Parser: Support sqlServer SEARCH unreserved word parsing
- SQL Parser: Add sql server MEMBER unreserved word
- Binder: Add ParameterMarkerSegmentBinder logic for Oracle MergeStatementBinder
- Federation: Support mysql cross join statement for sql federation
- Transaction: Add advice message in XATransactionCheckPrivilegeFailedException
- Sharding: Validate duplicate sharding actual data nodes
- Sharding: Support null condition value routing
- Proxy: Fix mysql client multi statements option in protocol
New Feature
- DistSQL: New syntax for query plugin (SPI) implementation
- DistSQL: New syntax for managing SQL_TRANSLATOR rule
Refactor
- JDBC: Merge shardingsphere-jdbc-core to shardingsphere-jdbc module
- Proxy: Rename server.yaml to global.yaml, config- to database- and keep compatible with old config
- Proxy: Rename
config-xxx.yaml
todatabase-xxx.yaml
- Proxy: Optimize MySQLComStmtPrepareExecutor logic
- Encrypt: Refactor encrypt interface and add encrypt algorithm metadata
- Infra: Create new algorithm module in infra and move snowflake, uuid to this module
- Infra: Modify unmarshal method to return a default object when yaml is empty
- Infra: Optimize StandardParameterBuilder logic
5.4.1
New Features
- Proxy: Supports dbcp and c3p0 connection pools
- Metadata: Standalone mode adapts to metadata new structure
- Governance: Governance supports register instance level data source
Enhancements
- Mode: Improve Standalone mode JDBC type impl reset data on initialization
- Scaling: Isolate ShardingSphereDataSource Standalone repository in pipeline
- Scaling: Disable system-schema-metadata-enabled in pipeline
- JDBC: Move jdbc core META-INF/services/java.sql.Driver from test to main
- Scaling: Add algorithm columns in SHOW MIGRATION CHECK ALGORITHMS DistSQL result
- Scaling: Add type_aliaes column in SHOW MIGRATION CHECK ALGORITHMS DistSQL result
- Encrypt: Add duplicate name check for derived columns and logical columns
- Encrypt: Deny DDL for cipher columns in Proxy
- Encrypt: Add the default type for derived columns to varchar(4000)
Bug Fixes
- Scaling: Fix get inventory position not correctly on breakpoint resuming when table names are similar
- Scaling: Fix CDC importer not start on breakpoint resuming when first inventory task is finished
- Single Table: Fix not switched active version when CREATE/DROP table modifies the Single rule configuration
- JDBC: Fix JDBC memory leak with MySQL in the 5.4.0 version
Change Log
5.4.0
API Changes
- Metadata: Change sharding broadcast tables to global broadcast tables
- JDBC: Remove exclamation mark (!) for global rules
- DistSQL: Simplify keywords ASSISTED_QUERY, LIKE_QUERY in encrypt DistSQL
- DistSQL: Optimize SQL_PARSER rule syntax
- Encrypt: Adjust encryption yaml API to distinguish between encrypt, like, and assisted query configurations
- Encrypt: Remove plain column and queryWithCipherColumn configuration in encrypt feature
- Readwrite-splitting: Refactor read/write splitting api
- Proxy: Remove property proxy-instance-type configuration
- Proxy: Remove property proxy-backend-executor-suitable
- Proxy: Remove property proxy-mysql-default-version
- Scaling: Refactor commit rollback streaming to drop streaming
- Sharding: Merge ShardingCacheRule into ShardingRule (Experimental)
New Features
- DistSQL: New syntax to manage SQL_FEDERATION rule
- Proxy: Support Unix Domain Socket
Enhancements
- Scaling: CDC supports pure incremental mode
- Scaling: CDC supports exporting data by transaction
- Scaling: CDC supports MySQL and PostgreSQL
- Scaling: CDC supports single table
- Scaling: CDC supports all openGauss data types
- Scaling: CDC supports replication reconnection
- Scaling: Remove DataConsistencyCalculateAlgorithmChooser
- Scaling: Improve performance of integer unique key table inventory data splitting
- Scaling: Adjust process configuration default value to reduce resource consumption
- Scaling: Auto refresh table metadata for migration
- Scaling: Compatible with openGauss existing replication slot reuse when database not existing
- Scaling: Show data consistency check status result should be empty when it's not completed
- Scaling: Enable concurrent CRC32 match on source and target
- Scaling: Pipeline job compatible with sharding rule audit strategy
- Metadata: Refactor metadata persistence structure
- Metadata: Optimize the process of loading single table metadata
- Metadata: Support MySQL/PostgreSQL/openGauss system tables empty query
- DistSQL: Add support for transactionalReadQueryStrategy for read/write splitting rule
- DistSQL: Enhanced algorithm properties check
- Transaction: Add privilege check
- Transaction: Remove the TransactionTypeHolder and only create the current transaction manager
- Parser: Support MySQL LOAD DATA and LOAD XML statement with single table or broadcast table
- Parser: Improve the parsing support of high-priority SQL statements in the test results of the MySQL test program
- Parser: Oracle dialect parser now supports Chinese comma
- Encrypt: Support query of encrypt column in projection subquery when use encrypt feature
- Kernel: Adds table existence metadata check for INSERT, DELETE, UPDATE and SELECT statements
- JDBC: Implement batch execution for ShardingSphereStatement
- Proxy: Frontend supports SSL/TLS
- Proxy: Support Flush message for PostgreSQL/openGauss Proxy
- Proxy: Support data type bit, bool for PostgreSQL Proxy
Bug Fixes
- Scaling: Fix pipeline job failure status persistence and usage
- Scaling: Fix CDC DELETE event Record.beforeList is null
- Scaling: Fix openGauss mppdb decoding plugin single quote issue
- Scaling: Fix execute engine not closed after job stopping
- Scaling: Fix stop job before task starting
- Metadata: Fix case sensitive issue when loading schema meta data with H2 database
- Metadata: Fix "object not found" exception when config PostgreSQL/openGauss schema name as database name
- DistSQL: Fix wrong result of check_table_metadata_enabled when execute SHOW DIST VARIABLE
- Encrypt: Fix SQL rewrite exception when use PostgreSQL/openGauss encrypt like feature
- Sharding: Support null sharding condition pass to sharding algorithm to allow user control null value route
- Parser: Support BETWEEN AND expression parsing in MySQL Projection
- Mask: Fix wrong mask result when config same value of from-x and to-y with KEEP_FROM_X_TO_Y
- Infra: Fix ClassNotFoundException may occur when missing pgjdbc
- Proxy: Fix MySQL packet out of order when client sending pipelining requests
Change Log
5.3.2
API Changes
- Proxy: Add property
system-log-level
, support dynamic change of log level by DistSQL. - DistSQL: Remove Hint-related DistSQL, users can use
SQL Hint
instead
New Features
- Scaling: Support any type of column unique key table
Enhancements
- Scaling: Use stream query for inventory dump and data consistency check
- Scaling: Compatible with
VARBINARY
column type in MySQL binlog parsing - Scaling: Refactor
AbstractSimplePipelineJob.execute
to blocking - Scaling: Improve table records count calculation
- Scaling: Support proxy sharding rule absent for migration job
- Scaling: Add
useServerPrepStmts=false
for pipeline job on MySQL - Scaling: Improve datetime/time parsing for MySQL binlog
- Scaling: Add global status for prepare stage
- Scaling: Add
netTimeoutForStreamingResults
for pipeline job on MySQL - Authority: Support specifying password authentication method
- Authority: Add md5 authentication support for openGauss protocol
- Agent: Add more metrics for JDBC
- Kernel: ShardingSphere Driver configuration supports Apollo
- Kernel: Adjust
SKIP_ENCRYPT_REWRITE SQL
Hint toSKIP_SQL_REWRITE
to support more scenarios - Kernel: Support openGauss
EXPLAIN PERFORMANCE
- Encrypt: Like supports concat function
Bug Fixes
- Scaling: Fix unicode char and special char decoding for PostgreSQL incremental task
- Scaling: Fix Migration not support PostgreSQL json type
- DistSQL:
CREATE SHARDING TABLE RULE
supportsNONE
strategy - Kernel: Fix use system database error when data sources are empty
- Kernel: Fix set worker-id does not take effect with Standalone mode
- Kernel: Clear storage node information when delete readwrite-splitting and database discovery rules
- Kernel: Fix the abnormal problem of Column index out of range in single table complex query
- Kernel: Fix PostgreSQL like lower case failed.
- Kernel: Fixed the exception of built-in metabase data collection when the front and back database types were inconsistent
- Kernel: Fix the problem of routing error reporting under certain table names
- Kernel: Fix MySQL create procedure parse error
- Kernel: Fix union extract table name NPE
- Kernel: Fix upper case table constraint not rewrite error
- Kernel: Fix failed to parse PostgreSQL / openGauss SQL contains money type
- Kernel: Fix PostgreSQL / openGauss positional parameter rule
- Kernel: Fix PostgreSQL / openGauss failed to parse const with type cast
- Kernel: Fix Chinese characters encode exception when execute select with sql federation engine
- Kernel: Fix
IndexOutOfBoundsException
when execute set variable statement int jdbc adapter - Kernel: Fix index does not exist exception when execute drop index statement
- Proxy: Properly handle number sign in MySQL Proxy binary protocol
- Proxy: Fix PostgreSQL Proxy failed to handle bytea data type
- Proxy: Fix PostgreSQL Proxy failed to parse time value with microseconds
- Proxy: Fix PostgreSQL protocol codec for date type in binary format
- Proxy: Fix possible CCE
PostgreSQLInt2BinaryProtocolValue
- Proxy: Fix possible error when client pass quoted charset to PostgreSQL/openGauss Proxy
Change Log
5.3.1
New Features
- Kernel: Add new data masking, dynamic data masking features, and built-in data masking algorithms
- Scaling: Basic support of CDC feature
- DistSQL: Add masking rule related DistSQL
Enhancements
- Kernel: Cluster mode avoids secondary refresh of metadata
- Kernel: SHOW COMPUTE NODES supports displaying the version number of each instance
- Kernel: System database add cluster information table
- Kernel: Standalone mode persistent metadata supports MySQL
- Kernel: SQL HINT performance improvement
- Kernel: Restore routing to the specified database with Hint
- Encrypt: Supports underscore wildcards for Encrypt CharDigestLikeEncryptAlgorithm
- Kernel: Support SQL federation SELECT NULLS LAST/FIRST statement
- Kernel: Refactor encrypt integration test logic and add more test cases
- Kernel: Add salt props for MD5MaskAlgorithm, MD5EncryptAlgorithm
- Kernel: Refactor ShardingConditionEngine to support SPI configuration
- DistSQL: Support defining column level
QUERY_WITH_CIPHER_COLUMN
whenCREATE ENCRYPT RULE
- DistSQL: Add algorithm type check for
CREATE SHARDING TABLE RULE
Bug Fixes
- Fix the problem of ZooKeeper cluster error reporting when ShardingSphere connects to Kubernetes
- Kernel: Fix use Consul in cluster mode start up failure
- DB Discovery: Close heartbeat job when drop discovery rule
- Kernel: Fix wrong decide result when execute same sharding condition subquery with SQL federation
- Kernel: Fix priority problem of UNION, INTERSECT, EXCEPT set operation in SQL Federation for PostgreSQL and openGuass dialect
- Kernel: Fix create view index out of range exception when view contains set operator
- Kernel: Add XA resource exceeds length check
- Kernel: Fix transaction support for spring requires_new
- Encrypt: Fix AESEncryptAlgorithm decrypt exception when config char type with PostgreSQL and openGauss
- Encrypt: Fix abnormal expansion result for shorthand when encrypt subquery contains an alias
- Kernel: Fix unsigned flag of column metadata was not serialized
- Kernel: Fix PostgreSQL / openGauss select fetch parsing issue to support federation execution engine
- Proxy: Fix packet sequence ID may be incorrect if error occurred in MySQL Proxy
- Proxy: Fix error occur in Proxy when using PostgreSQL composite type
- Proxy: Set proper column definition flag for MySQL COM_STMT_PREPARE
- Proxy: When querying PG metadata through Proxy and the result set is empty, the labels are lost
Change Log
5.3.0
API Changes
- DistSQL: Refactor syntax API, please refer to the user manual
- Proxy: Change the configuration style of global rule, remove the exclamation mark
- Proxy: Allow zero-configuration startup, enable the default account root/root when there is no Authority configuration
- Proxy: Remove the default logback.xml and use API initialization
- JDBC: Remove the Spring configuration and use Driver + YAML configuration instead
Enhancements
- DistSQL: New syntax REFRESH DATABASE METADATA, refresh logic database metadata
- Kernel: Support DistSQL REFRESH DATABASE METADATA to load configuration from the governance center and rebuild MetaDataContext
- Support postgresql/openGauss setting transaction isolation level
- Scaling: Increase inventory task progress update frequency
- Scaling: DATA_MATCH consistency check support breakpoint resume
- Scaling: Support drop consistency check job via DistSQL
- Scaling: Rename column from sharding_total_count to job_item_count in job list DistSQL response
- Scaling: Add sharding column in incremental task SQL to avoid broadcast routing
- Scaling: Sharding column could be updated when generating SQL
- Scaling: Improve column value reader for DATA_MATCH consistency check
- DistSQL: Encrypt DistSQL syntax optimization, support like query algorithm
- DistSQL: Add properties value check when REGISTER STORAGE UNIT
- DistSQL: Remove useless algorithms at the same time when DROP RULE
- DistSQL: EXPORT DATABASE CONFIGURATION supports broadcast tables
- DistSQL: REGISTER STORAGE UNIT supports heterogeneous data sources
- Encrypt: Support Encrypt LIKE feature
- Automatically start distributed transactions when executing DML statements across multiple shards
- Kernel: Support client \d for PostgreSQL and openGauss
- Kernel: Support select group by, order by statement when column contains null values
- Kernel: Support parse RETURNING clause of PostgreSQL/openGauss Insert
- Kernel: SQL HINT performance improvement
- Kernel: Support mysql case when then statement parse
- Kernel: Supporting data source level heterogeneous database gateway
- (Experimental) Sharding: Add sharding cache plugin
- Proxy: Support more PostgreSQL datetime formats
- Proxy: Support MySQL COM_RESET_CONNECTION
- Scaling: Improve MySQLBinlogEventType.valueOf to support unknown event type
- Kernel: Support case when for federation
Bug Fix
- Scaling: Fix barrier node created at job deletion
- Scaling: Fix part of columns value might be ignored in DATA_MATCH consistency check
- Scaling: Fix jdbc url parameters are not updated in consistency check
- Scaling: Fix tables sharding algorithm type INLINE is case-sensitive
- Scaling: Fix incremental task on MySQL require mysql system database permission
- Proxy: Fix the NPE when executing select SQL without storage node
- Proxy: Support DATABASE_PERMITTED permission verification in unicast scenarios
- Kernel: Fix the wrong value of worker-id in show compute nodes
- Kernel: Fix route error when the number of readable data sources and weight configurations of the Weight algorithm are not equal
- Kernel: Fix multiple groups of readwrite-splitting refer to the same load balancer name, and the load balancer fails problem
- Kernel: Fix can not disable and enable compute node problem
- JDBC: Fix data source is closed in ShardingSphereDriver cluster mode when startup problem
- Kernel: Fix wrong rewrite result when part of logical table name of the binding table is consistent with the actual table name, and some are inconsistent
- Kernel: Fix startup exception when use SpringBoot without configuring rules
- Encrypt: Fix null pointer exception when Encrypt value is null
- Kernel: Fix oracle parsing does not support varchar2 specified type
- Kernel: Fix serial flag judgment error within the transaction
- Kernel: Fix cursor fetch error caused by wasNull change
- Kernel: Fix alter transaction rule error when refresh metadata
- Encrypt: Fix EncryptRule cast to TransparentRule exception that occurs when the call procedure statement is executed in the Encrypt scenario
- Encrypt: Fix exception which caused by ExpressionProjection in shorthand projection
- Proxy: Fix PostgreSQL Proxy int2 negative value decoding incorrect
- Proxy: PostgreSQL/openGauss support describe insert returning clause
- Proxy: Fix gsql 3.0 may be stuck when connecting Proxy
- Proxy: Fix parameters are missed when checking SQL in Proxy backend
- Proxy: Enable MySQL Proxy to encode large packets
- Kernel: Fix oracle parse comment without whitespace error
- DistSQL: Fix show create table for encrypt table
Refactor
- Scaling: Reverse table name and column name when generating SQL if it's SQL keyword
- Scaling: Improve increamental task failure handling
- Kernel: Governance center node adjustment, unified hump to underscore
Change Log
5.2.1
New Feature
- Add ShardingSphere default system database to support global metadata management
- Support asynchronous data consistency check
- Added support for Consul governance center
- Added support for Nacos governance center
- Added support for the view function in the governance center
Enhancement
- SQL Federation engine adds ADVANCED executor and adapts to openGauss database
- Support ShardingSphere Proxy startup after read-write splitting read database is disabled
- SQL HINT supports force sharding route
- Show processlist supports showing Proxy connections (sleep, active)
- Optimized ShardingSphere-JDBC data source configuration error message
- ShardingSphere-JDBC supports SpringBoot 3.x version
- Support load MySQL, PostgreSQL, openGauss and SQLServer view metadata
- Update Snakeyaml to 1.33 and open YAML 3MB limit
- Reuse cached connections as possible when unicast sharding
- Support Parsing ALTER ROLE in Oracle
- Add support of ALTER RESOURCE COST for Oracle
- Support parsing Drop Materialized View in Oracle
- Support parsing DROP LIBRARY in Oracle
- Support parsing DROP JAVA in Oracle
- Support parsing DROP PLUGGABLE DATABASE in Oracle
- Support parsing DROP INDEX TYPE in Oracle
- Support Parsing ALTER PACKAGE in openGauss
- Support openGauss select offset, count statement parse and remove useless syntax in PostgreSQL grammar
- Add max_size to openGauss syntax
- Optimize alter view/drop view parse logic and fix alter view refresher bug
- Add sql parser error detail to ParseCancellationException
- Add support for parse OptOnConflict for postgres
- Enhance support for ALTER TABLE and ALTER VIEW in PostgreSQL
- Add a missing keyword in PostgreSQL Declare Statement
- Add json function support to mysql parser
- ShardingSphere-Proxy automatically adapts to cgroup memory limits in Docker environment
- Show migration status DistSQL respond new added error_message column
- Show migration status respond new added processed_records_count column
- Support MySQL 8 caching_sha2_password authentication in incremental dump
- Improve drop pipeline process configuration
- Support unique key table migration
- Support migrate table to new name table
- Improve thread pool usage in pipeline task and runner
- Support cancelable data consistency check
- DistSQL: When creating or altering readwrite-splitting rule, check duplicate write or read resources
- DistSQL: Add validity check for
ALTER SHARDING BINDING TABLE RULES
- Standalone mode H2 support persistent metadata
- Fix openGauss cursor execution in xa transaction
- Added transaction related exceptions
Bug Fix
- Generate proper placeholder for PostgreSQL when rewriting
- Fix opengauss update set parse exception
- Fix parse exception when execute insert statement with negative value
- Fix wrong connectDescriptorUrlPattern in OracleDataSourceMetaData
- Fix insert SQL garbled after sharding rewrote in special rules
- Fix exception when execute select * from information_schema.tables
- Fix exception when execute alter view rename
- Fix PostgreSQL check data source permission when using rolsuper
- DistSQL: fix NPE for
REFRESH TABLE METADATA
when there is no resource - Fix Unmodified table metadata when modify rules
- Fix database discovery
- The MySQL.NORMAL_REPLICATION algorithm cannot find primary node"
- Fixed using etcd to build a cluster event not aware
- Fix NPE occurred when transaction management is not created
- Fix routing exception that occurs when the sharding value of the InlineShardingAlgorithm algorithm exceeds Integer
API Changes
- SQL HINT syntax format adjust to SQL-style format
- DistSQL: Remove syntax
COUNT DATABASE RULES
- ShardingSphere mode remove overwrite configuration
- Agent: Optimize configuration of agent.yaml
Change Log
5.2.0
New Feature
- Support SQL audit for sharding feature
- Support MySQL show processlist and kill process list id feature
- Scaling: Add dedicated DistSQL for data migration
- Scaling: Basic support migrate data to heterogeneous database
- DistSQL: New syntax
CREATE MIGRATION PROCESS CONFIGURATION
- DistSQL: New syntax
ALTER MIGRATION PROCESS CONFIGURATION
- DistSQL: New syntax
SHOW MIGRATION PROCESS CONFIGURATION
- DistSQL: New syntax
ADD MIGRATION SOURCE RESOURCE
- DistSQL: New syntax
SHOW SQL_TRANSLATOR RULE
- DistSQL: New syntax
CREATE SHARDING AUDITOR
- DistSQL: New syntax
ALTER SHARDING AUDITOR
- DistSQL: New syntax
SHOW SHARDING AUDIT ALGORITHMS
Enhancement
- Support column visible feature for MySQL, Oracle, SQLServer and H2
- Support cartesian product configuration for read write splitting
- Support spring namespace and spring boot usage for sql translator
- Support JSR-310 Year and Month in IntervalShardingAlgorithm
- Support broadcast table update/delete limit statement
- Support create index on table(column) statement rewrite when config encrypt
- Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
- Support encrypt column rewrite when execute column is null in predicate
- Support encrypt show create table return logic columns
- Support create table with index statement rewrite when config encrypt
- Support PostgreSQL create operator statement parse
- Support PostgreSQL create materialized view statement parse
- Support PostgreSQL nested comments parse
- Support PostgreSQL alter subscription statement parse
- Support PostgreSQL create group statement parse
- Support PostgreSQL alter statictics statement parse
- Support PostgreSQL create foreign table statement parse
- Support PostgreSQL alter server statement parse
- Support PostgreSQL create foreign data wrapper statement parse
- Support PostgreSQL create event trigger statement parse
- Support PostgreSQL security label statement parse
- Support PostgreSQL reindex statement parse
- Support PostgreSQL reassign owned statement and refresh materialized view statement parse
- Support PostgreSQL prepare transaction statement parse
- Support PostgreSQL create collation statement parse
- Support PostgreSQL lock statement parse
- Support PostgreSQL alter rule statement parse
- Support PostgreSQL notify statement parse
- Support PostgreSQL unlisten statement parse
- Support Oracle alter function and alter hierarchy statement parse
- Support Oracle alter pluggable database statement parse
- Support Oracle alter materialized view log statement parse
- Support Oracle alter diskgroup statement parse
- Support Oracle alter operator statement parse
- Support oracle alter cluster statement parse
- Support oracle alter audit policy statement parse
- Support Oracle alter index type statement parse
- Support Oracle lock table statement parse
- Support Oracle alter java statement parse
- Support Oracle inline constraint statement parse
- Support openGauss geometric operator statement parse
- Optimize MySQL visible/invisible parse of create/alter table statements
- Support scope of variable prefixed with @@ in MySQL SET statement parse
- Support MySQL create procedure with create view parse
- Support column segments parse in create index on table statement
- Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
- Support encrypt column rewrite when execute column is null in predicate
- Support encrypt show create table return logic columns
- Support create table with index statement rewrite when config encrypt
- Support parsing ALTER LOCKDOWN PROFILE in Oracle
- Support parsing ALTER MATERIALIZED VIEW in Oracle
- Support parsing ALTER MATERIALIZED ZONEMAP in Oracle
- Support parsing ALTER LIBRARY in Oracle
- Support parsing ALTER INMEMORY JOIN GROUP in Oracle
- Support parsing DROP OPERATOR in Oracle
- Support parsing DROP RESTORE POINT in Oracle
- Support parsing CREATE RESTORE POINT in Oracle
- Support parsing DROP INMEMORY JOIN GROUP in Oracle
- Support parsing create_bit_xor_table in MySQL
- Support parsing MySQL DO statement
- Support parsing DropServer in openGauss
- Support parsing CREATE AGGREGATE In openGauss
- Support parsing ALTER ROUTINE in PostgreSQL
- Add PostgreSQL Create Cast Statement
- Add PostgreSQL Create Aggregate Statement
- Support fetch/move/close cursor statement in PostgreSQL
- Support Parsing ALTER PUBLICATION in PostgreSQL
- Add PostgreSQL Create Access Method Statement
- Support Parsing ALTER POLICY in PostgreSQL
- Support parsing ALTER OPERATOR in PostgreSQL
- Add PostgreSQL Copy Statement
- Add PostgreSQL Comment Statement
- Support listen statement in postgreSQL
- Support DECLARE cursor statement
- Add default serverConfig in helm charts
- Assemble openGauss JDBC Driver into Proxy distribution
- ShardingSphere-Proxy listen on specified IP addresses
- Support COM_STMT_SEND_LONG_DATA in MySQL Proxy
- SELECT VERSION() support alias in MySQL Proxy
- Fix openGauss Proxy could not be connected if no resource defined
- Support using JRE defined in JAVA_HOME in ShardingSphere-Proxy's startup script
- Avoid client blocked when OOM occurred in ShardingSphere-Proxy
- Support using c3p0 in ShardingSphere-JDBC
- Support SET NAMES with value quoted by double-quote
- Connection.prepareStatement with columns arguments is available in ShardingSphere-JDBC
- Scaling: Improve MySQL connect and reconnect
- Scaling: Fix MySQL json column may cause leak at incremental task
- Scaling: Add permission check for PostgreSQL data sources
- Scaling: Incremental migration support for MySQL MGR mode
- Scaling: Improve job progress persistence
- Scaling: Start job DistSQL execute and return synchronously
- Scaling: Inventory migration support table has primary key and unique key
- Scaling: Close unerlying ElasticJob when stopping job
- Scaling: Improve logical replication slot name generation for PostgreSQL and openGauss
- Scaling: Make query DistSQL could be executed when no database selected
- DistSQL: Add worker_id to the result set of
SHOW INSTANCE LIST
&SHOW INSTANCE INFO
- DistSQL: Improve the result of
EXPORT DATABASE CONFIG
- DistSQL: Support more databases for
FORMAT SQL
- DistSQL: Optimize the execution logic of
CREATE TRAFFIC RULE
- DistSQL: Add paramter
writeDataSourceQueryEnabled
for RDL READWRITE_SPLITTING RULE. - DistSQL: Support
assistEncryptor
for Encrypt RDL - DistSQL: Add sharding algorithm type check when
CREATE SHARDING TABLE RULE
- Support database discovery to configure multiple groups of high availability under the same logic database
- Support ShardingSphere-Proxy to start up under empty logic library
- Support for isolating EventBus events by instance
- Support the database to detect changes in the master node and restart the detection heartbeat task
- Support ShardingSphere-Proxy to generate new worker-id when re-registering in cluster mode
- Thrown exception when inserting expression value in shadow column on executing insert
- Support distributed transactions across multiple logical databases
- Support executing truncate in XA & PostgreSQL
- Support alter local transaction rule with DistSQL
- Support global transaction manager
- Delete support for branch transaction on proxy
Bug Fix
- Fix single table metadata refresh error caused by filtering DataSourceContainedRule
- Fix parsing exception caused by the null value of MySQL blob type
- Fix PostgreSQL/openGauss reset statement parse error
- Fix wrong parameter rewrite when use sharding and encrypt
- Fix the failed conversion of Month related classes on IntervalShardingAlgorithm
- Fix NullPointerException when execute select union statement contains subquery
- Fix wrong encrypt rewrite result due to incorrect order of metadata
- Fix MySQL trim function parse error
- Fix MySQL insert values with _binary parse error
- Fix MySQL syntax error cannot be thrown to client
- Avoid EventLoop blocked because of closing JDBC resources
- Correct server status flags returned by MySQL Proxy
- Fix a possible connection leak issue if Proxy client disconnected in transaction
- Fixed a possible consistency issue with the statement being executed when the Proxy client is disconnected
- Avoid pooled connection polluted by executing SET statements
- Make SHOW TABLES FROM work in ShardingSphere-Proxy
- Fix PostgreSQL DDL could not be executed by Extended Query
- Fix SHOW VARIABLES could not be executed in PostgreSQL Proxy without resource
- Fix FileNotFoundException when use ShardingSphere Driver with SpringBoot fatjar
- Scaling: Fix the problem that the table contains both primary key and unique index at inventory migration
- Scaling: Improve incremental migration, support the latest position in the middle of batch insert event
- Scaling: Fix the error caused by null field value in openGauss incremental migration
- DistSQL: Fix incorrect strategy name in result of
SHOW SHARDING TABLE RULES
- DistSQL: Fix current rule config is modified in advance when
ALTER SHARDING TABLE RULE
- DistSQL: Fix connection leak when
ALTER RESOURCE
- DistSQL: Fix
CREATE TRAFFIC RULE
failed when load balance algorithm is null - Fix that the monitoring heartbeat task was not stopped when the database was discovered and the logical library was deleted
- Fix cluster mode ShardingSphere-JDBC load all logic database
- Fix worker-id generated by SnowflakeKeyGenerateAlgorithm in cluster mode may exceed the maximum value
- Fix
DistSQL
adding shadow algorithm exception without shadow data source - Fix cross-database data source confusion caused by same data source name in multiple logical databases
1...
5.1.2
New Feature
- Kernel: Alpha version to support SQL dialect translate for MySQL and PostgreSQL
- Kernel: Support custom schema for PostgreSQL and openGauss
- Kernel: Support create/alter/drop view statement for PostgreSQL and openGauss
- Kernel: Support openGauss cursor statement
- Kernel: Support use customize system database
- Kernel: Support get openGauss and MySQL create SQL
- Kernel: Support get postgres create SQL
- Proxy: Official support for quickly deploying a ShardingSphere-Proxy cluster with a ZooKeeper cluster in Kubernetes using Helm
- JDBC: Support ShardingSphere JDBC Driver
- Scaling: Support PostgreSQL auto create table
- Scaling: Support scaling for table with customized schema in PostgreSQL and openGauss
- Scaling: Support scaling for table with text primary key and without integer primary key
- Mode: Registry center supports PG/openGauss three-level structure
- Mode: Registry center supports database-level distributed lock
Enhancement
- Kernel: Support copy statement for PostgreSQL and openGauss
- Kernel: Support alter/drop index statement for PostgreSQL
- Kernel: Support update force index statement for MySQL
- Kernel: Support create/alter/drop schema for openGauss
- Kernel: Optimize RoundRobinReplicaLoadBalanceAlgorithm and RoundRobinTrafficLoadBalanceAlgorithm logic
- Kernel: Optimize metadata loading logic when frontendDatabaseType and backendDatabaseType are different
- Kernel: Refactor meta data load logic
- Kernel: Optimize show processlist statement
- Kernel: Improve performance about large tables loaded
- Kernel: Support execute comment statement
- Kernel: Support view in sharding rule
- Kernel: Support parsing CREATE ROLLBACK SEGMENT in Oracle
- Kernel: Support Parsing DROP TYPE in openGauss
- Kernel: Support Parsing ALTER TYPE in openGauss
- Kernel: Support parsing DROP DISKGROUP in Oracle
- Kernel: Support parsing CREATE DISKGROUP in Oracle
- Kernel: Support parsing DROP FLASHBACK ARCHIVE in Oracle
- Kernel: Support Parsing CHECKPOINT in openGauss
- Kernel: Support parsing CREATE FLASHBACK ARCHIVE in Oracle
- Kernel: Add PostgreSQL Close Statement
- Kernel: Support Parsing DROP CAST in openGauss
- Kernel: Support parsing CREATE CAST in openGauss
- Kernel: Support parsing CREATE CONTROL FILE in Oracle
- Kernel: Support Parsing DROP DIRECTORY in openGauss
- Kernel: Support parsing ALTER DIRECTORY in openGauss
- Kernel: Support parsing CREATE DIRECTORY in openGauss
- Kernel: Add PostgreSQL Checkpoint Statement
- Kernel: Support parsing DROP SYNONYM in openGauss
- Kernel: Support parsing CREATE SYNONYM in openGauss
- Kernel: Support parsing ALTER SYNONYM in openGauss
- Kernel: Add PostgreSQL CALL Statement
- Kernel: Support parsing CREATE PFILE in Oracle
- Kernel: Support parsing CREATE SPFILE in Oracle
- Kernel: Support parsing ALTER SEQUENCE in Oracle
- Kernel: Support parsing CREATE CONTEXT in Oracle
- Kernel: Support Parsing ALTER PACKAGE in oracle
- Kernel: Support parsing CREATE SEQUENCE in Oracle
- Kernel: Support parsing ALTER ATTRIBUTE DIMENSION in Oracle
- Kernel: Support parsing ALTER ANALYTIC VIEW in Oracle
- Kernel: Use ShardingSphere SPI in SQLVisitorFacade
- Kernel: Use ShardingSphere SPI in DatabaseTypedSQLParserFacade
- Kernel: Support parsing ALTER OUTLINE in Oracle
- Kernel: Support parsing DROP OUTLINE in Oracle
- Kernel: Support parsing drop edition in oracle
- Kernel: Support WITH Common Table Expression of SQLServer
- Kernel: Exclude parenthesis from SubquerySegment's start and stop index in withClause
- Kernel: Refactor JoinTableSegment
- Kernel: Support parsing DROP SYNONYM in Oracle
- Kernel: Support parsing CREATE DIRECTORY in Oracle
- Kernel: Support parsing CREATE SYNONYM in Oracle
- Kernel: Support for XmlNamespaces Clause of SQLServer SELECT Statement
- Kernel: Support parsing Alter Database Dictionary in Oracle
- Kernel: Support FOR Clause of SQLServer SELECT Statement
- Kernel: Support Parsing ALTER DATABASE LINK in Oracle
- Kernel: Support CREATE EDITION Parsing in Oracle
- Kernel: Support parsing ALTER TRIGGER in Oracle
- Kernel: Add SQLServer REVERT Statement
- Kernel: Support Parsing DROP TEXT SEARCH in PostgreSQL
- Kernel: Add drop server for PostgreSQL
- Kernel: Support Parsing ALTER VIEW in Oracle
- Kernel: Add drop access method for PostgreSQL
- Kernel: Support Parsing DROP ROUTINE in PostgreSQL
- Kernel: Proofread SQLServer DROP USER Statement
- Kernel: Support parsing DROP TRIGGER in Oracle
- Kernel: Support parsing Drop subscription in PostgreSQL
- Kernel: Add drop operator class for PostgreSQL
- Kernel: Support parsing DROP PUBLICATION in PostgreSQL
- Kernel: Support Parsing DROP VIEW in Oracle
- Kernel: Support Parsing DROP TRIGGER in PostgreSQL
- Kernel: Support Parsing DROP DIRECTORY in Oracle
- Kernel: Support Parsing DROP STATISTICS for PostgreSQL
- Kernel: Add drop type SQL parser for PostgreSQL
- Kernel: Support Parsing DROP RULE in PostgreSQL
- Kernel: Proofread SQLServer ALTER LOGIN Statement
- Kernel: Support parsing PostgreSQL DROP FOREIGN DATA WRAPPER
- Kernel: Small changes to PostgreSQL DROP EVENT TRIGGER statement
- Proxy: ShardingSphere-Proxy MySQL supports receiving MySQL packet more than 16 MB
- Proxy: Supports netty parameter ChannelOption.SO_BACKLOG configurable in ShardingSphere-Proxy
- Proxy: Optimize so-reuseaddr in netty to solve the problem of port occupied
- Proxy: Docker image of ShardingSphere-Proxy supports aarch64 platform
- Proxy: Make server version configurable in ShardingSphere-Proxy MySQL
- Proxy: Supports more character sets in ShardingSphere-Proxy PostgreSQL/openGauss
- Proxy: Make default port configurable in ShardingSphere-Proxy
- Scaling: Compatible with HA ports for openGauss:3.0 when thread_pool enabled
- Scaling: Optimize ZooKeeper event handling in PipelineJobExecutor to avoid blocking ZooKeeper events
- Scaling: Make table name case-insensitive in whole process
- Scaling: Improve replication slot cleanup for PostgreSQL and openGauss
- Scaling: Improve lock protection for job preparation
- Scaling: Support PostgreSQL insert on conflict do update
- Scaling: Do not cache data source in GlobalDataSourceRegistry to avoid possible shared resource close issue
- Scaling: Reuse data source pool as more as possible to reduce working database connections
- DistSQL:
REFRESH TABLE METADATA
supports specifying PostgreSQL's schema - DistSQL:
ALTER SHARDING TABLE RULE
add validation of binding table - Mode: ShardingSphere-JDBC supports configuring database connection name
- Distributed Transaction: DistSQL is prohibited from executing within a transaction
- Distributed Transaction: autocommit = 0, DDL part of DML will automatically open the transaction
Bug Fix
- Kernel: Fix parsing error about show statement for PostgreSQL and openGauss
- Kernel: Fix parsing error about time extract function for for PostgreSQL and openGauss
- Kernel: Fix parsing error about select mod function for for PostgreSQL and openGauss
- Kernel: Fix PSQLException when execute join statement with multi schema in readwrite scenario
- Kernel: Fix wrong route result when execute create schema statement in encrypt scenario
- Kernel: Fix npe when execute drop schema if exist statement
- Kernel: Fix wrong route result when execute SELECT LAST_INSERT_ID() AS id; statement
- Kernel: Fix npe when execute use database when database doesn't contains datasource
- Kernel: Fix create function with set var
- Proxy: Fix NPE caused by column's case unmatched in PostgreSQLComDescribeExecutor
- Proxy: Complete command tags for schema DDL in ShardingSphere-Proxy PostgreSQL / openGauss
- Scaling: Fix MySQL unsigned type null value cause error during increment task
- Scaling: Fix resource leak caused by error occurred when creating DataSource in ShardingSphere-Scaling
- Scaling: Fix ShardingSphereDataSource creation ignoring other rules
- Scaling: Fix on preparation job could not be stopped
- Scaling: Fix data source property url and jdbcUrl compatibility
- Scaling: Fix openGauss logical replication slot creation, avoid possible incremental data loss
- Scaling: Update local job status before persisting job status to registry center, make sure it won't be overwritten later
- Scaling: Handling null value in TestDecodingPlugin for PostgreSQL
- DistSQL: Fix
SET VARIABLE
modification not taking effect in stand-alone and memory mode - DistSQL: Fix the inconsistency between
SHOW INSTANCE LIST
display data and actual data - DistSQL: Fix capitalization sensitivity in sharding DistSQL
- Mode: Fix the new version metadata lost data after the Scaling changes the table sharding rules
- Distributed Transaction: Fix getIndexInfo with catalog
API Changes
- DistSQL: Change
EXPORT SCHEMA CONFIG
toEXPORT DATABASE CONFIG
- DistSQL: Change
IMPORT SCHEMA CONFIG
toIMPORT DATABASE CONFIG
- DistSQL: Change
SHOW SCHEMA RESOURCES
toSHOW DATABASE RESOURCES
- DistSQL: Change
COUNT SCHEMA RULES
toCOUNT DATABASE RULES
- Mode: Adjust db-discovery algorithm configuration
- Authority: Authority provider
ALL_PRIVILEGES_PERMITTED
updated toALL_PERMITTED
- Authority: Authority provider
SCHEMA_PRIVILEGES_PERMITTED
updated toDATABASE_PERMITTED
Refactor
- Scaling: Refactor JobConfiguration, prepare for different types of jobs reuse and extension
- Mode: Optimize compute node structure of the registry center
- Mode: Use uuid instead of ip@port as instance id