Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[#1127] YSQL: Collation Support (part 2)
Summary:
Currently YSQL has very limited support for collation. Only C collation is
supported. Collation is about text data sorting rules and postgres provides
collation support via the underlining OS library including libc and libicu.
Simply put, given two text values (aka strings) s1 and s2, collation sort
comparison must be done via strcoll(s1, s2) rather than strcmp(s1, s2).
Postgres supports two types of collations: deterministic collations and
non-deterministic collations. Postgres 11.2 only supports deterministic
collations. Postgres 12 and beyond support both deterministic collations and
non-deterministic collations. Non-deterministic collation comparison of s1 and
s2 directly translates to strcoll(s1, s2), and deterministic collation
comparison uses strcmp(s1, s2) as a "tie-breaker" when strcoll(s1, s2) == 0.
This is the second change to support deterministic collation in YSQL to bring
YSQL collation support in sync with PG 11.2. With this change, --with-icu is now
always on when building postgres so that ICU code is always built into postgres.
The normal build are not expected to have any behavior change and YSQL will
continue to report collation related errors. We'll produce a special build with
ICU enabled via a new gflag: FLAGS_TEST_pg_collation_enabled=true. This gflag
is used during development to enable collation support. However, once collation
encoded data lands in docdb, turning off this gflag is not supported.
The plan is that once we fully support deterministic collations in YSQL, we'll
turn on FLAGS_TEST_pg_collation_enabled=true by default.
The high level changes are:
(1) In addition to type and modifier, for character data types that YSQL
expects, collation related information is now also passed from postgres
layer to PgGate when constructing any PgExpr. Collation info currently
includes:
```
typedef struct PgCollationInfo {
bool collate_is_valid_non_c;
const char *sortkey;
} YBCPgCollationInfo;
```
collate_is_valid_non_c indicates whether the collation is a valid non-C
collation. A non-C collation requires strcoll semantics. In contrast C collation
only requires strcmp semantics which is much more efficient.
sortkey is the collation sortkey that will be stored in docdb and used for
byte-wise comparison. It is null-terminated.
(2) When building PgExpr, collate_is_valid_non_c is recorded as
'collate_is_valid_non_c_' in PgExpr.
(3) When building PgConstant, sortkey is recorded as collation_sortkey_ in
PgConstant.
(4) For C collation PgConstant, collation_sortkey_ should be nullptr and we
continue to store the string value that postgres layer has passed down as we
currently do. In this way we try to minimize the performance penalty for text
data that do not use collation.
(5) For non-C collation PgConstant, we store a collation-encoded string value.
which is composed of a fixed leading bytes, followed by a null-terminated sort
key, followed by the original character string value.
(6) A sort key of a string value is computed such that strcmp(sortkey(s1),
sortkey(s2)) == strcoll(s1, s2). Two different s1 and s2 may have identical sort
keys. For deterministic collations, we need to use strcmp(s1, s2) as a
tie-breaker when strcmp(sortkey(s1), sortkey(s2)) == 0. That's why the original
string value is appended after the null-terminated sort key to complete a
collation-encoded string.
(7) The collation-encoded string value replaces the original value and is sent
to DocDB for storage.
(8) On the way back from DocDB, PgGate will use the null byte that separates the
sort key from the original string value to strip off the leading bytes and the
sort key, only hands the original string value over to postgres layer above.
(9) A new test only C constant kTestOnlyUseOSDefaultCollation is used for
testing purpose. It can only be set when FLAGS_pg_collation_enabled is true
(otherwise assertion fails). Once set to true, a clean build is needed which
includes initdb and CreateInitialSysCatalogSnapshot. All the initial databases
will have text columns default to en_US.UTF-8 collation. Therefore in regression
tests all text columns in user tables will also have en_US.UTF-8 collation.
Because this is a non-C collation, the new code path that stores
collation-encoded strings is tested.
(10) Planed optimization: store only the original value for a non-key column to
save storage space, also for hash key, consider store only the original value as
well if feasible.
(11) Known issue: a sort key is a null-terminated byte sequence (without any
embedded \0 byte). As a result the collation encoded string may contain invalid
UTF-8 characters. However it is set as a QLValue string field in place of the
original string value which is UTF-8. Protobuf reports invalid UTF-8 as an ERROR
even though the collation-encoded string is still sent across the wire without
any loss.
Test Plan:
1. Do default build and run regression tests. Verify:
(1.1) pg_collation has only 4 rows as before.
```
yugabyte=# select * from pg_collation;
select * from pg_collation;
collname | collnamespace | collowner | collprovider | collencoding |
collcollate
| collctype | collversion
-----------+---------------+-----------+--------------+--------------+-------------
+-----------+-------------
default | 11 | 10 | d | -1 |
| |
C | 11 | 10 | c | -1 | C
| C |
POSIX | 11 | 10 | c | -1 | POSIX
| POSIX |
ucs_basic | 11 | 10 | c | 6 | C
| C |
(4 rows)
```
(1.2) We continue to report collation related errors.
```
yugabyte=# create collation nd (provider = 'libc', locale='');
create collation nd (provider = 'libc', locale='');
ERROR: CREATE COLLATION not supported yet
LINE 1: create collation nd (provider = 'libc', locale='');
^
HINT: Please report the issue on https://github.com/YugaByte/yugabyte-db/issues
yugabyte=# create table foo(id text collate "C");
create table foo(id text collate "C");
ERROR: COLLATE not supported yet
LINE 1: create table foo(id text collate "C");
^
HINT: See #1127. Click '+' on the
description to raise its priority
```
(1.3) All initial databases have collation "C" as expected.
```
yugabyte=# select datname, encoding, datcollate from pg_database;
select datname, encoding, datcollate from pg_database;
datname | encoding | datcollate
-----------------+----------+------------
template1 | 6 | C
template0 | 6 | C
postgres | 6 | C
yugabyte | 6 | C
system_platform | 6 | C
(5 rows)
```
2. Do build with FLAGS_TEST_pg_collation_enabled=true, also in
YBIsCollationEnabled set default env value of
YBCIsEnvVarTrueWithDefault("FLAGS_TEST_pg_collation_enabled" to true,
run regression tests. Verify:
(2.1) pg_collation has many rows because it has imported many ICU collations
from OS provided libicu library.
```
yugabyte=# select * from pg_collation limit 10;
select * from pg_collation limit 10;
collname | collnamespace | collowner | collprovider | collencoding |
collcollat
e | collctype | collversion
-------------+---------------+-----------+--------------+--------------+-----------
--+------------+-------------
default | 11 | 10 | d | -1 |
| |
C | 11 | 10 | c | -1 | C
| C |
POSIX | 11 | 10 | c | -1 | POSIX
| POSIX |
ucs_basic | 11 | 10 | c | 6 | C
| C |
en_US.utf8 | 11 | 10 | c | 6 |
en_US.utf8
| en_US.utf8 |
en_US | 11 | 10 | c | 6 |
en_US.utf8
| en_US.utf8 |
und-x-icu | 11 | 10 | i | -1 | und
| und | 153.14
af-x-icu | 11 | 10 | i | -1 | af
| af | 153.14.37
af-NA-x-icu | 11 | 10 | i | -1 | af-NA
| af-NA | 153.14.37
af-ZA-x-icu | 11 | 10 | i | -1 | af-ZA
| af-ZA | 153.14.37
(10 rows)
yugabyte=# select count(*) from pg_collation;
select count(*) from pg_collation;
count
-------
789
(1 row)
```
(2.2) All initial databases have collation "C" as expected.
```
yugabyte=# select datname, encoding, datcollate from pg_database;
select datname, encoding, datcollate from pg_database;
datname | encoding | datcollate
-----------------+----------+------------
template1 | 6 | C
template0 | 6 | C
postgres | 6 | C
yugabyte | 6 | C
system_platform | 6 | C
(5 rows)
```
(2.3) We can now create collation and also support column collation.
```
yugabyte=# create collation nd (provider = 'libc', locale='');
create collation nd (provider = 'libc', locale='');
CREATE COLLATION
yugabyte=# create table foo(id text collate "C");
create table foo(id text collate "C");
CREATE TABLE
yugabyte=# create table bar(id text collate "en_US.utf8");
create table bar(id text collate "en_US.utf8");
```
(2.4) We can see collation "C" and "en_US.utf8" sort 'a' and 'A' differently
which is consistent with postgres.
```
yugabyte=# insert into foo values ('a');
insert into foo values ('a');
INSERT 0 1
yugabyte=# insert into foo values ('A');
insert into foo values ('A');
INSERT 0 1
yugabyte=# select id from foo order by id;
select id from foo order by id;
id
----
A
a
(2 rows)
yugabyte=# insert into bar values ('a');
insert into bar values ('a');
INSERT 0 1
yugabyte=# insert into bar values ('A');
insert into bar values ('A');
INSERT 0 1
yugabyte=# select id from bar order by id;
select id from bar order by id;
id
----
a
A
(2 rows)
```
3. Do build with same gflags settings as 2, also set
kTestOnlyUseOSDefaultCollation=true and run regression tests. Verify:
(3.1) Same as (2.1)
(3.2) All initial databases have collation "en_US.UTF-8".
```
yugabyte=# select datname, encoding, datcollate from pg_database;
select datname, encoding, datcollate from pg_database;
datname | encoding | datcollate
-----------------+----------+-------------
template1 | 6 | en_US.UTF-8
template0 | 6 | en_US.UTF-8
postgres | 6 | en_US.UTF-8
yugabyte | 6 | en_US.UTF-8
system_platform | 6 | en_US.UTF-8
(5 rows)
```
(3.3) Same as (2.3)
(3.4) Because "en_US.UTF-8" is the default collation, a simple text column will
use it implicitly and will be sorted according to collation "en_US.UTF-8".
```
yugabyte=# create table text_tab (id text);
create table text_tab (id text);
CREATE TABLE
yugabyte=# insert into text_tab values ('a');
insert into text_tab values ('a');
INSERT 0 1
yugabyte=# insert into text_tab values ('A');
insert into text_tab values ('A');
INSERT 0 1
yugabyte=# select id from text_tab order by id;
select id from text_tab order by id;
id
----
a
A
(2 rows)
```
(3.5) Explicitly specified collation "C" will be sorted according to collation
"C".
```
yugabyte=# create table text_tab2(id text collate "C");
create table text_tab2(id text collate "C");
CREATE TABLE
yugabyte=# insert into text_tab2 values ('a');
insert into text_tab2 values ('a');
INSERT 0 1
yugabyte=# insert into text_tab2 values ('A');
insert into text_tab2 values ('A');
INSERT 0 1
yugabyte=# select id from text_tab2 order by id;
select id from text_tab2 order by id;
id
----
A
a
(2 rows)
```
Reviewers: mihnea, mbautin, neil, dmitry
Reviewed By: dmitry
Subscribers: yql
Differential Revision: https://phabricator.dev.yugabyte.com/D12330- Loading branch information
