[chore](be) Add default timezone files #25097

zclllyybb · 2023-10-07T20:25:05Z

Proposed changes

Issue Number: close #xxx

It's about 600kb and 7.5m after unzipping.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-10-07T20:32:33Z

clang-tidy review says "All clean, LGTM! 👍"

zclllyybb · 2023-10-07T20:33:35Z

run buildall

github-actions · 2023-10-07T20:41:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-07T21:35:39Z

TeamCity be ut coverage result:
Function Coverage: 36.37% (8167/22458)
Line Coverage: 28.52% (65399/229326)
Region Coverage: 27.41% (33845/123470)
Branch Coverage: 24.09% (17256/71646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6f5d6a39545cbba6e9e4e11cdde1862a0d47dd96_6f5d6a39545cbba6e9e4e11cdde1862a0d47dd96/report/index.html

doris-robot · 2023-10-07T21:36:17Z

TeamCity be ut coverage result:
Function Coverage: 36.36% (8165/22458)
Line Coverage: 28.51% (65378/229326)
Region Coverage: 27.41% (33842/123470)
Branch Coverage: 24.08% (17251/71646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6f5d6a39545cbba6e9e4e11cdde1862a0d47dd96_6f5d6a39545cbba6e9e4e11cdde1862a0d47dd96/report/index.html

doris-robot · 2023-10-07T21:54:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.88 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162205296 Bytes

doris-robot · 2023-10-07T21:55:38Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.98 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162227370 Bytes

zclllyybb · 2023-10-11T18:14:43Z

run buildall

github-actions · 2023-10-11T18:22:23Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-11T19:26:15Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (8170/22500)
Line Coverage: 28.46% (65443/229965)
Region Coverage: 27.36% (33897/123898)
Branch Coverage: 24.05% (17267/71800)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9a38758ea5f18101d81d6ff99bf88c172a686f02_9a38758ea5f18101d81d6ff99bf88c172a686f02/report/index.html

doris-robot · 2023-10-11T19:41:25Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.77 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162129213 Bytes

HappenLee

LGTM

github-actions · 2023-10-12T03:34:28Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-12T03:34:30Z

PR approved by anyone and no changes requested.

build.sh

zclllyybb · 2023-10-14T18:46:07Z

run buildall

github-actions · 2023-10-14T18:54:17Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-14T20:00:46Z

TeamCity be ut coverage result:
Function Coverage: 36.72% (8246/22456)
Line Coverage: 28.86% (66139/229134)
Region Coverage: 27.57% (34300/124420)
Branch Coverage: 24.26% (17456/71952)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0b0c1a8e8b8ffda5b725d47eb9e3820450042b2f_0b0c1a8e8b8ffda5b725d47eb9e3820450042b2f/report/index.html

github-actions · 2023-10-20T04:53:10Z

PR approved by at least one committer and no changes requested.

Gabriel39

LGTM

CalvinKirs · 2024-01-18T06:46:58Z

I would say no, could you provide a license for this package? Also, we should not put it in the source package, or at least the source package in the released version should not include the binary package.

This reverts commit 9a675fc.

…ezone parsing (#37062) 1. revert #25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

…ezone parsing (apache#37062) 1. revert apache#25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

…imezone parsing (#37062) (#37269) pick #37062 1. revert #25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

…ezone parsing (#37062) 1. revert #25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

…imezone parsing (#37062) (#37270) pick #37062 1. revert #25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

…imezone parsing (apache#37062) (apache#37270) pick apache#37062 1. revert apache#25097. we decide to rely on OS. not maintain independent tzdata anymore to keep result consistency 2. refactor timezone load. removed rwlock. before: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (6.88 sec) ``` now: ```sql mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates; +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ | 16000000 | 16000000 | +-------------------------------------------------------------------------------------+--------------------------------------------------------+ 1 row in set (2.61 sec) ``` 3. now don't support timezone offset format string like 'UTC+8', like we already said in https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage 4. support case-insensitive timezone parsing in nereids. 5. a bug when parse timezone using nereids. should check DST by input, but wrongly by now before. now fixed. doc pr: apache/doris-website#810

zclllyybb force-pushed the fix_ut branch from 991b94e to 6f5d6a3 Compare October 7, 2023 20:32

zclllyybb force-pushed the fix_ut branch from 6f5d6a3 to 9a38758 Compare October 11, 2023 18:13

HappenLee previously approved these changes Oct 12, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 12, 2023

github-actions bot added the reviewed label Oct 12, 2023

yiguolei reviewed Oct 13, 2023

View reviewed changes

build.sh Outdated Show resolved Hide resolved

yiguolei reviewed Oct 13, 2023

View reviewed changes

build.sh Outdated Show resolved Hide resolved

zclllyybb added 3 commits October 15, 2023 02:45

all files

187ceed

packed to tar.gz

4e55d9b

fix

0b0c1a8

zclllyybb dismissed HappenLee’s stale review via 0b0c1a8 October 14, 2023 18:45

zclllyybb force-pushed the fix_ut branch from 9a38758 to 0b0c1a8 Compare October 14, 2023 18:45

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 14, 2023

Merge branch 'master' into fix_ut

68c1641

BiteTheDDDDt approved these changes Oct 20, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2023

Gabriel39 approved these changes Oct 20, 2023

View reviewed changes

Gabriel39 merged commit 9a675fc into apache:master Oct 20, 2023
3 of 4 checks passed

zclllyybb deleted the fix_ut branch October 20, 2023 05:13

xiaokang added dev/2.0.3 merge_conflict labels Oct 20, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023

[chore](be) Add default timezone files (apache#25097)

47073a4

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Nov 1, 2023

[chore](be) Add default timezone files (apache#25097)

80a659f

zclllyybb mentioned this pull request Nov 1, 2023

[branch-2.0](pick) pick "Add default timezone files (#25097)" #26255

Merged

xiaokang pushed a commit that referenced this pull request Nov 2, 2023

[chore](be) Add default timezone files (#25097) (#26255)

63a7e41

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 2, 2023

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023

[chore](be) Add default timezone files (apache#25097) (apache#26255)

81aad7e

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[chore](be) Add default timezone files (apache#25097)

9196a0f

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jul 1, 2024

Revert "[chore](be) Add default timezone files (apache#25097)"

d5b1edc

This reverts commit 9a675fc.

zclllyybb mentioned this pull request Jul 1, 2024

[Refactor](timezone) refactor tzdata load to accelerate and unify timezone parsing #37062

Merged

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jul 1, 2024

Revert "[chore](be) Add default timezone files (apache#25097)"

8434889

This reverts commit 9a675fc.

zclllyybb added a commit to zclllyybb/doris that referenced this pull request Jul 2, 2024

Revert "[chore](be) Add default timezone files (apache#25097)"

4d6e181

This reverts commit 9a675fc.

This was referenced Jul 4, 2024

[branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) #37269

Merged

[branch-2.0](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) #37270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore](be) Add default timezone files #25097

[chore](be) Add default timezone files #25097

zclllyybb commented Oct 7, 2023 •

edited

Loading

github-actions bot commented Oct 7, 2023

zclllyybb commented Oct 7, 2023

github-actions bot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

zclllyybb commented Oct 11, 2023

github-actions bot commented Oct 11, 2023

doris-robot commented Oct 11, 2023

doris-robot commented Oct 11, 2023

HappenLee left a comment

github-actions bot commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

zclllyybb commented Oct 14, 2023

github-actions bot commented Oct 14, 2023

doris-robot commented Oct 14, 2023

github-actions bot commented Oct 20, 2023

Gabriel39 left a comment

CalvinKirs commented Jan 18, 2024

[chore](be) Add default timezone files #25097

[chore](be) Add default timezone files #25097

Conversation

zclllyybb commented Oct 7, 2023 • edited Loading

Proposed changes

Further comments

github-actions bot commented Oct 7, 2023

zclllyybb commented Oct 7, 2023

github-actions bot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

doris-robot commented Oct 7, 2023

zclllyybb commented Oct 11, 2023

github-actions bot commented Oct 11, 2023

doris-robot commented Oct 11, 2023

doris-robot commented Oct 11, 2023

HappenLee left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 12, 2023

github-actions bot commented Oct 12, 2023

zclllyybb commented Oct 14, 2023

github-actions bot commented Oct 14, 2023

doris-robot commented Oct 14, 2023

github-actions bot commented Oct 20, 2023

Gabriel39 left a comment

Choose a reason for hiding this comment

CalvinKirs commented Jan 18, 2024

zclllyybb commented Oct 7, 2023 •

edited

Loading