Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

table: use golang version zstd to replace the old one to make badger and unistore portable #159

Merged
merged 2 commits into from
Jan 3, 2020

Conversation

jackysp
Copy link
Member

@jackysp jackysp commented Dec 30, 2019

Signed-off-by: Shuaipeng Yu jackysp@gmail.com
PTAL @coocood

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>
@coocood
Copy link
Member

coocood commented Dec 30, 2019

Have you run go test?

@bobotu
Copy link

bobotu commented Dec 31, 2019

In addition to ZSTD, the following issues prevent us from allowing running unit tests on Windows:

  1. The os.OpenFile miss a flag on Windows platform, make the behavior of delete an opened file different with POSIX environment. (See proposal: os: Create/Open/OpenFile() set FILE_SHARE_DELETE on windows golang/go#32088)
  2. ResourceManager and blobGCHandler doesn't close and free resources after DB.Close, so some test case will meet The process cannot access the file because it is being used by another process, The requested operation cannot be performed on a file with a user-mapped section open when reopen DB
  3. DirectWriter.Finish try to truncate the file to actual size, but Windows doesn't allow doing this on an O_DIRECT file, the returned error is truncate: The parameter is incorrect. Fortunately, this is easy to fix, we can open this file in normal mode and do truncate. (See https://docs.microsoft.com/zh-cn/windows/win32/fileio/file-buffering#alignment-and-file-access-requirements)
  4. We should use path/filepath instead of path to work on file's path.
  5. Seems there are some memory-leak in some test case. I have got OOM on my laptop.

@bobotu
Copy link

bobotu commented Dec 31, 2019

BTW, it seems like some workers will terminate when a filesystem error occurred, but these workers will never start again and the database becomes unavailable.

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>
@jackysp jackysp changed the title table: remove zstd to make badger and unistore portable table: use golang implement zstd to replace the old one to make badger and unistore portable Dec 31, 2019
@jackysp jackysp changed the title table: use golang implement zstd to replace the old one to make badger and unistore portable table: use golang version zstd to replace the old one to make badger and unistore portable Dec 31, 2019
@coocood coocood merged commit 0783c47 into pingcap:master Jan 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants