Skip to content

Commit

Permalink
Revert "Migrate to zlib-ng, part 3: Remove zlib and zlib-intel source…
Browse files Browse the repository at this point in the history
… code an…" (#105284)

This reverts commit 4d2d3d3.
  • Loading branch information
carlossanlop authored Jul 22, 2024
1 parent ea2ffbd commit 4399da8
Show file tree
Hide file tree
Showing 80 changed files with 53,338 additions and 6 deletions.
32 changes: 32 additions & 0 deletions THIRD-PARTY-NOTICES.TXT
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,38 @@ shall not be used in advertising or otherwise to promote the sale,
use or other dealings in these Data Files or Software without prior
written authorization of the copyright holder.

License notice for Zlib
-----------------------

https://github.com/madler/zlib
https://zlib.net/zlib_license.html

/* zlib.h -- interface of the 'zlib' general purpose compression library
version 1.3.1, January 22nd, 2024

Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler

This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu

*/

License notice for zlib-ng
-----------------------

Expand Down
8 changes: 5 additions & 3 deletions docs/workflow/requirements/linux-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,15 @@ Install the following packages for the toolchain:
* liblttng-ust-dev
* libssl-dev
* libkrb5-dev
* zlib1g-dev
* ninja-build (optional, enables building native code with ninja instead of make)

**NOTE**: If you have an Ubuntu version older than 22.04 LTS, or Debian version older than 12, don't install `cmake` using `apt` directly. Follow the note written down below.

```bash
sudo apt install -y cmake llvm lld clang build-essential \
python-is-python3 curl git lldb libicu-dev liblttng-ust-dev \
libssl-dev libkrb5-dev ninja-build
python-is-python3 curl git lldb libicu-dev liblttng-ust-dev \
libssl-dev libkrb5-dev zlib1g-dev ninja-build
```

**NOTE**: As of now, Ubuntu's `apt` only has until CMake version 3.16.3 if you're using Ubuntu 20.04 LTS (less in older Ubuntu versions), and version 3.18.4 in Debian 11 (less in older Debian versions). This is lower than the required 3.20, which in turn makes it incompatible with the repo. For this case, we can use the `snap` package manager or the _Kitware APT feed_ to get a new enough version of CMake.
Expand Down Expand Up @@ -97,12 +98,13 @@ Install the following packages for the toolchain:
* libicu-devel
* openssl-devel
* krb5-devel
* zlib-devel
* lttng-ust-devel
* ninja-build (optional, enables building native code with ninja instead of make)

```bash
sudo dnf install -y cmake llvm lld lldb clang python curl git libicu-devel openssl-devel \
krb5-devel lttng-ust-devel ninja-build
krb5-devel zlib-devel lttng-ust-devel ninja-build
```

### Gentoo
Expand Down
6 changes: 3 additions & 3 deletions eng/install-native-dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,13 @@ case "$os" in
apt update

apt install -y build-essential gettext locales cmake llvm clang lldb liblldb-dev libunwind8-dev libicu-dev liblttng-ust-dev \
libssl-dev libkrb5-dev
libssl-dev libkrb5-dev zlib1g-dev

localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
elif [ "$ID" = "fedora" ]; then
dnf install -y cmake llvm lld lldb clang python curl libicu-devel openssl-devel krb5-devel lttng-ust-devel
dnf install -y cmake llvm lld lldb clang python curl libicu-devel openssl-devel krb5-devel zlib-devel lttng-ust-devel
elif [ "$ID" = "alpine" ]; then
apk add build-base cmake bash curl clang llvm-dev krb5-dev lttng-ust-dev icu-dev openssl-dev
apk add build-base cmake bash curl clang llvm-dev krb5-dev lttng-ust-dev icu-dev zlib-dev openssl-dev
else
echo "Unsupported distro. distro: $ID"
exit 1
Expand Down
32 changes: 32 additions & 0 deletions src/installer/pkg/THIRD-PARTY-NOTICES.TXT
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,38 @@ shall not be used in advertising or otherwise to promote the sale,
use or other dealings in these Data Files or Software without prior
written authorization of the copyright holder.

License notice for Zlib
-----------------------

https://github.com/madler/zlib
https://zlib.net/zlib_license.html

/* zlib.h -- interface of the 'zlib' general purpose compression library
version 1.2.13, October 13th, 2022

Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler

This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu

*/

License notice for Mono
-------------------------------

Expand Down
20 changes: 20 additions & 0 deletions src/native/external/cgmanifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,26 @@
},
"DevelopmentDependency": false
},
{
"Component": {
"Type": "git",
"Git": {
"RepositoryUrl": "https://github.com/madler/zlib",
"CommitHash": "51b7f2abdade71cd9bb0e7a373ef2610ec6f9daf"
}
},
"DevelopmentDependency": false
},
{
"Component": {
"Type": "git",
"Git": {
"RepositoryUrl": "https://github.com/intel/zlib",
"CommitHash": "6160a8f20c3626aec2f8b0fda5bf2e65bfe31781"
}
},
"DevelopmentDependency": false
},
{
"Component": {
"Type": "git",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
From edabaf799fd071a328e0adb743a98628df6649f0 Mon Sep 17 00:00:00 2001
From: Levi Broderick <levib@microsoft.com>
Date: Mon, 28 Aug 2023 15:26:38 -0700
Subject: [PATCH] Make zlib-intel compile clean against C4244 clang equivalent
is "implicit-int-conversion" warning

The change to deflate.c is legal because 'len' has an upper bound of
MAX_STORED, which means it fits cleanly into a 16-bit integer. So
writing out 2x 8-bit values will not result in data loss.

The change to trees.c is legal because within this loop, 'count' is
intended to have an upper bound of 138, with the target assignment
only executing if 'count' is bounded by 4. Neither the 'count' local
in isolation nor the addition that's part of the target line is
expected to result in integer overflow. But even if it did, that's a
matter for a different warning code and doesn't impact the correctness
of the narrowing cast being considered here.

The change to slide_sse.c is legal because 'w_size' is limited to
1 << 15 (see deflateInit2_ in deflate.c), so this fits cleanly into
a 16-bit value.
---
src/native/external/zlib-intel/deflate.c | 8 ++++----
src/native/external/zlib-intel/slide_sse.c | 2 +-
src/native/external/zlib-intel/trees.c | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/native/external/zlib-intel/deflate.c b/src/native/external/zlib-intel/deflate.c
index bd5e95774a6..108b1a187af 100644
--- a/src/native/external/zlib-intel/deflate.c
+++ b/src/native/external/zlib-intel/deflate.c
@@ -1553,10 +1553,10 @@ local block_state deflate_stored(s, flush)
_tr_stored_block(s, (char *)0, 0L, last);

/* Replace the lengths in the dummy stored block with len. */
- s->pending_buf[s->pending - 4] = len;
- s->pending_buf[s->pending - 3] = len >> 8;
- s->pending_buf[s->pending - 2] = ~len;
- s->pending_buf[s->pending - 1] = ~len >> 8;
+ s->pending_buf[s->pending - 4] = (Bytef)len;
+ s->pending_buf[s->pending - 3] = (Bytef)(len >> 8);
+ s->pending_buf[s->pending - 2] = (Bytef)~len;
+ s->pending_buf[s->pending - 1] = (Bytef)(~len >> 8);

/* Write the stored block header bytes. */
flush_pending(s->strm);
diff --git a/src/native/external/zlib-intel/slide_sse.c b/src/native/external/zlib-intel/slide_sse.c
index 342fd562dd1..eb74202c5a0 100644
--- a/src/native/external/zlib-intel/slide_sse.c
+++ b/src/native/external/zlib-intel/slide_sse.c
@@ -18,7 +18,7 @@ void slide_hash_sse(deflate_state *s)
unsigned n;
Posf *p;
uInt wsize = s->w_size;
- z_const __m128i xmm_wsize = _mm_set1_epi16(s->w_size);
+ z_const __m128i xmm_wsize = _mm_set1_epi16((short)s->w_size);

n = s->hash_size;
p = &s->head[n] - 8;
diff --git a/src/native/external/zlib-intel/trees.c b/src/native/external/zlib-intel/trees.c
index 35462a1313a..f78b7d8c63e 100644
--- a/src/native/external/zlib-intel/trees.c
+++ b/src/native/external/zlib-intel/trees.c
@@ -717,7 +717,7 @@ local void scan_tree(s, tree, max_code)
if (++count < max_count && curlen == nextlen) {
continue;
} else if (count < min_count) {
- s->bl_tree[curlen].Freq += count;
+ s->bl_tree[curlen].Freq += (ush)count;
} else if (curlen != 0) {
if (curlen != prevlen) s->bl_tree[curlen].Freq++;
s->bl_tree[REP_3_6].Freq++;
--
2.42.0.windows.1

Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
From 86d96652ddd60f61dc7b0c94b601f6d156d34632 Mon Sep 17 00:00:00 2001
From: Levi Broderick <levib@microsoft.com>
Date: Mon, 28 Aug 2023 15:26:38 -0700
Subject: [PATCH] Make zlib compile clean against C4244 clang equivalent is
"implicit-int-conversion" warning

The change to deflate.c is legal because 'len' has an upper bound of
MAX_STORED, which means it fits cleanly into a 16-bit integer. So
writing out 2x 8-bit values will not result in data loss.

The change to trees.c is legal because within this loop, 'count' is
intended to have an upper bound of 138, with the target assignment
only executing if 'count' is bounded by 4. Neither the 'count' local
in isolation nor the addition that's part of the target line is
expected to result in integer overflow. But even if it did, that's a
matter for a different warning code and doesn't impact the correctness
of the narrowing cast being considered here.
---
src/native/external/zlib/deflate.c | 8 ++++----
src/native/external/zlib/trees.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/native/external/zlib/deflate.c b/src/native/external/zlib/deflate.c
index d2e1106ef5d..b7636639754 100644
--- a/src/native/external/zlib/deflate.c
+++ b/src/native/external/zlib/deflate.c
@@ -1738,10 +1738,10 @@ local block_state deflate_stored(s, flush)
_tr_stored_block(s, (char *)0, 0L, last);

/* Replace the lengths in the dummy stored block with len. */
- s->pending_buf[s->pending - 4] = len;
- s->pending_buf[s->pending - 3] = len >> 8;
- s->pending_buf[s->pending - 2] = ~len;
- s->pending_buf[s->pending - 1] = ~len >> 8;
+ s->pending_buf[s->pending - 4] = (Bytef)len;
+ s->pending_buf[s->pending - 3] = (Bytef)(len >> 8);
+ s->pending_buf[s->pending - 2] = (Bytef)~len;
+ s->pending_buf[s->pending - 1] = (Bytef)(~len >> 8);

/* Write the stored block header bytes. */
flush_pending(s->strm);
diff --git a/src/native/external/zlib/trees.c b/src/native/external/zlib/trees.c
index 5f305c47221..8a3eec559e5 100644
--- a/src/native/external/zlib/trees.c
+++ b/src/native/external/zlib/trees.c
@@ -721,7 +721,7 @@ local void scan_tree(s, tree, max_code)
if (++count < max_count && curlen == nextlen) {
continue;
} else if (count < min_count) {
- s->bl_tree[curlen].Freq += count;
+ s->bl_tree[curlen].Freq += (ush)count;
} else if (curlen != 0) {
if (curlen != prevlen) s->bl_tree[curlen].Freq++;
s->bl_tree[REP_3_6].Freq++;
--
2.42.0.windows.1

16 changes: 16 additions & 0 deletions src/native/external/zlib-intel-version.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
v1.2.13_jtk
(6160a8f20c3626aec2f8b0fda5bf2e65bfe31781)

https://github.com/intel/zlib/releases/tag/v1.2.13_jtk

Note: This library does not use a standard release lifecycle.
We're pointing to the latest available tagged version as of this writing.

We do not pull in every file from the intel/zlib repo. Instead, some years ago,
somebody determined what they believed was the minimum set that needed to be
copied into our repo in order to build. Since then, we've just updated only
those files in-place, ignoring other files in the intel/zlib repo. If new files
are introduced which are necessary for building the product, feel free to bring
those down as well.

We have also applied the custom patches under the patches/zlib-intel folder.
11 changes: 11 additions & 0 deletions src/native/external/zlib-intel/README.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Zlib-Intel is an optimized version of Zlib developed by Intel to take advantage of modern CPU features.

Brief Listing of Changes:
New deflate_medium() strategy for compression levels 4-6
Improved hashing using the SSE4.2 CRC32 instruction
SSE Saturated Subtractions for hash table shifting
Faster CRC32 calculations using SSE4.2 PCLMULQDQ instruction
Reduced loop unrolling

Detailed explanations of these optimizations available in the official whitepaper: http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/zlib-compression-whitepaper-copy.pdf

Loading

0 comments on commit 4399da8

Please sign in to comment.