Skip to content

Commit

Permalink
QATAPP-26660: update README
Browse files Browse the repository at this point in the history 
* update README

Change-Id: I6c8c73bd96a631eac1811bf5e50dc556e4ae6862
Signed-off-by: Chengfei Zhu <chengfei.zhu@intel.com>
  • Loading branch information
@cfzhu
cfzhu committed Mar 24, 2023
1 parent 73e8a23 commit 542c3e4
Showing 1 changed file with 84 additions and 147 deletions.
231 changes: 84 additions & 147 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,20 @@ QATzip is a user space library which builds on top of the Intel&reg; QuickAssist
Technology user space library, to provide extended accelerated compression and
decompression services by offloading the actual compression and decompression
request(s) to the Intel&reg; Chipset Series. QATzip produces data using the standard
gzip\* format (RFC1952) with extended headers or [lz4 blocks][*] with [lz4 frame format][**].
The data can be decompressed with a compliant gzip\* or lz4 implementation. QATzip is
gzip\* format (RFC1952) with extended headers or [lz4\* blocks][7] with [lz4\* frame format][8].
The data can be decompressed with a compliant gzip\* or lz4\* implementation. QATzip is
designed to take full advantage of the performance provided by Intel&reg; QuickAssist Technology.

[*]:https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md
[**]:https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md
The currently supported formats include:

|Data Format|Algorithm|QAT device|Description|
| :---------------: | :---------------: |:---------------: | :------------------------------------------------------------: |
| `QZ_DEFLATE_4B` | deflate\* | QAT1.x and QAT 2.0|Data is in DEFLATE\* with a 4 byte header|
| `QZ_DEFLATE_GZIP` | deflate\* | QAT1.x and QAT 2.0|Data is in DEFLATE\* wrapped by Gzip\* header and footer|
| `QZ_DEFLATE_GZIP_EXT` | deflate\* | QAT1.x and QAT 2.0|Data is in DEFLATE\* wrapped by Intel&reg; QAT Gzip\* extension header and footer|
| `QZ_DEFLATE_RAW` | deflate\* | QAT1.x and QAT 2.0|Data is in raw DEFLATE\* without any additional header. (Only support compression, decompression will fallback to software) |
| `QZ_LZ4` | lz4\* | QAT 2.0|Data is in LZ4\* wrapped by lz4\* frame |
| `QZ_LZ4S` | lz4s\* | QAT 2.0|Data is in LZ4S\* blocks |

## Licensing

Expand Down Expand Up @@ -64,20 +71,24 @@ contained in the file `LICENSE.GPL` within the `config_file` folder.
switch to software if there is insufficient system resources including acceleration
instances or memory. This feature allows for a common software stack between server
platforms that have acceleration devices and non-accelerated platforms.
* Automatic recovery from hardware compression failure.
* Provide streaming interface of compression and decompression to achieve better compression
ratio and throughput for data sets that are submitted piecemeal.
* 'qzip' utility supports compression from regular file, pipeline and block device.
* For standard GZIP format, try hardware decompression 1st before switch to software decompression.
* For QATzip GZIP\* format, try hardware decompression 1st before switch to software decompression.
* Enable adaptive polling mechanism to save CPU usage in stress mode.
* 'qzip' utility supports compression files and directories into 7z format.
* For 4xxx(QAT gen 4 devices), QATzip supports lz4 (de)compression algorithm.
* For 4xxx(QAT gen 4 devices), QATzip supports lz4s compression algorithm, the QATzip can generate
lz4s block with lz4 frame format, which can be used for software post-processing to generate
other compressed data format.
* For 4xxx(QAT gen 4 devices), QATzip supports ZSTD format compression, through import post-process
mechanism. this feature request to enable qzstd.
* Support QATzip Gzip\* format, it includes 10 bytes header and 8 bytes footer:

`| ID1 (1B) | ID2(0x8B) (1B) | Compression Method (8 = DEFLATE*) (1B) | Flags (1B) | Modification Time (4B) | Extra Flags (1B) | OS (1B) | Deflate Block| CRC32(4B)| ISIZE(4B)|`
* Support QATzip Gzip\* extended format. This consists of the standard 10 byte Gzip* header and follows RFC 1952 to extend the header by an additional 14 bytes. The extended headers structure is below:

`| Length of ext. header (2B) | SI1('Q') (1B) | SI2('Z') (1B) | Length of subheader (2B) | Intel(R) defined field 'Chunksize' (4B) | Intel(R) defined field 'Blocksize' (4B) | `
* Support Intel&reg; QATzip 4 byte header, the header indicates the length of the compressed block followed by the header.

`| Intel(R) defined Header (4B)|deflate\* block|`
* Support QATzip lz4* format. This format is structured as follows:

`| MagicNb(4B) |FLG(1B)|BD(1B)| CS(8B)|HC(1B)| |lz4\* Block | EndMark(4B)|`
## Hardware Requirements

This QATzip library supports compression and decompression offload to the following
Expand All @@ -87,31 +98,46 @@ acceleration devices:
* [Intel&reg; Communications Chipset 8925 to 8955 Series][2]
* [Intel&reg; Communications Chipset 8960 to 8970 Series][3]
* [Intel&reg; C3XXX Series Chipset][4]
* Intel&reg; 4XXX (QAT gen 4 devices)

[1]:https://www.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/purley/intel-xeon-scalable-processors.html
[2]:https://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-8950-brief.html
[3]:https://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-8960-8970-brief.html
[4]:https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/c3000-family-brief.html
* [Intel&reg; 4XXX Series][5]

## Software Requirements

This release was validated on the following:

* QATzip has been tested with the latest Intel&reg; QuickAssist Acceleration Driver.
Please download the QAT driver from the link https://developer.intel.com/quickassist
* QATzip has been tested by Intel&reg; on CentOS 7.8.2003 with kernel 3.10.0-1127.19.1.el7.x86\_64
Please download the QAT driver from the link [Intel&reg; QuickAssist Technology][6]
* QATzip has been tested by Intel&reg; on CentOS\* 7.8.2003 with kernel 3.10.0-1127.19.1.el7.x86\_64
* Zlib\* library of version 1.2.7 or higher
* Suggest GCC\* of version 4.8.5 or higher
* lz4\* library
* zstd\* static library


## Additional Information

The compression level in QATzip could be mapped to standard zlib\* as below:
* QATzip level 1 - 4, similar to zlib\* level 1 - 4.
* QATzip level 5 - 8, we map them to QATzip level 4.
* QATzip level 9, we will use software zlib\* to compress as level 9.
* For QAT1.x, the compression level in QATzip could be mapped to standard zlib\* as below:
* QATzip level 1 - 4, similar to zlib\* level 1 - 4.
* QATzip level 5 - 8, we map them to QATzip level 4.
* QATzip level 9, we will use software zlib\* to compress as level 9.
* For QAT2.0, the compression level in QATzip could be mapped to standard zlib\* or lz4\* as below:
* Will be updated in feature release.

* QATzip Compression Level Mapping:

| QATzip Level |QAT Level| QAT 2.0(deflate\*, LZ4\*, LZ4s\*) |QAT1.7/1.8(Deflate\*) |
| ---- | --- | ---- | ---- |
| 1 | CPA_DC_L1 |2(HW_L1) | DEPTH_1 |
| 2 | CPA_DC_L2 |2(HW_L1) | DEPTH_4 |
| 3 | CPA_DC_L3 |2(HW_L1) | DEPTH_8 |
| 4 | CPA_DC_L4 |2(HW_L1) | DEPTH_16 |
| 5 | CPA_DC_L5 |2(HW_L1) | DEPTH_16 |
| 6 | CPA_DC_L6 |8(HW_L6) | DEPTH_16 |
| 7 | CPA_DC_L7 |8(HW_L6) | DEPTH_16 |
| 8 | CPA_DC_L8 |8(HW_L6) | DEPTH_16 |
| 9 | CPA_DC_L9 |16(HW_L9) | DEPTH_16 |
| 10 | CPA_DC_L10 |16(HW_L9) | Unsupported |
| 11 | CPA_DC_L11 |16(HW_L9) | Unsupported |
| 12 | CPA_DC_L12 |16(HW_L9) | Unsupported |

## Limitations

Expand All @@ -127,14 +153,10 @@ The compression level in QATzip could be mapped to standard zlib\* as below:
* For 7z format, decompression only supports \*.7z archives compressed by qzip.
* For 7z format, decompression only supports software.
* For 7z format, the header compression is not supported.
* For lz4(s) compression, QATzip only supports 32KB history buffer.
* For zstd format compression, qzstd only supprots `hw_buffer_sz` which is less than 128KB.
* For lz4\* (de)compression, QATzip only supports 32KB history buffer.
* For zstd format compression, qzstd only supports `hw_buffer_sz` which is less than 128KB.
* Stream APIs only support "DEFLATE_GZIP", "DEFLATE_GZIP_EXT", "DEFLATE_RAW" for compression
and "DEFLATE_GZIP", "DEFLATE_GZIP_EXT" for decompression now.
* For compression(not stream), QATZip HW only support data format including "DEFLATE_4B", "DEFLATE_GZIP",
"DEFLATE_GZIP_EXT", "DEFLATE_RAW", "LZ4". Otherwise it will fallback to sw.
* For decompression(not stream), QATZip HW only support data format including "DEFLATE_GZIP", "DEFLATE_4B",
"DEFLATE_GZIP_EXT", "LZ4". Otherwise it will fallback to sw.

## Installation Instructions

Expand All @@ -143,17 +165,17 @@ The compression level in QATzip could be mapped to standard zlib\* as below:
Please follow the instructions contained in:

**For Intel&reg; C62X Series Chipset:**
Intel&reg; QuickAssist Technology Software for Linux\* - Getting Started Guide - HW version 1.7 (336212)

[Intel&reg; QuickAssist Technology Software for Linux\* - Getting Started Guide - HW version 1.7][11]

**For Intel&reg; Communications Chipset 89XX Series:**
Intel&reg; Communications Chipset 89xx Series Software for Linux\* - Getting
Started Guide (330750)

These instructions can be found on the 01.org website in the following section:
[Intel&reg; Communications Chipset 89xx Series Software for Linux\* - Getting Started Guide][12]

**For Intel&reg; 4XXX Series:**

[Intel&reg; Quickassist Technology][3]
[Intel&reg; QuickAssist Technology (Intel&reg; QAT) Software for Linux\* Getting Started Guide – Hardware Version 2.0][13]

[3]:https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches

### Install QATzip As Root User

Expand All @@ -178,7 +200,8 @@ These instructions can be found on the 01.org website in the following section:
./autogen.sh
./configure --with-ICP_ROOT=$ICP_ROOT
make clean
make all install
make
make install
```

For more configure options, please run "./configure -h" for help
Expand All @@ -190,13 +213,12 @@ ___Have to update those file, Otherwise QATzip will be Unavailable.___
QAT's the programmer’s guide which provides information on the architecture of the software
and usage guidelines. it allows customization of runtime operation.

* [Intel&reg; linux Programmer's Guide][4]

[4]:https://www.intel.com/content/www/us/en/content-details/710060/intel-quickassist-technology-software-for-linux-programmer-s-guide-hw-version-1-7.html
* [Intel&reg; QAT1.7 linux Programmer's Guide][9]
* [Intel&reg; QAT2.0 linux Programmer's Guide][10]

The Intel&reg; QATzip comes with some tuning example conf files to use. you can replace the
old conf file(under /etc/) by them. The detailed info about Configurable options, please
refer Programmer's Guide mannual.
refer Programmer's Guide manual.

The process section name(in configuration file) is the key change for QATzip.
there are two way to change:
Expand All @@ -214,7 +236,7 @@ Chipset, dh895xcc for Intel&reg; Communications Chipset 8925 to 8955 Series

`CONFIG_TYPE`: tuned configure file(s) for different usage,
`multiple_process_opt` for multiple process optimization,
`multiple_thread_opt` for multiple thread optimization
`multiple_thread_opt` for multiple thread optimization.

**Restart QAT driver**

Expand All @@ -225,76 +247,8 @@ Chipset, dh895xcc for Intel&reg; Communications Chipset 8925 to 8955 Series
With current configuration, each PCI-e device in C6XX platform could support
32 processes in maximum.

### Install QATzip As Non-root User

**Add Non-root user to qat group**

The installation of QAT driver package configures the driver to allow applications
to run as Non-root user. The users must be added to the 'qat' group after QAT drvier
is installed.

```bash
sudo usermod -g qat username # need to relogin
or
sudo newgrp qat # effective immediately
```

Change the amount of max locked memory for the username that is included in the group
name. This can be done by specifying the limit in /etc/security/limits.conf.
To set 500MB add a line like this in /etc/security/limits.conf:
```bash
cat /etc/security/limits.conf |grep qat
@qat - memlock 500000
```

**Enable huge page as root user**

```bash
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
rmmod usdm_drv
insmod $ICP_ROOT/build/usdm_drv.ko max_huge_pages=1024 max_huge_pages_per_process=48
```

**Update the configuration files as root user**

To update the configuration file, copy the configure file(s) from directory of
`$QZ_ROOT/config_file/$YOUR_PLATFORM/$CONFIG_TYPE/*.conf`
to directory of `/etc`

`YOUR_PLATFORM`: the QAT hardware platform, c6xx for Intel&reg; C62X Series
Chipset, dh895xcc for Intel&reg; Communications Chipset 8925 to 8955 Series

`CONFIG_TYPE`: tuned configure file(s) for different usage,
`multiple_process_opt` for multiple process optimization,
`multiple_thread_opt` for multiple thread optimization

**Restart the QAT driver as root user**

```bash
service qat_service restart
```

With current configuration, each PCI-e device in C6XX platform could support
32 processes in maximum

**Set below environment variable as non-root user**

`ICP_ROOT`: the root directory of your QAT driver source tree

`QZ_ROOT`: the root directory of your QATzip source tree

**Compile and install QATzip as root user**

```bash
cd $QZ_ROOT
./autogen.sh
./configure --with-ICP_ROOT=$ICP_ROOT
make clean
make all
make install
```

For more configure options, please run "./configure -h" for help
### Install QAT As Non-root User
Please refer to [Intel&reg; QuickAssist Technology (Intel&reg; QAT) Software for Linux\* Getting Started Guide][13] section "3.8 Running Applications as Non-Root User".

### Enable qzstd
If you want to enable lz4s + postprocessing pipeline, you have to compile qzstd. which
Expand All @@ -313,18 +267,7 @@ make sure that you have installed zstd static lib.
**test qzstd**

```bash
qzstd -h

it could support below options:
"Compress or uncompress a file with zstandard format.",
"",
" -d, decompress",
" -h, show help information",
" -L, set compression level of QAT",
" -o, set output file name",
" -C, set chunk size",
" -r, set max inflight request number",
" -P, set polling mode, only supports busy polling settings",
qzstd -k $your_input_file
```

### Test QATzip
Expand All @@ -333,30 +276,10 @@ Run the following command to check if the QATzip is setup correctly for
compressing or decompressing files:

```bash
qzip -k $your_input_file (add -h for help)
or
cat $your_input_file | qzip > $yout_output_file

This compression and decompression util could support below options:
" -A, --algorithm set algorithm type, currently only support deflate",
" -d, --decompress decompress",
" -f, --force force overwrite of output file and compress links",
" -h, --help give this help",
" -H, --huffmanhdr set huffman header type",
" -k, --keep keep (don't delete) input files",
" -V, --version display version number",
" -L, --level set compression level",
" -C, --chunksz set chunk size",
" -O, --output set output header format(gzip|gzipext|7z)",
" -r, set max inflight request number",
" -R, set Recursive mode for decompressing a directory
It only supports for gzip/gzipext format and
decompression operation",
" -o, set output file name",
" -P, --polling set polling mode, only supports busy polling settings"
qzip -k $your_input_file -O gzipext -A deflate
```

#### File compession in 7z:
#### File compression in 7z:
```bash
qzip -O 7z FILE1 FILE2 FILE3... -o result.7z
```
Expand Down Expand Up @@ -400,7 +323,7 @@ Known issues relating to the QATzip are described in this section.
|----------|:-------------
| Reference | QATAPP-26069 |
| Description | If the users call qzFree after qzMemDestory, they may encounter free memory error "free(): invalid pointe" |
| Implication | User use qzMalloc API to allocte continuous memory |
| Implication | User use qzMalloc API to allocate continuous memory |
| Resolution | Ensure qzMemDestory is invoked after qzFree, now we use attribute destructor to invoke qzMemDestory|
| Affected OS | Linux |

Expand Down Expand Up @@ -434,3 +357,17 @@ Intel, the Intel logo are trademarks of Intel Corporation in the U.S.
and/or other countries.

\*Other names and brands may be claimed as the property of others

[1]:https://www.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/purley/intel-xeon-scalable-processors.html
[2]:https://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-8950-brief.html
[3]:https://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-8960-8970-brief.html
[4]:https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/c3000-family-brief.html
[5]:https://www.intel.com/content/www/us/en/products/details/processors/xeon/scalable.html
[6]:https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html
[7]:https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md
[8]:https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md
[9]:https://www.intel.com/content/www/us/en/content-details/710060/intel-quickassist-technology-software-for-linux-programmer-s-guide-hw-version-1-7.html
[10]:https://www.intel.com/content/www/us/en/content-details/743912/intel-quickassist-technology-intel-qat-software-for-linux-programmers-guide-hardware-version-2-0.html
[11]:https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-customer-enabling-release.html
[12]:https://www.intel.com/content/www/us/en/content-details/710089/intel-communications-chipset-89xx-series-software-for-linux-getting-started-guide.html
[13]:https://www.intel.com/content/www/us/en/content-details/632506/intel-quickassist-technology-intel-qat-software-for-linux-getting-started-guide-hardware-version-2-0.html

0 comments on commit 542c3e4

Please sign in to comment.