Sync develop to stable (v.0.6) (#216)

* Performace improvement for vpicio (#211) * Paritial fix for the region transfer/wait performance issue * Committing clang-format changes * Improve the async processing for vpicio_mts_all, also fix a few compile issues * Committing clang-format changes * Minor change * Continue to optimize start_all performance for vpicio, add a few time related convinient functions * Committing clang-format changes * Fix hanging issue in CI testing * Committing clang-format changes * Disable debug prints * Revert back for non-all ops * Better pthread management * Better pthread management * Fix timeout issue with CI testing and clang-formatting * Committing clang-format changes * Test * Trigger test * Committing clang-format changes * Trigger CI * Committing clang-format changes * Trigger CI * Switch to static partition for vpicio * Replace vpicio_mts with new implementation --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Multi-thread fix and request merging (#205) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix an issue with region transfer request * Committing clang-format changes * Merge small requests when they are contiguous and 1D, change srun commands in run scripts to detect Perlmutter compute nodes * Merge only for REGION_LOCAL partition * Committing clang-format changes * Fix a bug that causes some tests to fail * Fix a couple of issues with start/wait all * Committing clang-format changes * Add aggregation support for contiguous read operations * Committing clang-format changes * Fix compile issue when multithread is enabled * Committing clang-format changes * minor change with test code * Committing clang-format changes * Remove metadata mutex for multi threading * Committing clang-format changes * Fix mutex * Committing clang-format changes * Fix an issue when closing an obj * Sync develop to stable (v.0.5) (#201) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * enable cache by default (#187) * Removing PDC macro (#189) * Removing gres option for ctest * Removing gres option from scripts * Update check for core * Remove PDC macro * Committing clang-format changes --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * BDCATS fix (#193) * Fix issues with bdcats_batch * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Update mpi_test.sh (#197) * Update .gitlab-ci.yml (#195) * Updates for latest integration with Jacamar and Gitlab tokens in CI * VPICIO bugfix (#196) * Fix VPICIO bug * Add more checks and error out when no server is selected * Committing clang-format changes * Add VPICIO and BDCATS to MPI test --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix vpicio_mts (#199) --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Committing clang-format changes * Fix rebase issue * Add timers * Committing clang-format changes * Add explict transfer start (all) with MPI communicator * Committing clang-format changes * MPI fix * remove debug msg * Committing clang-format changes * Add function comment for doc * Revert script changes * Committing clang-format changes * Revert script changes * Committing clang-format changes * Revert script setting --------- Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Update CMakeLists.txt to bump version number (#202) * Update CMakeLists.txt to bump version number * Update clang-format-check.yml * IDIOMS Update & BULKI v0.1 (#203) * fix cmake mercury_util not found issue * update for Julia support * fix hdf5.h not found for src/tools * update container config * add libhdf5-dev for Github Actions * update CMake for HDF5 in tools * update logic for finding HDF5 * update * remove use system hdf5 * delete useless find library * update findHDF5 * Feature/dart (#11) Update to avoid fixing compilation issue on src/tools (due to : HDF5 cannot be found) * Use cc on Perlmutter (#161) Dr. Tang fixed a compilation issue in NERSC CI where HDF5 cannot be detected even if the cray-parallel-hdf5 module is loaded on Perlmultter. * update with fixes on tools and llsm example * add gitignore for llsm * update gitignore * Feature/dart (#12) * fix formatting * update clangformat10 * update base dockerfile * Add clang-format10 to docker container. Also fixed clang-format. Add clang-format10 to docker container. Also fixed clang-format. * Fix pdc ls (#154) * pdc import, export, ls compiled successfully * removed requested files * formatting issues * changed install tools * gets checkpoint files * grabbing checkpoint files from within sub-directories, minor comments * Committing clang-format changes * Committing clang-format changes * Fix a few issues with pdc_ls * Committing clang-format changes --------- Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * update documentation * update document update document * sync branch sync branch * no UUID module is required * update document and make UUID an optional package * update docker repo name * updating docker repo name and make UUID optional * Complete support for Docker and Github Codespace (#157) Include support for Docker and Github Codespace so we can run our dev environment with the support of Docker. * SQLite and RocksDB support for KVtags (#165) SQLite and RocksDB support for KVtags * fix round for tag delete * update test * bulki update * BULKI base type worked * BULKI all tests done * new index code * update * update new test * update csv bench * update * update script * adding python scripts for generating large metadata set for LLSM application * update json schema * better json validator * update importer * update code for non-MPI compatibility * update llsm converter * update LLSM data converter * split files * update .gitignore * update * add timing info * update * update tag size * detect object creation failure * update * update object name with date * update for robustness * update * update JMD_DEBUG option * update output for overall output * update inttypes.h * update * update extractor * update inttypes.h * update converter * update importer information * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * fix issue * fixed search issues * update for infix * update * index persistence still needs improvement * update * enable cache by default (#187) * Removing PDC macro (#189) * Removing gres option for ctest * Removing gres option from scripts * Update check for core * Remove PDC macro * Committing clang-format changes --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * range query done' * range query local test passed * multi-condition in progress * clean up code * add comments * new benchmark * update * update range query test * update cmake: * update * update * update * update * someta range query * someta range query * someta range query * fix value serialization * update * update double free * update * update * update * fixed pointer issue * rb_tree delete fixed, now need to check index persistence * update * update * update * update * update * update * update * update * update * update * update * update * update * BDCATS fix (#193) * Fix issues with bdcats_batch * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * clean up code * update test sh * IDIOMS persistence DONE * update * remove old kvtag benchmarks * update * update * update changes * dart info * update * multi data type for the same key, supported now * Monitoring changes from feature/dart to develop (#18) Major changes: * IDIOMS -> affix-based query benchmark * IDIOMS -> Simulation Test * IDIOMS -> Multi data type supported for the same key * IDIOMS -> Range Query and Exact Query for Numeric Values * IDIOMS -> benchmark for numeric values (exact search and range query) * IDIOMS -> Index Persistence * BULKI -> A data serialization and deserialization mechanism. * fix CMakeLists.txt * update * update format * update BULKI interface order * BULKI API sorted * add idioms ci test * Feature/dart (#20) 1. add documentation about BULKI and IDIOMS query conditions 2. add ci test for IDIOMS 3. optimized BULKI to save space on its metadata fields. * Feature/dart (#22) update version * update * update * update * remove unnecessary .bin file * update * update --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> * Fix region transfer with object static partitioning (#214) * Update pdc_region_transfer.c * Committing clang-format changes * Update .gitlab-ci.yml Fix issue with Perlmutter CI libfabric module --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * EQSIM benchmark code and fixes (#213) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * enable cache by default (#187) * Benchmark code for EQSIM data * Committing clang-format changes * Minor adjustments * Committing clang-format changes * Updates * Committing clang-format changes * Change vpicio to use local server partitioning, add some debug prints * Committing clang-format changes * Add metadata query to benchmark code * Committing clang-format changes * Add ZFP compression for read and write * Committing clang-format changes * Add a option to use more ranks to read data so total data of each rank is less than the 4GB chunk limit * Committing clang-format changes * Add a data query code for EQSIM data * Committing clang-format changes * Minor adjustments for the HDF5 read code * Committing clang-format changes * Fix an issue with periodic data flush, minor changes to benchmark code * Committing clang-format changes * fix an issue with 3d read segfault * Committing clang-format changes * Fix compile issue * Update .gitlab-ci.yml * Update sleep time * Replace function * Replace function * Minor updates and doc changes * Committing clang-format changes * Update --------- Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Zhang Wei <zhangwei217245@lbl.gov> Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com>
hpc-io · Dec 3, 2024 · 5fe5929 · 5fe5929
1 parent 6999ac7
commit 5fe5929
Show file tree

Hide file tree

Showing 149 changed files with 21,153 additions and 3,825 deletions.
diff --git a/.docker/dev_base.Dockerfile b/.docker/dev_base.Dockerfile
@@ -4,13 +4,55 @@
 ARG ARCH
 FROM ${ARCH}ubuntu:jammy
 
-RUN echo "ARCH=${ARCH}" && sleep 3
-
 ARG ARCH_CODE
 
-RUN echo "ARCH_CODE=${ARCH_CODE}" && sleep 3
+ENV JULIA_HOME=/opt/julia
+ENV WORK_SPACE=/home/project
+ENV LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric
+ENV LIBFABRIC_DIR=$WORK_SPACE/install/libfabric
+ENV MERCURY_SRC_DIR=$WORK_SPACE/source/mercury
+ENV MERCURY_DIR=$WORK_SPACE/install/mercury
+
+ENV PDC_SRC_DIR=$WORK_SPACE/source/pdc
+ENV PDC_DIR=$WORK_SPACE/install/pdc
+
+ENV LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH"
+ENV PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH"
+ENV LD_LIBRARY_PATH="$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
+ENV PATH="$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
+
+
+ENV MERCURY_CMAKE_FLAGS="-DCMAKE_INSTALL_PREFIX=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DNA_USE_OFI=ON -DNA_USE_SM=OFF -DNA_OFI_TESTING_PROTOCOL=tcp "
+ENV CLANG_FORMAT_PATH=$WORK_SPACE/software/clang-format-lint-action/clang-format/clang-format10
+
 # Install necessary tools, MPICH, UUID library and developer files
-RUN apt-get update && apt-get install -y \
+RUN echo "ARCH=${ARCH}" && echo "ARCH_CODE=${ARCH_CODE}" && sleep 3 && mkdir -p $WORK_SPACE && \
+    mkdir -p $WORK_SPACE/software && \
+    mkdir -p $WORK_SPACE/source && \
+    mkdir -p $WORK_SPACE/install && \
+    mkdir -p $LIBFABRIC_SRC_DIR && \
+    mkdir -p $MERCURY_SRC_DIR && \
+    mkdir -p $PDC_SRC_DIR && \
+    mkdir -p $LIBFABRIC_DIR && \
+    mkdir -p $MERCURY_DIR && \
+    mkdir -p $PDC_DIR && \
+    rm -rf $LIBFABRIC_SRC_DIR/* && \
+    rm -rf $MERCURY_SRC_DIR/* && \
+    rm -rf $PDC_SRC_DIR/* && \
+    rm -rf $LIBFABRIC_DIR/* && \
+    rm -rf $MERCURY_DIR/* && \
+    rm -rf $PDC_DIR/* && \
+    echo "export LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric" > $WORK_SPACE/pdc_env.sh && \
+    echo "export LIBFABRIC_DIR=$WORK_SPACE/install/libfabric" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export MERCURY_SRC_DIR=$WORK_SPACE/source/mercury" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export MERCURY_DIR=$WORK_SPACE/install/mercury" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export PDC_SRC_DIR=$WORK_SPACE/source/pdc" >> $WORK_SPACE/pdc_env.sh && \
+    echo "export PDC_DIR=$WORK_SPACE/install/pdc" >> $WORK_SPACE/pdc_env.sh && \
+    echo 'export LD_LIBRARY_PATH=$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh  && \
+    echo 'export PATH=$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh && \
+    echo 'export LD_LIBRARY_PATH=$MERCURY_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh \
+    echo 'export PATH=$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh && \
+    apt-get update && apt-get install -y \
     build-essential \
     git \
     mpich \
@@ -28,107 +70,47 @@ RUN apt-get update && apt-get install -y \
     wget \
     axel \
     curl \
+    bc \
     vim \
     nano \
     gdb \
     cgdb \
     curl \
     valgrind \
-    python3 
-
-# Install Oh My Bash
-RUN bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohmybash/oh-my-bash/master/tools/install.sh)" && \
-    sed -i 's/OSH_THEME="font"/OSH_THEME="powerline-multiline"/g' ~/.bashrc
-
-# Install Julia
-
-RUN echo "https://julialang-s3.julialang.org/bin/linux/aarch64/1.6/julia-1.6.7-linux-aarch64.tar.gz" > /julia_url_arm64v8.txt && \
-    echo "https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.7-linux-x86_64.tar.gz" > /julia_url_amd64.txt
-
-RUN echo $(cat /julia_url_${ARCH_CODE}.txt) && sleep 3
-
-RUN mkdir -p /opt/julia && wget -O - $(cat /julia_url_${ARCH_CODE}.txt) | tar -xz -C /opt/julia --strip-components=1 && \
-    ln -s /opt/julia/bin/julia /usr/local/bin/julia
-
-RUN rm -rf /tmp/julia_url_*.txt
-
-ENV JULIA_HOME=/opt/julia
-
-# Install Rust
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-RUN echo 'source $HOME/.cargo/env' >> ~/.bashrc
-
-
-# Set WORK_SPACE environment variable and create necessary directories
-ENV WORK_SPACE=/home/project
-RUN mkdir -p $WORK_SPACE
-
-# Install clang-format repo
-RUN mkdir -p $WORK_SPACE/software
-RUN cd $WORK_SPACE/software && git clone https://github.com/DoozyX/clang-format-lint-action.git
-ENV CLANG_FORMAT_PATH=$WORK_SPACE/software/clang-format-lint-action/clang-format/clang-format10
-
-# Clone the repositories
-WORKDIR $WORK_SPACE/source
-RUN git clone https://github.com/ofiwg/libfabric.git && \
-    git clone https://github.com/mercury-hpc/mercury.git --recursive
-
-ENV LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric
-ENV LIBFABRIC_DIR=$WORK_SPACE/install/libfabric
-ENV MERCURY_SRC_DIR=$WORK_SPACE/source/mercury
-ENV MERCURY_DIR=$WORK_SPACE/install/mercury
-
-ENV PDC_SRC_DIR=$WORK_SPACE/source/pdc
-ENV PDC_DIR=$WORK_SPACE/install/pdc
-
-RUN mkdir -p $LIBFABRIC_SRC_DIR && \
-    mkdir -p $MERCURY_SRC_DIR && \
-    mkdir -p $PDC_SRC_DIR && \
-    mkdir -p $LIBFABRIC_DIR && \
-    mkdir -p $MERCURY_DIR && \
-    mkdir -p $PDC_DIR
-
-
-# Save the environment variables to a file
-RUN echo "export LIBFABRIC_SRC_DIR=$WORK_SPACE/source/libfabric" > $WORK_SPACE/pdc_env.sh && \
-    echo "export LIBFABRIC_DIR=$WORK_SPACE/install/libfabric" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export MERCURY_SRC_DIR=$WORK_SPACE/source/mercury" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export MERCURY_DIR=$WORK_SPACE/install/mercury" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export PDC_SRC_DIR=$WORK_SPACE/source/pdc" >> $WORK_SPACE/pdc_env.sh && \
-    echo "export PDC_DIR=$WORK_SPACE/install/pdc" >> $WORK_SPACE/pdc_env.sh
-
-
-# Build and install libfabric
-WORKDIR $LIBFABRIC_SRC_DIR
-RUN git checkout v1.18.0 && \
+    python3 && \
+    cd $WORK_SPACE/software && \
+    bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohmybash/oh-my-bash/master/tools/install.sh)" && \
+    sed -i 's/OSH_THEME="font"/OSH_THEME="powerline-multiline"/g' ~/.bashrc && \
+    echo "https://julialang-s3.julialang.org/bin/linux/aarch64/1.6/julia-1.6.7-linux-aarch64.tar.gz" > $WORK_SPACE/software/julia_url_arm64v8.txt && \
+    echo "https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.7-linux-x86_64.tar.gz" > $WORK_SPACE/software/julia_url_amd64.txt && \
+    echo $(cat $WORK_SPACE/software/julia_url_${ARCH_CODE}.txt) && sleep 3 && \
+    mkdir -p /opt/julia && wget -O - $(cat $WORK_SPACE/software/julia_url_${ARCH_CODE}.txt) | tar -xz -C /opt/julia --strip-components=1 && \
+    ln -s /opt/julia/bin/julia /usr/local/bin/julia && \
+    rm -rf $WORK_SPACE/software/julia_url_*.txt && \
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \
+    echo 'source $HOME/.cargo/env' >> ~/.bashrc && \
+    git clone https://github.com/DoozyX/clang-format-lint-action.git && \
+    git clone https://github.com/ofiwg/libfabric.git ${LIBFABRIC_SRC_DIR} && \
+    git clone https://github.com/mercury-hpc/mercury.git --recursive ${MERCURY_SRC_DIR} && \
+    cd $LIBFABRIC_SRC_DIR && \
+    git checkout v1.18.0 && \
     ./autogen.sh && \
     ./configure --prefix=$LIBFABRIC_DIR CC=mpicc CFLAG="-O2" && \
     make clean && \
     make -j 8 && make install && \
-    make check
-
-ENV LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH"
-ENV PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH"
-RUN echo 'export LD_LIBRARY_PATH=$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh  && \
-    echo 'export PATH=$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh
-
-
-# Build and install Mercury
-WORKDIR $MERCURY_SRC_DIR
-ENV MERCURY_CMAKE_FLAGS="-DCMAKE_INSTALL_PREFIX=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DNA_USE_OFI=ON -DNA_USE_SM=OFF -DNA_OFI_TESTING_PROTOCOL=tcp "
-RUN git checkout v2.2.0 \
-    mkdir -p build 
-WORKDIR ${MERCURY_SRC_DIR}/build
-RUN cmake $MERCURY_CMAKE_FLAGS ../ && \
+    make check && \
+    cd $MERCURY_SRC_DIR && \
+    git checkout v2.2.0 && \
+    mkdir -p build && \
+    cd ${MERCURY_SRC_DIR}/build && \
+    cmake $MERCURY_CMAKE_FLAGS ../ && \
     make -j 16 && make install && \
-    ctest
-
-# Set the environment variables
-ENV LD_LIBRARY_PATH="$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
-ENV PATH="$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
-RUN echo 'export LD_LIBRARY_PATH=$MERCURY_DIR/lib:$LD_LIBRARY_PATH' >> $WORK_SPACE/pdc_env.sh \
-    echo 'export PATH=$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH' >> $WORK_SPACE/pdc_env.sh
-
+    ctest && \
+    apt-get autoremove -y && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/* && \
+    rm -rf /tmp/* && \
+    rm -rf /var/tmp/* 
 
 ENV PDC_CMAKE_FLAGS="-DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX=$PDC_DIR -DPDC_ENABLE_MPI=ON -DMERCURY_DIR=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc -DMPI_RUN_CMD=mpiexec "
 

diff --git a/.github/workflows/clang-format-check.yml b/.github/workflows/clang-format-check.yml
@@ -9,7 +9,7 @@ jobs:
     steps:
     - uses: actions/checkout@v2
     - name: Run clang-format style check for C programs.
-      uses: DoozyX/clang-format-lint-action@v0.11
+      uses: DoozyX/clang-format-lint-action@v0.18.1
       with:
         source: '.'
         extensions: 'c,h,cpp,hpp'

diff --git a/.gitignore b/.gitignore
@@ -19,3 +19,9 @@ build
 
 # ignore docs build dir
 docs/build
+
+# ignore venv
+**/*venv
+
+# ignore .zed
+.zed
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -23,7 +23,7 @@ perlmutter-no-cache-build:
         SUPERCOMPUTER: "perlmutter"
         MERCURY_DIR: "/global/cfs/cdirs/${PDC_PROJECT}/pdc-perlmutter/mercury/install"
     script:
-        - module load libfabric/1.15.2.0
+        - module load libfabric
         - module list
         - mkdir -p ${PDC_BUILD_PATH}/perlmutter/no-cache
         - cd ${PDC_BUILD_PATH}/perlmutter/no-cache
@@ -51,7 +51,7 @@ perlmutter-cache-build:
         SUPERCOMPUTER: "perlmutter"
         MERCURY_DIR: "/global/cfs/cdirs/${PDC_PROJECT}/pdc-perlmutter/mercury/install"
     script:
-        - module load libfabric/1.15.2.0
+        - module load libfabric
         - module list
         - mkdir -p ${PDC_BUILD_PATH}/perlmutter/cache
         - cd ${PDC_BUILD_PATH}/perlmutter/cache
@@ -374,7 +374,7 @@ perlmutter-metrics-build:
         SUPERCOMPUTER: "perlmutter"
         MERCURY_DIR: "/global/cfs/cdirs/${PDC_PROJECT}/pdc-perlmutter/mercury/install"
     script:
-        - module load libfabric/1.15.2.0
+        - module load libfabric
         - module list
         - mkdir -p ${PDC_BUILD_PATH}/perlmutter/metrics
         - cd ${PDC_BUILD_PATH}/perlmutter/metrics

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -55,7 +55,7 @@ endif()
 # Version information
 #------------------------------------------------------------------------------
 set(PDC_VERSION_MAJOR "0")
-set(PDC_VERSION_MINOR "1")
+set(PDC_VERSION_MINOR "6")
 set(PDC_VERSION_PATCH "0")
 set(PDC_PACKAGE "pdc")
 set(PDC_PACKAGE_NAME "PDC")
@@ -198,6 +198,9 @@ if(SUPPRESS_IGNORABLE_WARNINGS)
     set(SUPPRESSED_LIST "-Wno-deprecated-non-prototype" ${SUPPRESSED_LIST})
   else()
     set(SUPPRESSED_LIST "-Wno-maybe-uninitialized" ${SUPPRESSED_LIST})
+    set(SUPPRESSED_LIST "-Wno-alloc-size-larger-than" ${SUPPRESSED_LIST})
+    # set(SUPPRESSED_LIST "-Wno-use-after-free" ${SUPPRESSED_LIST})
+    set(SUPPRESSED_LIST "-Wno-stringop-truncation" ${SUPPRESSED_LIST})
   endif()
   set(SUPPRESSED_LIST "-Wno-sign-compare" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-format" ${SUPPRESSED_LIST})
@@ -207,6 +210,8 @@ if(SUPPRESS_IGNORABLE_WARNINGS)
   set(SUPPRESSED_LIST "-Wno-unused-function" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-unused-result" ${SUPPRESSED_LIST})
   set(SUPPRESSED_LIST "-Wno-unused-but-set-variable" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-inline" ${SUPPRESSED_LIST})
+  set(SUPPRESSED_LIST "-Wno-array-bounds" ${SUPPRESSED_LIST})
 
   if(CMAKE_C_COMPILER_ID MATCHES "GNU|Clang")
     add_compile_options(${SUPPRESSED_LIST})
@@ -383,10 +388,10 @@ endif()
 option(PDC_SERVER_CACHE "Enable Server Caching." ON)
 if(PDC_SERVER_CACHE)
   set(PDC_SERVER_CACHE 1)
-  set(PDC_SERVER_CACHE_MAX_GB "3" CACHE STRING "Max GB for server cache")
-  set(PDC_SERVER_CACHE_FLUSH_TIME "30" CACHE STRING "Flush time for server cache")
+  set(PDC_SERVER_CACHE_MAX_GB "32" CACHE STRING "Max GB for server cache")
+  set(PDC_SERVER_IDLE_CACHE_FLUSH_TIME "3" CACHE STRING "Idle time to initiate flush from server cache")
 
-  add_compile_definitions(PDC_SERVER_CACHE_MAX_GB=${PDC_SERVER_CACHE_MAX_GB} PDC_SERVER_CACHE_FLUSH_TIME=${PDC_SERVER_CACHE_FLUSH_TIME})
+  add_compile_definitions(PDC_SERVER_CACHE_MAX_GB=${PDC_SERVER_CACHE_MAX_GB} PDC_SERVER_IDLE_CACHE_FLUSH_TIME=${PDC_SERVER_IDLE_CACHE_FLUSH_TIME})
 endif()
 
 
@@ -482,6 +487,14 @@ if(PDC_ENABLE_SQLITE3)
     set(ENABLE_SQLITE3 1)
 endif()
 
+#-----------------------------------------------------------------------------
+# ZFP option
+#-----------------------------------------------------------------------------
+option(PDC_ENABLE_ZFP "Enable ZFP." OFF)
+if(PDC_ENABLE_ZFP)
+    set(ENABLE_ZFP 1)
+endif()
+
 # Check availability of symbols
 #-----------------------------------------------------------------------------
 check_symbol_exists(malloc_usable_size "malloc.h" HAVE_MALLOC_USABLE_SIZE)
@@ -663,4 +676,4 @@ add_custom_target(format
   | xargs -0 clang-format -i -style=file && echo "... done"
   COMMENT "clang-format all source codes"
   VERBATIM 
-)
+)
diff --git a/benchmark/idioms/CMakeLists.txt b/benchmark/idioms/CMakeLists.txt
@@ -0,0 +1,68 @@
+cmake_minimum_required (VERSION 2.8.12)
+
+# Setup cmake policies.
+foreach(p
+    CMP0012
+    CMP0013
+    CMP0014
+    CMP0022 # CMake 2.8.12
+    CMP0025 # CMake 3.0
+    CMP0053 # CMake 3.1
+    CMP0054 # CMake 3.1
+    CMP0074 # CMake 3.12
+    CMP0075 # CMake 3.12
+    CMP0083 # CMake 3.14
+    CMP0093 # CMake 3.15
+  )
+  if(POLICY ${p})
+    cmake_policy(SET ${p} NEW)
+  endif()
+endforeach()
+
+project(METADATA_JSON_LOADER C)
+
+set(JMD_LDR_EXT_INCLUDE_DIRS "")
+set(JMD_LDR_EXT_LIBRARIES "")
+
+set(CMAKE_BUILD_TYPE RelWithDebInfo CACHE STRING "Choose the type of build." FORCE)
+# set(JMD_DEBUG ON CACHE BOOL "Enable debug mode.")
+option(JMD_DEBUG_MODE "Enable debug mode." ON)
+if(JMD_DEBUG_MODE)
+  add_definitions(-DJMD_DEBUG=1)
+endif()
+
+option(USE_SYSTEM_MPI "Use system-installed MPI." ON)
+if(USE_SYSTEM_MPI)
+  find_package(MPI)
+  if(MPI_FOUND)
+    add_definitions(-DJMD_LDR_ENABLE_MPI=1)
+    SET(CMAKE_C_COMPILER ${MPI_C_COMPILER})
+    SET(CMAKE_CXX_COMPILER ${MPI_CXX_COMPILER})
+    set(JMD_LDR_EXT_INCLUDE_DIRS ${MPI_C_INCLUDE_PATH}
+      ${JMD_LDR_EXT_INCLUDE_DIRS}
+      )
+    set(JMD_LDR_EXT_LIBRARIES ${MPI_C_LIBRARIES} ${JMD_LDR_EXT_LIBRARIES})
+  endif()
+endif()
+
+find_package(PDC REQUIRED)
+if(PDC_FOUND)
+    #message(STATUS "PDC include directory: ${PDC_INCLUDE_DIR}")
+  set(JMD_LDR_EXT_INCLUDE_DIRS ${PDC_INCLUDE_DIR}
+    ${JMD_LDR_EXT_INCLUDE_DIRS}
+    )
+  set(JMD_LDR_EXT_LIBRARIES pdc ${JMD_LDR_EXT_LIBRARIES})
+endif()
+
+
+add_library(cjson cjson/cJSON.c)
+
+add_executable(metadata_json_loader 
+            metadata_json_loader.c 
+            metadata_json_processor.h 
+            metadata_json_printer.c 
+            metadata_json_printer.h 
+            metadata_json_importer.c 
+            metadata_json_importer.h)
+target_link_libraries(metadata_json_loader ${PDC_EXT_LIB_DEPENDENCIES} pdc cjson ${JMD_LDR_EXT_LIBRARIES})
+target_include_directories(metadata_json_loader PUBLIC ${PDC_EXT_INCLUDE_DEPENDENCIES} ${JMD_LDR_EXT_INCLUDE_DIRS})