-
Notifications
You must be signed in to change notification settings - Fork 443
FileCompression
The BOINC client can handle compressed downloads using both the mod_gzip and mod_deflate mechanisms provided by Apache. However, resumption of interrupted downloads doesn't work with either of these methods, so we don't recommend using them.
For application version files, you can specify a <gzip/> flag. This will cause the file to be transferred in compressed form to 7.0+ clients.
If there is demand, we will extend this mechanism to work for job input files also.
If you include the <gzip_when_done>
tag in an output file description,
the client will gzip-compress the file after it has been generated.
The gzip_when_done is only supported in client version 5.8+. If you receive files from clients that do not support the gzip_when_done flag, then you should open the files with a function similar to this to your validator/assimilator:
int read_gzip_file_string(string file, string* result) {
FILE *infile;
char cmd[512];
char buf[4096];
result->erase();
sprintf(cmd, "gzip -dcf %s", file.c_str());
infile = popen(cmd, "r");
if (infile == NULL) {
return ERR_FOPEN;
}
while( fgets(buf, 4096, infile) != NULL ) {
result->append(buf);
}
result->append("\0");
if (pclose(infile) != 0) {
fprintf(stderr, "%s: pclose failed\n", file.c_str());
return 1;
}
return 0;
}
This will uncompress the file if it is compressed or will read it without modification if it is not compressed.
You can also do compression in your application. To assist this, BOINC provides a library boinc_zip, based on the Info-Zip libraries, but combines both zip & unzip functionality in one library. Any questions/comments please email Carl Christensen (carlgt1 at yahoo dot com)
This library can "co-exist" with zlib (libz) in case you need that too.
Basically, it will allow you to build a library that you can link
against to provide basic zip/unzip compression functionality.
It should only add a few hundred KB to your app
(basically like distributing zip
& unzip
executable binaries for different platforms).
The "unzip" functionality is there, that is you can unzip a file and it will create all directories & files in the zip file. The "zip" functionality has some limitations due to the cross-platform nature: mainly it doesn't provide zipping recursively (i.e. subdirectories); and wildcard handling is done using the "boinc_filelist" function which will be explained below.
For Windows, you can just add the project "boinc_zip" to your Visual Studio "Solution" or "Workspace." Basically just "Insert Existing Project" from the Visual Studio IDE, navigate over to the boinc/zip directory, and it should load the appropriate files. You can then build "Debug" and "Release" versions of the library. Then just add the appropriate reference to "boinc_zip.lib" (Release build) or "boinc_zipd.lib" (Debug build) in your app.
For Linux & Mac, you should be able to run "./configure" and then do a "make" to build the "libboinc_zip.a" lib that you will link against. In extreme cases, you may need to do an "aclocal && autoconf && automake" first, to build properly for your platform.
Also, please note that boinc_zip relies on some BOINC functions that you will need
(and will most likely be in your app already since they are handy) --
namely boinc/lib/filesys.cpp
and boinc/lib/util.cpp
.
Basically, you will need to #include "boinc_zip.h"
in your app
(of course your compiler will need to know where it is, i.e. -I../boinc/zip).
Then you can just call the function boinc_zip
with the appropriate arguments to zip or unzip.
There are three overloaded boinc_zip's provided:
int boinc_zip(int bZipType, const std::string szFileZip, const ZipFileList* pvectszFileIn);
int boinc_zip(int bZipType, const std::string szFileZip, const std::string szFileIn);
int boinc_zip(int bZipType, const char* szFileZip, const char* szFileIn);
bZipType
is ZIP_IT
or UNZIP_IT
(self-explanatory)
szFileZip
is the name of the zip file to create or extract
(I assume the user will provide it with the .zip extension)
The main differences are in the file parameter.
The zip library used was exhibiting odd behavior when "coexisting" with unzip,
particularly in the wildcard handling.
So a function was made that creates a ZipFileList
class,
which is basically a vector of filenames.
If you are just compressing a single file,
you can use either the std::string
or const char* szFileIn
overrides.
You can also just pass in a *
or a *.*
to zip up all files in a directory.
To zip multiple files in a "mix & match" fashion,
you can use the boinc_filelist
function provided.
Basically, it's a crude pattern matching of files in a directory,
but it has been useful for us on the CPDN project.
Just create a ZipFileList
instance, and then pass this into boinc_filelist
as follows:
bool boinc_filelist(
const std::string directory,
const std::string pattern,
ZipFileList* pList,
const unsigned char ucSort = SORT_NAME | SORT_DESCENDING,
const bool bClear = true
);
if you want to zip up all text (.txt) files in a directory,
just pass in: the directory as a std::string
, the pattern, i.e. ".txt", &yourZipList
The last two flags are the sort order of the file list
(CPDN files need to be in a certain order -- descending filenames, which is why that's the default).
The default is to "clear" your list,
you can set that to false
to keep adding files to your ZipFileList
.
When you have created your ZipFileList
just pass that pointer to boinc_zip
.
You will be able to add files in other directories this way.
There is a ziptest
Project for Windows provided to experiment,
which can also be run (the "ziptest.cpp") on Unix & Mac to experiment
with how boinc_zip
work (just g++ with the boinc/lib/filesys.cpp
& util.cpp
as described above).
These basic routines may be useful if you want to compress/decompress a file using the zlib library (usually called "libz.a" and available for most platforms). Include the header file below (qcn_gzip.h) in your program, and link against libz, and you will gain two simple to use functions for gzip'ing or gunzip'ing a file. This is for simple single file or file-by-file compression or decompression (i.e. one file that is to be compressed into a .gz or decompressed back to it's original uncompressed state). You can check for boinc client status if you want the ability to quit inside an operation etc.
qcn_gzip.h:
#ifndef _QCN_GZIP_H_
#define _QCN_GZIP_H_
#include <zlib.h>
int do_gzip(const char* strGZ, const char* strInput);
int do_gunzip(const char* strGZ, const char* strInput, bool bKeep = false);
#endif
qcn_gzip.cpp:
#include <stdio.h>
#include "filesys.h"
#include "qcn_gzip.h"
int do_gzip(const char* strGZ, const char* strInput) {
// take an input file (strInput) and turn it into a compressed file (strGZ)
// get rid of the input file after
FILE* fIn = boinc_fopen(strInput, "rb");
if (!fIn) return 1; //error
gzFile fOut = gzopen(strGZ, "wb");
if (!fOut) return 1; //error
fseek(fIn, 0, SEEK_SET); // go to the top of the files
gzseek(fOut, 0, SEEK_SET);
unsigned char buf[1024];
long lRead = 0, lWrite = 0;
while (!feof(fIn)) { // read 1KB at a time until end of file
memset(buf, 0x00, 1024);
lRead = 0;
lRead = (long) fread(buf, 1, 1024, fIn);
lWrite = (long) gzwrite(fOut, buf, lRead);
if (lRead != lWrite) break;
}
gzclose(fOut);
fclose(fIn);
if (lRead != lWrite) return 1; //error -- read bytes != written bytes
// if we made it here, it compressed OK, can erase strInput and leave
boinc_delete_file(strInput);
return 0;
}
// CMC - commented out status calls, are they too paranoid?
// if needed use sm->statusBOINC instead (for quit_request etc)
int do_gunzip(const char* strGZ, const char* strInput, bool bKeep) {
// take an input file (strInput) and turn it into a compressed file (strGZ)
// get rid of the input file after
//s.quit_request = 0;
//checkBOINCStatus();
FILE* fIn = boinc_fopen(strInput, "wb");
if (!fIn) return 1; //error
gzFile fOut = gzopen(strGZ, "rb");
if (!fOut) return 1; //error
fseek(fIn, 0, SEEK_SET); // go to the top of the files
gzseek(fOut, 0, SEEK_SET);
unsigned char buf[1024];
long lRead = 0, lWrite = 0;
while (!gzeof(fOut)) { // read 1KB at a time until end of file
memset(buf, 0x00, 1024);
lRead = 0;
lRead = (long) gzread(fOut,buf,1024);
lWrite = (long) fwrite(buf, 1, 1024, fIn);
if (lRead != lWrite) break;
//boinc_get_status(&s);
//if (s.quit_request | s.abort_request | s.no_heartbeat) break;
}
gzclose(fOut);
fclose(fIn);
//checkBOINCStatus();
if (lRead != lWrite) return 1; //error -- read bytes != written bytes
// if we made it here, it compressed OK, can erase strInput and leave
if (!bKeep) boinc_delete_file(strGZ);
return 0;
}