NEWS

Copyright (C) 2010, 2011, 2012, 2013, Steffen Knollmann

This file contains an overview of the major enhancement, bug fixes, or
other things for each version of ginnungagap since version 0.5.0.  If
relevant, the entries contain the short hash of the according commits to
the repository, smaller changes will not be listed individually but
paraphrased.


Version 0.7.0
-------------

This version brings a new tool to generate particle realisations of
velocity fields: generateICs, which supports HDF5 and Grafic inputs,
especially the former is relevant to support large resolutions which are
inefficient to write in Grafic format.

Changes from 0.6.3:
 134 files changed, 9523 insertions(+), 268 deletions(-)

(e7c24eb) [FIX] Missing doGas check
    The creation of gas particles should only happen, when gas particles are 
    requested.  There was an if statement missing...

(407c67d) [FIX] Replaces scaling with type generic function
    The scaling currently calculates in double space and only converts to 
    the target precision at the very end.  That potentially might lead to 
    rounding issues.  This change will compute in position in double, 
    convert to target precision and then apply the modulo operation in the 
    target precision.

(e0635b8) [FIX] Number of particles is overflowing
    When calculating the total number of particles from three 32bit 
    integers, they need to be cast to 64bit integers to prevent overflows. 
    Well, technically it could still overflow, but that is unlikely to 
    happen.

(33bf39e) [FIX] Histogram doesn't treat some datatypes
    The histogram needs to deal with float, int32 and int64.

(f40ca64) [WIP] More clean-up work in generateICs
    Now using libpart to create a partBunch structure that holds the 
    particles instead of having naked allocated arrays.  Now also supporting 
    the creation of gas particles and processing of more than one tile per 
    file.

(edf0466) [WIP] Starts cleaning up generateICs
    Generating Gadget ICs from HDF5 velocity fields works now in hand-tuned 
    circumstances.  Time to enable generateICs in the general makefile 
    system.

(45afa65) [WIP] Adds HDF5 reading
    The reader are generated from the ini file.  The ini file reading has 
    been revamped to leverage of the standard ginnungagap config (i.e. the 
    boxsize and zInit are read from the [Ginnungagap] section).

    The whole business with the fake mask is a bit fishy, to get the cell 
    count work out I had to hand-tune the hierarchy to start from 1x1x1 in 
    factors of 2.

(3841b08) [WIP] Restructures genics code
    Encapsulates some of the data used in the genics code: mode related 
    Booleans, resolution independent quantities (e.g. box-size, cosmological 
    model), and output related things (e.g. Gadget skeleton objects, number 
    of output files).

    Also fiddles with the mock code to read data from external files, 
    currently aiming at providing a version that can deal with the single 
    resolution, full-box situation (not working yet).  Copies code from 
    grafic2gadget.

(d45d1bd) Changes the way IDs are assigned in grafic2gadget
    The IDs will now start at 1 instead of 0.  Also, if gas particles are 
    present, their IDs will now start at 1, the DM particles start at 
    numGasParticles + 1 (change requested by Gustavo to be compatible with 
    previous runs).

    This will potentially break lareReaderLOI, which assumes IDs to start at 
    0. But since this reader isn't actually used that should be of no 
    consequence.  In any case the resolving of IDs to grid positions needs 
    to be investigated carefully.

(d078dfe) Merges work-in-progress for zooms into develop for more testing
    and more organised continued work on the zooms.


Version 0.6.3
-------------

This is a bug-fix release correcting issues with the longIDs options
of grafic2gadget appearing when trying to use more than 2^32 particles
and/or gas particles.

Changes from 0.6.2:
 1 file changed, 53 insertions(+), 37 deletions(-)

(d4bebd4) [FIX] Fixes integer overflows in grafic2gadget
    This is a modified version of a patch provided by Gustavo.

    When computing number of particles as a product of the components of a 
    32bit array, then the individual operation are performed in 32bit, thus 
    potentially overflowing, even though the result is assigned to an 64bit 
    integer.

    The code now uses an array of 64bit integers to store the number of 
    particles in each dimension, thus eliminating the intermediate 32bit 
    multiplications.  Additionally, the overflow check function will now 
    verify that the total number of particles fits in a 64bit integer (will 
    be some time before that limit is reached..) and terminate the 
    generation if a 64bit integer would be overflowing.

    The 32->64bit conversion is also done in the function initialising 
    positions/ids for 32bit IDs.  grafic2gadget does allow to generate IDs 
    in 32bit even though those are not enough, it merely emits a warning 
    that the IDs are overflowing.  So technically the same problem could 
    appear there.

    The function initialising positions and 64bit IDs also had a bug when 
    gas particles are used: The arrays were indexed by using numTotal 
    instead of numOffset, wildly running over array bounds and segfaulting.

Version 0.6.2
-------------

This is a bug-fix release correcting an issue with the longIDs options
of grafic2gadget.

Changes from 0.6.1:
 5 files changed, 68 insertions(+), 6 deletions(-)

(c308eda) [FIX] Fixes issue with long ids in Gadget module.
    When calculating the size of an element in a block, the ID block needs 
    to be treated separately from the floating-point blocks, i.e. the 
    previous check whether the file is written in double precision is not 
    correct to get the size of ID (which is quite independent on what the 
    floating point precision is).

    Fixed by checking if the block for to calculate the size is the ID
    block and then deducing the size per element from the state of the long
    ID flag of the header code structure (note that Gadget files do not
    store the width of the IDs explicitly, they have to be deduced from the
    block size).

    Adds a new test to check the sizes of elements in blocks to verify that 
    all combinations of single/double precision and 32/64 bit IDs report
    the correct sizes.

    Also corrects the description of the flag values in the doxygen comment 
    for gadgetHeader_setFlagDoublePrecision().

Version 0.6.1
-------------

This is the release of version 0.6.1 bringing the addition of small
command line tools to work with various file types (Gadget, CubePM,
ART) and options for grafic2gadget to change the velocity and positions
units (allowing to write files in kpc/h instead of Mpc/h).

Changes from 0.6.0:
 47 files changed, 9231 insertions(+), 362 deletions(-)

(56eb90a) Adds options to choose units in grafic2gadget.
    This adds the command line options --posFactor and --velFactor to 
    grafic2gadget, which work as multiplier when last touching the 
    position/velocity arrays.  The position factor is a bit special, as it 
    also is used to write the boxsize parameter of the Gadget file header
    in the appropriate unit.
    Using
      grafic2gadget ic_velx ic_vely ic_velz icMpc.gadget and
      grafic2gadget --posFactor 1000. ic_velx ic_vely ic_velz ickpc.gadget 
    will produce two Gadget files, the first one with positions in Mpc/h, 
    the second one with positions in kpc/h.

(ece51e9) Merge branch 'fancyGadget' into develop
    This overhauls the underlying code dealing with Gadget files,
    allowing for more flexibility in handling.

(2190bee) Adds the fileTools.
    This is a collection of small commandline tools to explore different 
    cosmological file types.  Currently supported are ART, CubePM, and 
    Gadget.

Version 0.6.0
-------------

This is the release of version 0.6.0 bringing major changes to the
grid writing and reading infrastructure (see below for more details).
Most notably, the HDF5 support has been finished and shows excellent
performance on curie (~5GB/s writing, ~10GB/s reading).

Additionally, this release finally fixes the permutation error of the
velocity fields in the OpenMP-only case (see 982f47b below).  On other
fronts the code documentation has been updated, now completely
documenting the code configuration files (mainly relevant config.h.in).
Ginnungagap is now available from: https://code.google.com/p/ginnungagap/

Changes from 0.5.3-alpha:
 157 files changed, 8238 insertions(+), 2274 deletions(-)

(982f47b) [FIX] Resolves the permutation error in OpenMP.
    This solves the problem that the velocity fields in the OpenMP only run 
    were permuted.  This was done by finally evaluating the actual 
    permutation of the grid instead of relying on a hardcoded permutation
    in g9pInit.c that was only correct for the MPI case.  The grid now has
    an additional function that returns the current dimenions (or rather,
    the index with 0 being x, 1, being y, and so on) of an original
    dimension.

(b2db841) Documentation update.
    This finishes the description of how to configure the build of 
    ginnungagap.  Additionally, it adds a quick-start example run-through
    to explain how to use ginnungagap.  The example ini-file in the
    'Writing Input Files' section has been adjusted to the slightly changed
    ini format.

(7ed0f30) [FIX] Adds missing ifdef guard around HDF5 things.
    The factory methods for the HDF5 reader should only be compiled if HDF5 
    is available.

(a012fba) Minor tweaks in the WN writing/reading.
    This changes the way the WN reader/writer are setup in ginnungagap, 
    instead of having them constructed from the [WhiteNoise] section, they 
    are now constructed from bespoke sections names there.

    Also, this adds two example ini files for use with ginnungagap.

(b5f9b8a) Removes the (dis-functional) SILO grid reader.
    The SILO grid reader is now removed from the code basis, it never
    worked anyways and there is no point in carrying aroung bit-rotting
    code.

(d51db07) Updates the grid reader infrastructure.
    The grid reader now uses a similar setup as the grid writer, common IO 
    functionality between reading and writing is outsourced to a separate 
    file.  This changes also changes the INI format for the grid reader, 
    most notably, the type is now specified as 'type = ..' instead of
    'readerType = ..'.  Additionally, all reader will now read a filename 
    split up into path, prefix, qualifier, and suffix; this replaces the
    old
    'graficFileName = ..' et al.

    Furthermore, the grid reading is now included in the unit testing of
    the grid library (at least for BOV and HDF5) and the code documentation
    has been updated.  The sample ini file for 'realSpaceConstraints' has
    been adjusted to work with the new reading routines.

    The main code (and tools) only had to be changed trivially to account 
    for the change.

(f9699a5) Overhauling the grid writer.
    The writer now uses a streamlined code structure under the hood and 
    allows now for more flexibility in writing.  The construction of
    writers from ini files has been factored out into a factory module and
    during that, the structure of the ini files changed slightly.

    The grafic writer now will not cycle through the _delta, _velx and so
    on names anymore but relies on the user to set the proper file name. 
    The HDF5 writer properly deals with multiple calls.  The Silo writer
    has been adjusted to reflect the changes in the code structure.

    As a main feature, the grid writer now will never overwrite a file 
    unless explicitly requested by the user.

    On a side, the whole code basis has been adjusted to now work with all 
    three grid writers for generating velocity fields et al.  This required 
    including explicit renaming calls in ginnungagap to keep the names in 
    the files (when the file type supports that: HDF5, SILO) consistent
    with what is actually in the file.  Also, ginnungagap will now write
    separate files for the overdensity and for each of the velocity
    components (possibly spilt over many subfiles, as in the case of
    Silo in MPI mode).

(8205b8f) Documenation updates.
    Now config.h and version.h are fully documented.  Actually, the 
    documentation is in the .in files, as they are used to auto-generate
    the actual files.

(9a18057) Adds a filename module to the utility library.
    Fri Jan 13 03:01:44 2012 +0000

(bb3df6d) Sets the path for HDF5 on curie.
    Wed Jan 11 02:52:55 2012 +0100

Version 0.5.3-alpha
-------------------

This is the alpha release of ginnungagap-0.5.3 focussing mostly on HDF5
support (working for WN data in read/write and for all other data in
write-only).  Additionally, the realSpaceConstraints tool is now MPI
parallel and some additions have been made for 2LPT corrections (not
fully supported yet).

Changes from 0.5.2-alpha:
 44 files changed, 1486 insertions(+), 281 deletions(-)

* (91eb7a3) Adds a gnuplot script for plotting grafic files.
    This is meant as an example that quickly references the required syntax 
    for GNUPlot (>= 4.5 required).

* (053c8f9) Major HDF5 update: supporting read/write.
    This merges the HDF5 writing and reading for ginnungagap.  Note that 
    while writing basically supports compression and checksumming, the 
    reader does not.  Also, the reader currently only works for white noise 
    fields generated by realSpaceConstraints.  This limitation is due to
    the ambiguity of how to map file datasets to memory datasets.

* (2d00902) Enables MPI capabilites for realSpaceConstraints.
    This essentially parallelizes the generation of white noise fields over 
    the z-direction of the lowest resolution grid.

* (144cd08) Adjusts validation of input parameters.
    The largest sigma that is expected to occur in the white noise field is 
    now calculated and used to estimate whether the starting redshift is 
    hight enough.  The starting redshift is high enough, if the largest 
    overdensity is less than one.  The largest overdensity is given by the 
    density variance in the box (sigmaBox) times the largest expected value 
    in the white noise field.  The code will now terminate, if the largest
    (expected) overdensity is larger than 1.

* (82b8279) Generating a histogram of the white noise.
    If histograms are enabled, this will now also produce a histogram for 
    the white noise data.  This requires two additional new fields in the 
    ini file to set the extreme values for the white noise and the name of 
    the file the white noise histogram is written to.  This is analogous to 
    the current handling of histograms.

* (0fb043c) Extra information in the printing of histograms.

* (c741eeb) Merges changes to allow gas ICs to be generated.
    This includes changes to grafic2gadget to allow the generation of gas 
    particles.  The IDs of the gas particles will be arrange in such a way 
    that they range after the DM IDs, this should keep the IDs for DM 
    particles identical between (full-box) runs with and without gas.  Gas 
    particles will have the same velocity as DM particles but are shifted
    by a constant fraction of a cell.

* (dca5951) Prevents calculation of P(k) for small grids.
    If the grid is too small, the power spectrum calculation may segfault
    as it uses some magic to estimate the slope at low and high
    frequencies. That requires a decent amount of samples which is not
    fulfilled for small grids.

* (ea0d78a) Now also writing z to Gadget headers.
    Only a was set explicitly in the grafic2gadget conversion utility.

* (1938740) 2LPT related changes and additions.
    This adds the calculation of the second derivatives to the IC part, the 
    required cosmology routines to calculate the second order growth terms 
    and changes the initGrid routine to only accept the values it needs to 
    work and nothing more.

* (5a76b03) Ensure correct type for large indices.
    Wed Nov 9 14:13:01 2011 +0100
    This changes the loop counters of the dimensions to 64bit integers to 
    ensure that the calculation of indices also in done with 64bit
    integers. The result always was a 64bit integer, but as a sum/product
    of 32bit integers which might silently overflow.

* (3dace56) Prepares for 2LPT corrections.
    Introduces a new key in the main section of the code to toggle between 
    the usage of second order corrections.  The default is to not calculate 
    the second order corrections.

Version 0.5.2-alpha
-------------------

This is the alpha release of ginnungagap-0.5.2.  This brings writing
support for HDF5 and improves the build system.  The biggest chance is
the way how MPI is handled in the build system.
Changes from 0.5.1-alpha:
 46 files changed, 2039 insertions(+), 406 deletions(-)

* (4d4de93) Adds curie to the systems.

* (17d9d4e) Adds erebos to the known systems.
    This sets the SPRNG and FFT details for erebos to use the libraries 
    installed in /z/knolli/opt/.  Setting the LD_LIBRARY_PATH is still the 
    responsibility of the user.  The system settings differ depending on
    the choice of --toolchain (intel and gcc are dealt with).

* (15c25b5) Fixes segfault related to histograms.
    When the calculation of histograms is switched off in the ini file (or 
    rather not switched on), then they really shouldn't be done.

* (40af9c2) Fixes filtering/chunking for the HDF5 writer.
    This now allows the writer to write to chunked data sets and apply 
    various filters to the data set.  Filters are, however, only available 
    in the serial version of the library, hence all filters are switched
    off the hard way, when the code is compiled with MPI.

* (a6261ee) HDF5 writer can now write grids.
    This implements the writing of grids (and, untested, patches). 
    Theoretically, it also supports compression and checksumming, albeit 
    untested.

* (1a84d88) Allows distributed grids to always get the rank.
    Retrieving the local rank of the calling process within the grid 
    distribution was a feature of the MPI-version of the distribution 
    module.  This is now always available and will return the rank 0 if the 
    distribution is not in MPI mode.  Test functions have been adjusted 
    accordingly.

* (94d6069) Fixes issue with MPI tests and adds OpenMP to MPI.
    Will set a Makefile variable for mpiexec which will include the path as 
    given by WITH_MPI_BIN_DIR.  Note that this directory must be either 
    given by the user, or it will be constructed by 'dirname' from
    'which mpicc'.

    Also, the build system now re-enables MPI+OpenMP by adding the required 
    OpenMP flags not only to CC but also to MPICC.

* (d951872) Fixes issues with the pdf documentation.
    When generating the latex documentation it seems that doxygen produces 
    broken TeX code if name-block contain any text.  This is circumvented 
    now by either moving the extra text to the file description, or
    deleting it.

* (ef7524c) Changes how MPI is handled in the build system.
    This will now explicitly set and use MPICC.  Makefiles that deal with 
    code that has to be compiled with MPI now will set CC to MPICC.  That 
    allows to compile code that has no dependency on MPI without any 
    reference to MPI.  Implementation of this change required adjusting
    most of the Makefiles and additionally a few irrelevant MPI
    dependencies have been removed from a few test functions.

    The way MPI is handled in by the configure is now:  --with-mpi will
    take an argument that is either no to not use MPI at all, or yes, which
    will use MPI and assume the MPI compiler to be 'mpicc', or any other
    word that will then be taken as the value for MPICC.  This should now
    be the executable without any path.  If the MPICC is in a non-standard 
    directory, then --with-mpi-bin-dir can be used to point to the proper 
    directory in which MPICC can be found.

    The configure script will now report which MPICC is being used and 
    include the full path in the output.  The path is either taken from the 
    user via --with-mpi-bin-dir or deduced via 'which MPICC' (that requires 
    MPICC to be in the PATH).

    All these changes are now used in the example configure.systems.own (in 
    doc/devel/) to allow for a compilation without sourcing any module. 
    Note that the build system does not try to resolve runtime dependency, 
    i.e. to run the code the user needs to ensure that all shared libraries 
    are actually found by the linker.

* (2bfbe5d) Adds a configure.systems.own for development.
    This is only for internal use, though users might find it helpful to 
    have a look at the file.

* (d1ebdd0) Documentation update for gridPatch.
    This will now completely document the gridPatch module.  Additionally,
    a few missing const qualifiers have been added to some functions.

* (d66541c) Adds a gridPatch function to retrieve data by var.
    This now allows to retrieve the data handle of a variable attached to a 
    patch by searching for the variable itself.  Previously, the only way
    to get the data handle was by passing the index of the data.

* (9823274) Fixes problems with HDF5 libs.
    The HDF5 libraries where not properly propagated through the build 
    chain.

* (cd2d61e) Changes the way configure deals with HDF5.
    Now that HDF5 is itself a supported output format of ginnungagap, the 
    configure script has been adapted so that the HDF5 backend of Silo is 
    not chosen automatically when --with-hdf5 is specified.  To use the
    HDF5 backend, --with-silo=hdf5 must be used now.


Version 0.5.1-alpha
-------------------

This is the alpha release of ginnungagap-0.5.1.  The release focuses
mainly on the build system and documentation updates.  The main addition
to the code base is the framework for the HDF5 reader/writer for the
grid library.
Changes from 0.5.0-alpha:
 58 files changed, 1932 insertions(+), 404 deletions(-)

* (2624744) Adds option for predefined systems to configure.
    This now allows to use --system=SYSTEM with the configure script to 
    configure the code according to a predefined set for a given machine. 
    No actual predefined system is included yet, only the system 'full' is 
    available, which will switch on all features the code has.

    The users may define their own systems by means of writing their own 
    definition in an extra file (configure.systems.own) that will be 
    evaluated if it exists.  An example of this file is included in 
    doc/examples/.

* (b4192e4) Adds a file to collect useful git commands.
    This is mainly meant to have a set of git commands related to querying
    the history of the repository for writing change logs.  Before they are
    forgotten yet again, it seems better to have a cheat sheet.

* (fbfbeaa) Adds an install target to the Makefile.
    This now allows to 'install' all generated binaries into a (possibly 
    user-defined) directory.  The default is to move the binaries from
    their respective source directory into bin/ in the top directory.

* (816658f) Updates doxygen configuration.
    Now also extracting static members (so that the call graph images work 
    smoothly) and adding a tag line to the project documentation.

* (b8fad9d) Adds stubs for the HDF5 libgrid-IO.
    This only provides a compilable set of files which at the moment do not 
    actually do anything.  They cause a bunch of compiler warnings about 
    unused variables.

* Numerous updates to the code documentation.


Version 0.5.0-alpha
-------------------

This is the alpha release of ginnungagap-0.5.0.  This is essentially
the version that was presented at the ginnungagap workshop in Madrid
on the 21th of September 2011.  This version includes two small fixes
to the realSpaceConstraints program that fix is to also work when
compiled with MPI.