NEWS

Noteworthy changes in release a.b
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* hts_srand48() now seeds the same POSIX-standard sequences of pseudo-random
  numbers regardless of platform, including on OpenBSD where plain srand48()
  produces a different cryptographically-strong non-deterministic sequence.


Noteworthy changes in release 1.10.2 (19th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is a release fix that corrects minor inconsistencies discovered in
previous deliverables.


Noteworthy changes in release 1.10.1 (17th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The support for 64-bit coordinates in VCF brought problems for files
not conforming to VCF/BCF specification. While previous versions would
make out-of-range values silently overflow creating nonsense values
but parseable file, the version 1.10 would silently create an invalid BCF.


Noteworthy changes in release 1.10 (6th December 2019)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Brief summary
-------------

There are many changes in this release, so the executive summary is:

* Addition of support for references longer than 2Gb (NB: SAM and VCF
  formats only, not their binary counterparts).  This may need changes
  in code using HTSlib.  See README.large_positions.md for more information.

* Added a SAM header API.

* Major speed up to SAM reading and writing.  This also now supports
  multi-threading.

* We can now auto-index on-the-fly while writing a file.  This also
  includes to bgzipped SAM.gz.

* Overhaul of the S3 interface, which now supports version 4
  signatures.  This also makes writing to S3 work.

These also required some ABI changes.  See below for full details.


Features / updates
------------------

* A new SAM/BAM/CRAM header API has been added to HTSlib, allowing header
  data to be updated without having to parse or rewrite large parts of the
  header text.  See htslib/sam.h for function definitions and
  documentation. (#812)

  The header typedef and several pre-existing functions have been renamed
  to have a sam_hdr_ prefix: sam_hdr_t, sam_hdr_init(), sam_hdr_destroy(),
  and sam_hdr_dup().  (The existing bam_hdr_-prefixed names are still
  provided for compatibility with existing code.) (#887, thanks to
  John Marshall)

* Changes to hfile_s3, which provides support for the AWS S3 API. (#839)

  - hfile_s3 now uses version 4 signatures by default.  Attempting to write to
    an S3 bucket will also now work correctly.  It is possible to force
    version 2 signatures by creating environment variable HTS_S3_V2 (the exact
    value does not matter, it just has to exist).  Note that writing depends
    on features that need version 4 signatures, so forcing version 2 will
    disable writes.

  - hfile_s3 will automatically retry requests where the region endpoint
    was not specified correctly, either by following the 301 redirect (when
    using path-style requests) or reading the 400 response (when using
    virtual-hosted style requests and version 4 signatures).  The first
    region to try can be set by using the AWS_DEFAULT_REGION environment
    variable, by setting "region" in ".aws/credentials" or by setting
    "bucket_location" in ".s3cfg".

  - hfile_s3 now percent-escapes the path component of s3:// URLs.  For
    backwards-compatibility it will ignore any paths that have already
    been escaped (detected by looking for '%' followed by two hexadecimal
    digits.)

  - New environment variables HTS_S3_V2, HTS_S3_HOST, HTS_S3_S3CFG
    and HTS_S3_PART_SIZE to force version-2 signatures, control the
    S3 server hostname, the configuration file and upload chunk
    sizes respectively.

* Numerous SAM format improvements.

  - Bgzipped SAM files can now be indexed and queried.  The library now
    recognises sam.gz as a format name to ease this usage. (#718, #916)

  - The SAM reader and writer now supports multi-threading via the
    thread-pool. (#916)

    Note that the multi-threaded SAM reader does not currently support seek
    operations.  Trying to do this (for example with an iterator range request)
    will result in the SAM readers dropping back to single-threaded mode.

  - Major speed up of SAM decoding and encoding, by around 2x. (#722)

  - SAM format can now handle 64-bit coordinates and references.  This
    has implications for the ABI too (see below).  Note BAM and CRAM
    currently cannot handle references longer than 2Gb, however given
    the speed and threading improvements SAM.gz is a viable workaround. (#709)

* We can now automatically build indices on-the-fly while writing
  SAM, BAM, CRAM, VCF and BCF files.  (Note for SAM and VCF this only
  works when bgzipped.) (#718)

* HTSlib now supports the @SQ-AN header field, which lists alternative names
  for reference sequences.  This means given "@SQ SN:1 AN:chr1", tools like
  samtools can accept requests for "1" or "chr1" equivalently.  (#931)

* Zero-length files are no longer considered to be valid SAM files
  (with no header and no alignments).  This has been changed so that pipelines
  such as `somecmd | samtools ...` with `somecmd` aborting before outputting
  anything will now propagate the error to the second command. (#721, thanks
  to John Marshall; #261 reported by Adrian Tan)

* Added support for use of non-standard index names by pasting the
  data filename and index filename with ##idx##.  For example
  "/path1/my_data.bam##idx##/path2/my_index.csi" will open bam file
  "/path1/my_data.bam" and index file "/path2/my_index.csi". (#884)

  This affects hts_idx_load() and hts_open() functions.

* Improved the region parsing code to handle colons in reference
  names.  Strings can be disambiguated by the use of braces, so for
  example when reference sequences called "chr1" and "chr1:100-200"
  are both present, the regions "{chr1}:100-200" and "{chr1:100-200}"
  unambiguously indicate which reference is being used. (#708)

  A new function hts_parse_region() has been added along with
  specialisations for sam_parse_region() and fai_parse_region().

* CRAM encoding now has additional checks for MD/NM validity.  If
  they are incorrect, it stores the (incorrect copy) verbatim so
  round-trips "work". (#792)

* Sped up decoding of CRAM by around 10% when the MD tag is being
  generated. (#874)

* CRAM REF_PATH now supports %Ns (where N is a single digit)
  expansion in http URLs, similar to how it already supported this
  for directories. (#791)

* BGZF now permits indexing and seeking using virtual offsets in
  completely uncompressed streams. (#904, thanks to Adam Novak)

* bgzip now asks for extra confirmation before decompressing files
  that don't have a known compression extension (e.g. .gz).  This avoids
  `bgzip -d foo.bam.bai` producing a foo.bam file that is very much not
  a BAM-formatted file. (#927, thanks to John Marshall)

* The htsfile utility can now copy files (including to/from URLs using
  HTSlib's remote access facilities) with the --copy option, in
  addition to its existing uses of identifying file formats and
  displaying sequence or variant data. (#756, thanks to John Marshall)

* Added tabix --min-shift option. (#752, thanks to Garrett Stevens)

* Tabix now has an -D option to disable storing a local copy of a
  remote index. (#870)

* Improved support for MSYS Windows compiler environment. (#966)

* External htslib plugins are now supported on Windows. (#966)


API additions and improvements
------------------------------

* New API functions bam_set_mempolicy() and bam_get_mempolicy() have
  been added.  These allow more control over the ownership of bam1_t
  alignment record data; see documentation in htslib/sam.h for more
  information. (#922)

* Added more HTS_RESULT_USED checks, this time for VCF I/O. (#805)

* khash can now hash kstrings.  This makes it easier to hash
  non-NUL-terminated strings. (#713)

* New haddextension() filename extension API function. (#788, thanks to
  John Marshall)

* New hts_resize() macro, designed to replace uses of hts_expand()
  and hts_expand0(). (#805)

* Added way of cleaning up unused jobs in the thread pool via the new
  hts_tpool_dispatch3() function. (#830)

* New API functions hts_reglist_create() and sam_itr_regarray() are added
  to create hts_reglist_t region lists from `chr:<from>-<to>` type region
  specifiers. (#836)

* Ksort has been improved to facilitate library use.  See KSORT_INIT2
  (adds scope / namespace capabilities) and KSORT_INIT_STATIC interfaces.
  (#851, thanks to John Marshall)

* New kstring functions (#879):
     KS_INITIALIZE   -  Initializer for structure assignment
     ks_initialize() -  Initializer for pointed-to kstrings
     ks_expand()     -  Increase kstring capacity by a given amount
     ks_clear()      -  Set kstring length to zero
     ks_free()       -  Free the underlying buffer
     ks_c_str()      -  Returns the kstring buffer as a const char *,
                        or an empty string if the length is zero.

* New API functions hts_idx_load3(), sam_index_load3(), tbx_index_load3()
  and bcf_index_load3() have been added.  These allow control of whether
  remote indexes should be cached locally, and allow the error message
  printed when the index does not exist to be suppressed. (#870)

* Improved hts_detect_format() so it no longer assumes all text is
  SAM unless positively identified otherwise.  It also makes a stab
  at detecting bzip2 format and identifying BED, FASTA and FASTQ
  files. (#721, thanks to John Marshall; #200, #719 both reported by
  Torsten Seemann)

* File format errors now set errno to EFTYPE (BSD, MacOS) when
  available instead of ENOEXEC. (#721)

* New API function bam_set_qname (#942)

* In addition to the existing hts_version() function, which reflects the
  HTSlib version being used at runtime, <htslib/hts.h> now also provides
  HTS_VERSION, a preprocessor macro reflecting the HTSlib version that
  a program is being compiled against.  (#951, thanks to John Marshall; #794)


ABI changes
-----------

This release contains a number of things which change the Application
Binary Interface (ABI).  This means code compiled against an earlier
library will require recompiling.  The shared library soversion has
been bumped.

* On systems that support it, the default symbol visibility has been
  changed to hidden and the only exported symbols are ones that form part
  of the officially supported ABI.  This is to make clear exactly which
  symbols are considered parts of the library interface.  It also
  helps packagers who want to check compatibility between HTSlib versions.
  (#946; see for example issues #311, #616, and #695)

* HTSlib now supports 64 bit reference positions.  This means several
  structures, function parameters, and return values have been made bigger
  to allow larger values to be stored.  While most code that uses
  HTSlib interfaces should still build after this change, some alterations
  may be needed - notably to printf() formats where the values of structure
  members are being printed. (#709)

  Due to file format limitations, large positions are only supported
  when reading and writing SAM and VCF files.

  See README.large_positions.md for more information.

* An extra field has been added to the kbitset_t struct so bitsets can
  be made smaller (and later enlarged) without involving memory allocation.
  (#710, thanks to John Marshall)

* A new field has been added to the bam_pileup1_t structure to keep track
  of which CIGAR operator is being processed.  This is used by a new
  bam_plp_insertion() function which can be used to return the sequence of
  any inserted bases at a given pileup location.  If the alignment includes
  CIGAR P operators, the returned sequence will include pads. (#699)

* The hts_itr_t and hts_itr_multi_t structures have been merged and can be
  used interchangeably.  Extra fields have been added to hts_itr_t to support
  this.  hts_itr_multi_t is now a typedef for hts_itr_t; sam_itr_multi_next()
  is now an alias for sam_itr_next() and hts_itr_multi_destroy() is an alias
  for hts_itr_destroy(). (#836)

* An improved regidx interface has been added.  To allow this, struct
  reg_t has been removed, regitr_t has been modified and various new
  API functions have been added to htslib/regidx.h.  While parts of
  the old regidx API have been retained for backwards compatibility,
  it is recommended that all code using regidx should be changed to use
  the new interface. (#761)

* Elements in the hts_reglist_t structure have been reordered slightly
  so that they pack together better. (#761)

* bgzf_utell() and bgzf_useek() now use type off_t instead of long for
  the offset.  This allows them to work correctly on files longer than
  2G bytes on Windows and 32-bit Linux. (#868)

* A number of functions that used to return void now return int so that
  they can report problems like memory allocation failures.  Callers
  should take care to check the return values from these functions. (#834)

  The affected functions are:
     ksort.h:             ks_introsort(), ks_mergesort()
     sam.h:               bam_mplp_init_overlaps()
     synced_bcf_reader.h: bcf_sr_regions_flush()
     vcf.h:               bcf_format_gt(), bcf_fmt_array(),
                            bcf_enc_int1(), bcf_enc_size(),
                          bcf_enc_vchar(), bcf_enc_vfloat(), bcf_enc_vint(),
                            bcf_hdr_set_version(), bcf_hrec_format()
     vcfutils.h:          bcf_remove_alleles()

* bcf_set_variant_type() now outputs VCF_OVERLAP for spanning
  deletions (ALT=*). (#726)

* A new field (hrecs) has been added to the bam_hdr_t structure for
  use by the new header API.  The old sdict field is now not used and
  marked as deprecated.  The l_text field has been changed from uint32_t
  to size_t, to allow for very large headers in SAM files.  The text
  and l_text fields have been left for backwards compatibility, but
  should not be accessed directly in code that uses the new header API.
  To access the header text, the new functions sam_hdr_length() and
  sam_hdr_str() should be used instead. (#812)

* The old cigar_tab field is now marked as deprecated; use the new
  bam_cigar_table[] instead. (#891, thanks to John Marshall)

* The bam1_core_t structure's l_qname and l_extranul fields have been
  rearranged and enlarged; l_qname still includes the extra NULs.
  (Almost all code should use bam_get_qname(), bam_get_cigar(), etc,
  and has no need to use these fields directly.)  HTSlib now supports
  the SAM specification's full 254 QNAME length again.  (#900, thanks
  to John Marshall; #520)

* bcf_index_load() no longer tries the '.tbi' suffix when looking for
  BCF index files (.tbi indexes are for text files, not binary BCF). (#870)

* htsFile has a new 'state' member to support SAM multi-threading. (#916)

* A new field has been added to the bam1_t structure, and others
  have been rearranged to remove structure holes. (#709; #922)


Bug fixes
---------

* Several BGZF format fixes:

  - Support for multi-member gzip files. (#744, thanks to Adam Novak; #742)

  - Fixed error handling code for native gzip formatted files. (64c4927)

  - CRCs checked when threading too (previously only when non-threaded). (#745)

  - Made bgzf_useek function work with threads. (#818)

  - Fixed rare threading deadlocks. (#831)

  - Reading of very short files (<28 bytes) that do not contain an EOF block.
    (#910)

* Fixed some thread pool deadlocks caused by race conditions. (#746, #906)

* Many additional memory allocation checks in VCF, BCF, SAM and CRAM
  code. This also changes the return type of some functions.  See ABI
  changes above. (#920 amongst others)

* Replace some sam parsing abort() calls with proper errors.
  (#721, thanks to John Marshall; #576)

* Fixed to permit SAM read names of length 252 to 254 (the maximum
  specified by the SAM specification). (#900, thanks to John Marshall)

* Fixed mpileup overlap detection heuristic to work with BAMs having
  long CIGARs (more than 65536 operations). (#802)

* Security fix: CIGAR strings starting with the "N" operation can no
  longer cause underflow on the bam CIGAR structure.  Similarly CIGAR
  strings that are entirely "D" ops could leak the contents of
  uninitialised variables. (#699)

* Fixed bug where alignments starting 0M could cause an invalid
  memory access in sam_prob_realn(). (#699)

* Fixed out of bounds memory access in mpileup when given a reference
  with binary characters (top-bit set). (#808, thanks to John Marshall)

* Fixed crash in mpileup overlap_push() function. (#882; #852 reported
  by Pierre Lindenbaum)

* Fixed various potential CRAM memory leaks when recovering from
  error cases.

* Fixed CRAM index queries for unmapped reads (#911; samtools/samtools#958
  reported by @acorvelo)

* Fixed the combination of CRAM embedded references and multiple
  slices per container.  This was incorrectly setting the header
  MD5sum.  (No impact on default CRAM behaviour.) (b2552fd)

* Removed unwanted explicit data flushing in CRAM writing, which on
  some OSes caused major slowdowns. (#883)

* Fixed inefficiencies in CRAM encoding when many small references
  occur within the middle of large chromosomes.  Previously it
  switched into multi-ref mode, but not back out of it which caused
  the read POS field to be stored poorly. (#896)

* Fixed CRAM handling of references when the order of sequences in a
  supplied fasta file differs to the order of the @SQ headers. (#935)

* Fixed BAM and CRAM multi-threaded decoding when used in conjunction
  with the multi-region iterator. (#830; #577, #822, #926 all reported by
  Brent Pedersen)

* Removed some unaligned memory accesses in CRAM encoder and
  undefined behaviour in BCF reading (#867, thanks to David Seifert)

* Repeated calling of bcf_empty() no longer crashes. (#741)

* Fixed bug where some 8 or 16-bit negative integers were stored using values
  reserved by the BCF specification.  These numbers are now promoted to the
  next size up, so -121 to -128 are stored using at least 16 bits, and -32761
  to -32768 are stored using 32 bits.

  Note that while BCF files affected by this bug are technically incorrect,
  it is still possible to read them.  When converting to VCF format,
  HTSlib (and therefore bcftools) will interpret the values as intended
  and write out the correct negative numbers. (#766, thanks to John Marshall;
  samtools/bcftools#874)

* Allow repeated invocations of bcf_update_info() and bcf_update_format_*()
  functions. (#856, thanks to John Marshall; #813 reported by Steffen Möller)

* Memory leak removed in knetfile's kftp_parse_url() function. (#759, thanks
  to David Alexander)

* Fixed various crashes found by libfuzzer (invalid data leading to
  errors), mostly but not exclusively in CRAM, VCF and BCF decoding. (#805)

* Improved robustness of BAI and CSI index creation and loading. (#870; #967)

* Prevent (invalid) creation of TBI indices for BCF files.
  (#837; samtools/bcftools#707)

* Better parsing of handling of remote URLs with ?param=val
  components and their interaction with remote index URLs. (#790; #784
  reported by Mark Ebbert)

* hts_idx_load() now checks locally for all possible index names before
  attempting to download a remote index.  It also checks that the remote
  file it downloads is actually an index before trying to save and use
  it.  (#870; samtools/samtools#1045 reported by Albert Vilella)

* hts_open_format() now honours the compression field, no longer also
  requiring an explicit "z" in the mode string.  Also fixed a 1 byte
  buffer overrun. (#880)

* Removed duplicate hts_tpool_process_flush prototype. (#816, reported by
  James S Blachly)

* Deleted defunct cram_tell declaration. (66c41e2; #915 reported by
  Martin Morgan)

* Fixed overly aggressive filename suffix checking in bgzip. (#927, thanks to
  John Marshall; #129, reported by @hguturu)

* Tabix and bgzip --help output now goes to standard output. (#754, thanks to
  John Marshall)

* Fixed bgzip index creation when using multiple threads. (#817)

* Made bgzip -b option honour -I (index filename). (#817)

* Bgzip -d no longer attempts to unlink(NULL) when decompressing stdin. (#718)


Miscellaneous other changes
---------------------------

* Integration with Google OSS fuzzing for automatic detection of
  more bugs. (Thanks to Google for their assistance and the bugs it
  has found.) (#796, thanks to Markus Kusano)

* aclocal.m4 now has the pkg-config macros. (6ec3b94d; #733 reported by
  Thomas Hickman)

* Improved C++ compatibility of some header files. (#772; #771 reported
  by @cwrussell)

* Improved strict C99 compatibility. (#860, thanks to John Marshall)

* Travis and AppVeyor improvements to aid testing. (#747; #773 thanks to
  Lennard Berger; #781; #809; #804; #860; #909)

* Various minor compiler warnings fixed. (#708; #765; #846, #860, thanks to
  John Marshall; #865; #966; #973)

* Various new and improved error messages.

* Documentation updates (mostly in the header files).

* Even more testing with "make check".

* Corrected many copyright dates. (#979)

* The default non-configure Makefile now uses libcurl instead of
  knet, so it can support https. (#895)






Noteworthy changes in release 1.9 (18th July 2018)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* If `./configure` fails, `make` will stop working until either configure
  is re-run successfully, or `make distclean` is used.  This makes
  configuration failures more obvious.  (#711, thanks to John Marshall)

* The default SAM version has been changed to 1.6.  This is in line with the
  latest version specification and indicates that HTSlib supports the
  CG tag used to store long CIGAR data in BAM format.

* bgzip integrity check option '--test' (#682, thanks to @sd4B75bJ, @jrayner)

* Faidx can now index fastq files as well as fasta.  The fastq index adds
  an extra column to the `.fai` index which gives the offset to the quality
  values.  New interfaces have been added to `htslib/faidx.h` to read the
  fastq index and retrieve the quality values.  It is possible to open
  a fastq index as if fasta (only sequences will be returned), but not
  the other way round. (#701)

* New API interfaces to add or update integer, float and array aux tags. (#694)

* Add `level=<number>` option to `hts_set_opt()` to allow the compression
  level to be set.  Setting `level=0` enables uncompressed output. (#715)

* Improved bgzip error reporting.

* Better error reporting when CRAM reference files can't be opened. (#706)

* Fixes to make tests work properly on Windows/MinGW - mainly to handle
  line ending differences. (#716)

* Efficiency improvements:

  - Small speed-up for CRAM indexing.

  - Reduce the number of unnecessary wake-ups in the thread pool. (#703)

  - Avoid some memory copies when writing data, notably for uncompressed
    BGZF output. (#703)

* Bug fixes:

  - Fix multi-region iterator bugs on CRAM files. (#684)

  - Fixed multi-region iterator bug that caused some reads to be skipped
    incorrectly when reading BAM files. (#687)

  - Fixed synced_bcf_reader() bug when reading contigs multiple times. (#691,
    reported by @freeseek)

  - Fixed bug where bcf_hdr_set_samples() did not update the sample dictionary
    when removing samples. (#692, reported by @freeseek)

  - Fixed bug where the VCF record ref length was calculated incorrectly
    if an INFO END tag was present. (71b00a)

  - Fixed warnings found when compiling with gcc 8.1.0. (#700)

  - sam_hdr_read() and sam_hdr_write() will now return an error code
    if passed a NULL file pointer, instead of crashing.

  - Fixed possible negative array look-up in sam_parse1() that somehow escaped
    previous fuzz testing. (CVE-2018-13845, #731, reported by @fCorleone)

  - Fixed bug where cram range queries could incorrectly report an error
    when using multiple threads. (#734, reported by Brent Pedersen)

  - Fixed very rare rANS normalisation bug that could cause an assertion
    failure when writing CRAM files.  (#739, reported by @carsonhh)

Noteworthy changes in release 1.8 (3rd April 2018)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* The URL to get sequences from the EBI reference server has been changed
  to https://.  This is because the EBI no longer serve sequences via
  plain HTTP - requests to the http:// endpoint just get redirected.
  HTSlib needs to be linked against libcurl to download https:// URLs,
  so CRAM users who want to get references from the EBI will need to
  run configure and ensure libcurl support is enabled using the
  --enable-libcurl option.

* Added libdeflate as a build option for alternative faster compression and
  decompression.  Results vary by CPU but compression should be twice as fast
  and decompression faster.

* It is now possible to set the compression level in bgzip.  (#675; thanks
  to Nathan Weeks).

* bgzip now gets its own manual page.

* CRAM encoding now stored MD and NM tags verbatim where the reference
  contains 'N' characters, to work around ambiguities in the SAM
  specification (samtools #717/762).
  Also added "store_md" and "store_nm" cram-options for forcing these
  tags to be stored at all locations.  This is best when combined with
  a subsequent decode_md=0 option while reading CRAM.

* Multiple CRAM bug fixes, including a fix to free and the subsequent reuse of
  references with `-T ref.fa`. (#654; reported by Chris Saunders)

* CRAM multi-threading bugs fixed: don't try to call flush on reading;
  processing of multiple range queries; problems with multi-slice containers.

* Fixed crashes caused when decoding some cramtools produced CRAM files.

* Fixed a couple of minor rANS issues with handling invalid data.

* Fixed bug where probaln_glocal() tried to allocate far more memory than
  needed when the query sequence was much longer than the reference.  This
  caused crashes in samtools and bcftools mpileup when used on data with very
  long reads. (#572, problem reported by Felix Bemm via minimap2).

* sam_prop_realn() now returns -1 (the same value as for unmapped reads)
  on reads that do not include at least one 'M', 'X' or '=' CIGAR operator,
  and no longer adds BQ or ZQ tags.  BAQ adjustments are only made to bases
  covered by these operators so there is no point in trying to align
  reads that do not have them. (#572)

Noteworthy changes in release 1.7 (26th January 2018)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* BAM: HTSlib now supports BAMs which include CIGARs with more than 
  65535 operations as per HTS-Specs 18th November (dab57f4 and 2f915a8).

* BCF/VCF:
  - Removed the need for long double in pileup calculations.
  - Sped up the synced reader in some situations.
  - Bug fixing: removed memory leak in bcf_copy.

* CRAM:
  - Added support for HTS_IDX_START in cram iterators.
  - Easier to build when lzma header files are absent.
  - Bug fixing: a region query with REQUIRED_FIELDS option to
    disable sequence retrieval now gives correct results.
  - Bug fixing: stop queries to regions starting after the last
    read on a chromosome from incorrectly reporting errors
    (#651, #653; reported by Imran Haque and @egafni via pysam).

* Multi-region iterator: The new structure takes a list of regions and
  iterates over all, deduplicating reads in the process, and producing a 
  full list of file offset intervals. This is usually much faster than 
  repeatedly using the old single-region iterator on a series of regions.

* Curl improvements:
  - Add Bearer token support via HTS_AUTH_LOCATION env (#600).
  - Use CURL_CA_BUNDLE environment variable to override the CA (#622; 
    thanks to Garret Kelly & David Alexander).
  - Speed up (removal of excessive waiting) for both http(s) and ftp.
  - Avoid repeatedly reconnecting by removal of unnecessary seeks.
  - Bug fixing: double free when libcurl_open fails.

* BGZF block caching, if enabled, now performs far better (#629; reported
  by Ram Yalamanchili).

* Added an hFILE layer for in-memory I/O buffers (#590; thanks to Thomas 
  Hickman).

* Tidied up the drand48 support (intended for systems that do not
  provide this function).

Noteworthy changes in release 1.6 (28th September 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Fixed bug where iterators on CRAM files did not propagate error return
  values to the caller correctly.  Thanks go to Chris Saunders.

* Overhauled Windows builds.  Building with msys2/mingw64 now works
  correctly and passes all tests.

* More improvements to logging output (thanks again to Anders Kaplan).

* Return codes from sam_read1() when reading cram have been made
  consistent with those returned when reading sam/bam.  Thanks to
  Chris Saunders (#575).

* BGZF CRC32 checksums are now always verified.

* It's now possible to set nthreads = 1 for cram files.

* hfile_libcurl has been modified to make it thread-safe.  It's also
  better at handling web servers that do not honour byte range requests
  when attempting to seek - it now sets errno to ESPIPE and keeps
  the existing connection open so callers can revert to streaming mode
  it they want to.

* hfile_s3 now recalculates access tokens if they have become stale.  This
  fixes a reported problem where authentication failed after a file
  had been in use for more than 15 minutes.

* Fixed bug where remote index fetches would fail to notice errors when
  writing files.

* bam_read1() now checks that the query sequence length derived from the
  CIGAR alignment matches the sequence length in the BAM record.

Noteworthy changes in release 1.5 (21st June 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Added a new logging API: hts_log(), along with hts_log_error(),
  hts_log_warn() etc. convenience macros.  Thanks go to Anders Kaplan
  for the implementation. (#499, #543, #551)

* Added a new file I/O option "block_size" (HTS_OPT_BLOCK_SIZE) to
  alter the hFILE buffer size.

* Fixed various bugs, including compilation issues samtools/bcftools#610,
  samtools/bcftools#611 and robustness to corrupted data #537, #538,
  #541, #546, #548, #549, #554.


Noteworthy changes in release 1.4.1  (8th May 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is primarily a security bug fix update.

* Fixed SECURITY (CVE-2017-1000206) issue with buffer overruns with
  malicious data. (#514)

* S3 support for non Amazon AWS endpoints. (#506)

* Support for variant breakpoints in bcftools. (#516)

* Improved handling of BCF NaNs. (#485)

* Compilation / portability improvements. (#255, #423, #498, #488)

* Miscellaneous bug fixes (#482, #521, #522, #523, #524).

* Sanitise headers (#509)


Release 1.4 (13 March 2017)

* Incompatible changes: several functions and data types have been changed
  in this release, and the shared library soversion has been bumped to 2.

  - bam_pileup1_t has an additional field (which holds user data)
  - bam1_core_t has been modified to allow for >64K CIGAR operations
    and (along with bam1_t) so that CIGAR entries are aligned in memory
  - hopen() has vararg arguments for setting URL scheme-dependent options
  - the various tbx_conf_* presets are now const
  - auxiliary fields in bam1_t are now always stored in little-endian byte
    order (previously this depended on if you read a bam, sam or cram file)
  - index metadata (accessible via hts_idx_get_meta()) is now always
    stored in little-endian byte order (previously this depended on if
    the index was in tbi or csi format)
  - bam_aux2i() now returns an int64_t value
  - fai_load() will no longer save local copies of remote fasta indexes
  - hts_idx_get_meta() now takes a uint32_t * for l_meta (was int32_t *)

* HTSlib now links against libbz2 and liblzma by default.  To remove these
  dependencies, run configure with options --disable-bz2 and --disable-lzma,
  but note that this may make some CRAM files produced elsewhere unreadable. 

* Added a thread pool interface and replaced the bgzf multi-threading
  code to use this pool.  BAM and CRAM decoding is now multi-threaded
  too, using the pool to automatically balance the number of threads
  between decode, encode and any data processing jobs.

* New errmod_cal(), probaln_glocal(), sam_cap_mapq(), and sam_prob_realn()
  functions, previously internal to SAMtools, have been added to HTSlib.

* Files can now be accessed via Google Cloud Storage using gs: URLs, when
  HTSlib is configured to use libcurl for network file access rather than
  the included basic knetfile networking.

* S3 file access now also supports the "host_base" setting in the
  $HOME/.s3cfg configuration file.

* Data URLs ("data:,text") now follow the standard RFC 2397 format and may
  be base64-encoded (when written as "data:;base64,text") or may include
  percent-encoded characters.  HTSlib's previous over-simplified "data:text"
  format is no longer supported -- you will need to add an initial comma.

* When plugins are enabled, S3 support is now provided by a separate
  hfile_s3 plugin rather than by hfile_libcurl itself as previously.
  When --enable-libcurl is used, by default both GCS and S3 support
  and plugins will also be built; they can be individually disabled
  via --disable-gcs and --disable-s3.

* The iRODS file access plugin has been moved to a separate repository.
  Configure no longer has a --with-irods option; instead build the plugin
  found at <https://github.com/samtools/htslib-plugins>.

* APIs to portably read and write (possibly unaligned) data in little-endian
  byte order have been added.

* New functions bam_auxB_len(), bam_auxB2i() and bam_auxB2f() have been
  added to make accessing array-type auxiliary data easier.  bam_aux2i()
  can now return the full range of values that can be stored in an integer
  tag (including unsigned 32 bit tags).  bam_aux2f() will return the value
  of integer tags (as a double) as well as floating-point ones.  All of
  the bam_aux2 and bam_auxB2 functions will set errno if the requested
  conversion is not valid.

* New functions fai_load3() and fai_build3() allow fasta indexes to be
  stored in a different location to the indexed fasta file.

* New functions bgzf_index_dump_hfile() and bgzf_index_load_hfile()
  allow bgzf index files (.gzi) to be written to / read from an existing
  hFILE handle.

* hts_idx_push() will report when trying to add a range to an index that
  is beyond the limits that the given index can handle.  This means trying
  to index chromosomes longer than 2^29 bases with a .bai or .tbi index
  will report an error instead of apparantly working but creating an invalid
  index entry.

* VCF formatting is now approximately 4x faster.  (Whether this is
  noticable depends on what was creating the VCF.)

* CRAM lossy_names mode now works with TLEN of 0 or TLEN within +/- 1
  of the computed value.  Note in these situations TLEN will be
  generated / fixed during CRAM decode.

* CRAM now supports bzip2 and lzma codecs.  Within htslib these are
  disabled by default, but can be enabled by specifying "use_bzip2" or
  "use_lzma" in an hts_opt_add() call or via the mode string of the
  hts_open_format() function.

Noteworthy changes in release 1.3.2  (13 September 2016)

* Corrected bin calculation when converting directly from CRAM to BAM.
  Previously a small fraction of converted reads would fail Picard's
  validation with "bin field of BAM record does not equal value computed"
  (SAMtools issue #574).

* Plugins can now signal to HTSlib which of RTLD_LOCAL and RTLD_GLOBAL
  they wish to be opened with -- previously they were always RTLD_LOCAL.


Noteworthy changes in release 1.3.1  (22 April 2016)

* Improved error checking and reporting, especially of I/O errors when
  writing output files (#17, #315, PR #271, PR #317).

* Build fixes for 32-bit systems; be sure to run configure to enable
  large file support and access to 2GiB+ files.

* Numerous VCF parsing fixes (#321, #322, #323, #324, #325; PR #370).
  Particular thanks to Kostya Kortchinsky of the Google Security Team
  for testing and numerous input parsing bug reports.

* HTSlib now prints an informational message when initially creating a
  CRAM reference cache in the default location under your $HOME directory.
  (No message is printed if you are using $REF_CACHE to specify a location.)

* Avoided rare race condition when caching downloaded CRAM reference sequence
  files, by using distinctive names for temporary files (in addition to O_EXCL,
  which has always been used).  Occasional corruption would previously occur
  when multiple tools were simultaneously caching the same reference sequences
  on an NFS filesystem that did not support O_EXCL (PR #320).

* Prevented race condition in file access plugin loading (PR #341).

* Fixed mpileup memory leak, so no more "[bam_plp_destroy] memory leak [...]
  Continue anyway" warning messages (#299).

* Various minor CRAM fixes.

* Fixed documentation problems #348 and #358.


Noteworthy changes in release 1.3  (15 December 2015)

* Files can now be accessed via HTTPS and Amazon S3 in addition to HTTP
  and FTP, when HTSlib is configured to use libcurl for network file access
  rather than the included basic knetfile networking.

* HTSlib can be built to use remote access hFILE backends (such as iRODS
  and libcurl) via a plugin mechanism.  This allows other backends to be
  easily added and facilitates building tools that use HTSlib, as they
  don't need to be linked with the backends' various required libraries.

* When writing CRAM output, sam_open() etc now default to writing CRAM v3.0
  rather than v2.1.

* fai_build() and samtools faidx now accept initial whitespace in ">"
  headers (e.g., "> chr1 description" is taken to refer to "chr1").

* tabix --only-header works again (was broken in 1.2.x; #249).

* HTSlib's configure script and Makefile now fully support the standard
  convention of allowing CC/CPPFLAGS/CFLAGS/LDFLAGS/LIBS to be overridden
  as needed.  Previously the Makefile listened to $(LDLIBS) instead; if you
  were overriding that, you should now override LIBS rather than LDLIBS.

* Fixed bugs #168, #172, #176, #197, #206, #225, #245, #265, #295, and #296.


Noteworthy changes in release 1.2.1  (3 February 2015)

* Reinstated hts_file_type() and FT_* macros, which were available until 1.1
  but briefly removed in 1.2.  This function is deprecated and will be removed
  in a future release -- you should use hts_detect_format() etc instead


Noteworthy changes in release 1.2  (2 February 2015)

* HTSlib now has a configure script which checks your build environment
  and allows for selection of optional extras.  See INSTALL for details

* By default, reference sequences are fetched from the EBI CRAM Reference
  Registry and cached in your $HOME cache directory.  This behaviour can
  be controlled by setting REF_PATH and REF_CACHE enviroment variables
  (see the samtools(1) man page for details)

* Numerous CRAM improvements:
  - Support for CRAM v3.0, an upcoming revision to CRAM supporting
    better compression and per-container checksums
  - EOF checking for v2.1 and v3.0 (similar to checking BAM EOF blocks)
  - Non-standard values for PNEXT and TLEN fields are now preserved
  - hts_set_fai_filename() now provides a reference file when encoding
  - Generated read names are now numbered from 1, rather than being
    labelled 'slice:record-in-slice'
  - Multi-threading and speed improvements

* New htsfile command for identifying file formats, and corresponding
  file format detection APIs

* New tabix --regions FILE, --targets FILE options for filtering via BED files

* Optional iRODS file access, disabled by default.  Configure with --with-irods
  to enable accessing iRODS data objects directly via 'irods:DATAOBJ'

* All occurences of 2^29 in the source have been eliminated, so indexing
  and querying against reference sequences larger than 512Mbp works (when
  using CSI indices)

* Support for plain GZIP compression in various places

* VCF header editing speed improvements

* Added seq_nt16_int[] (equivalent to the samtools API's bam_nt16_nt4_table)

* Reinstated faidx_fetch_nseq(), which was accidentally removed from 1.1.
  Now faidx_fetch_nseq() and faidx_nseq() are equivalent; eventually
  faidx_fetch_nseq() will be deprecated and removed [#156]

* Fixed bugs #141, #152, #155, #158, #159, and various memory leaks