SAM tools Files

------------------------------------------------------------------------------
htslib - changes v1.14
------------------------------------------------------------------------------

Features and Updates
--------------------

* Added a keep option to bgzip to leave the original file untouched.  This
  brings bgzip into line with gzip. (PR #1331, thanks to Alex Petty)

* "endpos" has been added to the filter language, giving the position of the
  rightmost mapped base as measured by the CIGAR string.  For unmapped reads
  it is the same as "pos". (PR #1307, thanks to John Marshall)

* Interfaces have been added to interpret the new base modification tags
  added to the SAMtags document in samtools/hts-specs#418.  (PR #1132)

* New API functions hts_flush()/sam_flush()/bcf_flush() for flushing output
  htsFile/samFile/vcfFile streams. (PR #1326, thanks to John Marshall)

* The synced_bcf_reader now sorts lines with symbolic alleles by END tag as
  well as POS.  (PR #1321)

* Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and
  BCF_SR_TARGETS_OVERLAP for better control of records that start outside the
  desired region but overlap it are handled.  Fixes samtools/bcftools#1420
  and samtools/bcftools#1421 raised by John Marshall.  (PR #1327)

* HTSlib will now accept long-cigar CG:B: tags made by htsjdk which don't
  quite follow the specification properly (using signed values instead of
  unsigned).  Thanks to Colin Diesh for reporting an example file. (PR #1317)

* The warning printed when the BGZF reader finds a file with no EOF block
  has been changed to be less alarming.  Unfortunately some third-party
  BGZF encoders don't write EOF blocks at the end of files.  Thanks to
  Keiran Raine for reporting an example file.  (PR #1323)

* The FASTA and FASTQ readers get an option to skip over the first item on
  the header line, and use the second as the read name.  It allows the
  original name to be restored on some of the fastq files served from the
  European Nucleotide Archive (ENA).  (PR #1325)

* HTSlib is now more strict when parsing the VCF samples line (beginning
  #CHROM).  It will only accept tabs between the mandatory field names and
  sample names must be separated with tabs. (PR #1328)

* HTSlib will now warn if it looks like the header has been corrupted by
  diagnostic messages from the program that made it.  This can happen when
  using `nohup`, which by default mixes stdout and stderr into the same
  stream.  (PR#1339, thanks to John Marshall)

* File format detection will now recognise signatures for XZ, Zstd and D4
  files (note that HTSlib will not read them yet).  (PR #1340, thanks to
  John Marshall)

Build changes
-------------

These are compiler, configuration and makefile based changes.

* Some redundant tests have been removed from the test harness, speeding it
  up. (PR #1308)

* The version.sh script now works better on shallow checkouts.  (PR #1324)

* A check-untracked Makefile target has been added to catch untracked files
  (mostly) left by the test harness.  (PR #1324)

Bug fixes
---------

* Fixed a case where flushing the thread pool could very occasionally cause a
  deadlock. (PR #1309)

* Fixed a bug where some CRAM files could fail to decode if the
  required_fields option was in use.  Thanks to Matt Sexton for reporting
  the issue. (PR #1314, fixes samtools/samtools#1475)

* Fixed a regression where the S3 plugin could not read public files
  unless you supplied some Amazon credentials.  Thanks to Chris Saunders
  for reporting. (PR #1332, fixes samtools/samtools#1491)

* Fixed a possible CRAM thread deadlock discovered by @ryancaicse. (PR #1330,
  fixes #1329)

* Some set-but-unused variables have been removed.  (PR #1334)

* Fixed a bug which prevented "flag.read2" from working in the filter
  language unless it was at the end of the expression.  Thanks to
  Vamsi Kodali for reporting the issue.  (PR #1342)

* Fixed a memory leak that could happen if CRAM fails to inflate a LZMA
  block. (PR #1340, thanks to John Marshall)

------------------------------------------------------------------------------
samtools - changes v1.14
------------------------------------------------------------------------------

Notice:

 * Samtools mpileup VCF and BCF output (deprecated in release 1.9) will be
   removed in the next release.  Please use bcftools mpileup instead.

New work and changes:

 * The legacy samtools API (libbam.a, bam_endian.h, sam.h and most of bam.h)
   has been removed.  We recommend coding against the HTSlib API directly.
   The legacy API had not been actively maintained since 2015. (#1483)

 * New "samtools samples" command to list the samples used in a SAM/BAM/CRAM
   file. (#1432; thanks to Pierre Lindenbaum)

 * "mpileup" now supports base modifications via the SAM Mm/MM auxiliary tag.
    Please see the "--output-mods" option. (#1311)

 * Added "mpileup --output-BP-5" option to output the BP field in 5' to 3'
   order instead of left to right. (#1484; fixes #1481)

 * Added "samtools view --rf" option as an additional FLAG filtering method. 
   This keeps records only if (FLAG & N) != 0. (#1508; fixes #1470)

 * New "samtools import -N" option to use the second word on a FASTQ header
   line, matching the SRA/ENA FASTQ variant. (#1485)

 * Improve "view -x" option to simplify specifying multiple tags, and added
   the reverse "--keep-tag" option to include rather than exclude. (#516)

 * Switched the processing order of "view" -x (tag filtering) and -e
   (expression) handling.  Expressions now happen first so we can filter
   on tags which are about to be deleted.  This is now consistent with
   the "view -d" behaviour too. (#1480; fixes #1476. Reported by
   William Rowell)

 * Added filter expression "endpos" keyword. (#1464.  Thanks to
   John Marshall)

 * "samtools view" errors now appear after any SAM output, improving their
   visibility. (#1490.  Thanks to John Marshall)

 * Improved "samtools sort" use of temporary files, both tidying up if it
   fails and recovery when facing pre-existing temporary files. (#1510; fixes
   #1035, #1503.  Reported by Vivek Rai and Maarten Kooyman)

 * Filtering in "samtools markdup" now sets the UNMAP BAM flag when given the
   "-p" option. (#1512; fixes #1469)

 * Make CRAM references shared during "samtools merge" so merging many files
   has a lower memory usage. (#471)

Bug fixes:

 * Prevent "samtools depth" from closing stdout when outputting to terminal,
   avoiding a bad interaction with PySam. (#1465.  Thanks to John Marshall)

 * In-place "samtools reheader" now works on CRAMs produced using a higher
   than default compression level. (#1479)

 * Fix setting of the dt tag in "markdup".  Optical duplicates were being
   marked too early, negating the tagging and counting elsewhere. (#1487;
   fixes #1486.  Reported by Kevin Lewis)

 * Reinstate the "samtools stats -I" option to filter by sample. (#1496;
   fixes #1489.  Reported by Matthias Bernt)

 * Fix "samtools fastq" handling of dual index tags on single-ended input.
   (#1474)

 * Improve "samtools coverage" documentation. (#1521; fixes #1504. Reported
   by Peter Menzel)

Non user-visible changes and build improvements:

 * Replace Curses mvprintw() with va_list-based equivalent. (#1509. Thanks to
   John Marshall and Andreas Tille)

 * Fixed some clang-13 warning messages. (#1506)

 * Improve quoting of options in "samtools import" tests. (#1466. Thanks to
   John Marshall)

 * Fixed a faulty test which caused test harness failures on NetBSD. (#1520)

------------------------------------------------------------------------------
bcftools - changes v1.14
------------------------------------------------------------------------------

Changes affecting the whole of bcftools, or multiple commands:

* New `--regions-overlap` and `--targets-overlap` options which address a
  long-standing design problem with subsetting VCF files by region. 
  BCFtools recognize two sets of options, one for streaming (`-t/-T`) and
  one for index-gumping (`-r/-R`). They behave differently, the first
  includes only records with POS coordinate within the regions, the other
  includes overlapping regions. The two new options allow to modify the
  default behaviour, see the man page for more details.

* The `--output-type` option can be used to override the default compression
  level

Changes affecting specific commands:

* bcftools annotate

    - when `--set-id` and `--remove` are combined, `--set-id` cannot use tags
      deleted by `--remove`. This is now detected and the program exists with
      an informative error message instead of segfaulting (#1540)

    - while non-symbolic variation are uniquely identified by
      POS,REF,ALT, symbolic alleles starting at the same position were
      indistinguishable. This prevented correct matching of records with
      the same positions and variant type but different length given by
      INFO/END (samtools/htslib@60977f2). When annotating from a VCF/BCF,
      the matching is done automatically. When annotating from a
      tab-delimited text file, this feature can be invoked by using `-c
      INFO/END`.

    - add a new '.' modifier to control whether missing values should be
      carried over from a tab-delimited file or not. For example: 

       -c TAG .. adds TAG if the source value is not missing. If TAG exists
       in the target file, it will be overwritten

       -c .TAG .. adds TAG even if the source value is missing. This can
       overwrite non-missing values with a missing value and can create empty
       VCF fields (`TAG=.`)

* bcftools +check-ploidy

    - by default missing genotypes are not used when determining ploidy.
      With the new option `-m, --use-missing` it is possible to use the
      information carried in the missing and half-missing genotypes (e.g.
      ".", "./." or "./1")

* bcftools concat

    - new `--ligate-force` and `--ligate-warn` options for finer control
      of `-l, --ligate` behaviour in imperfect overlaps. The new default
      is to throw an error when sites present in one chunk but absent in
      the other are encountered. To drop such sites and proceed, use the
      new `--ligate-warn` option (previously this was the default). To
      keep such sites, use the new `--ligate-force` option (#1567).

* bcftools consensus:

    - Apply mask even when the VCF has no notion about the chromosome. It
      was possible to encounter this problem when `contig` lines were not
      present in the VCF header and no variants were called on that
      chromosome (#1592)

* bcftools +contrast:

    - support for chunking within map/reduce framework allowing to collect
      NASSOC counts even for empty case/control sample sets (#1566)

* bcftools csq:

    - bug fix, compound indels were not recognised in some cases (#1536)

    - compound variants were incorrectly marked as 'inframe' even when stop
      codon would occur before the frame was restored (#1551)

    - bug fix, FORMAT/BCSQ bitmasks could have been assigned incorrectly
      to some samples at multiallelic sites, a superset of the correct
      consequences would have been set (#1539)

    - bug fix, the upstream stop could be falsely assigned to all samples in
      a multi-sample VCF even if the stop was relevant for a single sample
      only (#1578)

    - further improve the detection of mismatching chromosome naming (e.g.
      "chrX" vs "X") in the GFF, VCF and fasta files

* bcftools merge:

    - keep (sum) INFO/AN,AC values when merging VCFs with no samples (#1394)

* bcftools mpileup:

    - new --indel-size option which allows to increase the maximum considered
      indel size considered, large deletions in long read data are otherwise
      lost.

* bcftools norm:

    - atomization now supports Number=A,R string annotations (#1503)

    - assign as many alternate alleles to genotypes at multiallelic sites in
      the`-m +` mode, disregarding the phase.  Previously the program assumed
      to be executed as an inverse operation of `-m -`, but when that was not
      the case, reference alleles would have been filled instead of multiple
      alternate alleles (#1542)

* bcftools sort:

    - increase accuracy of the --max-mem option limit, previously the limit
      could be exceeded by more than 20% (#1576)

* bcftools +trio-dnm:

    - new `--with-pAD` option to allow processing of VCFs without FORMAT/QS.
      The existing `--ppl` option was changed to the analogous `--with-pPL`

* bcftools view:

    - the functionality of the option --compression-level lost in 1.12 has
      been restored