faidx: ignore lines starting with ';' before first record

[adamjohnwright]

We are making it so that people can store metadata in the header of
FASTA files and would like this library to be able to work with the
files that use the semicolon as the marker for the comments in the
header. Currently samtools faidx fails with error message unexpected
semicolon at first line. The semicolon was in the original Pearson
FASTA convention and would allow us to specify FASTA files with file
headers. With this change only lines starting with a semicolon before
the first '>' will be skipped. The lines after the first '>' will act
the same as they did before this change. The faidx.5 manpage has been
updated and the 'make check' passes. Three tests were added and FASTQ
is not affected. See #2103.

Assisted-by: Claude:claude-opus-5-5

[jkbonfield]

Rather than attempting to store meta data in FASTA, which is a rather badly supported format, would it not be better to converge on the same approach that htsjdk uses and add better support for the .dict format? That's basically SAM headers associated with the fasta, which means it can add on various key-value pairs in a structured way as well as providing an easier path for converting this into SQ headers when aligned against.

[adamjohnwright]

Thanks @jkbonfield and @whitwham for looking at this and for the explanation.

The reason we'd like the metadata at the top of the FASTA file itself, rather than in a sidecar such as a .dict, is that it then travels with the data: it can't get separated or lost when files are copied, mirrored or archived, and the FHR checksum covers the metadata and the sequence together. Sidecar forms (JSON/YAML) already exist in FHR for workflows that prefer them.

On the de facto standard: in practice it's mixed rather than settled. In our testing, many widely used tools already read FASTA files with leading ; lines, including BWA, minimap2, makeblastdb, MAFFT, kallisto, seqtk, and even samtools dict and pysam's FastxFile, while others (faidx, Picard, seqkit and some index builders) reject them.

We understand the concern about reviving ; comments in general. What we're asking for is deliberately narrow: only ; lines before the first >, with every other line read exactly as today. We're proposing the same narrow change to several of the tools that reject these files (htsjdk, BioPerl, seqkit, UCSC and others), and several of those PRs are open now. The more tools can read these files without stripping the header, the more useful the headers become, and if it becomes common practice we'd be glad to revisit it here.

Background on the standard: https://doi.org/10.1093/bib/bbae122

[molikd]

I don't find the “de facto standard” argument particularly convincing. Semicolon comments were part of the original Pearson/Lipman FASTA format, and there is no single, universally accepted authority defining the FASTA specification. Inconsistent support across implementations demonstrates an interoperability problem; it does not, by itself, establish a consensus that those comments are invalid.

In that context, accepting a leading semicolon comment block is support for an existing historical convention, rather than the invention of a new one. There may be practical reasons for a particular parser to decline that support, but I think those should be distinguished from an argument that the syntax falls outside FASTA.