Skip to content

Tutorial: Indexing 10 E. coli Assemblies

By the end you will have a queryable k-mer index of these assemblies built in three commands:
design → build → query.

Prerequisites

  • kmhelpers installed (see Installation)
  • Command wget to download the dataset
  • ~200 MB of free disk space for the dataset + resulting index

Dependencies

The following tools are installed alongside kmhelpers when using the conda environment:

  • kmindex - the underlying indexing engine
  • ntcard - k-mer counter used by list

Step 1 - Download the dataset

mkdir -p coli_dataset && cd coli_dataset
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/780/515/GCA_000780515.1_ASM78051v1/GCA_000780515.1_ASM78051v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/076/125/GCA_001076125.1_ASM107612v1/GCA_001076125.1_ASM107612v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/417/575/GCA_001417575.1_ASM141757v1/GCA_001417575.1_ASM141757v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/944/435/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/075/925/GCA_001075925.1_ASM107592v1/GCA_001075925.1_ASM107592v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/936/715/GCA_000936715.1_E8C1_assembly/GCA_000936715.1_E8C1_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/939/215/GCA_000939215.1_Ec57A_A7_MIRA_assembly/GCA_000939215.1_Ec57A_A7_MIRA_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/413/795/GCA_001413795.1_ASM141379v1/GCA_001413795.1_ASM141379v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/373/195/GCA_001373195.1_57A_A7_assembly/GCA_001373195.1_57A_A7_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/938/575/GCA_000938575.1_D1C4_assembly/GCA_000938575.1_D1C4_assembly_genomic.fna.gz"
cd ..
I/O

Input: 10 NCBI FTP URLs
Output: coli_dataset/

Step 2 - Create a file list

cat > coli_10.txt << 'EOF'
coli_dataset/GCA_000780515.1_ASM78051v1_genomic.fna.gz
coli_dataset/GCA_001076125.1_ASM107612v1_genomic.fna.gz
coli_dataset/GCA_001417575.1_ASM141757v1_genomic.fna.gz
coli_dataset/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly_genomic.fna.gz
coli_dataset/GCA_001075925.1_ASM107592v1_genomic.fna.gz
coli_dataset/GCA_000936715.1_E8C1_assembly_genomic.fna.gz
coli_dataset/GCA_000939215.1_Ec57A_A7_MIRA_assembly_genomic.fna.gz
coli_dataset/GCA_001413795.1_ASM141379v1_genomic.fna.gz
coli_dataset/GCA_001373195.1_57A_A7_assembly_genomic.fna.gz
coli_dataset/GCA_000938575.1_D1C4_assembly_genomic.fna.gz
EOF
I/O

Input: paths to the 10 .fna.gz files (written manually)
Output: coli_10.txt

Step 3 - Design the index (design)

kmhelpers design coli_10.txt \
    -o coli_db/ \
    -n coli \
    -S initial \
    -k 25 \
    -b 1.1 \
    -g 2
I/O

Input: coli_10.txt
Output: coli_db/list/ (JSONL), coli_db/profile/ (profile.yaml, groups.png), coli_db/compose/ (index definitions)

INFO

design runs list → profile → compose in a single command.
For a detailed walkthrough of each sub-command, see Step-by-step: list → profile → compose.

Step 4 - Build the index (build)

kmhelpers build coli_db/compose/coli/initial/coli.yaml -o coli_build/ --show-progress
I/O

Input: coli_db/compose/coli/initial/coli.yaml
Output: coli_build/index.json + sub-index data files in coli_build/kmindex_data/ - ready-to-query index

INFO

build runs plan → apply in a single command, validating all paths before starting the build.
For a detailed walkthrough, or to run plan first and apply later with bash or apply, see Step-by-step: plan → apply.

Step 5 - Query the index (query)

Extract the first contig of the first sample as a query sequence:

zcat coli_dataset/GCA_000780515.1_ASM78051v1_genomic.fna.gz \
    | awk '/^>/{n++} n==2{exit} {print}' > query.fa

Then query the index:

kmhelpers query -r coli_build/ -o results/ query.fa
I/O

Input: coli_build/ (index root), query.fa (first contig of GCA_000780515)
Output: results/

RESULT

query merges the results of all sub-indices into results/query/results.tsv, one row per query sequence and one column per matching sample:

seq GCA_000780515_1_ASM78051v1_genomic_fna
JSPL01000060.1  1.000

Each value is the fraction of query k-mers found in that sample. Samples scoring below the threshold (-R, default 0.05) are omitted. Use -f to select another output format (json, yaml, md, html). GCA_000780515 scores 1.0 -- a perfect match, as expected since query.fa was extracted from that assembly.

Next steps

To add samples to this index later without rebuilding it from scratch, see Update an existing index.

References

E. Coli colony

Stewart EJ, Madden R, Paul G, Taddei F (2005), CC BY-SA 4.0, via Wikimedia Commons