Skip to content

Tutorial: Indexing 10 E. coli Assemblies

By the end you will have a queryable k-mer index of these assemblies built in three commands:
design → build → query.

Prerequisites

  • kmhelpers installed (see Installation)
  • Command wget to download the dataset
  • ~200 MB of free disk space for the dataset + resulting index

Dependencies

The following tools are installed alongside kmhelpers when using the conda environment:

  • kmindex - the underlying indexing engine
  • ntcard - k-mer counter used by list

Step 1 - Download the dataset

mkdir -p coli_dataset && cd coli_dataset
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/780/515/GCA_000780515.1_ASM78051v1/GCA_000780515.1_ASM78051v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/076/125/GCA_001076125.1_ASM107612v1/GCA_001076125.1_ASM107612v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/417/575/GCA_001417575.1_ASM141757v1/GCA_001417575.1_ASM141757v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/944/435/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/075/925/GCA_001075925.1_ASM107592v1/GCA_001075925.1_ASM107592v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/936/715/GCA_000936715.1_E8C1_assembly/GCA_000936715.1_E8C1_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/939/215/GCA_000939215.1_Ec57A_A7_MIRA_assembly/GCA_000939215.1_Ec57A_A7_MIRA_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/413/795/GCA_001413795.1_ASM141379v1/GCA_001413795.1_ASM141379v1_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/373/195/GCA_001373195.1_57A_A7_assembly/GCA_001373195.1_57A_A7_assembly_genomic.fna.gz"
wget "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/938/575/GCA_000938575.1_D1C4_assembly/GCA_000938575.1_D1C4_assembly_genomic.fna.gz"
cd ..
I/O

Input: 10 NCBI FTP URLs
Output: coli_dataset/

Step 2 - Create a file list

cat > coli_10.txt << 'EOF'
coli_dataset/GCA_000780515.1_ASM78051v1_genomic.fna.gz
coli_dataset/GCA_001076125.1_ASM107612v1_genomic.fna.gz
coli_dataset/GCA_001417575.1_ASM141757v1_genomic.fna.gz
coli_dataset/GCA_000944435.1_Ec57A_E8C1_MIRA_assembly_genomic.fna.gz
coli_dataset/GCA_001075925.1_ASM107592v1_genomic.fna.gz
coli_dataset/GCA_000936715.1_E8C1_assembly_genomic.fna.gz
coli_dataset/GCA_000939215.1_Ec57A_A7_MIRA_assembly_genomic.fna.gz
coli_dataset/GCA_001413795.1_ASM141379v1_genomic.fna.gz
coli_dataset/GCA_001373195.1_57A_A7_assembly_genomic.fna.gz
coli_dataset/GCA_000938575.1_D1C4_assembly_genomic.fna.gz
EOF
I/O

Input: paths to the 10 .fna.gz files (written manually)
Output: coli_10.txt

Step 3 - Design the index (design)

kmhelpers design coli_10.txt \
    -o coli_db/ \
    -n coli \
    -S initial \
    -k 25 \
    -b 1.1 \
    -g 2
I/O

Input: coli_10.txt
Output: coli_db/list/ (JSONL), coli_db/profile/ (profile.yaml, groups.png), coli_db/compose/ (index definitions)

INFO

design runs list → profile → compose in a single command.
For a detailed walkthrough of each sub-command, see Step-by-step: list → profile → compose.

Step 4 - Build the index (build)

kmhelpers build coli_db/compose/coli/initial/coli.yaml -o coli_build/ --show-progress
I/O

Input: coli_db/compose/coli/initial/coli.yaml
Output: coli_build/index.json + sub-index data files in coli_build/kmindex_data/ - ready-to-query index

INFO

build runs plan → apply in a single command, validating all paths before starting the build.
For a detailed walkthrough, or to run plan first and apply later with bash or apply, see Step-by-step: plan → apply.

Step 5 - Query the index (query)

Extract the first contig of the first sample as a query sequence:

zcat coli_dataset/GCA_000780515.1_ASM78051v1_genomic.fna.gz \
    | awk '/^>/{n++} n==2{exit} {print}' > query.fa

Then query the index:

kmhelpers query -r coli_build/ -o results/ query.fa
I/O

Input: coli_build/ (index root), query.fa (first contig of GCA_000780515)
Output: results/

RESULT

query writes one JSON file per sub-index into results/query/result:

coli_g0.json 
{
    "coli_g0": {
        "JSPL01000060.1": {}
    }
}
coli_g1.json 
{
    "coli_g1": {
        "JSPL01000060.1": {
            "GCA_000780515_1_ASM78051v1_genomic_fna": 1.0
        }
    }
}

Each value is the fraction of query k-mers found in that sample. GCA_000780515 scores 1.0 -- a perfect match, as expected since query.fa was extracted from that assembly.

References

E. Coli colony

Stewart EJ, Madden R, Paul G, Taddei F (2005), CC BY-SA 4.0, via Wikimedia Commons