Using high-performance data formats – HDF5

VCF processing is very slow: if you do an empty for loop over a big VCF file, it can easily take days just to parse it. This is because text parsing is very demanding. Alternatively, using NumPy arrays is fast, but you are limited to whatever fits in memory. There are several alternatives to deal with both of these problems (and we will explore more than one in this chapter). Here, we will consider representing our data in HDF5 format.

We will use an existing HDF5 file that was exported from a VCF file and do some basic extraction of data.

Getting ready

We will revisit the Anopheles gambiae dataset that we used in previous chapters. There is a HDF5 version of the VCF file that we used previously. You can download chromosome arm 3L from ftp://ngs.sanger.ac.uk/production/ag1000g/phase1/AR3/variation/main/hdf5/ag1000g.phase1.ar3.pass.3L.h5. Remember that we are dealing with a VCF representation of 765 mosquitoes that can be carriers of Plasmodium falciparum...