Change search
ReferencesLink to record
Permanent link

Direct link
A quantitative assessment of the Hadoop framework for analyzing massively parallel DNA sequencing data
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology.
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology.
St Petersburg State Univ, Inst Chem, Dept Phys Chem, St Petersburg 199034, Russia.
Uppsala University, Disciplinary Domain of Medicine and Pharmacy, Faculty of Pharmacy, Department of Pharmaceutical Biosciences. Uppsala University, Science for Life Laboratory, SciLifeLab.ORCID iD: 0000-0002-8083-2864
2015 (English)In: GigaScience, ISSN 2047-217X, E-ISSN 2047-217X, Vol. 4, 26Article in journal (Refereed) Published
Abstract [en]

Background: New high-throughput technologies, such as massively parallel sequencing, have transformed the life sciences into a data-intensive field. The most common e-infrastructure for analyzing this data consists of batch systems that are based on high-performance computing resources; however, the bioinformatics software that is built on this platform does not scale well in the general case. Recently, the Hadoop platform has emerged as an interesting option to address the challenges of increasingly large datasets with distributed storage, distributed processing, built-in data locality, fault tolerance, and an appealing programming methodology. Results: In this work we introduce metrics and report on a quantitative comparison between Hadoop and a single node of conventional high-performance computing resources for the tasks of short read mapping and variant calling. We calculate efficiency as a function of data size and observe that the Hadoop platform is more efficient for biologically relevant data sizes in terms of computing hours for both split and un-split data files. We also quantify the advantages of the data locality provided by Hadoop for NGS problems, and show that a classical architecture with network-attached storage will not scale when computing resources increase in numbers. Measurements were performed using ten datasets of different sizes, up to 100 gigabases, using the pipeline implemented in Crossbow. To make a fair comparison, we implemented an improved preprocessor for Hadoop with better performance for splittable data files. For improved usability, we implemented a graphical user interface for Crossbow in a private cloud environment using the CloudGene platform. All of the code and data in this study are freely available as open source in public repositories. Conclusions: From our experiments we can conclude that the improved Hadoop pipeline scales better than the same pipeline on high-performance computing resources, we also conclude that Hadoop is an economically viable option for the common data sizes that are currently used in massively parallel sequencing. Given that datasets are expected to increase over time, Hadoop is a framework that we envision will have an increasingly important role in future biological data analysis.

Place, publisher, year, edition, pages
2015. Vol. 4, 26
Keyword [en]
Next generation sequencing; Massively parallel sequencing; Hadoop; High-performance computing; DNA-seq; Bioinformatics
National Category
Bioinformatics (Computational Biology)
URN: urn:nbn:se:uu:diva-254718DOI: 10.1186/s13742-015-0058-5ISI: 000365665200001PubMedID: 26045962OAI: diva2:819372
Science for Life Laboratory - a national resource center for high-throughput molecular bioscienceSwedish National Infrastructure for Computing (SNIC), p2013023

Erratum in GigaScience 2015, 4:61 DOI: 10.1186/s13742-015-0100-7

Available from: 2015-06-04 Created: 2015-06-10 Last updated: 2016-11-07Bibliographically approved

Open Access in DiVA

fulltext(1181 kB)38 downloads
File information
File name FULLTEXT02.pdfFile size 1181 kBChecksum SHA-512
Type fulltextMimetype application/pdf

Other links

Publisher's full textPubMedErratum

Search in DiVA

By author/editor
Siretskiy, AlexeySundqvist, ToreSpjuth, Ola
By organisation
Department of Information TechnologyDepartment of Pharmaceutical BiosciencesScience for Life Laboratory, SciLifeLab
In the same journal
Bioinformatics (Computational Biology)

Search outside of DiVA

GoogleGoogle Scholar
Total: 106 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 501 hits
ReferencesLink to record
Permanent link

Direct link