The encyclopedia · R&D & Science · Technical decision · 1998–2006
HDF5 rebuilt the scientific data format for big parallel datasets and won
NCSA's HDF5 (1998) kept HDF's self-describing files but added 64-bit sizes and parallel I/O; it became science's standard.
NCSA · The HDF Group
the move
The Hierarchical Data Format began at NCSA in the 1980s as a way to move scientific data between different computing platforms in a self-describing file. But by the late 1990s HDF4 had limits: files capped at 2GB, a data model with inconsistent classes, and no real parallel I/O.
In September 1998 NCSA released the HDF5 beta: a new experimental version with a 64-bit address space, a simpler object model built around datasets, spaces, types, groups and files, and effective parallel I/O using MPI-IO. The source was freely available, and the library was roughly a third the size of HDF4.
HDF5 kept the defining HDF idea — a portable, self-describing file where metadata travels with the data — and today the HDF Group maintains it under a BSD-style license as a high-performance storage suite running from laptops to massively parallel systems, with 700+ GitHub projects in fields from astronomy and earth science to genomics and finance.
why it works
- Self-describing files meant data carried its own meaning across platforms and teams.
- 64-bit addressing and parallel I/O fit the big-science computing that was arriving.
- Open-source licensing removed every barrier to adoption.
- A simpler object model made the format easier to implement and extend.
what transfers
When the old format hits its limits, redesign the model rather than patching it: a cleaner object model plus open code is what let one format serve decades of data.
what came after
HDF5 became a fixture of scientific computing and industry, from astronomy and earth science to genomics and finance, maintained under a BSD-style license with a large ecosystem of libraries and tools; the format continues to evolve for exascale systems and machine-learning workloads.
references
spotted an error? The archive wants to know.