HDF5 rebuilt the scientific data format for big parallel datasets and won

The Hierarchical Data Format began at NCSA in the 1980s as a way to move scientific data between different computing platforms in a self-describing file. But by the late 1990s HDF4 had limits: files capped at 2GB, a data model with inconsistent classes, and no real parallel I/O.

In September 1998 NCSA released the HDF5 beta: a new experimental version with a 64-bit address space, a simpler object model built around datasets, spaces, types, groups and files, and effective parallel I/O using MPI-IO. The source was freely available, and the library was roughly a third the size of HDF4.

HDF5 kept the defining HDF idea — a portable, self-describing file where metadata travels with the data — and today the HDF Group maintains it under a BSD-style license as a high-performance storage suite running from laptops to massively parallel systems, with 700+ GitHub projects in fields from astronomy and earth science to genomics and finance.

Redesign the format, keep the self-describing idea

Self-describing files meant data carried its own meaning across platforms and teams.

64-bit addressing and parallel I/O fit the big-science computing that was arriving.

Open-source licensing removed every barrier to adoption.

A simpler object model made the format easier to implement and extend.

When the old format hits its limits, redesign the model rather than patching it: a cleaner object model plus open code is what let one format serve decades of data.

HDF5 became a fixture of scientific computing and industry, from astronomy and earth science to genomics and finance, maintained under a BSD-style license with a large ecosystem of libraries and tools; the format continues to evolve for exascale systems and machine-learning workloads.