Hierarchical, multiresolution data representations enable interactive analysis and visualization of large-scale simulations. One promising application of these techniques is to store high performance computing simulation output in a hierarchical Z (HZ) ordering that translates data from a Cartesian coordinate scheme to a one-dimensional array ordered by locality at different resolution levels. However, when the dimensions of the simulation data are not an even power of 2, parallel HZ ordering produces sparse memory and network access patterns that inhibit I/O performance. This work presents a new technique for parallel HZ ordering of simulation datasets that restructures simulation data into large (power of 2) blocks to facilitate efficient I/O aggregation. We perform both weak and strong scaling experiments using the S3D combustion application on both Cray-XE6 (65,536 cores) and IBM Blue Gene/P (131,072 cores) platforms. We demonstrate that data can be written in a hierarchical, multiresolution format with performance competitive to that of native data-ordering methods.

More information on PIDX and links to downloads.


Collaborators at:

Argonne National Laboratory

Venkatram Vishwanath
Phil Carns
Robert Ross
Robert Latham
Michael papka 

Sandia National Laboratories

Hemanth Kolla
Ray Grout
Jackie Chen

Project Staff

Sidarth Kumar Sidarth Kumar - Graduate Student Research Assistant
John Edwards John Edwards - Postdoctoral Research Associate
Cameron Christensen Cameron Christensen - Software Developer