Reimplementing the Hierarchical Data System using HDF5

@article{Jenness2015ReimplementingTH,
  title={Reimplementing the Hierarchical Data System using HDF5},
  author={Tim Jenness},
  journal={Astron. Comput.},
  year={2015},
  volume={12},
  pages={221-228}
}
  • T. Jenness
  • Published 13 February 2015
  • Computer Science, Physics
  • Astron. Comput.
Abstract The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N -Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS… Expand
HDFITS: Porting the FITS data model to HDF5
TLDR
A comparison of FITS and HDF5 as a format for storage of astronomy datasets is presented, showing that HDFITS allows faster reading of data, and improved compression, and by only changing the import lines in Python-based FITS utilities,HDFITS formatted data can be presented transparently as an in-memory FITS equivalent. Expand
ASDF: A new data format for astronomy
TLDR
The proposed format is called the Advanced Scientific Data Format (ASDF) and is based on an existing text format, YAML, that the author believes removes most of the current problems with the FITS format. Expand
Astronomical data formats: What we have and how we got here
  • J. Mink
  • Computer Science, Physics
  • Astron. Comput.
  • 2015
TLDR
The evolution of the use of world coordinates in FITS is presented as an example and knowledge of the evolution and advantages of present standards is useful before the future of how astronomical data is formatted is discussed. Expand
Learning from FITS: Limitations in use in modern astronomical research
TLDR
This paper details some selected important problems which exist within the FITS standard today and may provide insight into deeper underlying issues which reside in the format and provides a discussion of some lessons learned. Expand
To Integrate Heterogeneous devices to provide Astroinformatics Data
The domain of astronomy is now explored by armatures, hobbyist and professionals alike. One of the requirement is to view, analyses and compute astroinformatic parameters is to access data fromExpand
AST: A library for modelling and manipulating coordinate systems
TLDR
A description of the data model used by the Starlink AST library, a mature system that has been in use for more than 17 years, may be useful as a means of informing development of similar systems in the future. Expand
The General Single-Dish Data format: A retrospective
TLDR
An overview of the GSDD requirements and associated data model is provided, the implementation of the resultant file formats are discussed, its usage in the observatories is described and a retrospective on the format is provided. Expand
cuFFS: A GPU-accelerated code for Fast Faraday rotation measure Synthesis
TLDR
This paper presents a GPU-accelerated code, called cuFFS or CUDA-ACcelerated Fast Faraday Synthesis, to perform Faraday rotation measure synthesis, and aims to reduce the time spent on disk I/O and support the faster HDFITS format in addition to the standard FITS format. Expand
Automated reduction of sub-millimetre single-dish heterodyne data from the James Clerk Maxwell Telescope using orac-dr
With the advent of modern multi-detector heterodyne instruments that can result in observations generating thousands of spectra per minute it is no longer feasible to reduce these data as individualExpand
DEVELOPMENT OF A METHOD TO DETERMINE CUTTING FORCES BASED ON PLANNING AND PROCESS DATA AS CONTRIBUTION FOR THE CREATION OF DIGITAL PROCESS TWINS
TLDR
The work shows the recording of both data sources and creation of a digital process twin for a milling application and the usage of technological models at the example of cutting force calculation based on drive currents in combination with a material removal simulation. Expand
...
1
2
...

References

SHOWING 1-10 OF 53 REFERENCES
Status of LOFAR Data in HDF5 Format
The Hierarchical Data Format, version 5 (HDF5) is a data model, library, and file format for storing and managing data. It is designed for flexible and efficient I/O and for high volume, complexExpand
HDFITS: Porting the FITS data model to HDF5
TLDR
A comparison of FITS and HDF5 as a format for storage of astronomy datasets is presented, showing that HDFITS allows faster reading of data, and improved compression, and by only changing the import lines in Python-based FITS utilities,HDFITS formatted data can be presented transparently as an in-memory FITS equivalent. Expand
The Evolution of the FIGARO Data Reduction System
TLDR
The Figaro data reduction system originated at Caltech around 1983 and was adopted as Starlink's main spectroscopic data reduction package, although it is by no means limited to spectra; it has operations for images and data cubes and even a few (very specialized) for four-dimensional data hypercubes. Expand
The Future of Astronomical Data Formats: Learning from FITS
TLDR
This paper details some selected important problems which exist within the FITS standard today and may provide insight into deeper underlying issues which reside in the format and provides a discussion of some lessons learned. Expand
Learning from 25 years of the extensible N-Dimensional Data Format
TLDR
An overview of the historical drivers for the development of NDF and the lessons learned from using a defined hierarchical data model for many years in data reduction software, data pipelines and in data acquisition systems is provided. Expand
Is HDF5 a good format to replace UVFITS
TLDR
It is shown that the underlying data model of FITS can be ported to HDF5, a first step toward achieving widerHDF5 support and a tool for converting between formats. Expand
ASDF: A new data format for astronomy
TLDR
The proposed format is called the Advanced Scientific Data Format (ASDF) and is based on an existing text format, YAML, that the author believes removes most of the current problems with the FITS format. Expand
H5MD: A structured, efficient, and portable file format for molecular data
Abstract We propose a new file format named “H5MD” for storing molecular simulation data, such as trajectories of particle positions and velocities, along with thermodynamic observables that areExpand
Data Management At the UKIRT and JCMT
TLDR
The history of software tools to simplify observing and make it possible to use the telescopes in many different operational modes are discussed and how the decision to combine software development at both telescopes led each to get features that they could not have justified if they were treated independently. Expand
Learning from FITS: Limitations in use in modern astronomical research
TLDR
This paper details some selected important problems which exist within the FITS standard today and may provide insight into deeper underlying issues which reside in the format and provides a discussion of some lessons learned. Expand
...
1
2
3
4
5
...