Document Type
Article
Journal/Book Title/Conference
Environmental Modelling & Software
Volume
102
Publisher
Pergamon Press
Publication Date
2-28-2018
First Page
233
Last Page
240
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Abstract
HydroShare (https://www.hydroshare.org) is an online collaborative system to support the open sharing of hydrologic data, analytical tools, and computer models. Hydrologic data and models are often large, extending to multi-gigabyte or terabyte scale, and as a result, the scalability of centralized data management poses challenges for a system such as HydroShare. A distributed data management framework that enables distributed physical data storage and management in multiple locations thus becomes a necessity. We use the iRODS (Integrated Rule-Oriented Data System) data grid middleware as the distributed data storage and management back end in HydroShare. iRODS provides a unified virtual file system for distributed physical storages in multiple locations and enables data federation across geographically dispersed institutions around the world. In this paper, we describe the iRODS-based distributed data management approaches implemented in HydroShare to provide a practical demonstration of a production system for supporting big data in the environmental sciences.
Recommended Citation
Yi, Hong, et al. "Advancing Distributed Data Management for the HydroShare Hydrologic Information System." Environmental Modelling & Software, vol. 102, 2018, pp. 233-240. https://doi.org/10.1016/j.envsoft.2017.12.008
