Session
Session 11: Big Data From Small Satellites 2
Abstract
As part of an Internal Research & Development (IR&D) project, Solers created and demonstrated a Cloud-Based Ingest & Processing Framework (I&PF) hosted within the Amazon Web Services (AWS) cloud infrastructure, as a mechanism to enable fast/easy integration of satellite (and other) data sources, data processing/product generation algorithms, and data consumers within a cloud-hosted workflow (or “data pipeline”) framework. This framework leverages AWS cloud services and open source software. It provides web-based user interfaces and RESTful web services for discovery and access of the ingested and processed data, as well as workflow monitoring and management.
During the IR&D project, Solers implemented 3 different NOAA-specific use cases to demonstrate the flexibility of the Cloud-Based I&PF for NOAA satellite and radar data integration and processing. Solers further extended the framework by applying it to a commercial project with OmniEarth, leveraging it to automate OmniEarth's previously manual satellite imagery ingest and land classification deep learning algorithm execution workflows within their AWS cloud infrastructure. OmniEarth leverages land classified satellite imagery to assist with water budget calculations for their commercial Water Resource Management product, and the automation provided by the Cloud-Based I&PF helped OmniEarth significantly reduce their ingest and algorithm execution timeline (from days to hours).
Presentation
Cloud-Based Ingest & Processing Framework (I&PF) - A Cloud-Hosted Framework for Satellite (and Other) Data Integration and Processing
As part of an Internal Research & Development (IR&D) project, Solers created and demonstrated a Cloud-Based Ingest & Processing Framework (I&PF) hosted within the Amazon Web Services (AWS) cloud infrastructure, as a mechanism to enable fast/easy integration of satellite (and other) data sources, data processing/product generation algorithms, and data consumers within a cloud-hosted workflow (or “data pipeline”) framework. This framework leverages AWS cloud services and open source software. It provides web-based user interfaces and RESTful web services for discovery and access of the ingested and processed data, as well as workflow monitoring and management.
During the IR&D project, Solers implemented 3 different NOAA-specific use cases to demonstrate the flexibility of the Cloud-Based I&PF for NOAA satellite and radar data integration and processing. Solers further extended the framework by applying it to a commercial project with OmniEarth, leveraging it to automate OmniEarth's previously manual satellite imagery ingest and land classification deep learning algorithm execution workflows within their AWS cloud infrastructure. OmniEarth leverages land classified satellite imagery to assist with water budget calculations for their commercial Water Resource Management product, and the automation provided by the Cloud-Based I&PF helped OmniEarth significantly reduce their ingest and algorithm execution timeline (from days to hours).
