START Conference Manager    

rasdaman: Big Data Cubes at Your Fingertips

Peter Baumann

(Submission #358)


Abstract

A paradigm shift is becoming reality. We begin to see the datacubes behind the millions of files. We start combining heterogeneous datacubes in anad-hoc fashion. And we begin to overcome the age-old, technology imposed divide between data and metadata, supported by query languages like OGC WCPS for geo datacubes and ISO SQL/MDA for general multi-dimensional arrays.

Rasdaman is a scalable datacube engine enabling simple ingestion, access, analytics, and fusion on any-size spatio-temporal datacubes. By implementing the OGC geo datacube analytics language, WCPS (Web Coverage Processing Service), rasdaman implements an "any query, any time, on any size" paradigm. However, except for experts this is not an end user interface - rather, it is a convenient client/server API. Users would remain in their comfort zone for data navigation (e.g., OpenLayers), Web GIS integration (e.g., QGIS, ArcGIS), or analytics (e.g., python and R). Among others, this unleashes direct data interaction, including 3-D visualization through NASA WorldWind, multi-cube fusion, what-if scenarios, common EO data processing, and timeseries analytics. Phase 1 of EarthServer has advanced scalable array database technology into 100+ TB services; in current phase 2, Petabyte datacubes are being built.

In the server, a series of advanced techniques boost rasdaman performance, including adaptive partitioning, semantic caching, automatic parallelization and distributed processing, and heterogeneous hardware. Current operational database sizes exceed 500 TB, and queries have been split across more than 1,000 AWS cloud nodes.

In standardization, rasdaman has acted as blueprint for datacubes in OGC, ISO, and INSPIRE, including the forthcoming ISO Array SQL standard; naturally, rasdaman has become OGC and INSPIRE WCS Reference Implementation. ESA in 2017 has spotted rasdaman as "the world-leading technology in this area". Still this year, rasdaman will be installed on an ESA satellite, OPS-SAT, to demonstrate feasibility of on-board datacube processing and integration with the emerging EarthServer federation where rasdaman acts as the standards-based technology platform.

In our talk, we will present the rasdaman architecture, relate it to the state of the art, and demonstrate large-scale deployments and how to federate these. In particular, we will show how rasdaman can enable INSPIRE service federation in a novel way.

Categories

Topic Area:  [2.5] Technical solutions for optimizing and/or extending the INSPIRE technical framework
Abstract Type:  Oral Presentation

START Conference Manager (V2.61.0 - Rev. 4840)