Processing the Geospatial Data on-demand

Processing and presenting vast amounts of geospatial data remains a challenge, especially when data needs to be prepared for different missions and platforms.

Traditional processing methods rely on preparing datasets in advance, but for Badb this would generate massive amounts of data and require substantial storage and computing capacity. Our missions vary, and the data ultimately needs to be used by unmanned systems with limited onboard computing resources.

Instead of processing every possible version beforehand, the Badb approach is to process data on demand – generating what is needed, in the form it is needed, for a specific mission.

Historically, the prevailing approach has been fairly simple: preprocess and prepare the data, store it, and then distribute or present it. In practice, data is received on the ground, processed into the required formats, and stored for later use in the final presentation or application layer.

The above diagram has been inspired from ESA Sentinel missions hence the Level or "L" labeling - the naming can vary from mission to mission.

Challenges with preprocessed datasets

As with any largely static architecture, this approach has familiar drawbacks. It is less flexible, requires ongoing maintenance, and can generate significant storage costs.

This does not mean that preprocessing is unsuitable for every use case. For example, serving a map of a country with a fixed coordinate reference system (CRS), predefined zoom levels and tiles, and temporally static datasets can be handled efficiently using traditional methods.

The situation becomes much more demanding when a mission produces data continuously over many years and covers a large geographical area. In such cases, the amount of data can become enormous, and storing multiple processed versions quickly becomes expensive.

To illustrate the scale, the Sentinel-1 mission was reported to generate up to 6 TB of products per day. If sustained throughout a year, this would amount to approximately 2.19 PB annually [1].

This figure refers to generated product volume rather than the size of a single raw acquisition. The systematic Sentinel-1 processing chain archives Level-0 data as well as Level-1 GRD/SLC and Level-2 OCN products [2, 3]. A single Sentinel-1 satellite, Sentinel-1A, was reported to produce around 3.5 TB per day, equivalent to approximately 1.28 PB per year when annualised [3].

The same issue can be seen with Sentinel-2. Level-1C and Level-2A user products disseminated in 2021 totalled around 4.7 PB [4]. More than 1 PB of this annual volume resulted from spatial duplication caused by the UTM/MGRS tiling grid. These publicly available Sentinel-2 products are generated downstream from Level-0, Level-1A and Level-1B data [5].

In other words, a relatively small amount of raw or Level-0 satellite data can ultimately result in a much larger volume of processed products.

Compression and more efficient storage strategies can reduce some of these costs, but they do not remove the underlying problem. For the commercial missions used in the Badb project, such as the Pléiades series and GEOSAT-2, comparable figures are not always publicly reported. However, ESA missions provide a useful indication of the scale of the challenge.

The solution

Throughout the history of computing, new approaches have emerged in response to growing volumes of data and increasing processing demands.

The evolution of the World Wide Web provides a useful parallel. Early websites contained relatively little interactive content. Today, web applications provide rich and computationally demanding features that compete for processing power and network capacity. This has become possible largely because computing hardware and infrastructure have evolved continuously.

A similar development can be applied to Earth observation data.

Today, there are technologies capable of processing data very quickly, including GPUs, cloud computing platforms and high-speed memory. However, the total volume of data remains too large for inexpensive, fast storage at every processing level.

For Badb, there is an additional challenge: the required data products depend on the mission. Different missions may require different coordinate reference systems, tile sizes, contrast enhancements, data sources and other parameters. The number of possible combinations is therefore extremely large.

Instead of preparing and storing every possible version in advance, the solution is to process the data on demand.

The approach developed by KappaZeta is based on three main principles:

• Store only the root level of the data.

• Use modern infrastructure technologies to scale computing resources according to mission requirements.

• Develop processors and algorithms that can generate the required products on demand.

The graphic above illustrates this concept, although it does not show the cluster, cloud or hardware scaling components. The main development focus is therefore on processor development and infrastructure configuration.

Compared with traditional preprocessing architectures, this approach reduces the number of datasets and storage components that need to be maintained.

It also makes the system easier to improve over time. When processors or algorithms are updated, there is no need to launch large-scale reprocessing campaigns. Similarly, if quality checks identify problems in previously generated outputs, the entire archive does not need to be rebuilt because the data is generated again when requested.

This flexibility is particularly important for Badb. UAVs and UGVs can have very different data requirements, while battlefield conditions and the varying quality of commercial satellite imagery add further variables. The number of possible data configurations is therefore effectively too large to prepare and store in advance.

With on-demand processing, data can instead be generated according to the specific requirements of each mission and crew.

The underlying data can be stored in a secure cloud environment or kept offline, including directly at a ground control station. Processing resources can be located close to the data, while remote processing can make use of high-speed data transfers. Fast connectivity is therefore one of the key prerequisites for effective on-demand processing.

Most importantly, there is no need to maintain an archive containing every possible processed combination. Resources that would otherwise be spent on storing and maintaining these datasets can instead be invested in faster storage, networking and computing capacity.

References

[1] Copernicus Data Space Ecosystem, Annual report 2023. Annual reports.

[2] Copernicus Sentinel Wiki. Sentinel-1 mission and product information.

[3] Potin et al. (2016). Sentinel-1 mission and data processing information.

[4] Bauer-Marschallinger and Falkner (2023). Sentinel-2 data dissemination and product volume information.

[5] Meoni et al. (2024). Sentinel-2 processing levels and product generation.