Example Use Case

For this example use case we are going to try and query 1 year of Sentinel 2 data over an AOI with 10x10km.

Let’s do some napkin math to figure out how much data that will be. To ground it in reality let’s actually query the CDSE STAC catalog.

import datetime
import pickle

import pystac_client
import shapely
from shapely.ops import transform
from pyproj import Transformer

point = shapely.geometry.Point(14.39903, 47.03920)
event = datetime.date(2022, 1, 1)
start = datetime.date(event.year - 1, event.month, event.day)

crs = "EPSG:3035"
# WGS84 -> LAEA Europe (EPSG:3035)
transformer = Transformer.from_crs("EPSG:4326", crs, always_xy=True)
point_3035 = transform(transformer.transform, point)

# 5 km square buffer -> 10x10 km box centered on the point
box_10km = point_3035.buffer(5000, cap_style="square")
bounds = box_10km.bounds

# for search:
back = Transformer.from_crs(crs, "EPSG:4326", always_xy=True)
box_4326 = transform(back.transform, box_10km)

# We'll also pickle those file for later use
# write out the geoms so they can be used by other notebooks
with open("params.pkl", "wb") as fp:
    pickle.dump(
        {
            "start": start,
            "end": event,
            "point": point,
            "point_3035": point_3035,
            "box_3035": box_10km,
            "box_4326": box_4326,
        },
        fp,
    )

URL = "https://stac.dataspace.copernicus.eu/v1"
cat = pystac_client.Client.open(URL)
cat.add_conforms_to("ITEM_SEARCH")

params = {
    "limit": 100,
    "collections": "sentinel-2-l2a",
    "datetime": f"{start.strftime('%Y-%m-%d')}/{event.strftime('%Y-%m-%d')}",
    "bbox": box_4326.bounds,
    "query": {"eo:cloud_cover": {"lte": 80}},
    "sortby": "properties.eo:cloud_cover",
    "fields": {"exclude": ["geometry"]},
}

items = list(cat.search(**params).items_as_dicts())
len(items)

89

It’s 89 acquisitions in total for 1 year with a cloud cover of less than or equal 80%.

For this example we’ll just pretend to want to use NDMI and would like to get 3 bands: B08, B11 and SCL.

B08 is at 10m resolution, B11 and SCL at 20m resolution. For a 10x10km AOI this works out to around 276MB of uncompressed data.

At the maximum 20MB/s download speed that Creodias S3 provides, this should only take a few seconds. Let’s see how long it actually takes.

Results

¹ only queried 48 acquisitions for some reason
² not including pre-computing/downloading TLM headers

From these results we can see that Sentinel Hub is by far the fastest option to get the full data cube. The fastest local option is the JP2 TLM Virtualizarr, this however is not an operational option (see details below).

The biggest disappointment is xcube: It takes very long to intitialize even the lazy cube, and does not seem to be set up for dask multi-threading, taking as long as the odc single threaded approach.

But why is everything else so slow?

The devil (details)

There’s a few complications we are glossing over which can add a lot of overhead. To understand them we first need to understand how Sentinel-2 data is provided by CDSE. We’ll be talking about tiles, blocks and what we need to do to get those blocks. First a high level visual overview. We’ll go into detail for each level below.

The largest level of tiling which Sentinel-2 provides is the MGRS tiles. These are 110x110km in size, with around 10km of overlap between tiles (this can be more or less depending on latitude and longitude). This leads us to:

Direct Access vs Hosted Services

All of this is most relevant when collecting your datacube directly from the data on a remote S3 bucket. Libraries which do it in this way are:

• odc-stac
• stackstac
• xcube-stac

All of these options only rely on a STAC catalog and a file server.

There is another option. CDSE also provides hosted services. These are services running on computing infrastructure in the same datacenter as where the Sentinel data is hosted. These services have the big advantage that they are a lot closer to the data than we are. Requests for data are handled entirely within the datacenter. This means that more bandwidth is available and requests complete faster.

The two hosted services available are:

• SentinelHub
• openEO

These services can assemble the finished datacube entirely within the datacenter, carrying out reprojection and mosaicing for you. Data from extra blocks which aren’t needed can be already discarded on the server and only the finished datacube can be sent to you. If the hosted service is efficient in doing this, it should be faster than compiling the data cube through direct access.

The disadvantage is that this pre-processing on the server has a cost. Because of this cost resources are limited per month. SentinelHub offers 20.000 Processing Units per month and 10.000 Credits for openEO for free users. How far this gets you depends on the amount of requested data.

For these hosted services the absolute best way to use them is to implement your entire workflow to run on the cloud. However for fast prototyping, having the data local is often much quicker.