Data access and discovery#

Authors & Contributors#

Authors#

Pier Lorenzo Marasco, Ispra (Italy), @pl-marasco
Alejandro Coca-Castro, The Alan Turing Institute (United Kingdom), @acocac
Anne Fouilloux, Simula Research Laboratory (Norway), @annefou

Contributors#

Tina Odaka, Ifremer (France), @tinaok

Overview

Questions

How to access online (remote) datasets?
How to prepare and discover online geoscience datasets?
What is Analysis Ready, Cloud Optimized data (ARCO)?
What is pangeo-forge?
What is stac?

Objectives

Learn to access datasets from online object storage
Learn about preparing and discovery online datasets
Learn about Analysis Cloud Optimized (ARCO) data
Learn about the Pangeo Forge initiative
Learn about stac

Context#

We will be using CMIP6 data and access them through S3-compatible storage using Pangeo Catalog.

We will also discuss the generation of Cloud-Optimised Datasets by introducing the pangeo-forge initiative. Finally, we will explore Sentinel-2 Cloud-Optimised Dataset online through SpatioTemporal Asset Catalogs (STAC).

Setup#

This episode uses the following main Python packages:

xarray [HH17] with netCDF4 and h5netcdf engines
intake-esm []
s3fs [S3FsDTeam16]

Please install these packages if not already available in your Python environment.

Packages#

In this episode, Python packages are imported when we start to use them. However, for best software practices, we recommend you to install and import all the necessary libraries at the top of your Jupyter notebook.

Introduction to CMIP6 data#

Coupled Model Intercomparison project Phase 6

Project under World Climate Research Programme (WCRP)
Since 1995 CMIP has coordinated climate model experiments
Defines common experiment protocols, forcings and output.
33 model groups participate

Very useful links:#

Database for data request: http://clipc-services.ceda.ac.uk/dreq/index.html
- –>SEARCH FOR VARIABLES: http://clipc-services.ceda.ac.uk/dreq/mipVars.html
- Search for experiments: http://clipc-services.ceda.ac.uk/dreq/experiments.html
Overview: https://www.wcrp-climate.org/wgcm-cmip/wgcm-cmip6
ES-DOCs: https://search.es-doc.org/

Advantages:#

Homogenized and standardized outputs
Same variable name
Same experiments

Experiments (DECK)#

S3-compatible Object Storage to access online data#

Up to now we have downloaded data locally and then opened with Xarray open_dataset. When willing to manipulate large amount of data, this approach is not optimal (since it requires a lot of unnecessary local downloads). Sharing data online as Object Storage allows for data sharing and access to much larger amounts of data.

One of the most popular methods to access online remote data is through Amazon Simple Storage Service (S3) and you don’t necessarily need to use Amazon services to benefit from S3 object storage. Many other providers offer S3-compatible object storage that can be accessed in a very similar way.

When using S3-compatible object storage, you still need to list all the files you would like to access. With such amount of data, it would be very cumbersome. This is why a catalog is created: this catalog is a text file (json format) which describes where and which data to get. It adds additional metadata too. CMIP6 Pangeo online catalog can be loaded using intake-esm Python package.

Introduction to the Pangeo CMIP6 online catalog#

import xarray as xr
xr.set_options(display_style='html')
import intake
import cftime

Open CMIP6 online catalog#

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(cat_url)
col

pangeo-cmip6 catalog with 7674 dataset(s) from 514818 asset(s):

	unique
activity_id	18
institution_id	36
source_id	88
experiment_id	170
member_id	657
table_id	37
variable_id	700
grid_label	10
zstore	514818
dcpp_init_year	60
version	736

Search corresponding data#

cat = col.search(source_id=['CESM2'], experiment_id=['historical'], table_id=['Amon'], variable_id=['tas'], member_id=['r1i1p1f1'])
cat.df

	activity_id	institution_id	source_id	experiment_id	member_id	table_id	variable_id	grid_label	zstore	dcpp_init_year	version
0	CMIP	NCAR	CESM2	historical	r1i1p1f1	Amon	tas	gn	gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1...	NaN	20190308

Create dictionary from the list of datasets we found#

This step may take several minutes so be patient!

dset_dict = cat.to_dataset_dict(zarr_kwargs={'use_cftime':True})

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'

100.00% [1/1 00:00<00:00]

dataset_list = list(dset_dict.keys())
dataset_list

['CMIP.NCAR.CESM2.historical.Amon.gn']

Open dataset#

Use xarray python package to analyze netCDF dataset
open_dataset allows to get all the metadata without loading data into memory.
with xarray, we only load into memory what is needed.

dset = dset_dict[dataset_list[0]]
dset

<xarray.Dataset>
Dimensions:    (lat: 192, nbnd: 2, lon: 288, member_id: 1, time: 1980)
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
    lat_bnds   (lat, nbnd) float32 dask.array<chunksize=(192, 2), meta=np.ndarray>
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
    lon_bnds   (lon, nbnd) float32 dask.array<chunksize=(288, 2), meta=np.ndarray>
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
    time_bnds  (time, nbnd) object dask.array<chunksize=(1980, 2), meta=np.ndarray>
  * member_id  (member_id) <U8 'r1i1p1f1'
Dimensions without coordinates: nbnd
Data variables:
    tas        (member_id, time, lat, lon) float32 dask.array<chunksize=(1, 600, 192, 288), meta=np.ndarray>
Attributes: (12/50)
    Conventions:             CF-1.7 CMIP-6.2
    activity_id:             CMIP
    branch_method:           standard
    branch_time_in_child:    674885.0
    branch_time_in_parent:   219000.0
    case_id:                 15
    ...                      ...
    variant_label:           r1i1p1f1
    status:                  2019-10-25;created;by nhn2@columbia.edu
    netcdf_tracking_ids:     hdl:21.14100/d9a7225a-49c3-4470-b7ab-a8180926f839
    version_id:              v20190308
    intake_esm_varname:      ['tas']
    intake_esm_dataset_key:  CMIP.NCAR.CESM2.historical.Amon.gn

Get metadata corresponding to near-surface air temperature (tas)#

print(dset['tas'])

<xarray.DataArray 'tas' (member_id: 1, time: 1980, lat: 192, lon: 288)>
dask.array<broadcast_to, shape=(1, 1980, 192, 288), dtype=float32, chunksize=(1, 600, 192, 288), chunktype=numpy.ndarray>
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
  * member_id  (member_id) <U8 'r1i1p1f1'
Attributes:
    cell_measures:  area: areacella
    cell_methods:   area: time: mean
    comment:        near-surface (usually, 2 meter) air temperature
    description:    near-surface (usually, 2 meter) air temperature
    frequency:      mon
    id:             tas
    long_name:      Near-Surface Air Temperature
    mipTable:       Amon
    out_name:       tas
    prov:           Amon ((isd.003))
    realm:          atmos
    standard_name:  air_temperature
    time:           time
    time_label:     time-mean
    time_title:     Temporal mean
    title:          Near-Surface Air Temperature
    type:           real
    units:          K
    variable_id:    tas

dset.time.values

array([cftime.DatetimeNoLeap(1850, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 2, 14, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 3, 15, 12, 0, 0, 0, has_year_zero=True),
       ...,
       cftime.DatetimeNoLeap(2014, 10, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 11, 15, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 12, 15, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)

dset.sel(lat=60, lon=10.75, method='nearest').sel(time='1850-12-15')['tas'].values

array([[269.032]], dtype=float32)

Warning

The same dataset can be available from different locations e.g. ESGF, Zenodo, S3-compatible Object storage, etc. How do you know if it corresponds to the very same dataset? You cannot know except if the datasets have a persistent identifier such as a Digital Object Identifier. It is therefore recommended 1) to be extra careful about where you get your datasets, and 2) to double check that the content is exactly what you expect (for instance, you can perform basic quality checks).

Pangeo Tutorial for the eScience Course 2022

Data access and discovery

Contents