Data access and discovery#

Authors & Contributors#

Authors#

  • Pier Lorenzo Marasco, Ispra (Italy), @pl-marasco

  • Alejandro Coca-Castro, The Alan Turing Institute (United Kingdom), @acocac

  • Anne Fouilloux, Simula Research Laboratory (Norway), @annefou

Contributors#

  • Tina Odaka, Ifremer (France), @tinaok

Overview

Questions
  • How to access online (remote) datasets?
  • How to prepare and discover online geoscience datasets?
  • What is Analysis Ready, Cloud Optimized data (ARCO)?
  • What is pangeo-forge?
  • What is stac?
Objectives
  • Learn to access datasets from online object storage
  • Learn about preparing and discovery online datasets
  • Learn about Analysis Cloud Optimized (ARCO) data
  • Learn about the Pangeo Forge initiative
  • Learn about stac

Context#

We will be using CMIP6 data and access them through S3-compatible storage using Pangeo Catalog.

We will also discuss the generation of Cloud-Optimised Datasets by introducing the pangeo-forge initiative. Finally, we will explore Sentinel-2 Cloud-Optimised Dataset online through SpatioTemporal Asset Catalogs (STAC).

Setup#

This episode uses the following main Python packages:

Please install these packages if not already available in your Python environment.

Packages#

In this episode, Python packages are imported when we start to use them. However, for best software practices, we recommend you to install and import all the necessary libraries at the top of your Jupyter notebook.

Introduction to CMIP6 data#

Coupled Model Intercomparison project Phase 6

  • Project under World Climate Research Programme (WCRP)

  • Since 1995 CMIP has coordinated climate model experiments

  • Defines common experiment protocols, forcings and output.

  • 33 model groups participate

Advantages:#

  • Homogenized and standardized outputs

  • Same variable name

  • Same experiments

Experiments (DECK)#

image

S3-compatible Object Storage to access online data#

Up to now we have downloaded data locally and then opened with Xarray open_dataset. When willing to manipulate large amount of data, this approach is not optimal (since it requires a lot of unnecessary local downloads). Sharing data online as Object Storage allows for data sharing and access to much larger amounts of data.

One of the most popular methods to access online remote data is through Amazon Simple Storage Service (S3) and you don’t necessarily need to use Amazon services to benefit from S3 object storage. Many other providers offer S3-compatible object storage that can be accessed in a very similar way.

When using S3-compatible object storage, you still need to list all the files you would like to access. With such amount of data, it would be very cumbersome. This is why a catalog is created: this catalog is a text file (json format) which describes where and which data to get. It adds additional metadata too. CMIP6 Pangeo online catalog can be loaded using intake-esm Python package.

Introduction to the Pangeo CMIP6 online catalog#

import xarray as xr
xr.set_options(display_style='html')
import intake
import cftime

Open CMIP6 online catalog#

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(cat_url)
col

pangeo-cmip6 catalog with 7674 dataset(s) from 514818 asset(s):

unique
activity_id 18
institution_id 36
source_id 88
experiment_id 170
member_id 657
table_id 37
variable_id 700
grid_label 10
zstore 514818
dcpp_init_year 60
version 736

Search corresponding data#

cat = col.search(source_id=['CESM2'], experiment_id=['historical'], table_id=['Amon'], variable_id=['tas'], member_id=['r1i1p1f1'])
cat.df
activity_id institution_id source_id experiment_id member_id table_id variable_id grid_label zstore dcpp_init_year version
0 CMIP NCAR CESM2 historical r1i1p1f1 Amon tas gn gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1... NaN 20190308

Create dictionary from the list of datasets we found#

  • This step may take several minutes so be patient!

dset_dict = cat.to_dataset_dict(zarr_kwargs={'use_cftime':True})
--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
100.00% [1/1 00:00<00:00]
dataset_list = list(dset_dict.keys())
dataset_list
['CMIP.NCAR.CESM2.historical.Amon.gn']

Open dataset#

  • Use xarray python package to analyze netCDF dataset

  • open_dataset allows to get all the metadata without loading data into memory.

  • with xarray, we only load into memory what is needed.

dset = dset_dict[dataset_list[0]]
dset
<xarray.Dataset>
Dimensions:    (lat: 192, nbnd: 2, lon: 288, member_id: 1, time: 1980)
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
    lat_bnds   (lat, nbnd) float32 dask.array<chunksize=(192, 2), meta=np.ndarray>
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
    lon_bnds   (lon, nbnd) float32 dask.array<chunksize=(288, 2), meta=np.ndarray>
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
    time_bnds  (time, nbnd) object dask.array<chunksize=(1980, 2), meta=np.ndarray>
  * member_id  (member_id) <U8 'r1i1p1f1'
Dimensions without coordinates: nbnd
Data variables:
    tas        (member_id, time, lat, lon) float32 dask.array<chunksize=(1, 600, 192, 288), meta=np.ndarray>
Attributes: (12/50)
    Conventions:             CF-1.7 CMIP-6.2
    activity_id:             CMIP
    branch_method:           standard
    branch_time_in_child:    674885.0
    branch_time_in_parent:   219000.0
    case_id:                 15
    ...                      ...
    variant_label:           r1i1p1f1
    status:                  2019-10-25;created;by nhn2@columbia.edu
    netcdf_tracking_ids:     hdl:21.14100/d9a7225a-49c3-4470-b7ab-a8180926f839
    version_id:              v20190308
    intake_esm_varname:      ['tas']
    intake_esm_dataset_key:  CMIP.NCAR.CESM2.historical.Amon.gn

Get metadata corresponding to near-surface air temperature (tas)#

print(dset['tas'])
<xarray.DataArray 'tas' (member_id: 1, time: 1980, lat: 192, lon: 288)>
dask.array<broadcast_to, shape=(1, 1980, 192, 288), dtype=float32, chunksize=(1, 600, 192, 288), chunktype=numpy.ndarray>
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
  * member_id  (member_id) <U8 'r1i1p1f1'
Attributes:
    cell_measures:  area: areacella
    cell_methods:   area: time: mean
    comment:        near-surface (usually, 2 meter) air temperature
    description:    near-surface (usually, 2 meter) air temperature
    frequency:      mon
    id:             tas
    long_name:      Near-Surface Air Temperature
    mipTable:       Amon
    out_name:       tas
    prov:           Amon ((isd.003))
    realm:          atmos
    standard_name:  air_temperature
    time:           time
    time_label:     time-mean
    time_title:     Temporal mean
    title:          Near-Surface Air Temperature
    type:           real
    units:          K
    variable_id:    tas
dset.time.values
array([cftime.DatetimeNoLeap(1850, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 2, 14, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 3, 15, 12, 0, 0, 0, has_year_zero=True),
       ...,
       cftime.DatetimeNoLeap(2014, 10, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 11, 15, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 12, 15, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
dset.sel(lat=60, lon=10.75, method='nearest').sel(time='1850-12-15')['tas'].values
array([[269.032]], dtype=float32)

Warning

The same dataset can be available from different locations e.g. ESGF, Zenodo, S3-compatible Object storage, etc. How do you know if it corresponds to the very same dataset? You cannot know except if the datasets have a persistent identifier such as a Digital Object Identifier. It is therefore recommended 1) to be extra careful about where you get your datasets, and 2) to double check that the content is exactly what you expect (for instance, you can perform basic quality checks).

Access remote files from S3-compatible Object Storage#

The Pangeo CMIP6 online catalog is very user-friendly. However, the complete ESGF CMIP6 catalog is close to 20 PB and so far only a very tiny amount of cloud-optimised CMIP6 data has been generated (approximately 1PB). Therefore, you may have to download CMIP6 data or other kind of data for instance when comparing CMIP6 model outputs with observations.

Rather than downloading datasets locally and individually, it is recommended to share what you download to everyone. Additional datasets are being stored and made publicly available in OpenStack Object storage (Swift).

import s3fs
import xarray as xr
fs = s3fs.S3FileSystem(anon=True,
      client_kwargs={
         'endpoint_url': 'https://object-store.cloud.muni.cz'
      })

Tip

The parameter anon is for anonymous and is set to True because the data we have stored at https://object-store.cloud.muni.cz is public

List files and folders in existing buckets#

Instead of organizing files in various folders, object storage systems store files in a flat organization of containers (called “buckets”).

fs.ls('MODIS')
['MODIS/MOD08_M3.A2000032.061.2017276183309.nc',
 'MODIS/MOD08_M3.A2000061.061.2017272215822.nc',
 'MODIS/MOD08_M3.A2000092.061.2017276174940.nc',
 'MODIS/MOD08_M3.A2000122.061.2017275191641.nc',
 'MODIS/MOD08_M3.A2000153.061.2017276072839.nc',
 'MODIS/MOD08_M3.A2000183.061.2017276075622.nc',
 'MODIS/MOD08_M3.A2000214.061.2017276050502.nc',
 'MODIS/MOD08_M3.A2000245.061.2017276075932.nc',
 'MODIS/MOD08_M3.A2000275.061.2017276173030.nc',
 'MODIS/MOD08_M3.A2000306.061.2017276190346.nc',
 'MODIS/MOD08_M3.A2000336.061.2017276203640.nc',
 'MODIS/MOD08_M3.A2001001.061.2017277122349.nc',
 'MODIS/MOD08_M3.A2001032.061.2017277112301.nc',
 'MODIS/MOD08_M3.A2001060.061.2017277125646.nc',
 'MODIS/MOD08_M3.A2001091.061.2017277145429.nc',
 'MODIS/MOD08_M3.A2001121.061.2017277185847.nc',
 'MODIS/MOD08_M3.A2001152.061.2017277193228.nc',
 'MODIS/MOD08_M3.A2001182.061.2017277232025.nc',
 'MODIS/MOD08_M3.A2001213.061.2017278042002.nc',
 'MODIS/MOD08_M3.A2001244.061.2017278143507.nc',
 'MODIS/MOD08_M3.A2001274.061.2017278155329.nc',
 'MODIS/MOD08_M3.A2001305.061.2017278180906.nc',
 'MODIS/MOD08_M3.A2001335.061.2017278195919.nc',
 'MODIS/MOD08_M3.A2002001.061.2018116210818.nc',
 'MODIS/MOD08_M3.A2002032.061.2018086185625.nc',
 'MODIS/MOD08_M3.A2002060.061.2018116231409.nc',
 'MODIS/MOD08_M3.A2002091.061.2017279185034.nc',
 'MODIS/MOD08_M3.A2002121.061.2017279235924.nc',
 'MODIS/MOD08_M3.A2002152.061.2017280023717.nc',
 'MODIS/MOD08_M3.A2002182.061.2017280205715.nc',
 'MODIS/MOD08_M3.A2002213.061.2017280223026.nc',
 'MODIS/MOD08_M3.A2002244.061.2017280234438.nc',
 'MODIS/MOD08_M3.A2002274.061.2017281015551.nc',
 'MODIS/MOD08_M3.A2002305.061.2017281025115.nc',
 'MODIS/MOD08_M3.A2002335.061.2017281035150.nc',
 'MODIS/MOD08_M3.A2003001.061.2018117155743.nc',
 'MODIS/MOD08_M3.A2003032.061.2018116235452.nc',
 'MODIS/MOD08_M3.A2003060.061.2018117002319.nc',
 'MODIS/MOD08_M3.A2003091.061.2018117160200.nc',
 'MODIS/MOD08_M3.A2003121.061.2017283030440.nc',
 'MODIS/MOD08_M3.A2003152.061.2017283052308.nc',
 'MODIS/MOD08_M3.A2003182.061.2017283075355.nc',
 'MODIS/MOD08_M3.A2003213.061.2017283082726.nc',
 'MODIS/MOD08_M3.A2003244.061.2017283173012.nc',
 'MODIS/MOD08_M3.A2003274.061.2017283173119.nc',
 'MODIS/MOD08_M3.A2003305.061.2017283174124.nc',
 'MODIS/MOD08_M3.A2003335.061.2017283174833.nc',
 'MODIS/MOD08_M3.A2004001.061.2017284131754.nc',
 'MODIS/MOD08_M3.A2004032.061.2017284231722.nc',
 'MODIS/MOD08_M3.A2004061.061.2017285135054.nc',
 'MODIS/MOD08_M3.A2004092.061.2017285200445.nc',
 'MODIS/MOD08_M3.A2004122.061.2017286064100.nc',
 'MODIS/MOD08_M3.A2004153.061.2017286155932.nc',
 'MODIS/MOD08_M3.A2004183.061.2017287000057.nc',
 'MODIS/MOD08_M3.A2004214.061.2017287092018.nc',
 'MODIS/MOD08_M3.A2004245.061.2017288152313.nc',
 'MODIS/MOD08_M3.A2004275.061.2017288212405.nc',
 'MODIS/MOD08_M3.A2004306.061.2017288212156.nc',
 'MODIS/MOD08_M3.A2004336.061.2017289083810.nc',
 'MODIS/MOD08_M3.A2005001.061.2019072160440.nc',
 'MODIS/MOD08_M3.A2005032.061.2019072154406.nc',
 'MODIS/MOD08_M3.A2005060.061.2017291031953.nc',
 'MODIS/MOD08_M3.A2005091.061.2017290210757.nc',
 'MODIS/MOD08_M3.A2005121.061.2019072154947.nc',
 'MODIS/MOD08_M3.A2005152.061.2017298025833.nc',
 'MODIS/MOD08_M3.A2005182.061.2017292152251.nc',
 'MODIS/MOD08_M3.A2005213.061.2017293045950.nc',
 'MODIS/MOD08_M3.A2005244.061.2017293143200.nc',
 'MODIS/MOD08_M3.A2005274.061.2017293234404.nc',
 'MODIS/MOD08_M3.A2005305.061.2017294072541.nc',
 'MODIS/MOD08_M3.A2005335.061.2017296182027.nc',
 'MODIS/MOD08_M3.A2006001.061.2017272042922.nc',
 'MODIS/MOD08_M3.A2006032.061.2017272120606.nc',
 'MODIS/MOD08_M3.A2006060.061.2017272165438.nc',
 'MODIS/MOD08_M3.A2006091.061.2017272222236.nc',
 'MODIS/MOD08_M3.A2006121.061.2017276144021.nc',
 'MODIS/MOD08_M3.A2006152.061.2017277133958.nc',
 'MODIS/MOD08_M3.A2006182.061.2017275215927.nc',
 'MODIS/MOD08_M3.A2006213.061.2017279005924.nc',
 'MODIS/MOD08_M3.A2006244.061.2017277022732.nc',
 'MODIS/MOD08_M3.A2006274.061.2017278000639.nc',
 'MODIS/MOD08_M3.A2006305.061.2017278200246.nc',
 'MODIS/MOD08_M3.A2006335.061.2017279033150.nc',
 'MODIS/MOD08_M3.A2007001.061.2017282123406.nc',
 'MODIS/MOD08_M3.A2007032.061.2017282123335.nc',
 'MODIS/MOD08_M3.A2007060.061.2017282123411.nc',
 'MODIS/MOD08_M3.A2007091.061.2017282123415.nc',
 'MODIS/MOD08_M3.A2007121.061.2017283040324.nc',
 'MODIS/MOD08_M3.A2007152.061.2017283231630.nc',
 'MODIS/MOD08_M3.A2007182.061.2017285083949.nc',
 'MODIS/MOD08_M3.A2007213.061.2017285131028.nc',
 'MODIS/MOD08_M3.A2007244.061.2017285234236.nc',
 'MODIS/MOD08_M3.A2007274.061.2017286113952.nc',
 'MODIS/MOD08_M3.A2007305.061.2017287030845.nc',
 'MODIS/MOD08_M3.A2007335.061.2017287173138.nc',
 'MODIS/MOD08_M3.A2008001.061.2017288092622.nc',
 'MODIS/MOD08_M3.A2008032.061.2017289180803.nc',
 'MODIS/MOD08_M3.A2008061.061.2017291151011.nc',
 'MODIS/MOD08_M3.A2008092.061.2017291002057.nc',
 'MODIS/MOD08_M3.A2008122.061.2017291141050.nc',
 'MODIS/MOD08_M3.A2008153.061.2017292153358.nc',
 'MODIS/MOD08_M3.A2008183.061.2019072155257.nc',
 'MODIS/MOD08_M3.A2008214.061.2017293113821.nc',
 'MODIS/MOD08_M3.A2008245.061.2017294002156.nc',
 'MODIS/MOD08_M3.A2008275.061.2017297203335.nc',
 'MODIS/MOD08_M3.A2008306.061.2017295071904.nc',
 'MODIS/MOD08_M3.A2008336.061.2017296110255.nc',
 'MODIS/MOD08_M3.A2009001.061.2017298012528.nc',
 'MODIS/MOD08_M3.A2009032.061.2017298063546.nc',
 'MODIS/MOD08_M3.A2009060.061.2017298174606.nc',
 'MODIS/MOD08_M3.A2009091.061.2017299003246.nc',
 'MODIS/MOD08_M3.A2009121.061.2017299201650.nc',
 'MODIS/MOD08_M3.A2009152.061.2017303185329.nc',
 'MODIS/MOD08_M3.A2009182.061.2017304132125.nc',
 'MODIS/MOD08_M3.A2009213.061.2017319011248.nc',
 'MODIS/MOD08_M3.A2009244.061.2017306062357.nc',
 'MODIS/MOD08_M3.A2009274.061.2017307010542.nc',
 'MODIS/MOD08_M3.A2009305.061.2017307213730.nc',
 'MODIS/MOD08_M3.A2009335.061.2017308144544.nc',
 'MODIS/MOD08_M3.A2010001.061.2017310220725.nc',
 'MODIS/MOD08_M3.A2010032.061.2017311073823.nc',
 'MODIS/MOD08_M3.A2010060.061.2017311183851.nc',
 'MODIS/MOD08_M3.A2010091.061.2017312081103.nc',
 'MODIS/MOD08_M3.A2010121.061.2017312231425.nc',
 'MODIS/MOD08_M3.A2010152.061.2017313162026.nc',
 'MODIS/MOD08_M3.A2010182.061.2017314041948.nc',
 'MODIS/MOD08_M3.A2010213.061.2017318023643.nc',
 'MODIS/MOD08_M3.A2010244.061.2017318110207.nc',
 'MODIS/MOD08_M3.A2010274.061.2017318133158.nc',
 'MODIS/MOD08_M3.A2010305.061.2017318184613.nc',
 'MODIS/MOD08_M3.A2010335.061.2017320004241.nc',
 'MODIS/MOD08_M3.A2011001.061.2017320154950.nc',
 'MODIS/MOD08_M3.A2011032.061.2017321185442.nc',
 'MODIS/MOD08_M3.A2011060.061.2017322121213.nc',
 'MODIS/MOD08_M3.A2011091.061.2017323161255.nc',
 'MODIS/MOD08_M3.A2011121.061.2017324112918.nc',
 'MODIS/MOD08_M3.A2011152.061.2017325012251.nc',
 'MODIS/MOD08_M3.A2011182.061.2017325182247.nc',
 'MODIS/MOD08_M3.A2011213.061.2017326102442.nc',
 'MODIS/MOD08_M3.A2011244.061.2017327040751.nc',
 'MODIS/MOD08_M3.A2011274.061.2017328001250.nc',
 'MODIS/MOD08_M3.A2011305.061.2017328173108.nc',
 'MODIS/MOD08_M3.A2011335.061.2017330001115.nc',
 'MODIS/MOD08_M3.A2012001.061.2017332005900.nc',
 'MODIS/MOD08_M3.A2012032.061.2017332052720.nc',
 'MODIS/MOD08_M3.A2012061.061.2017332125551.nc',
 'MODIS/MOD08_M3.A2012092.061.2017333070458.nc',
 'MODIS/MOD08_M3.A2012122.061.2017334103546.nc',
 'MODIS/MOD08_M3.A2012153.061.2017335121954.nc',
 'MODIS/MOD08_M3.A2012183.061.2017336234843.nc',
 'MODIS/MOD08_M3.A2012214.061.2017338084514.nc',
 'MODIS/MOD08_M3.A2012245.061.2017339022214.nc',
 'MODIS/MOD08_M3.A2012275.061.2017339215337.nc',
 'MODIS/MOD08_M3.A2012306.061.2017340192241.nc',
 'MODIS/MOD08_M3.A2012336.061.2017341184807.nc',
 'MODIS/MOD08_M3.A2013001.061.2017298193816.nc',
 'MODIS/MOD08_M3.A2013032.061.2017297150003.nc',
 'MODIS/MOD08_M3.A2013060.061.2017298024454.nc',
 'MODIS/MOD08_M3.A2013091.061.2017298192533.nc',
 'MODIS/MOD08_M3.A2013121.061.2017299031432.nc',
 'MODIS/MOD08_M3.A2013152.061.2017299180328.nc',
 'MODIS/MOD08_M3.A2013182.061.2017300185420.nc',
 'MODIS/MOD08_M3.A2013213.061.2017304210739.nc',
 'MODIS/MOD08_M3.A2013244.061.2017304161740.nc',
 'MODIS/MOD08_M3.A2013274.061.2017305070652.nc',
 'MODIS/MOD08_M3.A2013305.061.2017306211731.nc',
 'MODIS/MOD08_M3.A2013335.061.2017308042052.nc',
 'MODIS/MOD08_M3.A2014001.061.2018116210846.nc',
 'MODIS/MOD08_M3.A2014032.061.2018116210744.nc',
 'MODIS/MOD08_M3.A2014060.061.2017310230130.nc',
 'MODIS/MOD08_M3.A2014091.061.2017311200138.nc',
 'MODIS/MOD08_M3.A2014121.061.2017312151500.nc',
 'MODIS/MOD08_M3.A2014152.061.2017313093158.nc',
 'MODIS/MOD08_M3.A2014182.061.2017314034734.nc',
 'MODIS/MOD08_M3.A2014213.061.2017314130249.nc',
 'MODIS/MOD08_M3.A2014244.061.2017317000542.nc',
 'MODIS/MOD08_M3.A2014274.061.2017317101857.nc',
 'MODIS/MOD08_M3.A2014305.061.2017317202015.nc',
 'MODIS/MOD08_M3.A2014335.061.2017318225533.nc',
 'MODIS/MOD08_M3.A2015001.061.2017319074723.nc',
 'MODIS/MOD08_M3.A2015032.061.2017319153036.nc',
 'MODIS/MOD08_M3.A2015060.061.2017320125759.nc',
 'MODIS/MOD08_M3.A2015091.061.2017320205832.nc',
 'MODIS/MOD08_M3.A2015121.061.2017321074517.nc',
 'MODIS/MOD08_M3.A2015152.061.2017321195647.nc',
 'MODIS/MOD08_M3.A2015182.061.2017322063945.nc',
 'MODIS/MOD08_M3.A2015213.061.2017322192741.nc',
 'MODIS/MOD08_M3.A2015244.061.2017323024455.nc',
 'MODIS/MOD08_M3.A2015274.061.2017323115746.nc',
 'MODIS/MOD08_M3.A2015305.061.2017323221600.nc',
 'MODIS/MOD08_M3.A2015335.061.2017324201044.nc',
 'MODIS/MOD08_M3.A2016001.061.2017325101827.nc',
 'MODIS/MOD08_M3.A2016032.061.2017325143021.nc',
 'MODIS/MOD08_M3.A2016061.061.2017326005020.nc',
 'MODIS/MOD08_M3.A2016092.061.2017326123542.nc',
 'MODIS/MOD08_M3.A2016122.061.2017326222614.nc',
 'MODIS/MOD08_M3.A2016153.061.2017327071005.nc',
 'MODIS/MOD08_M3.A2016183.061.2017327181105.nc',
 'MODIS/MOD08_M3.A2016214.061.2017328034102.nc',
 'MODIS/MOD08_M3.A2016245.061.2017328135749.nc',
 'MODIS/MOD08_M3.A2016275.061.2017353013021.nc',
 'MODIS/MOD08_M3.A2016306.061.2017329084922.nc',
 'MODIS/MOD08_M3.A2016336.061.2017329140013.nc',
 'MODIS/MOD08_M3.A2017001.061.2017316214128.nc',
 'MODIS/MOD08_M3.A2017032.061.2017313053532.nc',
 'MODIS/MOD08_M3.A2017060.061.2017314102914.nc',
 'MODIS/MOD08_M3.A2017091.061.2017314221045.nc',
 'MODIS/MOD08_M3.A2017121.061.2017315132821.nc',
 'MODIS/MOD08_M3.A2017152.061.2017315220820.nc',
 'MODIS/MOD08_M3.A2017182.061.2017317043341.nc',
 'MODIS/MOD08_M3.A2017213.061.2017318193955.nc',
 'MODIS/MOD08_M3.A2017244.061.2017285132541.nc',
 'MODIS/MOD08_M3.A2017274.061.2017305195215.nc',
 'MODIS/MOD08_M3.A2017305.061.2017335084629.nc',
 'MODIS/MOD08_M3.A2017335.061.2019072155425.nc',
 'MODIS/MOD08_M3.A2018001.061.2018032091034.nc',
 'MODIS/MOD08_M3.A2018032.061.2018064190408.nc',
 'MODIS/MOD08_M3.A2018060.061.2018093210347.nc',
 'MODIS/MOD08_M3.A2018091.061.2018121152525.nc',
 'MODIS/MOD08_M3.A2018121.061.2018152090257.nc',
 'MODIS/MOD08_M3.A2018152.061.2018182091458.nc',
 'MODIS/MOD08_M3.A2018182.061.2018213092526.nc',
 'MODIS/MOD08_M3.A2018213.061.2018247180628.nc',
 'MODIS/MOD08_M3.A2018244.061.2018278214421.nc',
 'MODIS/MOD08_M3.A2018274.061.2018312191357.nc',
 'MODIS/MOD08_M3.A2018305.061.2018335100840.nc',
 'MODIS/MOD08_M3.A2018335.061.2019072155558.nc',
 'MODIS/MOD08_M3.A2019001.061.2019038190346.nc',
 'MODIS/MOD08_M3.A2019032.061.2019063205813.nc',
 'MODIS/MOD08_M3.A2019060.061.2019091091201.nc',
 'MODIS/MOD08_M3.A2019091.061.2019121094111.nc',
 'MODIS/MOD08_M3.A2019121.061.2019152091235.nc',
 'MODIS/MOD08_M3.A2019213.061.2019247234417.nc',
 'MODIS/MOD08_M3.A2019244.061.2019275214642.nc',
 'MODIS/MOD08_M3.A2019274.061.2019305094830.nc',
 'MODIS/MOD08_M3.A2019305.061.2019336215510.nc',
 'MODIS/MOD08_M3.A2019335.061.2020002234053.nc',
 'MODIS/MOD08_M3.A2020001.061.2020032093438.nc',
 'MODIS/MOD08_M3.A2020032.061.2020063080211.nc',
 'MODIS/MOD08_M3.A2020061.061.2020092090542.nc',
 'MODIS/MOD08_M3.A2020092.061.2020125212701.nc',
 'MODIS/MOD08_M3.A2020122.061.2020153091858.nc',
 'MODIS/MOD08_M3.A2020153.061.2020183102654.nc',
 'MODIS/MOD08_M3.A2020183.061.2020214100214.nc',
 'MODIS/MOD08_M3.A2020214.061.2020245105303.nc',
 'MODIS/MOD08_M3.A2020245.061.2020275103907.nc',
 'MODIS/MOD08_M3.A2020275.061.2020306092007.nc',
 'MODIS/MOD08_M3.A2020306.061.2020336212521.nc',
 'MODIS/MOD08_M3.A2020336.061.2021001085852.nc',
 'MODIS/MOD08_M3.A2021001.061.2021032165629.nc',
 'MODIS/MOD08_M3.A2021032.061.2021060103941.nc',
 'MODIS/MOD08_M3.A2021060.061.2021092223659.nc',
 'MODIS/MOD08_M3.A2021091.061.2021122121127.nc',
 'MODIS/MOD08_M3.A2021121.061.2021152100135.nc',
 'MODIS/MOD08_M3.A2021152.061.2021183012452.nc',
 'MODIS/MOD08_M3.A2021182.061.2021213142147.nc',
 'MODIS/MOD08_M3.A2021213.061.2021244103656.nc',
 'MODIS/MOD08_M3.A2021244.061.2021276153158.nc']
fs.ls('MODIS/MOD08_M3.A2000183.061.2017276075622.nc')
['MODIS/MOD08_M3.A2000183.061.2017276075622.nc']

Access remote files from S3-compatible Object Storage#

s3path = 's3://MODIS/MOD08_M3.A2000183.061.2017276075622.nc'
modis = xr.open_dataset(fs.open(s3path))
modis
<xarray.Dataset>
Dimensions:                                              (YDim:mod08: 180,
                                                          XDim:mod08: 360,
                                                          Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08: 7)
Dimensions without coordinates: YDim:mod08, XDim:mod08,
                                Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08
Data variables: (12/13)
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean           (YDim:mod08, XDim:mod08) float32 ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean     (YDim:mod08, XDim:mod08) float32 ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean        (Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08, YDim:mod08, XDim:mod08) float32 ...
    Cloud_Top_Temperature_Mean_Mean                      (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Effective_Emissivity_Mean_Mean                 (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Fraction_Mean_Mean                             (YDim:mod08, XDim:mod08) float32 ...
    ...                                                   ...
    Cloud_Effective_Radius_Liquid_Mean_Mean              (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Water_Path_Liquid_Mean_Mean                    (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Water_Path_Ice_Mean_Mean                       (YDim:mod08, XDim:mod08) float32 ...
    Effective_Optical_Depth_Average_Ocean_Micron_Levels  (Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08) float32 ...
    XDim                                                 (XDim:mod08) float32 ...
    YDim                                                 (YDim:mod08) float32 ...
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Rename dimensions and variables#

modis = modis.rename_dims({'YDim:mod08': 'lat', 'XDim:mod08':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08':'levels'})
modis = modis.rename_vars({'YDim':'lat', 'XDim':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels': 'levels'})
modis
<xarray.Dataset>
Dimensions:                                           (lat: 180, lon: 360,
                                                       levels: 7)
Dimensions without coordinates: lat, lon, levels
Data variables: (12/13)
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean        (lat, lon) float32 nan ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean  (lat, lon) float32 nan ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean     (levels, lat, lon) float32 ...
    Cloud_Top_Temperature_Mean_Mean                   (lat, lon) float32 -1.4...
    Cloud_Effective_Emissivity_Mean_Mean              (lat, lon) float32 0.93...
    Cloud_Fraction_Mean_Mean                          (lat, lon) float32 0.88...
    ...                                                ...
    Cloud_Effective_Radius_Liquid_Mean_Mean           (lat, lon) float32 12.3...
    Cloud_Water_Path_Liquid_Mean_Mean                 (lat, lon) float32 146....
    Cloud_Water_Path_Ice_Mean_Mean                    (lat, lon) float32 249....
    levels                                            (levels) float32 0.47 ....
    lon                                               (lon) float32 -179.5 .....
    lat                                               (lat) float32 89.5 ... ...
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Coordinates#

x = modis.lon.squeeze().reset_coords(drop=True)
y = modis.lat.squeeze().reset_coords(drop=True)
z = modis.levels.squeeze().reset_coords(drop=True)
modis = modis.assign_coords({"lon": x, "lat": y, 'levels': z})
modis
<xarray.Dataset>
Dimensions:                                           (lat: 180, lon: 360,
                                                       levels: 7)
Coordinates:
  * levels                                            (levels) float32 0.47 ....
  * lon                                               (lon) float32 -179.5 .....
  * lat                                               (lat) float32 89.5 ... ...
Data variables:
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean        (lat, lon) float32 nan ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean  (lat, lon) float32 nan ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean     (levels, lat, lon) float32 ...
    Cloud_Top_Temperature_Mean_Mean                   (lat, lon) float32 -1.4...
    Cloud_Effective_Emissivity_Mean_Mean              (lat, lon) float32 0.93...
    Cloud_Fraction_Mean_Mean                          (lat, lon) float32 0.88...
    Cloud_Optical_Thickness_Liquid_Mean_Mean          (lat, lon) float32 20.9...
    Cloud_Effective_Radius_Liquid_Mean_Mean           (lat, lon) float32 12.3...
    Cloud_Water_Path_Liquid_Mean_Mean                 (lat, lon) float32 146....
    Cloud_Water_Path_Ice_Mean_Mean                    (lat, lon) float32 249....
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org
modis.sel(lat=60, lon=10.75, method='nearest')['Aerosol_Optical_Depth_Land_Ocean_Mean_Mean'].values
array(0.134, dtype=float32)

Access multiple remote files#

s3path = 's3://MODIS/*.nc'
remote_files = fs.glob(s3path)
remote_files
['MODIS/MOD08_M3.A2000032.061.2017276183309.nc',
 'MODIS/MOD08_M3.A2000061.061.2017272215822.nc',
 'MODIS/MOD08_M3.A2000092.061.2017276174940.nc',
 'MODIS/MOD08_M3.A2000122.061.2017275191641.nc',
 'MODIS/MOD08_M3.A2000153.061.2017276072839.nc',
 'MODIS/MOD08_M3.A2000183.061.2017276075622.nc',
 'MODIS/MOD08_M3.A2000214.061.2017276050502.nc',
 'MODIS/MOD08_M3.A2000245.061.2017276075932.nc',
 'MODIS/MOD08_M3.A2000275.061.2017276173030.nc',
 'MODIS/MOD08_M3.A2000306.061.2017276190346.nc',
 'MODIS/MOD08_M3.A2000336.061.2017276203640.nc',
 'MODIS/MOD08_M3.A2001001.061.2017277122349.nc',
 'MODIS/MOD08_M3.A2001032.061.2017277112301.nc',
 'MODIS/MOD08_M3.A2001060.061.2017277125646.nc',
 'MODIS/MOD08_M3.A2001091.061.2017277145429.nc',
 'MODIS/MOD08_M3.A2001121.061.2017277185847.nc',
 'MODIS/MOD08_M3.A2001152.061.2017277193228.nc',
 'MODIS/MOD08_M3.A2001182.061.2017277232025.nc',
 'MODIS/MOD08_M3.A2001213.061.2017278042002.nc',
 'MODIS/MOD08_M3.A2001244.061.2017278143507.nc',
 'MODIS/MOD08_M3.A2001274.061.2017278155329.nc',
 'MODIS/MOD08_M3.A2001305.061.2017278180906.nc',
 'MODIS/MOD08_M3.A2001335.061.2017278195919.nc',
 'MODIS/MOD08_M3.A2002001.061.2018116210818.nc',
 'MODIS/MOD08_M3.A2002032.061.2018086185625.nc',
 'MODIS/MOD08_M3.A2002060.061.2018116231409.nc',
 'MODIS/MOD08_M3.A2002091.061.2017279185034.nc',
 'MODIS/MOD08_M3.A2002121.061.2017279235924.nc',
 'MODIS/MOD08_M3.A2002152.061.2017280023717.nc',
 'MODIS/MOD08_M3.A2002182.061.2017280205715.nc',
 'MODIS/MOD08_M3.A2002213.061.2017280223026.nc',
 'MODIS/MOD08_M3.A2002244.061.2017280234438.nc',
 'MODIS/MOD08_M3.A2002274.061.2017281015551.nc',
 'MODIS/MOD08_M3.A2002305.061.2017281025115.nc',
 'MODIS/MOD08_M3.A2002335.061.2017281035150.nc',
 'MODIS/MOD08_M3.A2003001.061.2018117155743.nc',
 'MODIS/MOD08_M3.A2003032.061.2018116235452.nc',
 'MODIS/MOD08_M3.A2003060.061.2018117002319.nc',
 'MODIS/MOD08_M3.A2003091.061.2018117160200.nc',
 'MODIS/MOD08_M3.A2003121.061.2017283030440.nc',
 'MODIS/MOD08_M3.A2003152.061.2017283052308.nc',
 'MODIS/MOD08_M3.A2003182.061.2017283075355.nc',
 'MODIS/MOD08_M3.A2003213.061.2017283082726.nc',
 'MODIS/MOD08_M3.A2003244.061.2017283173012.nc',
 'MODIS/MOD08_M3.A2003274.061.2017283173119.nc',
 'MODIS/MOD08_M3.A2003305.061.2017283174124.nc',
 'MODIS/MOD08_M3.A2003335.061.2017283174833.nc',
 'MODIS/MOD08_M3.A2004001.061.2017284131754.nc',
 'MODIS/MOD08_M3.A2004032.061.2017284231722.nc',
 'MODIS/MOD08_M3.A2004061.061.2017285135054.nc',
 'MODIS/MOD08_M3.A2004092.061.2017285200445.nc',
 'MODIS/MOD08_M3.A2004122.061.2017286064100.nc',
 'MODIS/MOD08_M3.A2004153.061.2017286155932.nc',
 'MODIS/MOD08_M3.A2004183.061.2017287000057.nc',
 'MODIS/MOD08_M3.A2004214.061.2017287092018.nc',
 'MODIS/MOD08_M3.A2004245.061.2017288152313.nc',
 'MODIS/MOD08_M3.A2004275.061.2017288212405.nc',
 'MODIS/MOD08_M3.A2004306.061.2017288212156.nc',
 'MODIS/MOD08_M3.A2004336.061.2017289083810.nc',
 'MODIS/MOD08_M3.A2005001.061.2019072160440.nc',
 'MODIS/MOD08_M3.A2005032.061.2019072154406.nc',
 'MODIS/MOD08_M3.A2005060.061.2017291031953.nc',
 'MODIS/MOD08_M3.A2005091.061.2017290210757.nc',
 'MODIS/MOD08_M3.A2005121.061.2019072154947.nc',
 'MODIS/MOD08_M3.A2005152.061.2017298025833.nc',
 'MODIS/MOD08_M3.A2005182.061.2017292152251.nc',
 'MODIS/MOD08_M3.A2005213.061.2017293045950.nc',
 'MODIS/MOD08_M3.A2005244.061.2017293143200.nc',
 'MODIS/MOD08_M3.A2005274.061.2017293234404.nc',
 'MODIS/MOD08_M3.A2005305.061.2017294072541.nc',
 'MODIS/MOD08_M3.A2005335.061.2017296182027.nc',
 'MODIS/MOD08_M3.A2006001.061.2017272042922.nc',
 'MODIS/MOD08_M3.A2006032.061.2017272120606.nc',
 'MODIS/MOD08_M3.A2006060.061.2017272165438.nc',
 'MODIS/MOD08_M3.A2006091.061.2017272222236.nc',
 'MODIS/MOD08_M3.A2006121.061.2017276144021.nc',
 'MODIS/MOD08_M3.A2006152.061.2017277133958.nc',
 'MODIS/MOD08_M3.A2006182.061.2017275215927.nc',
 'MODIS/MOD08_M3.A2006213.061.2017279005924.nc',
 'MODIS/MOD08_M3.A2006244.061.2017277022732.nc',
 'MODIS/MOD08_M3.A2006274.061.2017278000639.nc',
 'MODIS/MOD08_M3.A2006305.061.2017278200246.nc',
 'MODIS/MOD08_M3.A2006335.061.2017279033150.nc',
 'MODIS/MOD08_M3.A2007001.061.2017282123406.nc',
 'MODIS/MOD08_M3.A2007032.061.2017282123335.nc',
 'MODIS/MOD08_M3.A2007060.061.2017282123411.nc',
 'MODIS/MOD08_M3.A2007091.061.2017282123415.nc',
 'MODIS/MOD08_M3.A2007121.061.2017283040324.nc',
 'MODIS/MOD08_M3.A2007152.061.2017283231630.nc',
 'MODIS/MOD08_M3.A2007182.061.2017285083949.nc',
 'MODIS/MOD08_M3.A2007213.061.2017285131028.nc',
 'MODIS/MOD08_M3.A2007244.061.2017285234236.nc',
 'MODIS/MOD08_M3.A2007274.061.2017286113952.nc',
 'MODIS/MOD08_M3.A2007305.061.2017287030845.nc',
 'MODIS/MOD08_M3.A2007335.061.2017287173138.nc',
 'MODIS/MOD08_M3.A2008001.061.2017288092622.nc',
 'MODIS/MOD08_M3.A2008032.061.2017289180803.nc',
 'MODIS/MOD08_M3.A2008061.061.2017291151011.nc',
 'MODIS/MOD08_M3.A2008092.061.2017291002057.nc',
 'MODIS/MOD08_M3.A2008122.061.2017291141050.nc',
 'MODIS/MOD08_M3.A2008153.061.2017292153358.nc',
 'MODIS/MOD08_M3.A2008183.061.2019072155257.nc',
 'MODIS/MOD08_M3.A2008214.061.2017293113821.nc',
 'MODIS/MOD08_M3.A2008245.061.2017294002156.nc',
 'MODIS/MOD08_M3.A2008275.061.2017297203335.nc',
 'MODIS/MOD08_M3.A2008306.061.2017295071904.nc',
 'MODIS/MOD08_M3.A2008336.061.2017296110255.nc',
 'MODIS/MOD08_M3.A2009001.061.2017298012528.nc',
 'MODIS/MOD08_M3.A2009032.061.2017298063546.nc',
 'MODIS/MOD08_M3.A2009060.061.2017298174606.nc',
 'MODIS/MOD08_M3.A2009091.061.2017299003246.nc',
 'MODIS/MOD08_M3.A2009121.061.2017299201650.nc',
 'MODIS/MOD08_M3.A2009152.061.2017303185329.nc',
 'MODIS/MOD08_M3.A2009182.061.2017304132125.nc',
 'MODIS/MOD08_M3.A2009213.061.2017319011248.nc',
 'MODIS/MOD08_M3.A2009244.061.2017306062357.nc',
 'MODIS/MOD08_M3.A2009274.061.2017307010542.nc',
 'MODIS/MOD08_M3.A2009305.061.2017307213730.nc',
 'MODIS/MOD08_M3.A2009335.061.2017308144544.nc',
 'MODIS/MOD08_M3.A2010001.061.2017310220725.nc',
 'MODIS/MOD08_M3.A2010032.061.2017311073823.nc',
 'MODIS/MOD08_M3.A2010060.061.2017311183851.nc',
 'MODIS/MOD08_M3.A2010091.061.2017312081103.nc',
 'MODIS/MOD08_M3.A2010121.061.2017312231425.nc',
 'MODIS/MOD08_M3.A2010152.061.2017313162026.nc',
 'MODIS/MOD08_M3.A2010182.061.2017314041948.nc',
 'MODIS/MOD08_M3.A2010213.061.2017318023643.nc',
 'MODIS/MOD08_M3.A2010244.061.2017318110207.nc',
 'MODIS/MOD08_M3.A2010274.061.2017318133158.nc',
 'MODIS/MOD08_M3.A2010305.061.2017318184613.nc',
 'MODIS/MOD08_M3.A2010335.061.2017320004241.nc',
 'MODIS/MOD08_M3.A2011001.061.2017320154950.nc',
 'MODIS/MOD08_M3.A2011032.061.2017321185442.nc',
 'MODIS/MOD08_M3.A2011060.061.2017322121213.nc',
 'MODIS/MOD08_M3.A2011091.061.2017323161255.nc',
 'MODIS/MOD08_M3.A2011121.061.2017324112918.nc',
 'MODIS/MOD08_M3.A2011152.061.2017325012251.nc',
 'MODIS/MOD08_M3.A2011182.061.2017325182247.nc',
 'MODIS/MOD08_M3.A2011213.061.2017326102442.nc',
 'MODIS/MOD08_M3.A2011244.061.2017327040751.nc',
 'MODIS/MOD08_M3.A2011274.061.2017328001250.nc',
 'MODIS/MOD08_M3.A2011305.061.2017328173108.nc',
 'MODIS/MOD08_M3.A2011335.061.2017330001115.nc',
 'MODIS/MOD08_M3.A2012001.061.2017332005900.nc',
 'MODIS/MOD08_M3.A2012032.061.2017332052720.nc',
 'MODIS/MOD08_M3.A2012061.061.2017332125551.nc',
 'MODIS/MOD08_M3.A2012092.061.2017333070458.nc',
 'MODIS/MOD08_M3.A2012122.061.2017334103546.nc',
 'MODIS/MOD08_M3.A2012153.061.2017335121954.nc',
 'MODIS/MOD08_M3.A2012183.061.2017336234843.nc',
 'MODIS/MOD08_M3.A2012214.061.2017338084514.nc',
 'MODIS/MOD08_M3.A2012245.061.2017339022214.nc',
 'MODIS/MOD08_M3.A2012275.061.2017339215337.nc',
 'MODIS/MOD08_M3.A2012306.061.2017340192241.nc',
 'MODIS/MOD08_M3.A2012336.061.2017341184807.nc',
 'MODIS/MOD08_M3.A2013001.061.2017298193816.nc',
 'MODIS/MOD08_M3.A2013032.061.2017297150003.nc',
 'MODIS/MOD08_M3.A2013060.061.2017298024454.nc',
 'MODIS/MOD08_M3.A2013091.061.2017298192533.nc',
 'MODIS/MOD08_M3.A2013121.061.2017299031432.nc',
 'MODIS/MOD08_M3.A2013152.061.2017299180328.nc',
 'MODIS/MOD08_M3.A2013182.061.2017300185420.nc',
 'MODIS/MOD08_M3.A2013213.061.2017304210739.nc',
 'MODIS/MOD08_M3.A2013244.061.2017304161740.nc',
 'MODIS/MOD08_M3.A2013274.061.2017305070652.nc',
 'MODIS/MOD08_M3.A2013305.061.2017306211731.nc',
 'MODIS/MOD08_M3.A2013335.061.2017308042052.nc',
 'MODIS/MOD08_M3.A2014001.061.2018116210846.nc',
 'MODIS/MOD08_M3.A2014032.061.2018116210744.nc',
 'MODIS/MOD08_M3.A2014060.061.2017310230130.nc',
 'MODIS/MOD08_M3.A2014091.061.2017311200138.nc',
 'MODIS/MOD08_M3.A2014121.061.2017312151500.nc',
 'MODIS/MOD08_M3.A2014152.061.2017313093158.nc',
 'MODIS/MOD08_M3.A2014182.061.2017314034734.nc',
 'MODIS/MOD08_M3.A2014213.061.2017314130249.nc',
 'MODIS/MOD08_M3.A2014244.061.2017317000542.nc',
 'MODIS/MOD08_M3.A2014274.061.2017317101857.nc',
 'MODIS/MOD08_M3.A2014305.061.2017317202015.nc',
 'MODIS/MOD08_M3.A2014335.061.2017318225533.nc',
 'MODIS/MOD08_M3.A2015001.061.2017319074723.nc',
 'MODIS/MOD08_M3.A2015032.061.2017319153036.nc',
 'MODIS/MOD08_M3.A2015060.061.2017320125759.nc',
 'MODIS/MOD08_M3.A2015091.061.2017320205832.nc',
 'MODIS/MOD08_M3.A2015121.061.2017321074517.nc',
 'MODIS/MOD08_M3.A2015152.061.2017321195647.nc',
 'MODIS/MOD08_M3.A2015182.061.2017322063945.nc',
 'MODIS/MOD08_M3.A2015213.061.2017322192741.nc',
 'MODIS/MOD08_M3.A2015244.061.2017323024455.nc',
 'MODIS/MOD08_M3.A2015274.061.2017323115746.nc',
 'MODIS/MOD08_M3.A2015305.061.2017323221600.nc',
 'MODIS/MOD08_M3.A2015335.061.2017324201044.nc',
 'MODIS/MOD08_M3.A2016001.061.2017325101827.nc',
 'MODIS/MOD08_M3.A2016032.061.2017325143021.nc',
 'MODIS/MOD08_M3.A2016061.061.2017326005020.nc',
 'MODIS/MOD08_M3.A2016092.061.2017326123542.nc',
 'MODIS/MOD08_M3.A2016122.061.2017326222614.nc',
 'MODIS/MOD08_M3.A2016153.061.2017327071005.nc',
 'MODIS/MOD08_M3.A2016183.061.2017327181105.nc',
 'MODIS/MOD08_M3.A2016214.061.2017328034102.nc',
 'MODIS/MOD08_M3.A2016245.061.2017328135749.nc',
 'MODIS/MOD08_M3.A2016275.061.2017353013021.nc',
 'MODIS/MOD08_M3.A2016306.061.2017329084922.nc',
 'MODIS/MOD08_M3.A2016336.061.2017329140013.nc',
 'MODIS/MOD08_M3.A2017001.061.2017316214128.nc',
 'MODIS/MOD08_M3.A2017032.061.2017313053532.nc',
 'MODIS/MOD08_M3.A2017060.061.2017314102914.nc',
 'MODIS/MOD08_M3.A2017091.061.2017314221045.nc',
 'MODIS/MOD08_M3.A2017121.061.2017315132821.nc',
 'MODIS/MOD08_M3.A2017152.061.2017315220820.nc',
 'MODIS/MOD08_M3.A2017182.061.2017317043341.nc',
 'MODIS/MOD08_M3.A2017213.061.2017318193955.nc',
 'MODIS/MOD08_M3.A2017244.061.2017285132541.nc',
 'MODIS/MOD08_M3.A2017274.061.2017305195215.nc',
 'MODIS/MOD08_M3.A2017305.061.2017335084629.nc',
 'MODIS/MOD08_M3.A2017335.061.2019072155425.nc',
 'MODIS/MOD08_M3.A2018001.061.2018032091034.nc',
 'MODIS/MOD08_M3.A2018032.061.2018064190408.nc',
 'MODIS/MOD08_M3.A2018060.061.2018093210347.nc',
 'MODIS/MOD08_M3.A2018091.061.2018121152525.nc',
 'MODIS/MOD08_M3.A2018121.061.2018152090257.nc',
 'MODIS/MOD08_M3.A2018152.061.2018182091458.nc',
 'MODIS/MOD08_M3.A2018182.061.2018213092526.nc',
 'MODIS/MOD08_M3.A2018213.061.2018247180628.nc',
 'MODIS/MOD08_M3.A2018244.061.2018278214421.nc',
 'MODIS/MOD08_M3.A2018274.061.2018312191357.nc',
 'MODIS/MOD08_M3.A2018305.061.2018335100840.nc',
 'MODIS/MOD08_M3.A2018335.061.2019072155558.nc',
 'MODIS/MOD08_M3.A2019001.061.2019038190346.nc',
 'MODIS/MOD08_M3.A2019032.061.2019063205813.nc',
 'MODIS/MOD08_M3.A2019060.061.2019091091201.nc',
 'MODIS/MOD08_M3.A2019091.061.2019121094111.nc',
 'MODIS/MOD08_M3.A2019121.061.2019152091235.nc',
 'MODIS/MOD08_M3.A2019213.061.2019247234417.nc',
 'MODIS/MOD08_M3.A2019244.061.2019275214642.nc',
 'MODIS/MOD08_M3.A2019274.061.2019305094830.nc',
 'MODIS/MOD08_M3.A2019305.061.2019336215510.nc',
 'MODIS/MOD08_M3.A2019335.061.2020002234053.nc',
 'MODIS/MOD08_M3.A2020001.061.2020032093438.nc',
 'MODIS/MOD08_M3.A2020032.061.2020063080211.nc',
 'MODIS/MOD08_M3.A2020061.061.2020092090542.nc',
 'MODIS/MOD08_M3.A2020092.061.2020125212701.nc',
 'MODIS/MOD08_M3.A2020122.061.2020153091858.nc',
 'MODIS/MOD08_M3.A2020153.061.2020183102654.nc',
 'MODIS/MOD08_M3.A2020183.061.2020214100214.nc',
 'MODIS/MOD08_M3.A2020214.061.2020245105303.nc',
 'MODIS/MOD08_M3.A2020245.061.2020275103907.nc',
 'MODIS/MOD08_M3.A2020275.061.2020306092007.nc',
 'MODIS/MOD08_M3.A2020306.061.2020336212521.nc',
 'MODIS/MOD08_M3.A2020336.061.2021001085852.nc',
 'MODIS/MOD08_M3.A2021001.061.2021032165629.nc',
 'MODIS/MOD08_M3.A2021032.061.2021060103941.nc',
 'MODIS/MOD08_M3.A2021060.061.2021092223659.nc',
 'MODIS/MOD08_M3.A2021091.061.2021122121127.nc',
 'MODIS/MOD08_M3.A2021121.061.2021152100135.nc',
 'MODIS/MOD08_M3.A2021152.061.2021183012452.nc',
 'MODIS/MOD08_M3.A2021182.061.2021213142147.nc',
 'MODIS/MOD08_M3.A2021213.061.2021244103656.nc',
 'MODIS/MOD08_M3.A2021244.061.2021276153158.nc']

We need to add a time dimension to concatenate data. For this, we define a function that will be called for each remote file (via the preprocess parameter of Xarray open_mfdataset.)

from datetime import datetime
def paths_to_datetimeindex(paths):
    return  [datetime.strptime(date.split('.A')[-1].split('.')[0], '%Y%j') for date in paths]

Xarray open_mfdataset allows opening multiple files at the same time.

# Iterate through remote_files to create a fileset
fileset = [fs.open(file) for file in remote_files]

When opening remote files, you can also select the variables you wish to analyze.

# Create variable used for time axis
time_var = xr.Variable('time', paths_to_datetimeindex(remote_files))
time_var
<xarray.Variable (time: 258)>
array(['2000-02-01T00:00:00.000000000', '2000-03-01T00:00:00.000000000',
       '2000-04-01T00:00:00.000000000', '2000-05-01T00:00:00.000000000',
       '2000-06-01T00:00:00.000000000', '2000-07-01T00:00:00.000000000',
       '2000-08-01T00:00:00.000000000', '2000-09-01T00:00:00.000000000',
       '2000-10-01T00:00:00.000000000', '2000-11-01T00:00:00.000000000',
       '2000-12-01T00:00:00.000000000', '2001-01-01T00:00:00.000000000',
       '2001-02-01T00:00:00.000000000', '2001-03-01T00:00:00.000000000',
       '2001-04-01T00:00:00.000000000', '2001-05-01T00:00:00.000000000',
       '2001-06-01T00:00:00.000000000', '2001-07-01T00:00:00.000000000',
       '2001-08-01T00:00:00.000000000', '2001-09-01T00:00:00.000000000',
       '2001-10-01T00:00:00.000000000', '2001-11-01T00:00:00.000000000',
       '2001-12-01T00:00:00.000000000', '2002-01-01T00:00:00.000000000',
       '2002-02-01T00:00:00.000000000', '2002-03-01T00:00:00.000000000',
       '2002-04-01T00:00:00.000000000', '2002-05-01T00:00:00.000000000',
       '2002-06-01T00:00:00.000000000', '2002-07-01T00:00:00.000000000',
       '2002-08-01T00:00:00.000000000', '2002-09-01T00:00:00.000000000',
       '2002-10-01T00:00:00.000000000', '2002-11-01T00:00:00.000000000',
       '2002-12-01T00:00:00.000000000', '2003-01-01T00:00:00.000000000',
       '2003-02-01T00:00:00.000000000', '2003-03-01T00:00:00.000000000',
       '2003-04-01T00:00:00.000000000', '2003-05-01T00:00:00.000000000',
...
       '2018-06-01T00:00:00.000000000', '2018-07-01T00:00:00.000000000',
       '2018-08-01T00:00:00.000000000', '2018-09-01T00:00:00.000000000',
       '2018-10-01T00:00:00.000000000', '2018-11-01T00:00:00.000000000',
       '2018-12-01T00:00:00.000000000', '2019-01-01T00:00:00.000000000',
       '2019-02-01T00:00:00.000000000', '2019-03-01T00:00:00.000000000',
       '2019-04-01T00:00:00.000000000', '2019-05-01T00:00:00.000000000',
       '2019-08-01T00:00:00.000000000', '2019-09-01T00:00:00.000000000',
       '2019-10-01T00:00:00.000000000', '2019-11-01T00:00:00.000000000',
       '2019-12-01T00:00:00.000000000', '2020-01-01T00:00:00.000000000',
       '2020-02-01T00:00:00.000000000', '2020-03-01T00:00:00.000000000',
       '2020-04-01T00:00:00.000000000', '2020-05-01T00:00:00.000000000',
       '2020-06-01T00:00:00.000000000', '2020-07-01T00:00:00.000000000',
       '2020-08-01T00:00:00.000000000', '2020-09-01T00:00:00.000000000',
       '2020-10-01T00:00:00.000000000', '2020-11-01T00:00:00.000000000',
       '2020-12-01T00:00:00.000000000', '2021-01-01T00:00:00.000000000',
       '2021-02-01T00:00:00.000000000', '2021-03-01T00:00:00.000000000',
       '2021-04-01T00:00:00.000000000', '2021-05-01T00:00:00.000000000',
       '2021-06-01T00:00:00.000000000', '2021-07-01T00:00:00.000000000',
       '2021-08-01T00:00:00.000000000', '2021-09-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
# Load in and concatenate all individual GeoTIFFs
modis = xr.concat([xr.open_mfdataset([i],) for i in fileset],
                        dim=time_var)
modis
<xarray.Dataset>
Dimensions:                                              (time: 258,
                                                          YDim:mod08: 180,
                                                          XDim:mod08: 360,
                                                          Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08: 7)
Coordinates:
  * time                                                 (time) datetime64[ns] ...
Dimensions without coordinates: YDim:mod08, XDim:mod08,
                                Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08
Data variables: (12/13)
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean           (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean     (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean        (time, Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 7, 180, 360), meta=np.ndarray>
    Cloud_Top_Temperature_Mean_Mean                      (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Effective_Emissivity_Mean_Mean                 (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Fraction_Mean_Mean                             (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    ...                                                   ...
    Cloud_Effective_Radius_Liquid_Mean_Mean              (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Water_Path_Liquid_Mean_Mean                    (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Water_Path_Ice_Mean_Mean                       (time, YDim:mod08, XDim:mod08) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Effective_Optical_Depth_Average_Ocean_Micron_Levels  (time, Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08) float32 dask.array<chunksize=(1, 7), meta=np.ndarray>
    XDim                                                 (time, XDim:mod08) float32 dask.array<chunksize=(1, 360), meta=np.ndarray>
    YDim                                                 (time, YDim:mod08) float32 dask.array<chunksize=(1, 180), meta=np.ndarray>
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Apply the same pre-processing as earlier#

modis = modis.rename_dims({'YDim:mod08': 'lat', 'XDim:mod08':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08':'levels'})
modis = modis.rename_vars({'YDim':'lat', 'XDim':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels': 'levels'})
modis = modis.assign_coords({"longitude": x, "latitude": y, 'levels': z})
modis
<xarray.Dataset>
Dimensions:                                           (time: 258, lat: 180,
                                                       lon: 360, levels: 7)
Coordinates:
  * levels                                            (levels) float32 0.47 ....
    lon                                               (time, lon) float32 dask.array<chunksize=(1, 360), meta=np.ndarray>
    lat                                               (time, lat) float32 dask.array<chunksize=(1, 180), meta=np.ndarray>
  * time                                              (time) datetime64[ns] 2...
    longitude                                         (lon) float32 -179.5 .....
    latitude                                          (lat) float32 89.5 ... ...
Data variables:
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean        (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean  (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean     (time, levels, lat, lon) float32 dask.array<chunksize=(1, 7, 180, 360), meta=np.ndarray>
    Cloud_Top_Temperature_Mean_Mean                   (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Effective_Emissivity_Mean_Mean              (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Fraction_Mean_Mean                          (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Optical_Thickness_Liquid_Mean_Mean          (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Effective_Radius_Liquid_Mean_Mean           (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Water_Path_Liquid_Mean_Mean                 (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    Cloud_Water_Path_Ice_Mean_Mean                    (time, lat, lon) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Tip

If you use one of xarray’s open methods such as xarray.open_dataset to load netCDF files with the default engine, it is recommended to use decode_coords=”all”. This will load the grid mapping variable into coordinates for compatibility with rioxarray. See rioxarray documentation.

Preparing and discover online datasets#

With the plethora of cloud storage, there are many available online datasets. To ease the preparation and discovery of such datasets, we describe emerging community-driven initiatives promoting standards suited to both geospatial and geoscience communities. Most of the material below is adapted from a previous Pangeo 101 training [GAF2)].

Tip

While we provide a general intro to some initiatives, we suggest below a list of FOSS4G 2022 talks with very interesting developments to prepare and discover spatio-temporal datasets in the cloud. Enjoy!

Analysis Ready, cloud optimized data (ARCO)#

When analyzing data at scale, the data format used is key. For years, the main data format was netCDF e.g. Network Common Data Form but with the use of cloud computing and interest in Open Science, different formats are often more suitable.

Formats for analyzing data from the cloud are refered to as “Analysis Ready, Cloud Optimized” data formats or in short ARCO. Find further info about ARCO datasets in [SAH+22].

What is “Analysis Ready”?

  • Think in terms of “Datasets” not “data files”

  • No need for tedious homogenizing / cleaning setup guides

  • Curated and cataloged

What is “Cloud Optimized”?

  • Compatible with object storage e.g. access via HTTP

  • Supports lazy access and intelligent subsetting

  • Integrates with high-level analysis libraries and distributed frameworks

Instead of having a big dataset, ARCO datasets are chunked appropriately for analysis and have rich metadata (See Figure 1).

Fig 1. Example of an ARCO dataset. Source: [GAF2)].

The Pangeo forge initiative#

Pangeo Forge is an open source platform for data Extraction, Transformation, and Loading (ETL). The goal of Pangeo Forge is to make it easy to extract data from traditional repositories and deposit this data in cloud object storage in an analysis-ready, cloud optimized (ARCO) format [GAF2)].

Pangeo Forge is inspired directly by Conda Forge, a community-led collection of recipes for building conda packages.

It is under active development and the Pangeo community hopes it will play a role in democratizing the publication of datasets in ARCO format.

How does Pangeo Forge work?#

Pangeo Forge defines the concept of a recipe, which specifies the logic for transforming a specific data archive into an ARCO data store. All contributions to Pangeo Forge must include an executable Python module, named recipe.py or similar, in which the data transformation logic is embedded (Figure 2). The recipe contributor is expected to use one of a predefined set of template algorithms defined by Pangeo Forge. Each of these templated algorithms is designed to transform data of a particular source type into a corresponding ARCO format, and requires only that the contributor populate the template with information unique to their specific data transformation, including the location of the source files and the way in which they should be aligned in the resulting ARCO data store [SAH+22].

The diagram below looks complicated but like for conda forge most of the process is automated.

Fig 2. A recipe in relation to Pangeo Forge architecture. Source: [SAH+22].

The next step after preparing the dataset is then to tell the community where and how to access to your transformed dataset.

This is done by creating a catalog.

Spatio Temporal Asset Catalogs (STAC)#

The STAC specification is a common language to describe geospatial information, so it can more easily be worked with, indexed, and discovered.

Why STAC?#

  • Each provider has its own catalog and interface (APIs).

  • Every time you want to access a new catalog, you need to change your program.

  • We have lots of data providers and each with a bespoke interface.

  • It is becoming quickly difficult for programmers who need to design a new data connector each time.

Features#

  • STAC catalogs are extremely simple.

  • They are composed of three layers:

    • Catalogs

      • Collections

        • Items

  • STAC is very popular for Earth Observation satellite imagery.

  • For instance it can be used to access Sentinel-2 in AWS (see Figure 3).

Fig 3. Example of STAC collection of Sentinel-2 images hosted in AWS. Source: [GAF2)].

STAC and Pangeo Forge#

  • Pangeo-forge supports the creation of analysis-ready cloud optimized (ARCO) data in cloud object storage from “classical” data repositories.

  • STAC is used to create catalogs and goes beyond the Pangeo ecosystem.

  • Work is ongoing to figure out the best way to expose Pangeo-Forge-generated data assets via STAC catalogs.

Tip

Pangeo members, Scott Henderson (University of Washington) and Tom Augspurger (Microsoft), provided a great workshop in FOSS4G 2021 covering STAC.

Feel free to explore the GitHub repository of the here.

Key Points
  • Access to remote dataset
  • ARCO datasets
  • Pangeo Forge
  • STAC

References#

GAF2)(1,2,3,4)

Basile Goussard, Ryan Abernathey, and Anne Fouilloux. The pangeo ecosystem. https://training.galaxyproject.org/training-material/topics/climate/tutorials/pangeo-notebook/slides.html#1, 2022 (accessed August 7, 2022).

SAH+22(1,2,3)

Charles Stern, Ryan Abernathey, Joseph Hamman, Rachel Wegener, Chiara Lepore, Sean Harkins, and Alexander Merose. Pangeo forge: crowdsourcing analysis-ready, cloud optimized data production. Frontiers in Climate, 2022. URL: https://www.frontiersin.org/articles/10.3389/fclim.2021.782909, doi:10.3389/fclim.2021.782909.

Packages citation#

HH17

S. Hoyer and J. Hamman. Xarray: N-D labeled arrays and datasets in Python. Journal of Open Research Software, 2017. URL: https://doi.org/10.5334/jors.148, doi:10.5334/jors.148.

S3FsDTeam16

S3Fs Development Team. S3Fs. 2016. URL: https://github.com/fsspec/s3fs/.