Data access and discovery#

Authors & Contributors#

Authors#

  • Pier Lorenzo Marasco, Ispra (Italy), @pl-marasco

  • Alejandro Coca-Castro, The Alan Turing Institute (United Kingdom), @acocac

  • Anne Fouilloux, Simula Research Laboratory (Norway), @annefou

Contributors#

  • Tina Odaka, Ifremer (France), @tinaok

Overview

Questions
  • How to access online (remote) datasets?
  • How to prepare and discover online geoscience datasets?
  • What is Analysis Ready, Cloud Optimized data (ARCO)?
  • What is pangeo-forge?
  • What is stac?
Objectives
  • Learn to access datasets from online object storage
  • Learn about preparing and discovery online datasets
  • Learn about Analysis Cloud Optimized (ARCO) data
  • Learn about the Pangeo Forge initiative
  • Learn about stac

Context#

We will be using CMIP6 data and access them through S3-compatible storage using Pangeo Catalog.

We will also discuss the generation of Cloud-Optimised Datasets by introducing the pangeo-forge initiative. Finally, we will explore Sentinel-2 Cloud-Optimised Dataset online through SpatioTemporal Asset Catalogs (STAC).

Setup#

This episode uses the following main Python packages:

Please install these packages if not already available in your Python environment.

Packages#

In this episode, Python packages are imported when we start to use them. However, for best software practices, we recommend you to install and import all the necessary libraries at the top of your Jupyter notebook.

Introduction to CMIP6 data#

Coupled Model Intercomparison project Phase 6

  • Project under World Climate Research Programme (WCRP)

  • Since 1995 CMIP has coordinated climate model experiments

  • Defines common experiment protocols, forcings and output.

  • 33 model groups participate

Advantages:#

  • Homogenized and standardized outputs

  • Same variable name

  • Same experiments

Experiments (DECK)#

image

S3-compatible Object Storage to access online data#

Up to now we have downloaded data locally and then opened with Xarray open_dataset. When willing to manipulate large amount of data, this approach is not optimal (since it requires a lot of unnecessary local downloads). Sharing data online as Object Storage allows for data sharing and access to much larger amounts of data.

One of the most popular methods to access online remote data is through Amazon Simple Storage Service (S3) and you don’t necessarily need to use Amazon services to benefit from S3 object storage. Many other providers offer S3-compatible object storage that can be accessed in a very similar way.

When using S3-compatible object storage, you still need to list all the files you would like to access. With such amount of data, it would be very cumbersome. This is why a catalog is created: this catalog is a text file (json format) which describes where and which data to get. It adds additional metadata too. CMIP6 Pangeo online catalog can be loaded using intake-esm Python package.

Introduction to the Pangeo CMIP6 online catalog#

import xarray as xr
xr.set_options(display_style='html')
import intake
import cftime

Open CMIP6 online catalog#

cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(cat_url)
col

pangeo-cmip6 catalog with 7674 dataset(s) from 514818 asset(s):

unique
activity_id 18
institution_id 36
source_id 88
experiment_id 170
member_id 657
table_id 37
variable_id 700
grid_label 10
zstore 514818
dcpp_init_year 60
version 736

Search corresponding data#

cat = col.search(source_id=['CESM2'], experiment_id=['historical'], table_id=['Amon'], variable_id=['tas'], member_id=['r1i1p1f1'])
cat.df
activity_id institution_id source_id experiment_id member_id table_id variable_id grid_label zstore dcpp_init_year version
0 CMIP NCAR CESM2 historical r1i1p1f1 Amon tas gn gs://cmip6/CMIP6/CMIP/NCAR/CESM2/historical/r1... NaN 20190308

Create dictionary from the list of datasets we found#

  • This step may take several minutes so be patient!

dset_dict = cat.to_dataset_dict(zarr_kwargs={'use_cftime':True})
--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
100.00% [1/1 00:00<00:00]
dataset_list = list(dset_dict.keys())
dataset_list
['CMIP.NCAR.CESM2.historical.Amon.gn']

Open dataset#

  • Use xarray python package to analyze netCDF dataset

  • open_dataset allows to get all the metadata without loading data into memory.

  • with xarray, we only load into memory what is needed.

dset = dset_dict[dataset_list[0]]
dset
<xarray.Dataset>
Dimensions:    (lat: 192, nbnd: 2, lon: 288, member_id: 1, time: 1980)
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
    lat_bnds   (lat, nbnd) float32 dask.array<chunksize=(192, 2), meta=np.ndarray>
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
    lon_bnds   (lon, nbnd) float32 dask.array<chunksize=(288, 2), meta=np.ndarray>
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
    time_bnds  (time, nbnd) object dask.array<chunksize=(1980, 2), meta=np.ndarray>
  * member_id  (member_id) <U8 'r1i1p1f1'
Dimensions without coordinates: nbnd
Data variables:
    tas        (member_id, time, lat, lon) float32 dask.array<chunksize=(1, 600, 192, 288), meta=np.ndarray>
Attributes: (12/50)
    Conventions:             CF-1.7 CMIP-6.2
    activity_id:             CMIP
    branch_method:           standard
    branch_time_in_child:    674885.0
    branch_time_in_parent:   219000.0
    case_id:                 15
    ...                      ...
    variant_label:           r1i1p1f1
    status:                  2019-10-25;created;by nhn2@columbia.edu
    netcdf_tracking_ids:     hdl:21.14100/d9a7225a-49c3-4470-b7ab-a8180926f839
    version_id:              v20190308
    intake_esm_varname:      ['tas']
    intake_esm_dataset_key:  CMIP.NCAR.CESM2.historical.Amon.gn

Get metadata corresponding to near-surface air temperature (tas)#

print(dset['tas'])
<xarray.DataArray 'tas' (member_id: 1, time: 1980, lat: 192, lon: 288)>
dask.array<broadcast_to, shape=(1, 1980, 192, 288), dtype=float32, chunksize=(1, 600, 192, 288), chunktype=numpy.ndarray>
Coordinates:
  * lat        (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
  * lon        (lon) float64 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.2 357.5 358.8
  * time       (time) object 1850-01-15 12:00:00 ... 2014-12-15 12:00:00
  * member_id  (member_id) <U8 'r1i1p1f1'
Attributes:
    cell_measures:  area: areacella
    cell_methods:   area: time: mean
    comment:        near-surface (usually, 2 meter) air temperature
    description:    near-surface (usually, 2 meter) air temperature
    frequency:      mon
    id:             tas
    long_name:      Near-Surface Air Temperature
    mipTable:       Amon
    out_name:       tas
    prov:           Amon ((isd.003))
    realm:          atmos
    standard_name:  air_temperature
    time:           time
    time_label:     time-mean
    time_title:     Temporal mean
    title:          Near-Surface Air Temperature
    type:           real
    units:          K
    variable_id:    tas
dset.time.values
array([cftime.DatetimeNoLeap(1850, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 2, 14, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1850, 3, 15, 12, 0, 0, 0, has_year_zero=True),
       ...,
       cftime.DatetimeNoLeap(2014, 10, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 11, 15, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2014, 12, 15, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
dset.sel(lat=60, lon=10.75, method='nearest').sel(time='1850-12-15')['tas'].values
array([[269.032]], dtype=float32)

Warning

The same dataset can be available from different locations e.g. ESGF, Zenodo, S3-compatible Object storage, etc. How do you know if it corresponds to the very same dataset? You cannot know except if the datasets have a persistent identifier such as a Digital Object Identifier. It is therefore recommended 1) to be extra careful about where you get your datasets, and 2) to double check that the content is exactly what you expect (for instance, you can perform basic quality checks).

Access remote files from S3-compatible Object Storage#

The Pangeo CMIP6 online catalog is very user-friendly. However, the complete ESGF CMIP6 catalog is close to 20 PB and so far only a very tiny amount of cloud-optimised CMIP6 data has been generated (approximately 1PB). Therefore, you may have to download CMIP6 data or other kind of data for instance when comparing CMIP6 model outputs with observations.

Rather than downloading datasets locally and individually, it is recommended to share what you download to everyone. Additional datasets are being stored and made publicly available in OpenStack Object storage (Swift).

import s3fs
import xarray as xr
fs = s3fs.S3FileSystem(anon=True,
      client_kwargs={
         'endpoint_url': 'https://object-store.cloud.muni.cz'
      })

Tip

The parameter anon is for anonymous and is set to True because the data we have stored at https://object-store.cloud.muni.cz is public

List files and folders in existing buckets#

Instead of organizing files in various folders, object storage systems store files in a flat organization of containers (called “buckets”).

fs.ls('MODIS')
['MODIS/MOD08_M3.A2000032.061.2017276183309.nc',
 'MODIS/MOD08_M3.A2000061.061.2017272215822.nc',
 'MODIS/MOD08_M3.A2000092.061.2017276174940.nc',
 'MODIS/MOD08_M3.A2000122.061.2017275191641.nc',
 'MODIS/MOD08_M3.A2000153.061.2017276072839.nc',
 'MODIS/MOD08_M3.A2000183.061.2017276075622.nc',
 'MODIS/MOD08_M3.A2000214.061.2017276050502.nc',
 'MODIS/MOD08_M3.A2000245.061.2017276075932.nc',
 'MODIS/MOD08_M3.A2000275.061.2017276173030.nc',
 'MODIS/MOD08_M3.A2000306.061.2017276190346.nc',
 'MODIS/MOD08_M3.A2000336.061.2017276203640.nc',
 'MODIS/MOD08_M3.A2001001.061.2017277122349.nc',
 'MODIS/MOD08_M3.A2001032.061.2017277112301.nc',
 'MODIS/MOD08_M3.A2001060.061.2017277125646.nc',
 'MODIS/MOD08_M3.A2001091.061.2017277145429.nc',
 'MODIS/MOD08_M3.A2001121.061.2017277185847.nc',
 'MODIS/MOD08_M3.A2001152.061.2017277193228.nc',
 'MODIS/MOD08_M3.A2001182.061.2017277232025.nc',
 'MODIS/MOD08_M3.A2001213.061.2017278042002.nc',
 'MODIS/MOD08_M3.A2001244.061.2017278143507.nc',
 'MODIS/MOD08_M3.A2001274.061.2017278155329.nc',
 'MODIS/MOD08_M3.A2001305.061.2017278180906.nc',
 'MODIS/MOD08_M3.A2001335.061.2017278195919.nc',
 'MODIS/MOD08_M3.A2002001.061.2018116210818.nc',
 'MODIS/MOD08_M3.A2002032.061.2018086185625.nc',
 'MODIS/MOD08_M3.A2002060.061.2018116231409.nc',
 'MODIS/MOD08_M3.A2002091.061.2017279185034.nc',
 'MODIS/MOD08_M3.A2002121.061.2017279235924.nc',
 'MODIS/MOD08_M3.A2002152.061.2017280023717.nc',
 'MODIS/MOD08_M3.A2002182.061.2017280205715.nc',
 'MODIS/MOD08_M3.A2002213.061.2017280223026.nc',
 'MODIS/MOD08_M3.A2002244.061.2017280234438.nc',
 'MODIS/MOD08_M3.A2002274.061.2017281015551.nc',
 'MODIS/MOD08_M3.A2002305.061.2017281025115.nc',
 'MODIS/MOD08_M3.A2002335.061.2017281035150.nc',
 'MODIS/MOD08_M3.A2003001.061.2018117155743.nc',
 'MODIS/MOD08_M3.A2003032.061.2018116235452.nc',
 'MODIS/MOD08_M3.A2003060.061.2018117002319.nc',
 'MODIS/MOD08_M3.A2003091.061.2018117160200.nc',
 'MODIS/MOD08_M3.A2003121.061.2017283030440.nc',
 'MODIS/MOD08_M3.A2003152.061.2017283052308.nc',
 'MODIS/MOD08_M3.A2003182.061.2017283075355.nc',
 'MODIS/MOD08_M3.A2003213.061.2017283082726.nc',
 'MODIS/MOD08_M3.A2003244.061.2017283173012.nc',
 'MODIS/MOD08_M3.A2003274.061.2017283173119.nc',
 'MODIS/MOD08_M3.A2003305.061.2017283174124.nc',
 'MODIS/MOD08_M3.A2003335.061.2017283174833.nc',
 'MODIS/MOD08_M3.A2004001.061.2017284131754.nc',
 'MODIS/MOD08_M3.A2004032.061.2017284231722.nc',
 'MODIS/MOD08_M3.A2004061.061.2017285135054.nc',
 'MODIS/MOD08_M3.A2004092.061.2017285200445.nc',
 'MODIS/MOD08_M3.A2004122.061.2017286064100.nc',
 'MODIS/MOD08_M3.A2004153.061.2017286155932.nc',
 'MODIS/MOD08_M3.A2004183.061.2017287000057.nc',
 'MODIS/MOD08_M3.A2004214.061.2017287092018.nc',
 'MODIS/MOD08_M3.A2004245.061.2017288152313.nc',
 'MODIS/MOD08_M3.A2004275.061.2017288212405.nc',
 'MODIS/MOD08_M3.A2004306.061.2017288212156.nc',
 'MODIS/MOD08_M3.A2004336.061.2017289083810.nc',
 'MODIS/MOD08_M3.A2005001.061.2019072160440.nc',
 'MODIS/MOD08_M3.A2005032.061.2019072154406.nc',
 'MODIS/MOD08_M3.A2005060.061.2017291031953.nc',
 'MODIS/MOD08_M3.A2005091.061.2017290210757.nc',
 'MODIS/MOD08_M3.A2005121.061.2019072154947.nc',
 'MODIS/MOD08_M3.A2005152.061.2017298025833.nc',
 'MODIS/MOD08_M3.A2005182.061.2017292152251.nc',
 'MODIS/MOD08_M3.A2005213.061.2017293045950.nc',
 'MODIS/MOD08_M3.A2005244.061.2017293143200.nc',
 'MODIS/MOD08_M3.A2005274.061.2017293234404.nc',
 'MODIS/MOD08_M3.A2005305.061.2017294072541.nc',
 'MODIS/MOD08_M3.A2005335.061.2017296182027.nc',
 'MODIS/MOD08_M3.A2006001.061.2017272042922.nc',
 'MODIS/MOD08_M3.A2006032.061.2017272120606.nc',
 'MODIS/MOD08_M3.A2006060.061.2017272165438.nc',
 'MODIS/MOD08_M3.A2006091.061.2017272222236.nc',
 'MODIS/MOD08_M3.A2006121.061.2017276144021.nc',
 'MODIS/MOD08_M3.A2006152.061.2017277133958.nc',
 'MODIS/MOD08_M3.A2006182.061.2017275215927.nc',
 'MODIS/MOD08_M3.A2006213.061.2017279005924.nc',
 'MODIS/MOD08_M3.A2006244.061.2017277022732.nc',
 'MODIS/MOD08_M3.A2006274.061.2017278000639.nc',
 'MODIS/MOD08_M3.A2006305.061.2017278200246.nc',
 'MODIS/MOD08_M3.A2006335.061.2017279033150.nc',
 'MODIS/MOD08_M3.A2007001.061.2017282123406.nc',
 'MODIS/MOD08_M3.A2007032.061.2017282123335.nc',
 'MODIS/MOD08_M3.A2007060.061.2017282123411.nc',
 'MODIS/MOD08_M3.A2007091.061.2017282123415.nc',
 'MODIS/MOD08_M3.A2007121.061.2017283040324.nc',
 'MODIS/MOD08_M3.A2007152.061.2017283231630.nc',
 'MODIS/MOD08_M3.A2007182.061.2017285083949.nc',
 'MODIS/MOD08_M3.A2007213.061.2017285131028.nc',
 'MODIS/MOD08_M3.A2007244.061.2017285234236.nc',
 'MODIS/MOD08_M3.A2007274.061.2017286113952.nc',
 'MODIS/MOD08_M3.A2007305.061.2017287030845.nc',
 'MODIS/MOD08_M3.A2007335.061.2017287173138.nc',
 'MODIS/MOD08_M3.A2008001.061.2017288092622.nc',
 'MODIS/MOD08_M3.A2008032.061.2017289180803.nc',
 'MODIS/MOD08_M3.A2008061.061.2017291151011.nc',
 'MODIS/MOD08_M3.A2008092.061.2017291002057.nc',
 'MODIS/MOD08_M3.A2008122.061.2017291141050.nc',
 'MODIS/MOD08_M3.A2008153.061.2017292153358.nc',
 'MODIS/MOD08_M3.A2008183.061.2019072155257.nc',
 'MODIS/MOD08_M3.A2008214.061.2017293113821.nc',
 'MODIS/MOD08_M3.A2008245.061.2017294002156.nc',
 'MODIS/MOD08_M3.A2008275.061.2017297203335.nc',
 'MODIS/MOD08_M3.A2008306.061.2017295071904.nc',
 'MODIS/MOD08_M3.A2008336.061.2017296110255.nc',
 'MODIS/MOD08_M3.A2009001.061.2017298012528.nc',
 'MODIS/MOD08_M3.A2009032.061.2017298063546.nc',
 'MODIS/MOD08_M3.A2009060.061.2017298174606.nc',
 'MODIS/MOD08_M3.A2009091.061.2017299003246.nc',
 'MODIS/MOD08_M3.A2009121.061.2017299201650.nc',
 'MODIS/MOD08_M3.A2009152.061.2017303185329.nc',
 'MODIS/MOD08_M3.A2009182.061.2017304132125.nc',
 'MODIS/MOD08_M3.A2009213.061.2017319011248.nc',
 'MODIS/MOD08_M3.A2009244.061.2017306062357.nc',
 'MODIS/MOD08_M3.A2009274.061.2017307010542.nc',
 'MODIS/MOD08_M3.A2009305.061.2017307213730.nc',
 'MODIS/MOD08_M3.A2009335.061.2017308144544.nc',
 'MODIS/MOD08_M3.A2010001.061.2017310220725.nc',
 'MODIS/MOD08_M3.A2010032.061.2017311073823.nc',
 'MODIS/MOD08_M3.A2010060.061.2017311183851.nc',
 'MODIS/MOD08_M3.A2010091.061.2017312081103.nc',
 'MODIS/MOD08_M3.A2010121.061.2017312231425.nc',
 'MODIS/MOD08_M3.A2010152.061.2017313162026.nc',
 'MODIS/MOD08_M3.A2010182.061.2017314041948.nc',
 'MODIS/MOD08_M3.A2010213.061.2017318023643.nc',
 'MODIS/MOD08_M3.A2010244.061.2017318110207.nc',
 'MODIS/MOD08_M3.A2010274.061.2017318133158.nc',
 'MODIS/MOD08_M3.A2010305.061.2017318184613.nc',
 'MODIS/MOD08_M3.A2010335.061.2017320004241.nc',
 'MODIS/MOD08_M3.A2011001.061.2017320154950.nc',
 'MODIS/MOD08_M3.A2011032.061.2017321185442.nc',
 'MODIS/MOD08_M3.A2011060.061.2017322121213.nc',
 'MODIS/MOD08_M3.A2011091.061.2017323161255.nc',
 'MODIS/MOD08_M3.A2011121.061.2017324112918.nc',
 'MODIS/MOD08_M3.A2011152.061.2017325012251.nc',
 'MODIS/MOD08_M3.A2011182.061.2017325182247.nc',
 'MODIS/MOD08_M3.A2011213.061.2017326102442.nc',
 'MODIS/MOD08_M3.A2011244.061.2017327040751.nc',
 'MODIS/MOD08_M3.A2011274.061.2017328001250.nc',
 'MODIS/MOD08_M3.A2011305.061.2017328173108.nc',
 'MODIS/MOD08_M3.A2011335.061.2017330001115.nc',
 'MODIS/MOD08_M3.A2012001.061.2017332005900.nc',
 'MODIS/MOD08_M3.A2012032.061.2017332052720.nc',
 'MODIS/MOD08_M3.A2012061.061.2017332125551.nc',
 'MODIS/MOD08_M3.A2012092.061.2017333070458.nc',
 'MODIS/MOD08_M3.A2012122.061.2017334103546.nc',
 'MODIS/MOD08_M3.A2012153.061.2017335121954.nc',
 'MODIS/MOD08_M3.A2012183.061.2017336234843.nc',
 'MODIS/MOD08_M3.A2012214.061.2017338084514.nc',
 'MODIS/MOD08_M3.A2012245.061.2017339022214.nc',
 'MODIS/MOD08_M3.A2012275.061.2017339215337.nc',
 'MODIS/MOD08_M3.A2012306.061.2017340192241.nc',
 'MODIS/MOD08_M3.A2012336.061.2017341184807.nc',
 'MODIS/MOD08_M3.A2013001.061.2017298193816.nc',
 'MODIS/MOD08_M3.A2013032.061.2017297150003.nc',
 'MODIS/MOD08_M3.A2013060.061.2017298024454.nc',
 'MODIS/MOD08_M3.A2013091.061.2017298192533.nc',
 'MODIS/MOD08_M3.A2013121.061.2017299031432.nc',
 'MODIS/MOD08_M3.A2013152.061.2017299180328.nc',
 'MODIS/MOD08_M3.A2013182.061.2017300185420.nc',
 'MODIS/MOD08_M3.A2013213.061.2017304210739.nc',
 'MODIS/MOD08_M3.A2013244.061.2017304161740.nc',
 'MODIS/MOD08_M3.A2013274.061.2017305070652.nc',
 'MODIS/MOD08_M3.A2013305.061.2017306211731.nc',
 'MODIS/MOD08_M3.A2013335.061.2017308042052.nc',
 'MODIS/MOD08_M3.A2014001.061.2018116210846.nc',
 'MODIS/MOD08_M3.A2014032.061.2018116210744.nc',
 'MODIS/MOD08_M3.A2014060.061.2017310230130.nc',
 'MODIS/MOD08_M3.A2014091.061.2017311200138.nc',
 'MODIS/MOD08_M3.A2014121.061.2017312151500.nc',
 'MODIS/MOD08_M3.A2014152.061.2017313093158.nc',
 'MODIS/MOD08_M3.A2014182.061.2017314034734.nc',
 'MODIS/MOD08_M3.A2014213.061.2017314130249.nc',
 'MODIS/MOD08_M3.A2014244.061.2017317000542.nc',
 'MODIS/MOD08_M3.A2014274.061.2017317101857.nc',
 'MODIS/MOD08_M3.A2014305.061.2017317202015.nc',
 'MODIS/MOD08_M3.A2014335.061.2017318225533.nc',
 'MODIS/MOD08_M3.A2015001.061.2017319074723.nc',
 'MODIS/MOD08_M3.A2015032.061.2017319153036.nc',
 'MODIS/MOD08_M3.A2015060.061.2017320125759.nc',
 'MODIS/MOD08_M3.A2015091.061.2017320205832.nc',
 'MODIS/MOD08_M3.A2015121.061.2017321074517.nc',
 'MODIS/MOD08_M3.A2015152.061.2017321195647.nc',
 'MODIS/MOD08_M3.A2015182.061.2017322063945.nc',
 'MODIS/MOD08_M3.A2015213.061.2017322192741.nc',
 'MODIS/MOD08_M3.A2015244.061.2017323024455.nc',
 'MODIS/MOD08_M3.A2015274.061.2017323115746.nc',
 'MODIS/MOD08_M3.A2015305.061.2017323221600.nc',
 'MODIS/MOD08_M3.A2015335.061.2017324201044.nc',
 'MODIS/MOD08_M3.A2016001.061.2017325101827.nc',
 'MODIS/MOD08_M3.A2016032.061.2017325143021.nc',
 'MODIS/MOD08_M3.A2016061.061.2017326005020.nc',
 'MODIS/MOD08_M3.A2016092.061.2017326123542.nc',
 'MODIS/MOD08_M3.A2016122.061.2017326222614.nc',
 'MODIS/MOD08_M3.A2016153.061.2017327071005.nc',
 'MODIS/MOD08_M3.A2016183.061.2017327181105.nc',
 'MODIS/MOD08_M3.A2016214.061.2017328034102.nc',
 'MODIS/MOD08_M3.A2016245.061.2017328135749.nc',
 'MODIS/MOD08_M3.A2016275.061.2017353013021.nc',
 'MODIS/MOD08_M3.A2016306.061.2017329084922.nc',
 'MODIS/MOD08_M3.A2016336.061.2017329140013.nc',
 'MODIS/MOD08_M3.A2017001.061.2017316214128.nc',
 'MODIS/MOD08_M3.A2017032.061.2017313053532.nc',
 'MODIS/MOD08_M3.A2017060.061.2017314102914.nc',
 'MODIS/MOD08_M3.A2017091.061.2017314221045.nc',
 'MODIS/MOD08_M3.A2017121.061.2017315132821.nc',
 'MODIS/MOD08_M3.A2017152.061.2017315220820.nc',
 'MODIS/MOD08_M3.A2017182.061.2017317043341.nc',
 'MODIS/MOD08_M3.A2017213.061.2017318193955.nc',
 'MODIS/MOD08_M3.A2017244.061.2017285132541.nc',
 'MODIS/MOD08_M3.A2017274.061.2017305195215.nc',
 'MODIS/MOD08_M3.A2017305.061.2017335084629.nc',
 'MODIS/MOD08_M3.A2017335.061.2019072155425.nc',
 'MODIS/MOD08_M3.A2018001.061.2018032091034.nc',
 'MODIS/MOD08_M3.A2018032.061.2018064190408.nc',
 'MODIS/MOD08_M3.A2018060.061.2018093210347.nc',
 'MODIS/MOD08_M3.A2018091.061.2018121152525.nc',
 'MODIS/MOD08_M3.A2018121.061.2018152090257.nc',
 'MODIS/MOD08_M3.A2018152.061.2018182091458.nc',
 'MODIS/MOD08_M3.A2018182.061.2018213092526.nc',
 'MODIS/MOD08_M3.A2018213.061.2018247180628.nc',
 'MODIS/MOD08_M3.A2018244.061.2018278214421.nc',
 'MODIS/MOD08_M3.A2018274.061.2018312191357.nc',
 'MODIS/MOD08_M3.A2018305.061.2018335100840.nc',
 'MODIS/MOD08_M3.A2018335.061.2019072155558.nc',
 'MODIS/MOD08_M3.A2019001.061.2019038190346.nc',
 'MODIS/MOD08_M3.A2019032.061.2019063205813.nc',
 'MODIS/MOD08_M3.A2019060.061.2019091091201.nc',
 'MODIS/MOD08_M3.A2019091.061.2019121094111.nc',
 'MODIS/MOD08_M3.A2019121.061.2019152091235.nc',
 'MODIS/MOD08_M3.A2019213.061.2019247234417.nc',
 'MODIS/MOD08_M3.A2019244.061.2019275214642.nc',
 'MODIS/MOD08_M3.A2019274.061.2019305094830.nc',
 'MODIS/MOD08_M3.A2019305.061.2019336215510.nc',
 'MODIS/MOD08_M3.A2019335.061.2020002234053.nc',
 'MODIS/MOD08_M3.A2020001.061.2020032093438.nc',
 'MODIS/MOD08_M3.A2020032.061.2020063080211.nc',
 'MODIS/MOD08_M3.A2020061.061.2020092090542.nc',
 'MODIS/MOD08_M3.A2020092.061.2020125212701.nc',
 'MODIS/MOD08_M3.A2020122.061.2020153091858.nc',
 'MODIS/MOD08_M3.A2020153.061.2020183102654.nc',
 'MODIS/MOD08_M3.A2020183.061.2020214100214.nc',
 'MODIS/MOD08_M3.A2020214.061.2020245105303.nc',
 'MODIS/MOD08_M3.A2020245.061.2020275103907.nc',
 'MODIS/MOD08_M3.A2020275.061.2020306092007.nc',
 'MODIS/MOD08_M3.A2020306.061.2020336212521.nc',
 'MODIS/MOD08_M3.A2020336.061.2021001085852.nc',
 'MODIS/MOD08_M3.A2021001.061.2021032165629.nc',
 'MODIS/MOD08_M3.A2021032.061.2021060103941.nc',
 'MODIS/MOD08_M3.A2021060.061.2021092223659.nc',
 'MODIS/MOD08_M3.A2021091.061.2021122121127.nc',
 'MODIS/MOD08_M3.A2021121.061.2021152100135.nc',
 'MODIS/MOD08_M3.A2021152.061.2021183012452.nc',
 'MODIS/MOD08_M3.A2021182.061.2021213142147.nc',
 'MODIS/MOD08_M3.A2021213.061.2021244103656.nc',
 'MODIS/MOD08_M3.A2021244.061.2021276153158.nc']
fs.ls('MODIS/MOD08_M3.A2000183.061.2017276075622.nc')
['MODIS/MOD08_M3.A2000183.061.2017276075622.nc']

Access remote files from S3-compatible Object Storage#

s3path = 's3://MODIS/MOD08_M3.A2000183.061.2017276075622.nc'
modis = xr.open_dataset(fs.open(s3path))
modis
<xarray.Dataset>
Dimensions:                                              (YDim:mod08: 180,
                                                          XDim:mod08: 360,
                                                          Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08: 7)
Dimensions without coordinates: YDim:mod08, XDim:mod08,
                                Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08
Data variables: (12/13)
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean           (YDim:mod08, XDim:mod08) float32 ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean     (YDim:mod08, XDim:mod08) float32 ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean        (Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08, YDim:mod08, XDim:mod08) float32 ...
    Cloud_Top_Temperature_Mean_Mean                      (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Effective_Emissivity_Mean_Mean                 (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Fraction_Mean_Mean                             (YDim:mod08, XDim:mod08) float32 ...
    ...                                                   ...
    Cloud_Effective_Radius_Liquid_Mean_Mean              (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Water_Path_Liquid_Mean_Mean                    (YDim:mod08, XDim:mod08) float32 ...
    Cloud_Water_Path_Ice_Mean_Mean                       (YDim:mod08, XDim:mod08) float32 ...
    Effective_Optical_Depth_Average_Ocean_Micron_Levels  (Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08) float32 ...
    XDim                                                 (XDim:mod08) float32 ...
    YDim                                                 (YDim:mod08) float32 ...
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Rename dimensions and variables#

modis = modis.rename_dims({'YDim:mod08': 'lat', 'XDim:mod08':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels:mod08':'levels'})
modis = modis.rename_vars({'YDim':'lat', 'XDim':'lon', 'Effective_Optical_Depth_Average_Ocean_Micron_Levels': 'levels'})
modis
<xarray.Dataset>
Dimensions:                                           (lat: 180, lon: 360,
                                                       levels: 7)
Dimensions without coordinates: lat, lon, levels
Data variables: (12/13)
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean        (lat, lon) float32 nan ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean  (lat, lon) float32 nan ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean     (levels, lat, lon) float32 ...
    Cloud_Top_Temperature_Mean_Mean                   (lat, lon) float32 -1.4...
    Cloud_Effective_Emissivity_Mean_Mean              (lat, lon) float32 0.93...
    Cloud_Fraction_Mean_Mean                          (lat, lon) float32 0.88...
    ...                                                ...
    Cloud_Effective_Radius_Liquid_Mean_Mean           (lat, lon) float32 12.3...
    Cloud_Water_Path_Liquid_Mean_Mean                 (lat, lon) float32 146....
    Cloud_Water_Path_Ice_Mean_Mean                    (lat, lon) float32 249....
    levels                                            (levels) float32 0.47 ....
    lon                                               (lon) float32 -179.5 .....
    lat                                               (lat) float32 89.5 ... ...
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org

Coordinates#

x = modis.lon.squeeze().reset_coords(drop=True)
y = modis.lat.squeeze().reset_coords(drop=True)
z = modis.levels.squeeze().reset_coords(drop=True)
modis = modis.assign_coords({"lon": x, "lat": y, 'levels': z})
modis
<xarray.Dataset>
Dimensions:                                           (lat: 180, lon: 360,
                                                       levels: 7)
Coordinates:
  * levels                                            (levels) float32 0.47 ....
  * lon                                               (lon) float32 -179.5 .....
  * lat                                               (lat) float32 89.5 ... ...
Data variables:
    Aerosol_Optical_Depth_Land_Ocean_Mean_Mean        (lat, lon) float32 nan ...
    AOD_550_Dark_Target_Deep_Blue_Combined_Mean_Mean  (lat, lon) float32 nan ...
    Aerosol_Optical_Depth_Average_Ocean_Mean_Mean     (levels, lat, lon) float32 ...
    Cloud_Top_Temperature_Mean_Mean                   (lat, lon) float32 -1.4...
    Cloud_Effective_Emissivity_Mean_Mean              (lat, lon) float32 0.93...
    Cloud_Fraction_Mean_Mean                          (lat, lon) float32 0.88...
    Cloud_Optical_Thickness_Liquid_Mean_Mean          (lat, lon) float32 20.9...
    Cloud_Effective_Radius_Liquid_Mean_Mean           (lat, lon) float32 12.3...
    Cloud_Water_Path_Liquid_Mean_Mean                 (lat, lon) float32 146....
    Cloud_Water_Path_Ice_Mean_Mean                    (lat, lon) float32 249....
Attributes:
    HDFEOSVersion:                     HDFEOS_V2.19
    StructMetadata.0:                  GROUP=SwathStructure\nEND_GROUP=SwathS...
    StructMetadata.1:                  Min"\n\t\t\t\tDataType=DFNT_INT16\n\t\...
    StructMetadata.2:                  BJECT=DataField_295\n\t\t\tOBJECT=Data...
    StructMetadata.3:                  NT16\n\t\t\t\tDimList=("YDim","XDim")\...
    StructMetadata.4:                  \t\tDataFieldName="Cloud_Retrieval_Fra...
    StructMetadata.5:                  List=("YDim","XDim")\n\t\t\tEND_OBJECT...
    StructMetadata.6:                  CT=DataField_1037\n\t\t\t\tDataFieldNa...
    CoreMetadata.0:                    \nGROUP                  = INVENTORYME...
    ArchiveMetadata.0:                 \nGROUP                  = ARCHIVEDMET...
    identifier_product_doi:            10.5067/MODIS/MOD08_M3.061
    identifier_product_doi_authority:  http://dx.doi.org
modis.sel(lat=60, lon=10.75, method='nearest')['Aerosol_Optical_Depth_Land_Ocean_Mean_Mean'].values
array(0.134, dtype=float32)

Access multiple remote files#

s3path = 's3://MODIS/*.nc'
remote_files = fs.glob(s3path)
remote_files
['MODIS/MOD08_M3.A2000032.061.2017276183309.nc',
 'MODIS/MOD08_M3.A2000061.061.2017272215822.nc',
 'MODIS/MOD08_M3.A2000092.061.2017276174940.nc',
 'MODIS/MOD08_M3.A2000122.061.2017275191641.nc',
 'MODIS/MOD08_M3.A2000153.061.2017276072839.nc',
 'MODIS/MOD08_M3.A2000183.061.2017276075622.nc',
 'MODIS/MOD08_M3.A2000214.061.2017276050502.nc',
 'MODIS/MOD08_M3.A2000245.061.2017276075932.nc',
 'MODIS/MOD08_M3.A2000275.061.2017276173030.nc',
 'MODIS/MOD08_M3.A2000306.061.2017276190346.nc',
 'MODIS/MOD08_M3.A2000336.061.2017276203640.nc',
 'MODIS/MOD08_M3.A2001001.061.2017277122349.nc',
 'MODIS/MOD08_M3.A2001032.061.2017277112301.nc',
 'MODIS/MOD08_M3.A2001060.061.2017277125646.nc',
 'MODIS/MOD08_M3.A2001091.061.2017277145429.nc',
 'MODIS/MOD08_M3.A2001121.061.2017277185847.nc',
 'MODIS/MOD08_M3.A2001152.061.2017277193228.nc',
 'MODIS/MOD08_M3.A2001182.061.2017277232025.nc',
 'MODIS/MOD08_M3.A2001213.061.2017278042002.nc',
 'MODIS/MOD08_M3.A2001244.061.2017278143507.nc',
 'MODIS/MOD08_M3.A2001274.061.2017278155329.nc',
 'MODIS/MOD08_M3.A2001305.061.2017278180906.nc',
 'MODIS/MOD08_M3.A2001335.061.2017278195919.nc',
 'MODIS/MOD08_M3.A2002001.061.2018116210818.nc',
 'MODIS/MOD08_M3.A2002032.061.2018086185625.nc',
 'MODIS/MOD08_M3.A2002060.061.2018116231409.nc',
 'MODIS/MOD08_M3.A2002091.061.2017279185034.nc',
 'MODIS/MOD08_M3.A2002121.061.2017279235924.nc',
 'MODIS/MOD08_M3.A2002152.061.2017280023717.nc',
 'MODIS/MOD08_M3.A2002182.061.2017280205715.nc',
 'MODIS/MOD08_M3.A2002213.061.2017280223026.nc',
 'MODIS/MOD08_M3.A2002244.061.2017280234438.nc',
 'MODIS/MOD08_M3.A2002274.061.2017281015551.nc',
 'MODIS/MOD08_M3.A2002305.061.2017281025115.nc',
 'MODIS/MOD08_M3.A2002335.061.2017281035150.nc',
 'MODIS/MOD08_M3.A2003001.061.2018117155743.nc',
 'MODIS/MOD08_M3.A2003032.061.2018116235452.nc',
 'MODIS/MOD08_M3.A2003060.061.2018117002319.nc',
 'MODIS/MOD08_M3.A2003091.061.2018117160200.nc',
 'MODIS/MOD08_M3.A2003121.061.2017283030440.nc',
 'MODIS/MOD08_M3.A2003152.061.2017283052308.nc',
 'MODIS/MOD08_M3.A2003182.061.2017283075355.nc',
 'MODIS/MOD08_M3.A2003213.061.2017283082726.nc',
 'MODIS/MOD08_M3.A2003244.061.2017283173012.nc',
 'MODIS/MOD08_M3.A2003274.061.2017283173119.nc',
 'MODIS/MOD08_M3.A2003305.061.2017283174124.nc',
 'MODIS/MOD08_M3.A2003335.061.2017283174833.nc',
 'MODIS/MOD08_M3.A2004001.061.2017284131754.nc',
 'MODIS/MOD08_M3.A2004032.061.2017284231722.nc',
 'MODIS/MOD08_M3.A2004061.061.2017285135054.nc',
 'MODIS/MOD08_M3.A2004092.061.2017285200445.nc',
 'MODIS/MOD08_M3.A2004122.061.2017286064100.nc',
 'MODIS/MOD08_M3.A2004153.061.2017286155932.nc',
 'MODIS/MOD08_M3.A2004183.061.2017287000057.nc',
 'MODIS/MOD08_M3.A2004214.061.2017287092018.nc',
 'MODIS/MOD08_M3.A2004245.061.2017288152313.nc',
 'MODIS/MOD08_M3.A2004275.061.2017288212405.nc',
 'MODIS/MOD08_M3.A2004306.061.2017288212156.nc',
 'MODIS/MOD08_M3.A2004336.061.2017289083810.nc',
 'MODIS/MOD08_M3.A2005001.061.2019072160440.nc',
 'MODIS/MOD08_M3.A2005032.061.2019072154406.nc',
 'MODIS/MOD08_M3.A2005060.061.2017291031953.nc',
 'MODIS/MOD08_M3.A2005091.061.2017290210757.nc',
 'MODIS/MOD08_M3.A2005121.061.2019072154947.nc',
 'MODIS/MOD08_M3.A2005152.061.2017298025833.nc',
 'MODIS/MOD08_M3.A2005182.061.2017292152251.nc',
 'MODIS/MOD08_M3.A2005213.061.2017293045950.nc',
 'MODIS/MOD08_M3.A2005244.061.2017293143200.nc',
 'MODIS/MOD08_M3.A2005274.061.2017293234404.nc',
 'MODIS/MOD08_M3.A2005305.061.2017294072541.nc',
 'MODIS/MOD08_M3.A2005335.061.2017296182027.nc',
 'MODIS/MOD08_M3.A2006001.061.2017272042922.nc',
 'MODIS/MOD08_M3.A2006032.061.2017272120606.nc',
 'MODIS/MOD08_M3.A2006060.061.2017272165438.nc',
 'MODIS/MOD08_M3.A2006091.061.2017272222236.nc',
 'MODIS/MOD08_M3.A2006121.061.2017276144021.nc',
 'MODIS/MOD08_M3.A2006152.061.2017277133958.nc',
 'MODIS/MOD08_M3.A2006182.061.2017275215927.nc',
 'MODIS/MOD08_M3.A2006213.061.2017279005924.nc',
 'MODIS/MOD08_M3.A2006244.061.2017277022732.nc',
 'MODIS/MOD08_M3.A2006274.061.2017278000639.nc',
 'MODIS/MOD08_M3.A2006305.061.2017278200246.nc',
 'MODIS/MOD08_M3.A2006335.061.2017279033150.nc',
 'MODIS/MOD08_M3.A2007001.061.2017282123406.nc',
 'MODIS/MOD08_M3.A2007032.061.2017282123335.nc',
 'MODIS/MOD08_M3.A2007060.061.2017282123411.nc',
 'MODIS/MOD08_M3.A2007091.061.2017282123415.nc',
 'MODIS/MOD08_M3.A2007121.061.2017283040324.nc',
 'MODIS/MOD08_M3.A2007152.061.2017283231630.nc',
 'MODIS/MOD08_M3.A2007182.061.2017285083949.nc',
 'MODIS/MOD08_M3.A2007213.061.2017285131028.nc',
 'MODIS/MOD08_M3.A2007244.061.2017285234236.nc',
 'MODIS/MOD08_M3.A2007274.061.2017286113952.nc',
 'MODIS/MOD08_M3.A2007305.061.2017287030845.nc',
 'MODIS/MOD08_M3.A2007335.061.2017287173138.nc',
 'MODIS/MOD08_M3.A2008001.061.2017288092622.nc',
 'MODIS/MOD08_M3.A2008032.061.2017289180803.nc',
 'MODIS/MOD08_M3.A2008061.061.2017291151011.nc',
 'MODIS/MOD08_M3.A2008092.061.2017291002057.nc',
 'MODIS/MOD08_M3.A2008122.061.2017291141050.nc',
 'MODIS/MOD08_M3.A2008153.061.2017292153358.nc',
 'MODIS/MOD08_M3.A2008183.061.2019072155257.nc',
 'MODIS/MOD08_M3.A2008214.061.2017293113821.nc',
 'MODIS/MOD08_M3.A2008245.061.2017294002156.nc',
 'MODIS/MOD08_M3.A2008275.061.2017297203335.nc',
 'MODIS/MOD08_M3.A2008306.061.2017295071904.nc',
 'MODIS/MOD08_M3.A2008336.061.2017296110255.nc',
 'MODIS/MOD08_M3.A2009001.061.2017298012528.nc',
 'MODIS/MOD08_M3.A2009032.061.2017298063546.nc',
 'MODIS/MOD08_M3.A2009060.061.2017298174606.nc',
 'MODIS/MOD08_M3.A2009091.061.2017299003246.nc',
 'MODIS/MOD08_M3.A2009121.061.2017299201650.nc',
 'MODIS/MOD08_M3.A2009152.061.2017303185329.nc',
 'MODIS/MOD08_M3.A2009182.061.2017304132125.nc',
 'MODIS/MOD08_M3.A2009213.061.2017319011248.nc',
 'MODIS/MOD08_M3.A2009244.061.2017306062357.nc',
 'MODIS/MOD08_M3.A2009274.061.2017307010542.nc',
 'MODIS/MOD08_M3.A2009305.061.2017307213730.nc',
 'MODIS/MOD08_M3.A2009335.061.2017308144544.nc',
 'MODIS/MOD08_M3.A2010001.061.2017310220725.nc',
 'MODIS/MOD08_M3.A2010032.061.2017311073823.nc',
 'MODIS/MOD08_M3.A2010060.061.2017311183851.nc',
 'MODIS/MOD08_M3.A2010091.061.2017312081103.nc',
 'MODIS/MOD08_M3.A2010121.061.2017312231425.nc',
 'MODIS/MOD08_M3.A2010152.061.2017313162026.nc',
 'MODIS/MOD08_M3.A2010182.061.2017314041948.nc',
 'MODIS/MOD08_M3.A2010213.061.2017318023643.nc',
 'MODIS/MOD08_M3.A2010244.061.2017318110207.nc',
 'MODIS/MOD08_M3.A2010274.061.2017318133158.nc',
 'MODIS/MOD08_M3.A2010305.061.2017318184613.nc',
 'MODIS/MOD08_M3.A2010335.061.2017320004241.nc',
 'MODIS/MOD08_M3.A2011001.061.2017320154950.nc',
 'MODIS/MOD08_M3.A2011032.061.2017321185442.nc',
 'MODIS/MOD08_M3.A2011060.061.2017322121213.nc',
 'MODIS/MOD08_M3.A2011091.061.2017323161255.nc',
 'MODIS/MOD08_M3.A2011121.061.2017324112918.nc',
 'MODIS/MOD08_M3.A2011152.061.2017325012251.nc',
 'MODIS/MOD08_M3.A2011182.061.2017325182247.nc',
 'MODIS/MOD08_M3.A2011213.061.2017326102442.nc',
 'MODIS/MOD08_M3.A2011244.061.2017327040751.nc',
 'MODIS/MOD08_M3.A2011274.061.2017328001250.nc',
 'MODIS/MOD08_M3.A2011305.061.2017328173108.nc',
 'MODIS/MOD08_M3.A2011335.061.2017330001115.nc',
 'MODIS/MOD08_M3.A2012001.061.2017332005900.nc',
 'MODIS/MOD08_M3.A2012032.061.2017332052720.nc',
 'MODIS/MOD08_M3.A2012061.061.2017332125551.nc',
 'MODIS/MOD08_M3.A2012092.061.2017333070458.nc',
 'MODIS/MOD08_M3.A2012122.061.2017334103546.nc',
 'MODIS/MOD08_M3.A2012153.061.2017335121954.nc',
 'MODIS/MOD08_M3.A2012183.061.2017336234843.nc',
 'MODIS/MOD08_M3.A2012214.061.2017338084514.nc',
 'MODIS/MOD08_M3.A2012245.061.2017339022214.nc',
 'MODIS/MOD08_M3.A2012275.061.2017339215337.nc',
 'MODIS/MOD08_M3.A2012306.061.2017340192241.nc',
 'MODIS/MOD08_M3.A2012336.061.2017341184807.nc',
 'MODIS/MOD08_M3.A2013001.061.2017298193816.nc',
 'MODIS/MOD08_M3.A2013032.061.2017297150003.nc',
 'MODIS/MOD08_M3.A2013060.061.2017298024454.nc',
 'MODIS/MOD08_M3.A2013091.061.2017298192533.nc',
 'MODIS/MOD08_M3.A2013121.061.2017299031432.nc',
 'MODIS/MOD08_M3.A2013152.061.2017299180328.nc',
 'MODIS/MOD08_M3.A2013182.061.2017300185420.nc',
 'MODIS/MOD08_M3.A2013213.061.2017304210739.nc',
 'MODIS/MOD08_M3.A2013244.061.2017304161740.nc',
 'MODIS/MOD08_M3.A2013274.061.2017305070652.nc',
 'MODIS/MOD08_M3.A2013305.061.2017306211731.nc',
 'MODIS/MOD08_M3.A2013335.061.2017308042052.nc',
 'MODIS/MOD08_M3.A2014001.061.2018116210846.nc',
 'MODIS/MOD08_M3.A2014032.061.2018116210744.nc',
 'MODIS/MOD08_M3.A2014060.061.2017310230130.nc',
 'MODIS/MOD08_M3.A2014091.061.2017311200138.nc',
 'MODIS/MOD08_M3.A2014121.061.2017312151500.nc',
 'MODIS/MOD08_M3.A2014152.061.2017313093158.nc',
 'MODIS/MOD08_M3.A2014182.061.2017314034734.nc',
 'MODIS/MOD08_M3.A2014213.061.2017314130249.nc',
 'MODIS/MOD08_M3.A2014244.061.2017317000542.nc',
 'MODIS/MOD08_M3.A2014274.061.2017317101857.nc',
 'MODIS/MOD08_M3.A2014305.061.2017317202015.nc',
 'MODIS/MOD08_M3.A2014335.061.2017318225533.nc',
 'MODIS/MOD08_M3.A2015001.061.2017319074723.nc',
 'MODIS/MOD08_M3.A2015032.061.2017319153036.nc',
 'MODIS/MOD08_M3.A2015060.061.2017320125759.nc',
 'MODIS/MOD08_M3.A2015091.061.2017320205832.nc',
 'MODIS/MOD08_M3.A2015121.061.2017321074517.nc',
 'MODIS/MOD08_M3.A2015152.061.2017321195647.nc',
 'MODIS/MOD08_M3.A2015182.061.2017322063945.nc',
 'MODIS/MOD08_M3.A2015213.061.2017322192741.nc',
 'MODIS/MOD08_M3.A2015244.061.2017323024455.nc',
 'MODIS/MOD08_M3.A2015274.061.2017323115746.nc',
 'MODIS/MOD08_M3.A2015305.061.2017323221600.nc',
 'MODIS/MOD08_M3.A2015335.061.2017324201044.nc',
 'MODIS/MOD08_M3.A2016001.061.2017325101827.nc',
 'MODIS/MOD08_M3.A2016032.061.2017325143021.nc',
 'MODIS/MOD08_M3.A2016061.061.2017326005020.nc',
 'MODIS/MOD08_M3.A2016092.061.2017326123542.nc',
 'MODIS/MOD08_M3.A2016122.061.2017326222614.nc',
 'MODIS/MOD08_M3.A2016153.061.2017327071005.nc',
 'MODIS/MOD08_M3.A2016183.061.2017327181105.nc',
 'MODIS/MOD08_M3.A2016214.061.2017328034102.nc',
 'MODIS/MOD08_M3.A2016245.061.2017328135749.nc',
 'MODIS/MOD08_M3.A2016275.061.2017353013021.nc',
 'MODIS/MOD08_M3.A2016306.061.2017329084922.nc',
 'MODIS/MOD08_M3.A2016336.061.2017329140013.nc',
 'MODIS/MOD08_M3.A2017001.061.2017316214128.nc',
 'MODIS/MOD08_M3.A2017032.061.2017313053532.nc',
 'MODIS/MOD08_M3.A2017060.061.2017314102914.nc',
 'MODIS/MOD08_M3.A2017091.061.2017314221045.nc',
 'MODIS/MOD08_M3.A2017121.061.2017315132821.nc',
 'MODIS/MOD08_M3.A2017152.061.2017315220820.nc',
 'MODIS/MOD08_M3.A2017182.061.2017317043341.nc',
 'MODIS/MOD08_M3.A2017213.061.2017318193955.nc',
 'MODIS/MOD08_M3.A2017244.061.2017285132541.nc',
 'MODIS/MOD08_M3.A2017274.061.2017305195215.nc',
 'MODIS/MOD08_M3.A2017305.061.2017335084629.nc',
 'MODIS/MOD08_M3.A2017335.061.2019072155425.nc',
 'MODIS/MOD08_M3.A2018001.061.2018032091034.nc',
 'MODIS/MOD08_M3.A2018032.061.2018064190408.nc',
 'MODIS/MOD08_M3.A2018060.061.2018093210347.nc',
 'MODIS/MOD08_M3.A2018091.061.2018121152525.nc',
 'MODIS/MOD08_M3.A2018121.061.2018152090257.nc',
 'MODIS/MOD08_M3.A2018152.061.2018182091458.nc',
 'MODIS/MOD08_M3.A2018182.061.2018213092526.nc',
 'MODIS/MOD08_M3.A2018213.061.2018247180628.nc',
 'MODIS/MOD08_M3.A2018244.061.2018278214421.nc',
 'MODIS/MOD08_M3.A2018274.061.2018312191357.nc',
 'MODIS/MOD08_M3.A2018305.061.2018335100840.nc',
 'MODIS/MOD08_M3.A2018335.061.2019072155558.nc',
 'MODIS/MOD08_M3.A2019001.061.2019038190346.nc',
 'MODIS/MOD08_M3.A2019032.061.2019063205813.nc',
 'MODIS/MOD08_M3.A2019060.061.2019091091201.nc',
 'MODIS/MOD08_M3.A2019091.061.2019121094111.nc',
 'MODIS/MOD08_M3.A2019121.061.2019152091235.nc',
 'MODIS/MOD08_M3.A2019213.061.2019247234417.nc',
 'MODIS/MOD08_M3.A2019244.061.2019275214642.nc',
 'MODIS/MOD08_M3.A2019274.061.2019305094830.nc',
 'MODIS/MOD08_M3.A2019305.061.2019336215510.nc',
 'MODIS/MOD08_M3.A2019335.061.2020002234053.nc',
 'MODIS/MOD08_M3.A2020001.061.2020032093438.nc',
 'MODIS/MOD08_M3.A2020032.061.2020063080211.nc',
 'MODIS/MOD08_M3.A2020061.061.2020092090542.nc',
 'MODIS/MOD08_M3.A2020092.061.2020125212701.nc',
 'MODIS/MOD08_M3.A2020122.061.2020153091858.nc',
 'MODIS/MOD08_M3.A2020153.061.2020183102654.nc',
 'MODIS/MOD08_M3.A2020183.061.2020214100214.nc',
 'MODIS/MOD08_M3.A2020214.061.2020245105303.nc',
 'MODIS/MOD08_M3.A2020245.061.2020275103907.nc',
 'MODIS/MOD08_M3.A2020275.061.2020306092007.nc',
 'MODIS/MOD08_M3.A2020306.061.2020336212521.nc',
 'MODIS/MOD08_M3.A2020336.061.2021001085852.nc',
 'MODIS/MOD08_M3.A2021001.061.2021032165629.nc',
 'MODIS/MOD08_M3.A2021032.061.2021060103941.nc',
 'MODIS/MOD08_M3.A2021060.061.2021092223659.nc',
 'MODIS/MOD08_M3.A2021091.061.2021122121127.nc',
 'MODIS/MOD08_M3.A2021121.061.2021152100135.nc',
 'MODIS/MOD08_M3.A2021152.061.2021183012452.nc',
 'MODIS/MOD08_M3.A2021182.061.2021213142147.nc',
 'MODIS/MOD08_M3.A2021213.061.2021244103656.nc',
 'MODIS/MOD08_M3.A2021244.061.2021276153158.nc']

We need to add a time dimension to concatenate data. For this, we define a function that will be called for each remote file (via the preprocess parameter of Xarray open_mfdataset.)

from datetime import datetime
def paths_to_datetimeindex(paths):
    return  [datetime.strptime(date.split('.A')[-1].split('.')[0], '%Y%j') for date in paths]

Xarray open_mfdataset allows opening multiple files at the same time.

# Iterate through remote_files to create a fileset
fileset = [fs.open(file) for file in remote_files]

When opening remote files, you can also select the variables you wish to analyze.

# Create variable used for time axis
time_var = xr.Variable('time', paths_to_datetimeindex(remote_files))
time_var
<xarray.Variable (time: 258)>
array(['2000-02-01T00:00:00.000000000', '2000-03-01T00:00:00.000000000',
       '2000-04-01T00:00:00.000000000', '2000-05-01T00:00:00.000000000',
       '2000-06-01T00:00:00.000000000', '2000-07-01T00:00:00.000000000',
       '2000-08-01T00:00:00.000000000', '2000-09-01T00:00:00.000000000',
       '2000-10-01T00:00:00.000000000', '2000-11-01T00:00:00.000000000',
       '2000-12-01T00:00:00.000000000', '2001-01-01T00:00:00.000000000',
       '2001-02-01T00:00:00.000000000', '2001-03-01T00:00:00.000000000',
       '2001-04-01T00:00:00.000000000', '2001-05-01T00:00:00.000000000',
       '2001-06-01T00:00:00.000000000', '2001-07-01T00:00:00.000000000',
       '2001-08-01T00:00:00.000000000', '2001-09-01T00:00:00.000000000',
       '2001-10-01T00:00:00.000000000', '2001-11-01T00:00:00.000000000',
       '2001-12-01T00:00:00.000000000', '2002-01-01T00:00:00.000000000',
       '2002-02-01T00:00:00.000000000', '2002-03-01T00:00:00.000000000',
       '2002-04-01T00:00:00.000000000', '2002-05-01T00:00:00.000000000',
       '2002-06-01T00:00:00.000000000', '2002-07-01T00:00:00.000000000',
       '2002-08-01T00:00:00.000000000', '2002-09-01T00:00:00.000000000',
       '2002-10-01T00:00:00.000000000', '2002-11-01T00:00:00.000000000',
       '2002-12-01T00:00:00.000000000', '2003-01-01T00:00:00.000000000',
       '2003-02-01T00:00:00.000000000', '2003-03-01T00:00:00.000000000',
       '2003-04-01T00:00:00.000000000', '2003-05-01T00:00:00.000000000',
...
       '2018-06-01T00:00:00.000000000', '2018-07-01T00:00:00.000000000',
       '2018-08-01T00:00:00.000000000', '2018-09-01T00:00:00.000000000',
       '2018-10-01T00:00:00.000000000', '2018-11-01T00:00:00.000000000',
       '2018-12-01T00:00:00.000000000', '2019-01-01T00:00:00.000000000',
       '2019-02-01T00:00:00.000000000', '2019-03-01T00:00:00.000000000',
       '2019-04-01T00:00:00.000000000', '2019-05-01T00:00:00.000000000',
       '2019-08-01T00:00:00.000000000', '2019-09-01T00:00:00.000000000',
       '2019-10-01T00:00:00.000000000', '2019-11-01T00:00:00.000000000',
       '2019-12-01T00:00:00.000000000', '2020-01-01T00:00:00.000000000',
       '2020-02-01T00:00:00.000000000', '2020-03-01T00:00:00.000000000',
       '2020-04-01T00:00:00.000000000', '2020-05-01T00:00:00.000000000',
       '2020-06-01T00:00:00.000000000', '2020-07-01T00:00:00.000000000',
       '2020-08-01T00:00:00.000000000', '2020-09-01T00:00:00.000000000',
       '2020-10-01T00:00:00.000000000', '2020-11-01T00:00:00.000000000',
       '2020-12-01T00:00:00.000000000', '2021-01-01T00:00:00.000000000',
       '2021-02-01T00:00:00.000000000', '2021-03-01T00:00:00.000000000',
       '2021-04-01T00:00:00.000000000', '2021-05-01T00:00:00.000000000',
       '2021-06-01T00:00:00.000000000', '2021-07-01T00:00:00.000000000',
       '2021-08-01T00:00:00.000000000', '2021-09-01T00:00:00.000000000'],
      dtype='datetime64[ns]')