Notebook example on how to read-only with xarray files on the metno s3 store

key, secret key and endpoint can also be used to browse the store in jupyterlab bucket explorer

import s3fs
import xarray as xr
import s3fs
s3 = s3fs.S3FileSystem(key="K1CQ7M1DMTLUFK182APD", 
                       secret="3JuZAQm5I03jtpijCpHOdkAsJDNLNfZxBpM15Pi0", 
                       client_kwargs=dict(endpoint_url="https://rgw.met.no"))
s3.ls('escience2022')

['escience2022/Ada',
 'escience2022/Antoine',
 'escience2022/Dominic',
 'escience2022/ESA_SMOS_sss',
 'escience2022/Remy',
 'escience2022/Sara',
 'escience2022/Zhihong']

s3path = "s3://escience2022/Ada/OBS-ESACCI-OC/OBS_ESACCI-OC_sat_fv3.1_Omon_chl_199709-201712.nc"

esacci = xr.open_dataset(s3.open(s3path))
esacci

<xarray.Dataset>
Dimensions:    (time: 244, lat: 720, lon: 1440, bnds: 2)
Coordinates:
  * time       (time) datetime64[ns] 1997-09-04 1997-10-01 ... 2017-12-01
  * lat        (lat) float32 -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88
  * lon        (lon) float32 0.125 0.375 0.625 0.875 ... 359.1 359.4 359.6 359.9
    depth      float64 1.0
Dimensions without coordinates: bnds
Data variables:
    chl        (time, lat, lon) float32 ...
    time_bnds  (time, bnds) datetime64[ns] 1997-08-21T12:00:00 ... 2017-12-16
    lat_bnds   (lat, bnds) float64 -90.0 -89.75 -89.75 ... 89.75 89.75 90.0
    lon_bnds   (lon, bnds) float64 0.0 0.25 0.25 0.5 ... 359.5 359.8 359.8 360.0
Attributes: (12/13)
    comment:         Data binned using  6 by 6 cells average
    history:         Created on 2019-05-22 14:28:23
    host:            n020.cluster.net
    mip:             Omon
    modeling_realm:  sat
    project_id:      OBS
    ...              ...
    source:          ftp://oc-cci-data:ELaiWai8ae@oceancolour.org/occci-v3.1/...
    tier:            2
    title:           ESACCI-OC data reformatted for ESMValTool v2.0a1
    user:            ans033
    version:         fv3.1
    Conventions:     CF-1.5

xarray.Dataset

• Dimensions:
  • time: 244
  • lat: 720
  • lon: 1440
  • bnds: 2
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    1997-09-04 ... 2017-12-01

    axis :

    T

    bounds :

    time_bnds

    standard_name :

    time

    array(['1997-09-04T00:00:00.000000000', '1997-10-01T00:00:00.000000000',
           '1997-11-01T00:00:00.000000000', ..., '2017-10-01T00:00:00.000000000',
           '2017-11-01T00:00:00.000000000', '2017-12-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    -89.88 -89.62 ... 89.62 89.88

    axis :

    Y

    bounds :

    lat_bnds

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude coordinate

    array([-89.875, -89.625, -89.375, ...,  89.375,  89.625,  89.875],
          dtype=float32)

  • lon
    (lon)
    float32
    0.125 0.375 0.625 ... 359.6 359.9

    axis :

    X

    bounds :

    lon_bnds

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude coordinate

    array([1.25000e-01, 3.75000e-01, 6.25000e-01, ..., 3.59375e+02, 3.59625e+02,
           3.59875e+02], dtype=float32)

  • depth
    ()
    float64
    ...

    units :

    m

    standard_name :

    depth

    long_name :

    depth

    positive :

    down

    array(1.)

• Data variables: (4)
  • chl
    (time, lat, lon)
    float32
    ...

    standard_name :

    mass_concentration_of_phytoplankton_expressed_as_chlorophyll_in_sea_water

    long_name :

    Total Chlorophyll Mass Concentration at Surface

    units :

    kg m-3

    [252979200 values with dtype=float32]

  • time_bnds
    (time, bnds)
    datetime64[ns]
    ...

    array([['1997-08-21T12:00:00.000000000', '1997-09-17T12:00:00.000000000'],
           ['1997-09-17T12:00:00.000000000', '1997-10-16T12:00:00.000000000'],
           ['1997-10-16T12:00:00.000000000', '1997-11-16T00:00:00.000000000'],
           ...,
           ['2017-09-16T00:00:00.000000000', '2017-10-16T12:00:00.000000000'],
           ['2017-10-16T12:00:00.000000000', '2017-11-16T00:00:00.000000000'],
           ['2017-11-16T00:00:00.000000000', '2017-12-16T00:00:00.000000000']],
          dtype='datetime64[ns]')

  • lat_bnds
    (lat, bnds)
    float64
    ...

    array([[-90.  , -89.75],
           [-89.75, -89.5 ],
           [-89.5 , -89.25],
           ...,
           [ 89.25,  89.5 ],
           [ 89.5 ,  89.75],
           [ 89.75,  90.  ]])

  • lon_bnds
    (lon, bnds)
    float64
    ...

    array([[0.0000e+00, 2.5000e-01],
           [2.5000e-01, 5.0000e-01],
           [5.0000e-01, 7.5000e-01],
           ...,
           [3.5925e+02, 3.5950e+02],
           [3.5950e+02, 3.5975e+02],
           [3.5975e+02, 3.6000e+02]])

• Attributes: (13)

  comment :

  Data binned using 6 by 6 cells average

  history :

  Created on 2019-05-22 14:28:23

  host :

  n020.cluster.net

  mip :

  Omon

  modeling_realm :

  sat

  project_id :

  OBS

  reference :

  Volpe, G., et al., 2018. PRODUCT USER MANUAL. http://resources.marine.copernicus.eu/documents/PUM/CMEMS-OC-PUM-009-ALL.pdf

  source :

  ftp://oc-cci-data:ELaiWai8ae@oceancolour.org/occci-v3.1/geographic/netcdf/monthly/chlor_a/

  tier :

  2

  title :

  ESACCI-OC data reformatted for ESMValTool v2.0a1

  user :

  ans033

  version :

  fv3.1

  Conventions :

  CF-1.5