Save zarr files to EOSC (CESNET) using s3fs

• This example show you how to save your zarr file to object storage https://object-store.cloud.muni.cz

import s3fs
import xarray as xr
import zarr

Get a sample file

ds = xr.tutorial.open_dataset("air_temperature.nc").rename({"air": "Tair"})

Save your results to Remote object storage

• If not done, create your credentials by follwoing this link

• Verify your credentials in $HOME/.aws/credentials It should look like

[default]
aws_access_key_id=xxxxx
aws_secret_access_key=yyyy
aws_endpoint_url=https://object-store.cloud.muni.cz

It is important to save your results in 'your' bucket. [The credential created here ](../EOSC_to_bucket.md) is a common space for pangeo-eosc cloud users. You need to not to 'over write' data on other users

Define your s3 storage parameters

your_name='yourname'
path='escience/'+your_name
s3_prefix =  "s3://"+path
print(s3_prefix)
access_key = !aws configure get aws_access_key_id
access_key = access_key[0]
secret_key = !aws configure get aws_secret_access_key
secret_key = secret_key[0]
client_kwargs={'endpoint_url': 'https://object-store.cloud.muni.cz'}

s3://escience/yourname

set your zarr store (with S3Map)

zarr_file_name= 'S3Map_zarr'
s3 = s3fs.S3FileSystem(client_kwargs=client_kwargs,key=access_key, secret=secret_key)
uri = f"{s3_prefix}/{zarr_file_name}"
store_s3= s3fs.S3Map(root=uri,s3=s3,check=False)

Write your file down to zarr

%time ds.to_zarr(store=store_s3, mode='w', consolidated=True)

/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/dataset.py:2060: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs
  return to_zarr(  # type: ignore

CPU times: user 714 ms, sys: 70.3 ms, total: 784 ms
Wall time: 50 s

<xarray.backends.zarr.ZarrStore at 0x7ff66f37ce40>

Verify that your ‘zarr’ file is at your storage

target = s3fs.S3FileSystem(anon=False,client_kwargs=client_kwargs)
target.ls(path)#, detail=True, refresh=True)

['escience/yourname/S3Map_zarr']

Open your zarr storage

xr.open_zarr(store_s3)

<xarray.Dataset>
Dimensions:  (time: 2920, lat: 25, lon: 53)
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Data variables:
    Tair     (time, lat, lon) float32 dask.array<chunksize=(730, 13, 27), meta=np.ndarray>
Attributes:
    Conventions:  COARDS
    description:  Data is from NMC initialized reanalysis\n(4x/day).  These a...
    platform:     Model
    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...
    title:        4x daily NMC reanalysis (1948)

xarray.Dataset

• Dimensions:
  • time: 2920
  • lat: 25
  • lon: 53
• Coordinates: (3)
  • lat
    (lat)
    float32
    75.0 72.5 70.0 ... 20.0 17.5 15.0

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([75. , 72.5, 70. , 67.5, 65. , 62.5, 60. , 57.5, 55. , 52.5, 50. , 47.5,
           45. , 42.5, 40. , 37.5, 35. , 32.5, 30. , 27.5, 25. , 22.5, 20. , 17.5,
           15. ], dtype=float32)

  • lon
    (lon)
    float32
    200.0 202.5 205.0 ... 327.5 330.0

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. ], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2013-01-01 ... 2014-12-31T18:00:00

    long_name :

    Time

    standard_name :

    time

    array(['2013-01-01T00:00:00.000000000', '2013-01-01T06:00:00.000000000',
           '2013-01-01T12:00:00.000000000', ..., '2014-12-31T06:00:00.000000000',
           '2014-12-31T12:00:00.000000000', '2014-12-31T18:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • Tair
    (time, lat, lon)
    float32
    dask.array<chunksize=(730, 13, 27), meta=np.ndarray>

    GRIB_id :

    11

    GRIB_name :

    TMP

    actual_range :

    [185.16000366210938, 322.1000061035156]

    dataset :

    NMC Reanalysis

    level_desc :

    Surface

    long_name :

    4xDaily Air temperature at sigma level 995

    parent_stat :

    Other

    precision :

    2

    statistic :

    Individual Obs

    units :

    degK

    var_desc :

    Air temperature

    Array Chunk Bytes 14.76 MiB 0.98 MiB Shape (2920, 25, 53) (730, 13, 27) Count 2 Graph Layers 16 Chunks Type float32 numpy.ndarray

• Attributes: (5)

  Conventions :

  COARDS

  description :

  Data is from NMC initialized reanalysis (4x/day). These are the 0.9950 sigma level values.

  platform :

  Model

  references :

  http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html

  title :

  4x daily NMC reanalysis (1948)

Delete the zarr file you created

target.ls(path)

['escience/yourname/S3Map_zarr']

target.rm(path+'/S3Map_zarr', recursive=True)

target.ls('escience/')

['escience/jeani',
 'escience/michaels',
 'escience/mschulz',
 'escience/tinaok',
 'escience/yourname-here']

one users can delete another user's file.... So please be carefull especially when using recursive=True