Using ADAM-API to access UERRA regional reanalysis#

you need to get an account to https://reliance.adamplatform.eu/ (use ORCID to authenticate) and key your ADAM API key
make sure you save your ADAM API key in a file $HOME/adam-key

!pip install adamapi

WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: adamapi in /opt/conda/lib/python3.8/site-packages (2.0.11)
Requirement already satisfied: requests>=2.22.0 in /opt/conda/lib/python3.8/site-packages (from adamapi) (2.25.1)
Requirement already satisfied: imageio in /opt/conda/lib/python3.8/site-packages (from adamapi) (2.9.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests>=2.22.0->adamapi) (1.26.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests>=2.22.0->adamapi) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests>=2.22.0->adamapi) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests>=2.22.0->adamapi) (2.10)
Requirement already satisfied: pillow in /opt/conda/lib/python3.8/site-packages (from imageio->adamapi) (8.1.2)
Requirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from imageio->adamapi) (1.20.1)

import os
import glob
import pathlib
import zipfile
import adamapi as adam
import xarray as xr
from datetime import datetime
import matplotlib
import cartopy.crs as ccrs
from cmcrameri import cm

adam_key = open(os.path.join(os.environ['HOME'],"adam-key")).read().rstrip()

a = adam.Auth()

a.setKey(adam_key)
a.setAdamCore('https://reliance.adamplatform.eu')
a.authorize()

{'expires_at': '2021-10-19T15:58:51.934Z',
 'access_token': 'cfe0ed4d1ce94333814478fc77c323b4',
 'refresh_token': '93f51e708ce4470092a426ed8c071523',
 'expires_in': 3600}

Discover UERRA datasets#

This step is useful to get the dataset identifier (unique for a given datacube)

def discoverDasasets(a, search_name):
    datasets = adam.Datasets(a)
    catalogue = datasets.getDatasets()
    #Extracting the size of the catalogue
    total = catalogue['properties']['totalResults']
    items = catalogue['properties']['itemsPerPage']
    pages = total//items
    
    print('----------------------------------------------------------------------')
    print('\033[1m' + 'List of available datasets:')
    print ('\033[0m')

    #Extracting the list of datasets across the whole catalogue
    for i in range(0,pages):
        page = datasets.getDatasets(page = i)
        for element in page['content']: 
            if search_name in element['title'] :
                print(element['title'] + "\033[1m" + " --> datasetId "+ "\033[0m" + "= " + element['datasetId'])
    return datasets

datasets = discoverDasasets(a, 'UERRA')

----------------------------------------------------------------------
List of available datasets:

UERRA_2M_TEMPERATURE --> datasetId = 71114:UERRA_2M_TEMPERATURE
UERRA_ALBEDO --> datasetId = 80984:UERRA_ALBEDO
UERRA_OROGRAPHY --> datasetId = 80985:UERRA_OROGRAPHY
UERRA_SKIN_TEMPERATURE --> datasetId = 80982:UERRA_SKIN_TEMPERATURE
UERRA_SNOW_DENSITY --> datasetId = 71100:UERRA_SNOW_DENSITY

Get metadata from Snow density#

datasetID = '71100:UERRA_SNOW_DENSITY'

print('\033[1;34m' + 'Metadata of ' + datasetID + ':')
print ('\033[0;0m')

paged = datasets.getDatasets(datasetID)
for i in paged.items():
    print("\033[1m" +  str(i[0]) + "\033[0m" + ': ' + str(i[1]))

Metadata of 71100:UERRA_SNOW_DENSITY:

datasetId: 71100:UERRA_SNOW_DENSITY
creationDate: 2018-07-31T00:00:00Z
dataType: Float32
epsg: 4326
keywords: []
license: {'documentationUrl': '', 'dataProviderName': 'ECMWF', 'dataProviderUrl': '', 'licenseId': '', 'dataPolicy': '', 'doi': '', 'credits': ''}
maxValue: [300.0]
minValue: [0.0]
noDataValue: -9999
numberOfRecords: 60251
profile: {'profileSchema': 'eo_profile_schema.json', 'name': 'Earth Observation', 'mission': 'UERRA', 'sensor': 'UERRA', 'processingLevel': 'L1', 'instrument': '', 'platform': ''}
resolutionUnit: degree
temporalResolution: Daily
unit: 
unitDescription: 
updateDate: 2021-08-31T09:29:26Z
geometry: {'type': 'Polygon', 'coordinates': [[[-179.9940193, -88.9158823], [180.0, -88.9158823], [180.0, 89.0841177], [-179.9940193, 89.0841177], [-179.9940193, -88.9158823]]]}
resolutions: [0.1]
anyText: 4326,ESA,Daily,ESA
applications: ['Atmosphere']
datasetManager: mantovani@meeo.it
datasetManagerOrganisation: meeo
description: UERRA_SNOW_DENSITY
title: UERRA_SNOW_DENSITY
timeReferenceSystem: UTC
units: kg m-3
unitsDescription: Density
services: ['MWCS', 'ADAM', 'AdamApi']
technicalManager: govoni@meeo.it
filtersEnabled: {'type': 'object', 'title': 'Dataset filters', 'properties': {'startDate': {'title': 'Start Date', 'type': 'string', 'format': 'date', 'text_rule': 'false', 'ops_only': 'false', 'math_rule': 'false', 'order_rule': 'false', 'pattern': '^d{4}-[01]d-[0-3]d(T[0-2]d:[0-9]d:[0-9]dZ)?$'}, 'endDate': {'title': 'End Date', 'type': 'string', 'format': 'date', 'ops_only': 'false', 'text_rule': 'false', 'math_rule': 'false', 'order_rule': 'false', 'pattern': '^d{4}-[01]d-[0-3]d(T[0-2]d:[0-9]d:[0-9]dZ)?$'}}}
starDate: 2014-08-01T00:00:00Z
endDate: 2019-07-31T18:00:00Z
accounting: {'unit': 0.0032}

Discover and select products from a dataset#

for a given time range and spatial coverage

Get data over the Nordics countries#

The geometry field is extracted from a GeoJSON object , retrieving the value of the “feature” element.

Search data#

only print the first 10 products
UERRA reanalysis are provided 4 times a day (00, 06, 12, 18 UTC)

!pip install geojson_rewind

WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: geojson_rewind in /opt/conda/lib/python3.8/site-packages (1.0.1)

from adamapi import Search
from geojson_rewind import rewind
import json

The GeoJson object needs to be rearranged according to the counterclockwise winding order.This operation is executed in the next few lines to obtain a geometry that meets the requirements of the method. Geom_1 is the final result to be used in the discovery operation.

you can go to https://geojson.io/ to draw an area of interest (save the produced geojson to a file)

with open('nordics.geojson') as f:
    geom_dict = json.load(f)
output = rewind(geom_dict)    
geom_1 = str(geom_dict['features'][0]['geometry'])

geom_1

"{'type': 'Polygon', 'coordinates': [[[2.021484375, 54.36775852406841], [42.626953125, 54.36775852406841], [42.626953125, 71.88357830131248], [2.021484375, 71.88357830131248], [2.021484375, 54.36775852406841]]]}"

start_date = '2015-01-01T00:00:00Z'
end_date = '2015-12-31T00:00:00Z'

search = Search( a )
results = search.getProducts(
    datasetID, 
    geometry= geom_1,
    startDate = start_date,
    endDate = end_date
 )

# Printing the results

print('\033[1m' + 'List of available products:')
print ('\033[0m')
count = 1
for i in results['content']:

        print("\033[1;31;1m" + "#" + str(count))
        print ('\033[0m')
        for k in i.items():
            print(str(k[0]) + ': ' + str(k[1]))
        count = count+1
        print('------------------------------------')

List of available products:

#1

_id: {'$oid': '612df2f0d57e43c6680b3b62'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-31T00:00:00Z
productId: UERRA_SNOW_DENSITY_20151231000000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:24Z
status: Online
------------------------------------
#2

_id: {'$oid': '612df2efd57e43c6680b3b16'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-30T18:00:00Z
productId: UERRA_SNOW_DENSITY_20151230180000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:23Z
status: Online
------------------------------------
#3

_id: {'$oid': '612df2efd57e43c6680b3b1a'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-30T12:00:00Z
productId: UERRA_SNOW_DENSITY_20151230120000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:23Z
status: Online
------------------------------------
#4

_id: {'$oid': '612df2efd57e43c6680b3b07'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-30T06:00:00Z
productId: UERRA_SNOW_DENSITY_20151230060000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:23Z
status: Online
------------------------------------
#5

_id: {'$oid': '612df2efd57e43c6680b3b03'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-30T00:00:00Z
productId: UERRA_SNOW_DENSITY_20151230000000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:23Z
status: Online
------------------------------------
#6

_id: {'$oid': '612df2eed57e43c6680b3ab5'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-29T18:00:00Z
productId: UERRA_SNOW_DENSITY_20151229180000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:22Z
status: Online
------------------------------------
#7

_id: {'$oid': '612df2eed57e43c6680b3ab8'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-29T12:00:00Z
productId: UERRA_SNOW_DENSITY_20151229120000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:22Z
status: Online
------------------------------------
#8

_id: {'$oid': '612df2eed57e43c6680b3aae'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-29T06:00:00Z
productId: UERRA_SNOW_DENSITY_20151229060000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:22Z
status: Online
------------------------------------
#9

_id: {'$oid': '612df2eed57e43c6680b3aa4'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-29T00:00:00Z
productId: UERRA_SNOW_DENSITY_20151229000000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:22Z
status: Online
------------------------------------
#10

_id: {'$oid': '612df2edd57e43c6680b3a07'}
datasetId: UERRA_SNOW_DENSITY
productDate: 2015-12-28T18:00:00Z
productId: UERRA_SNOW_DENSITY_20151228180000.tif
geometry: {'geometries': [{'type': 'Polygon', 'coordinates': [[[-0.0940193, -88.9158823], [178.9059807, -88.9158823], [178.9059807, 89.0841177], [-0.0940193, 89.0841177], [-0.0940193, -88.9158823]]]}], 'type': 'GeometryCollection'}
insertDate: 2021-08-31T09:14:21Z
status: Online
------------------------------------

Get data#

be aware that you alwasy get daily average from ADAM-API

def getZipData(auth, dataset_info):
    if not (pathlib.Path(pathlib.Path(dataset_info['outputFname']).stem).exists() or pathlib.Path(dataset_info['outputFname']).exists()):
        data = adam.GetData(auth)
        image = data.getData(
        datasetId = dataset_info['datasetID'],
        startDate = dataset_info['startDate'],
        endDate = dataset_info['endDate'],
        geometry = dataset_info['geometry'],
        outputFname = dataset_info['outputFname'])
        print(image)

%%time

output_file = './UERRA_SNOW_DENSITY_SWE_ADAMAPI_' + start_date + '-' + end_date + '.zip'

datasetInfo = {
    'datasetID'   : datasetID,
    'startDate'   : start_date,
    'endDate'     : end_date,
    'geometry'    : geom_1,
    'outputFname' : output_file
    }
getZipData(a, datasetInfo)

CPU times: user 148 µs, sys: 0 ns, total: 148 µs
Wall time: 154 µs

Data analysis and Visualization#

Unzip data#

def unzipData(filename):
    with zipfile.ZipFile(filename, 'r') as zip_ref:
        zip_ref.extractall(path = pathlib.Path(filename).stem)

if not pathlib.Path(pathlib.Path(output_file).stem).exists():
    unzipData(output_file)

Pangeo Tutorial at CLIVAR CMIP6 Bootcamp 2022

Using ADAM-API to access UERRA regional reanalysis

Contents