Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load into memory without saving to disk #25

Closed
lgloege opened this issue Oct 1, 2020 · 2 comments
Closed

Load into memory without saving to disk #25

lgloege opened this issue Oct 1, 2020 · 2 comments

Comments

@lgloege
Copy link

lgloege commented Oct 1, 2020

Is there any way to load the requested data into memory without saving to disk? I am trying to load the data as netcdf and do something like this:

import cdsapi

c = cdsapi.Client()

params =  {
    'product_type': 'reanalysis',
    'variable': 'total_precipitation',
    'year': '2019',
    'month': '01',
    'day': '01',
    'time': '00:00',
    'format': 'netcdf',
    'grid':[1.0, 1.0],
     } 

data = c.retrieve('reanalysis-era5-single-levels',  params)
ds = data.load() # doesn't exist, but could load data into memory

# process the data below here

I know .download() will save to disk. but I am curious if there is any way to implement something like a .load() method which would load the data into memory as an xarray dataset. This way one could load the data and process it in one script. The current solution is to download the data and just throw away the original dataset after processing, but I feel like this process could be streamlined.

I looked at the download method but couldn't figure out a way to implement some type of load method. Any thoughts?

Thanks for making this package. This has saved me so much time!

@lgloege
Copy link
Author

lgloege commented Mar 27, 2021

I figured out a solution using urlopen. Here is a small working example

import cdsapi
import xarray as xr
from urllib.request import urlopen

c = cdsapi.Client()

# API request
params = {'format': 'netcdf',
        'product_type': 'monthly_averaged_reanalysis',
        'variable': '2m_temperature',
        'year': list(map(str, range(2000, 2022))),
        'month': list(map("{:02d}".format, range(1,13))),
        'time': '00:00',
        'grid': [1.0, 1.0],}

# retrieve the location of the file
fl = c.retrieve('reanalysis-era5-single-levels-monthly-means', params) 

# load into memory 
with urlopen(fl.location) as f:
    ds = xr.open_dataset(f.read())

@EddyCMWF
Copy link
Collaborator

Hi,
Thanks @lgloege for providing a solution.
Eddy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants