-
Notifications
You must be signed in to change notification settings - Fork 0
WorldPeatland Workflow
To generate WorldPeatland BioGeoPhysical Vegetation parameters, from now on referred to as BGPV params, you can follow the steps below. The BGPV params are based on two different data sources with different spatial resolution and for a different time period. BGVP params generated from MODIS and/or from Sentinel data.
There is a template configuration file WorldPeatland/code/template_config.yml which governs which data to fetch and process for all sites. To add a new site we use the template config to generate a site specific config.
By default we process a time period of 10 years from 2013 to 2023. You may wish to edit start_date and end_date in the template config. Do this before running any downloaders. Do not edit the generated site specific config.
This step will download all the MODIS data needed to generate the coarse resolution BGPV params. Downloaders could take a couple of days to run, thus it is best to run it in the background (with nohup or screen) to avoid breaking the workflow.
Navigate to your code directory and run
python -m WorldPeatland.code.download_modis <site-geojson>where site-geojson is the path to the geojson file which defines your site of interest.
The downloader will create a site-specific data directory inside the ROOT_DATA_DIR that you define in your settings.py file.
You will need to use this site-specific root data directory in later steps.
After running, your ROOT_DATA_DIR should have the following structure
├── <site>
│ ├── MODIS
│ │ ├── <product>
│ │ │ └── <tile>
│ │ │ ├── <symlink>.hdf
│ │ │ └── ...
│ │ └── ...
│ └── <site>_config.yml
└── raw_tiles
└── MODIS
├── <product>
│ └── <tile>
│ ├── <data>.hdf
│ └── ...
└── ...
The file <site>_config.yml is the site specific config, generated from the template config.
We store the modis data in raw_tiles directory so that it can be shared across multiple sites.
This step will create the time series for all the datasets downloaded:
python -m WorldPeatland.code.ts_generator_wp <path-to-site-data> <site-geojson>where <path-to-site-data> is the path ROOT_DATA_DIR/<site>
This will extract each variable from the MODIS data products and build a datacube (virtual raster dataset), adding directories to ROOT_DATA_DIR:
├── <site>
│ ├── MODIS
│ │ ├── <product>
│ │ │ └── <tile>
│ │ │ ├── <variable>
│ │ │ │ ├── <variable>.vrt
│ │ │ │ ├── <first>.tif
│ │ │ │ ├── <second>.tif
│ │ │ │ ├── ...
For a first run you can use the default QA settings. But if you think a lot of data is missing or the default QA setting is too strict then you can refer to how to change QA settings in the git wiki.
python -m WorldPeatland.code.apply_qa <path-to-site-data> <path-to-qa-settings>where <path-to-qa-settings> is a directory, and might be WorldPeatland/QA_settings to use the default QA settings.
This step will add interpolated/ directories to ROOT_DATA_DIR:
├── <site>
│ ├── MODIS
│ │ ├── <product>
│ │ │ └── <tile>
│ │ │ └── <variable>
│ │ │ └── interpolated
│ │ │ └── <single-file>.tif
python -m WorldPeatland.code.smoothing <path-to-site-data>This step will add an additional file to the interpolated/ directories.
├── <site>
│ ├── MODIS
│ │ ├── <product>
│ │ │ └── <tile>
│ │ │ └── <variable>
│ │ │ └── interpolated
│ │ │ ├── <file>.tif
│ │ │ └── <file>.<smooth>.tif
This script will descale the data, with the second argument removing seasonal trends.
python -m WorldPeatland.code.pixel_ts <path-to-site-data> False This step will add an additional file to the interpolated/ directories.
├── <site>
│ ├── MODIS
│ │ ├── <product>
│ │ │ └── <tile>
│ │ │ └── <variable>
│ │ │ └── interpolated
│ │ │ ├── <file>.tif
│ │ │ ├── <file>.<smooth>.tif
│ │ │ └── <file>.<smooth>.<descaled>.tif
The final outputs are placed in ROOT_DATA_DIR/<site>/MODIS/timeSeries/ for convenience.
And now you should have a geotiff file for each data products for BGPV params cleaned and descaled in the form of a timeseries over the time period indicated in the config file.
- Google Earth Engine
- Creodias