Skip to content

Commit

Permalink
geocode SLC integration test (#46)
Browse files Browse the repository at this point in the history
* download test data from zenodo in parallel
* run integration test in circleci
* adding pytest as requirement and to docker specifile
* updated specfile
* account for multiburst, new burst ID, geocode entire burst
* use main instead of tag because of missing features
* clone from https not ssh
* template yaml for test run
* validation to check for presense of reflectors

Co-authored-by: Scott Staniewicz <[email protected]>
  • Loading branch information
LiangJYu and scottstanie authored Mar 6, 2023
1 parent 31e6959 commit 8e0545e
Show file tree
Hide file tree
Showing 6 changed files with 303 additions and 31 deletions.
10 changes: 9 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -151,4 +151,12 @@ cython_debug/
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/


# unit test files/directories
tests/data/geo_cslc_s1.yaml
tests/data/2022-10-16_0000_Rosamond-corner-reflectors.csv
tests/data/S1A_IW_SLC__1SDV_20221016T015043_20221016T015111_045461_056FC0_6681.zip
tests/data/orbits
tests/data/test_dem.tiff
tests/data/test_burst_map.sqlite3
tests/product
tests/scratch
56 changes: 31 additions & 25 deletions docker/specifile.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab
https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/font-ttf-source-code-pro-2.038-h77eed37_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/font-ttf-ubuntu-0.83-hab24e00_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.39-hcc3a1bd_1.conda
https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-h41732ed_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libgfortran5-12.2.0-h337968e_19.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-12.2.0-h46fd767_19.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/poppler-data-0.4.11-hd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/poppler-data-0.4.12-hd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.11-3_cp311.conda
https://conda.anaconda.org/conda-forge/noarch/tzdata-2022g-h191b570_0.conda
https://conda.anaconda.org/conda-forge/noarch/fonts-conda-forge-1-0.tar.bz2
Expand All @@ -29,7 +29,7 @@ https://conda.anaconda.org/conda-forge/linux-64/geos-3.11.1-h27087fc_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gettext-0.21.1-h27087fc_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/giflib-5.2.1-h36c2ea0_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/icu-70.1-h27087fc_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/jpeg-9e-h166bdaf_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/jpeg-9e-h0b41bf4_3.conda
https://conda.anaconda.org/conda-forge/linux-64/json-c-0.16-hc379101_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/lerc-4.0.0-h27087fc_0.tar.bz2
Expand All @@ -43,10 +43,10 @@ https://conda.anaconda.org/conda-forge/linux-64/libopenblas-0.3.21-pthreads_h78a
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libwebp-base-1.2.4-h166bdaf_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.13-h166bdaf_4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.9.3-h9c3ff4c_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.9.4-hcb278e6_0.conda
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h27087fc_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/nspr-4.35-h27087fc_0.conda
https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.7-h0b41bf4_1.conda
https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.8-h0b41bf4_0.conda
https://conda.anaconda.org/conda-forge/linux-64/pixman-0.40.0-h36c2ea0_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h36c2ea0_1001.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/snappy-1.1.9-hbd366e4_2.tar.bz2
Expand All @@ -56,7 +56,7 @@ https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.0.10-h7f98852_0.ta
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxau-1.0.9-h7f98852_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.3-h7f98852_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-renderproto-0.11.1-h7f98852_1002.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-xextproto-7.3.0-h7f98852_1002.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-xextproto-7.3.0-h0b41bf4_1003.conda
https://conda.anaconda.org/conda-forge/linux-64/xorg-xproto-7.0.31-h7f98852_1007.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/yaml-0.2.5-h7f98852_2.tar.bz2
Expand All @@ -75,7 +75,7 @@ https://conda.anaconda.org/conda-forge/linux-64/readline-8.1.2-h0f457ee_0.tar.bz
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.3-hd9c2040_1000.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.13-h166bdaf_4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.2-h3eb15da_5.conda
https://conda.anaconda.org/conda-forge/linux-64/zstd-1.5.2-h3eb15da_6.conda
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.21.3-hafa529b_0.conda
https://conda.anaconda.org/conda-forge/linux-64/boost-cpp-1.78.0-h75c5d50_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/freetype-2.12.1-hca18f0e_1.conda
Expand All @@ -85,7 +85,8 @@ https://conda.anaconda.org/conda-forge/linux-64/libcblas-3.9.0-16_linux64_openbl
https://conda.anaconda.org/conda-forge/linux-64/libglib-2.74.1-h606061b_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/liblapack-3.9.0-16_linux64_openblas.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.5.0-h6adf6a1_2.conda
https://conda.anaconda.org/conda-forge/linux-64/nss-3.82-he02c5a1_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libxslt-1.1.37-h873f0b0_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/nss-3.88-he45b914_0.conda
https://conda.anaconda.org/conda-forge/linux-64/python-3.11.0-he550d4f_1_cpython.conda
https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.40.0-h4ff8645_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.7.2-h7f98852_0.tar.bz2
Expand All @@ -97,56 +98,61 @@ https://conda.anaconda.org/conda-forge/noarch/certifi-2022.12.7-pyhd8ed1ab_0.con
https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-2.1.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/exceptiongroup-1.1.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.14.1-hc2a2eb6_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.14.2-h14ed4e7_0.conda
https://conda.anaconda.org/conda-forge/noarch/idna-3.4-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/lcms2-2.14-hfd0df8a_1.conda
https://conda.anaconda.org/conda-forge/linux-64/libcurl-7.87.0-hdc1c0ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libcurl-7.88.1-hdc1c0ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libkml-1.3.0-h37653c0_1015.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libpq-15.1-hb675445_3.conda
https://conda.anaconda.org/conda-forge/linux-64/numpy-1.24.1-py311hbde0eaa_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libpq-15.2-hb675445_0.conda
https://conda.anaconda.org/conda-forge/linux-64/lxml-4.9.2-py311h14a6109_0.conda
https://conda.anaconda.org/conda-forge/linux-64/numpy-1.24.2-py311h8e6699e_0.conda
https://conda.anaconda.org/conda-forge/linux-64/openjpeg-2.5.0-hfec8fc6_2.conda
https://conda.anaconda.org/conda-forge/noarch/packaging-23.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pluggy-1.0.0-pyhd8ed1ab_5.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pycparser-2.21-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pysocks-1.7.1-pyha2e5f31_6.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pytz-2022.7.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/pyyaml-6.0-py311hd4cff14_5.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ruamel.yaml.clib-0.2.7-py311h2582759_1.conda
https://conda.anaconda.org/conda-forge/noarch/setuptools-66.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/setuptools-67.4.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tomli-2.0.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wheel-0.38.4-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.4-h7f98852_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.4-h0b41bf4_2.conda
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.10-h7f98852_1003.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/cached-property-1.5.2-hd8ed1ab_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/cairo-1.16.0-ha61ee94_1014.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/cffi-1.15.1-py311h409f033_3.conda
https://conda.anaconda.org/conda-forge/linux-64/cfitsio-4.2.0-hd9d235c_0.conda
https://conda.anaconda.org/conda-forge/linux-64/curl-7.87.0-hdc1c0ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/curl-7.88.1-hdc1c0ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.12.2-nompi_h4df4325_101.conda
https://conda.anaconda.org/conda-forge/noarch/pip-22.3.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/postgresql-15.1-h3248436_3.conda
https://conda.anaconda.org/conda-forge/noarch/pip-23.0.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/postgresql-15.2-h3248436_0.conda
https://conda.anaconda.org/conda-forge/linux-64/proj-9.1.1-h8ffa02c_2.conda
https://conda.anaconda.org/conda-forge/noarch/pytest-7.2.1-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.8.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ruamel.yaml-0.17.21-py311hd4cff14_2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/shapely-2.0.0-py311h0f577a2_0.conda
https://conda.anaconda.org/conda-forge/linux-64/shapely-2.0.1-py311h0f577a2_0.conda
https://conda.anaconda.org/conda-forge/linux-64/xerces-c-3.2.4-h55805fa_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/yamale-4.0.4-pyh6c4a22f_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/brotlipy-0.7.0-py311hd4cff14_1005.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/cryptography-39.0.0-py311h9b4c7bb_0.conda
https://conda.anaconda.org/conda-forge/linux-64/cryptography-39.0.1-py311h9b4c7bb_0.conda
https://conda.anaconda.org/conda-forge/linux-64/geotiff-1.7.1-h7a142b4_6.conda
https://conda.anaconda.org/conda-forge/linux-64/h5py-3.7.0-nompi_py311hbe7f6d8_102.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/h5py-3.8.0-nompi_py311h1db17ec_100.conda
https://conda.anaconda.org/conda-forge/linux-64/kealib-1.5.0-ha7026e8_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libnetcdf-4.8.1-nompi_h261ec11_106.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/libnetcdf-4.9.1-nompi_h34a3ff0_100.conda
https://conda.anaconda.org/conda-forge/linux-64/libspatialite-5.0.1-h221c8f1_23.conda
https://conda.anaconda.org/conda-forge/linux-64/pandas-1.5.3-py311h2872171_0.conda
https://conda.anaconda.org/conda-forge/linux-64/poppler-23.01.0-h091648b_0.conda
https://conda.anaconda.org/conda-forge/linux-64/pyproj-3.4.1-py311h945b3ca_1.conda
https://conda.anaconda.org/conda-forge/linux-64/pyre-1.11.2-py311hbbb8f27_3.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/tiledb-2.13.2-hd532e3d_0.conda
https://conda.anaconda.org/conda-forge/linux-64/libgdal-3.6.2-h8c90c07_5.conda
https://conda.anaconda.org/conda-forge/linux-64/libgdal-3.6.2-hea5766e_7.conda
https://conda.anaconda.org/conda-forge/noarch/pyopenssl-23.0.0-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/linux-64/gdal-3.6.2-py311hadb6153_5.conda
https://conda.anaconda.org/conda-forge/linux-64/gdal-3.6.2-py311hadb6153_7.conda
https://conda.anaconda.org/conda-forge/noarch/urllib3-1.26.14-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/requests-2.28.2-pyhd8ed1ab_0.conda
https://conda.anaconda.org/conda-forge/noarch/pooch-1.6.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.10.0-py311h8e6699e_0.conda
https://conda.anaconda.org/conda-forge/linux-64/isce3-0.9.0-py311h0802494_0.conda
https://conda.anaconda.org/conda-forge/linux-64/scipy-1.10.0-py311h8e6699e_2.conda
https://conda.anaconda.org/conda-forge/linux-64/isce3-0.10.0-py311h0802494_0.conda
5 changes: 4 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
python>=3.9 # sentinel1-reader requirement
numpy # sentinel1-reader requirement
lxml # sentinel1-reader requirement
gdal>=3
#isce3 # since the conda-installed isce3 is not the most updated version, installing isce3 from stratch is recommended, to stay in sync with isce3 development.
#journal # as of Mar 2022, journal from conda does not support python3.9; since it is included during isce3 installation above, comment this out temporarily.
lxml
pandas
pyproj
pytest
ruamel.yaml
scipy
yamale
h5py
shapely
requests
100 changes: 96 additions & 4 deletions tests/conftest.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,102 @@
import multiprocessing as mp
import os
import pytest
import pathlib
import types

import pytest
import requests
from s1reader.s1_orbit import check_internet_connection

from compass.utils import iono


def download_if_needed(local_path):
'''
Check if given path to file exists. Download if it from zenodo does not.
Parameters
----------
local_path: str
Path to file
'''
# return if file is found
if os.path.isfile(local_path):
return

check_internet_connection()

dst_dir, file_name = os.path.split(local_path)

# create destination directory if it does not exist
if dst_dir:
os.makedirs(dst_dir, exist_ok=True)

# download data
dataset_url = 'https://zenodo.org/record/7668411/files/'
target_url = f'{dataset_url}/{file_name}'
with open(local_path, 'wb') as f:
f.write(requests.get(target_url).content)


@pytest.fixture(scope="session")
def geocode_slc_params():
'''
Parameters to be used by geocode SLC unit test
Returns
-------
test_params: SimpleNamespace
SimpleNamespace containing geocode SLC unit test parameters
'''
test_params = types.SimpleNamespace()

# burst ID and date of burst
burst_id = 't064_135523_iw2'
burst_date = '20221016'

# get test working directory
test_path = pathlib.Path(__file__).parent.resolve()

# set other paths relative to working directory
test_data_path = f'{test_path}/data'

# paths for template and actual runconfig
gslc_template_path = f'{test_data_path}/geo_cslc_s1_template.yaml'
test_params.gslc_cfg_path = f'{test_data_path}/geo_cslc_s1.yaml'

# read runconfig template, replace pieces, write to runconfig
with open(gslc_template_path, 'r') as f_template, \
open(test_params.gslc_cfg_path, 'w') as f_cfg:
cfg = f_template.read().replace('@TEST_PATH@', str(test_path)).\
replace('@DATA_PATH@', test_data_path).\
replace('@BURST_ID@', burst_id)
f_cfg.write(cfg)

# files needed for geocode SLC unit test
test_files = ['S1A_IW_SLC__1SDV_20221016T015043_20221016T015111_045461_056FC0_6681.zip',
'orbits/S1A_OPER_AUX_POEORB_OPOD_20221105T083813_V20221015T225942_20221017T005942.EOF',
'test_dem.tiff', 'test_burst_map.sqlite3',
'2022-10-16_0000_Rosamond-corner-reflectors.csv']
test_files = [f'{test_data_path}/{test_file}' for test_file in test_files]

# parallel download of test files (if necessary)
with mp.Pool(len(test_files)) as pool:
pool.map(download_if_needed, test_files)

# path to file containing corner reflectors
test_params.corner_coord_csv_path = test_files[-1]

# path the output HDF5
output_path = f'{test_path}/product/{burst_id}/{burst_date}'
output_file_name = f'{burst_id}_{burst_date}.h5'
test_params.output_hdf5_path = f'{output_path}/{output_file_name}'

# path to groups and datasets in output HDF5
test_params.grid_group_path = '/science/SENTINEL1/CSLC/grids'
test_params.raster_path = f'{test_params.grid_group_path}/VV'

return test_params

@pytest.fixture(scope='session')
def ionex_params(download_data=True):
'''
Expand All @@ -17,9 +110,8 @@ def ionex_params(download_data=True):
Returns
-------
tec_file: str
Path to local or downloaded TEC file to
use in the unit test
test_params : SimpleNamespace
SimpleNamespace containing parameters needed for ionex unit test
'''
test_params = types.SimpleNamespace()

Expand Down
43 changes: 43 additions & 0 deletions tests/data/geo_cslc_s1_template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
runconfig:
groups:
dynamic_ancillary_file_group:
dem_file: @DATA_PATH@/test_dem.tiff
static_ancillary_file_group:
burst_database_file: @DATA_PATH@/test_burst_map.sqlite3
input_file_group:
burst_id:
- @BURST_ID@
orbit_file_path:
- @DATA_PATH@/orbits/S1A_OPER_AUX_POEORB_OPOD_20221105T083813_V20221015T225942_20221017T005942.EOF
safe_file_path:
- @DATA_PATH@/S1A_IW_SLC__1SDV_20221016T015043_20221016T015111_045461_056FC0_6681.zip
pge_name_group:
pge_name: CSLC_S1_PGE
primary_executable:
product_type: CSLC_S1
processing:
rdr2geo:
enabled: false
geo2rdr:
lines_per_block: 1000
numiter: 25
threshold: 1.0e-08
geocoding:
flatten: true
lines_per_block: 1000
x_posting: 5
x_snap: null
y_posting: 10
y_snap: null
polarization: co-pol
range_split_spectrum:
enabled: False
product_path_group:
product_path: @TEST_PATH@/product
sas_output_file: @TEST_PATH@/product
scratch_path: @TEST_PATH@/scratch
worker:
gpu_enabled: false
gpu_id: 0
internet_access: false
name: cslc_s1_workflow_default
Loading

0 comments on commit 8e0545e

Please sign in to comment.