Updated validation READMEs

validation/README.md

@@ -11,13 +11,13 @@ Validation datasets will be made available on cloudstor once they are finalised.
     2. Build database for the events in the `DER_disturbance_analysis/validation/data` directory using `build_validation_databases.R`
     3. Run tool using metadata from the event directory as config json
     4. Batch save results in the same directory as the data, entering `ref` as the file name
-    5. (optional) (work in progress) - Instead of steps 3 and 4, use `run_validation_analysis.R` to generate the files.
+    5. (optional) - Instead of steps 3 and 4, use `run_validation_analysis.R` to generate the files.
 2. Run analysis on sample data with version of tool to be tested
     1. Checkout target branch
     2. Build database for the events in the `DER_disturbance_analysis/validation/data` directory using `build_validation_databases.R`
     3. Run tool using metadata from the event directory as config json. __IMPORTANT__ - make sure any changes in configuration in the target branch (new fields/values etc.) are reflected in the test meta data.
     4. Batch save results in the same directory as the data, entering `test` as the file name
-    5. (optional) (work in progress) - Instead of steps 3 and 4, use `run_validation_analysis.R` to generate the files.
+    5. (optional) - Instead of steps 3 and 4, use `run_validation_analysis.R` to generate the files.
 3. Compare reference and test results using `validate_results.R`
     1. Identify if results match
     2. Check any discrepencies against expected impact of test version of tool
@@ -30,9 +30,15 @@ Pass positional arguments of test and reference branch e.g.
 ./full_validation.sh branch_to_validate master
 ```
 
+## Common issues / FAQ
+
+-  `Error: Result must have length 36, not 0` or `Error in read.table(...) first five rows are empty: giving up` when running validation
+    - This is likely caused by new or modified tool settings. Ensure that any new settings are reflected in the `test_meta_data.json` file for each event. Once this validation is complete ensure that `ref_meta_data.json` files are updated and uploaded to cloudstor for future use.
+
+
 ## Choosing validation datasets
 
-A representative set of validation data is required. In order to capture this we are currently planning to use the data from 2021-05-25 from QLD, 2020-01-31 from SA, and at least one event using Tesla data (TBD)
+A representative set of validation data is required. In order to capture this we are currently uising the data from 2021-05-25 from QLD, 2020-01-31 from SA, and one event using Tesla data (2021-03-12)
 
 For each event the following process will need to be followed:
 

validation/data/.gitignore

@@ -2,3 +2,4 @@
 *
 */
 !.gitignore
+!README.md

validation/data/README.md

@@ -0,0 +1,7 @@
+# Validation datasets
+
+For description of how to use, see the README in DER_disturbace_analysis/validation.
+
+These data sets are currently for project MATCH/AEMO usage only. They are derived from datasets acquired under Projec MATCH NDAs with the relevant sources (Solar Analytics/Tesla).
+
+If we want to use these elsewhere in future or make them publicly available we will need to negotiate that with the relevant parties.