Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate support for multiple runs #278

Open
gctucker opened this issue Jun 9, 2023 · 0 comments
Open

Investigate support for multiple runs #278

gctucker opened this issue Jun 9, 2023 · 0 comments
Labels
roadmap API & Pipeline Roadmap

Comments

@gctucker
Copy link
Contributor

gctucker commented Jun 9, 2023

Some tests may need to be run several times with exactly the same combination of parameters to gather additional results and produce statistics. This is particularly useful for tests results that aren't stable or to get an average with performance runs. The paths and all the meta-data will be the same for each run, except some fields that will be used to distinguish results that are coming from the same run.

For example, we could have an "iteration" field for this which would be automatically incremented every time an identical data set is being sent.

Here's a typical scenario with first the data being sent to the API:

{
  "name": "foobar",
  "path": ["parent", "foobar"],
  "result": "pass",
  "data": {
    "score": "1234"
  },
}
{
  "name": "foobar",
  "path": ["parent", "foobar"],
  "result": "pass",
  "data": {
    "score": "1432"
  },
}

Then the data stored in the database would add "iteration": 1 to the first entry and "iteration": 2 to the second one. One issue with this is that the client can't determine the iteration value when submitting the data, so the data read back from the API is not the same as what was sent.

Another approach is to not implement anything, several nodes with the same parent and same path and same attributes can be retrieved with a query as a list. So the default approach for now would be to see whether the existing data model can already cope with this or if we need some additional features. This should become more apparent when trying to use the data to compute stats or detect performance regressions etc.

@gctucker gctucker added the roadmap API & Pipeline Roadmap label Jun 9, 2023
@gctucker gctucker added this to the Advanced features milestone Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
roadmap API & Pipeline Roadmap
Projects
Status: Todo
Development

No branches or pull requests

1 participant