Add features to facilitate debugging and testing pipeline services

[r-c-n]

Problems to solve

• Every time that a developer comes new to the project, they need to go through lots of obstacles to do any kind of useful improvement due to the lack of local testing and debugging facilities.
• Even more seasoned developers often spend a non-trivial amount of time developing small features (that should be trivial to do) because of the lack of developer-oriented features. Examples: selectable traces, debug probes.
• The provided tools and documentation don't address the specific needs of developers at all

Description of the situation

kernelci-api and kernelci-pipeline weren't designed with debugging, tracing and incremental development as part of its core features. This makes them way harder to develop and debug than they should be for a system that had flexibility and malleability as its reasons of being.

We're finding time and time again that individual contributors end up writing their own hacks to help them debug and test certain things, which leads to lots of wasted time a) writing these hacks and b) falling into the same obstacles that other people had in the past.

This issue will be at the core of improving the situation for the kernelci pipeline, both in terms of tools and documentation, and the time spent on it hopefully will bring greater returns as more development is being done in the pipeline.

Goals

• Refactor the existing pipeline services to achieve the following:
  • The possibility of fully running each service in isolation:
  • Improved traceability, especially in the transactions with the API and API helpers: make the data paths and flows easier to probe
• Enable the necessary tools to aid testing, such as:
  • Data generators: for example, node builders
• Facilitate testing each pipeline service fully locally
  • Make it possible to collect the output and artifacts of a service run independently from other services (eg. job code generation, node generation, logs and tarballs or their placeholders, etc)
• Integrate new developer tools and features into the kci tool
• Document these features for developers