[Library] Automate quality assurance testing of libraries

[s-celles]

Hello,

In #6646 I mentioned that library manager should display license for each library.

I think a more general issue is that Arduino should be able to automate quality assurance of libraries.
http://downloads.arduino.cc/libraries/library_index.json could help

The presence of a license file is one point, but assurance that the code is still compilable is an other important point. Then, assurance, thanks to unit tests, that library have some chance to be still usable is a last point.

In JuliaLang we have https://pkg.julialang.org/ which list all libraries (outside of any IDE).
So, thanks to continuous integration and this quality assurance testing, we know if a library is compilable or not and if tests of a library are passing or not...

https://github.com/ianfixes/arduino_ci provides scripts to ensure that when a commit is made, it doesn't break library (prevents him from compiling) and that unit tests are still passing.

All this may be insufficient, we could still have broken eggs, but that's a begining.

Require developers to tag their release (with semantic versionning) and set up requirements (with a minimal version number for each dependencies) in a configuration file could be an other important point (discussed in #5795 )

Kind regards

[s-celles]

Maybe some developpers with knowledge of C/C++ dependency manager could comment on the last point about requirements.

Code linting could also be checked.

A last, but important part of code quality assurance is code coverage but we are probably still far away from this point.

[per1234]

Here's the only way I can see this working:

• When the Library Manager indexer job finds a new tag it looks for the CI configuration file for any standard CI service (e.g. Travis CI, Circle CI, GitLab CI) and runs (or checks existing?) CI build for that tag, then records the result of that build in http://downloads.arduino.cc/libraries/library_index.json.
• Filter checkboxes are added to the Arduino IDE's Library Manager GUI:
  • Tested: CI build passed.
  • Untested: No CI build was done.
  • Test Failed: CI build failed.

The creation of the CI configuration file and which tests will be done on the library will be up to the library maintainer. That could include compile testing, unit testing, and linting. It is not feasible for Arduino to generate tests for 3rd party libraries.

[s-celles]

As written in #6646 (comment) detect license inconsistencies (ie a different license in library.properties and LICENSE file) could be part of the Arduino libraries quality assurance process

[s-celles]

Pinging @ianfixes about unit tests / continuous integration for Arduino libraries

[ianfixes]

Thanks for bringing me into this discussion, and I'm somewhat flattered by this attention towards my testing library. I still consider arduino_ci to have an alpha-level feature set, but so far the concept has proven to be fairly solid.

• Unit testing works, period. There's room for the mocks to become more elegant, but there's a lot already supported (e.g. faking a hardware modem via serial). You do not need hardware present to run the unit tests, regardless of architecture.
• Memory management issues can be exposed and annotated (via compiler features): Prevent segfaults by implementing the rule of three  adafruit/Adafruit-WS2801-Library#20.
• Library examples can be targeted to specific architectures via the config file, as an alternative to using preprocessor directives to work around compilation issues.
• Extra architectures (any board that can be installed via the arduino CLI) can be declared and configured via the config file
• Dependent libraries are installed automatically (although for now they must be declared manually in the config file)

Metadata tests (e.g. your comment about the LICENSE file) don't exist now, but I could certainly integrate them into the ruby side of the test runner. That would either be its own standalone script (similar to arduino_ci_remote.rb, the entry point for CI), or baked into that script and controlled via config.

Should we pick an Arduino library to use as a demo?

[per1234]

@ianfixes do you agree with my previous statement?:

The creation of the CI configuration file and which tests will be done on the library will be up to the library maintainer. That could include compile testing, unit testing, and linting. It is not feasible for Arduino to generate tests for 3rd party libraries.

Or do you think there is a way to do completely automated unit testing via arduino_ci without any configuration on the part of the library author?

As for testing of metadata and library structure, this is possible to do automatically (and in fact I'm already doing that independently). It's actually even more simple with the libraries in the Library Manager index because we know they're in the repository root and thus don't need to deal with the headache of searching for the library in subfolders. Here is the bash script I use to check metadata and library structure:
https://github.com/per1234/arduino-ci-script/blob/1.1.0/arduino-ci-script.sh#L1212-L2326

The Library Manager indexer actually already does a lot of verification of the library.properties and some of library structure (must be in repo root, must not contain executables). Any release that doesn't pass its verification is not added to the Library Manager index

[PaulStoffregen]

It is not feasible for Arduino to generate tests for 3rd party libraries.

Why not? Anyone can select a widely used board, manually open each of library's examples and click Verify. Why can't a CI script do the same?

And really, why isn't doing substantially more considered to be feasible? Even a rather elaborate test which runs the libraries on real hardware and compares against a known-good result is the sort of thing you might expect take 1 person-month to initially create, and then maybe 2-3 hours per library to set up. While there are thousands of libraries, only a few hundred are very widely used. I just don't understand why something like this is assumed to be so far out of reach, when it could be done pretty effectively by a few summer interns working with direction and occasional assistance from a seasoned engineer / developer.

[ianfixes]

It is not feasible for Arduino to generate tests for 3rd party libraries.

For unit tests, I agree completely: the library maintainers would need to take full responsibility for writing tests as well as maintaining any given CI solution as a whole. Period.

For compiling the examples provided by the library, I disagree slightly: anyone can load & compile those (via the GUI or CLI), including a server-side process. That said, such an effort would fail since not every library works for every architecture. You would need some configuration file format to sort that out (I can humbly offer my own YAML format for consideration), and I can't say whether it would be worth it to roll that out and then support it.

The same reasoning would apply to adopting a system like NPM modules or RubyGems -- formats that include a metadata field for the git repo URL. Although this could enable querying the a given commit to see test status (e.g. on GitHub), this would be impractical. You'd need to correlate the library with a repository, and the library version with a specific commit, and it would rely on a developer base that is not primarily software engineers to spend time maintaining it.

But if you'll allow me take a step back... I took the spirit of this issue to be "enable automated testing" -- that's a prerequisite, no matter who performs the QA¹. A few things currently block such testing, which I had to work around or simply brute-force my way though as I built my library. I would love to work with you to make a more official solution to these.

I guess my question for you would be: What impact would it have on the Arduino ecosystem if we could give library maintainers more confidence in accepting code contributions (i.e. if they could easily set up pull-request testing on GitHub, to automatically verify those contributions)?

¹ minor nitpick @scls19fr : it's "Quality Assurance", not "Insurance"; not sure if that will affect the searchability of this issue

[ianfixes]

@PaulStoffregen

Even a rather elaborate test which runs the libraries on real hardware and compares against a known-good result is the sort of thing you might expect take 1 person-month to initially create, and then maybe 2-3 hours per library to set up

See my Arduino CI library: https://github.com/ianfixes/arduino_ci

It mocks all aspects of the hardware for unit testing (including clock), does not require hardware to run (i.e. can be run on Travis or Appveyor CI), and takes roughly 0.5-2 hours to set up on any given library.

• Arduino-Queue.h
• Adafruit-WS2801
• Adafruit_FONA
• AcceleratedLED

That setup and maintenance should fall to the individual library maintainers, not the Arduino team.