Skip to content

Latest commit

 

History

History
108 lines (85 loc) · 6.53 KB

HEALTHCHECKS.md

File metadata and controls

108 lines (85 loc) · 6.53 KB
Raw

Open Source Project Health Checks

This page describes the VMware Tanzu Community Engagement team’s open source project health check process. Building on the team’s commitment to developing thriving communities, we’ve created this process to work with project teams to understand what a healthy project looks like, give feedback on how to improve health, and track project health over time.

Our approach is to focus on measuring and reporting metrics for open source projects we are heavily invested in. We want to steer clear of vanity metrics, and focus on actionable items that project teams can use to improve the project’s overall health and public image. Metrics include tracking things like documentation quality, maintainer attentiveness, and community involvement in the project.

Goals of the health checks:

  • Provide guidelines for project teams around what makes a healthy open source community experience
  • Highlight areas where improvements can be made
  • Spark discussions, collaboration, and new ideas to make a better community experience for everyone
  • Continuous evaluation of projects as they develop within the ecosystem, allowing teams the opportunity to hear about new guidelines and adjust on-going processes
  • Catch and make corrections for slow-downs or lapses as the projects progress

Non-goals of health checks:

  • Providing “we’re done” metrics - open source community engagements are never “done”
  • Implement a pass/fail rating system for projects
  • Create single “cookie-cutter” path for every project

Health check metrics

Open source projects with assigned community managers will be assessed with "Shoot for the Stars" measurements:

  • Project Maturity

    • Following the guidelines outlined in our Guidelines
    • Following CII best practices
    • Release velocity
      • Time between releases. Regular releases are a reliability indicator.
    • Roadmap
      • Existence and quality of roadmap.

  • Project Viability

    • Documentation
      • What is the thoroughness, and accessibility of documentation according to a set of criteria?
    • Low elephant factor - when applicable
      • Related to the bus/lottery factor, but measured at a corporation level. How viable would the project be if the largest organization pulled out its contributors? We use this measurement for multi-vendor projects such as CNCF project.

  • Growth of Community

    • Role definitions
      • Existence and quality of role definitions
    • Path to Maintainership
      • Published path to maintainership. How can someone go from user, to contributor, to maintainer.
    • Continually onboarding new contributors
      • How many casual/regular contributors are we onboarding/retaining?
    • Distribution of Work
      • How much of a project’s recent activity was distributed between community members (all maintainer driven, split, all community driven)?
    • Decision distribution
      • Central vs. distributed decision making. Shows scalability of community.
    • Communication channels are in place and used effectively
      • Slack, distribution lists, GitHub issues & PRs, community meetings, public meeting notes, recorded videos

  • Diversity of Community

    • Number of contributors
      • Having a stable number of contributors over time.
    • Contributing organizations
      • Having a stable number of organizations contributing over time.
    • Non-code contributions
      • Track contributions like running tests in test environments, writing blog posts, producing videos, and giving talks.
    • Contributor breadth
      • Ratio of non-core committers (regular and casual committers).

  • Attentiveness of Maintainers

    • Recognition of contributors
      • Shout-outs, recognition, and mentions in pull-requests or change logs.
    • Retrospectives
      • Existence of a retrospective meeting after a release. Collect lessons learned, improve processes, recognize contributors.
    • Release Note Completeness
      • Functionality changes and bug fixes should be represented in release notes. Good for users, and shows diligence of the community.
    • Transparency
      • Decisions are made public and communicated where appropriate. Discussion is occurring openly as much as possible.

  • User Community Engagement

    • User documentation (visible and being updated)
    • Communications channel for user questions are used actively by both users and maintainers
    • Obvious process for bug reports and feature requests
    • Meetings for user engagement, office hours, best practices

Health check measurement scale

In general each metric should be reported using the following guidelines:

  • Working well (Green)

    • The team and community are progressing or engaging at a good level. There is no immediate need to make improvements.

  • Some pieces are missing (Yellow)

    • There is still some work to be done to implement guidelines or recommended practices. This should be addressed when there is time.

  • Needs attention (Red)

    • Large pieces of process, content, or community interaction are missing. The project team should take steps to improve as soon as possible.

How to perform a health check

The Community Engagement team performs health checks every 6 months on projects with dedicated community managers.

We have an example sheet available that you can use to track your project's measurements.

  1. Review each metric and update the Status field according to the current state of the project.
  2. Research the current state of the project by
    1. Reviewing the previous Advice and Comments to see if progress has been made
    2. Talking to the project engineering leads
    3. Any metric graded below Working Well should have Advice or Notes filled in with your reasoning.
  3. Once you have reviewed each metric, schedule a meeting with the engineering leads of the project to go over your findings.
  4. Create a presentation of your findings
    1. This presentation should capture the key takeaways and potential action items from your evaluation of the project and discussions with project lead(s).
    2. Schedule time to present to the entire open source project team. This presentation is meant to share your findings of the current project health, and also be an open discussion with the project team. The team should agree on any action items and how to take the next steps to improve the health for the next review.
  5. Continously review action items until the next review cycle