Countless projects emerge in the open source world every day, but accurately assessing the health of an open source project has always been a challenge. Fortunately, we now have the open source project's health check doctor - OSS Compass, and recently its open source ecosystem evaluation system and SaaS services have undergone comprehensive upgrades! Last week, we introduced the Contributor Persona Model and Project Deep Dive Insight Solution. Now, let's take a look at the Three-Dimensional Evaluation System Graph and Compass Collaboration together.

Countless projects emerge in the open source world every day, but accurately assessing the health of an open source project has always been a challenge. Fortunately, we now have the open source project's health check doctor - OSS Compass, and recently its open source ecosystem evaluation system and SaaS services have undergone comprehensive upgrades! Let's take a look at the exciting updates together.

On the afternoon of December 13, 2023, amidst heavy snowfall in Beijing, OSS Compass Community 2023 Annual Meeting, co-organized by OSS Compass Community, Peking University, and Huawei Open Source, took place as scheduled. Despite the cold weather and slight traffic congestion, the enthusiasm of the experts remained unstoppable as they braved the snowy conditions, together igniting this feast of technology and art!

On the morning of December 13th, OSS Compass (hereinafter referred to as "Compass") community board held its fourth-quarter meeting at Peking University to explore the community's new value. The meeting was attended by 10 board members, including Minghui Zhou from Peking University, Xianping Tao and Liang Wang from Nanjing University, Hong Shu from OSChina, Hongwei Ma from Baidu, Kun Gao and Yehui Wang from Huawei, Zhongyi Tan from StarTogether Community, and Wenxuan Long from Checode attending the meeting in-person, while Shengxiang Zhang from OSChina participating in the meeting online. Three board members were absent due to unforeseen circumstances.

In the previous two articles, "Reflections on the Evaluation and Measurement of Open Source Ecosystem (1) - Evolution and Trends" and "Reflections on the Evaluation and Measurement of Open Source Ecosystem (2) - The Multidimensional Space of Evaluation Systems", I summarized three main directions of open source community evaluation and measurement: open source softwares, open source projects, and open source ecosystems. I also introduced a three-dimensional space of evaluation systems (Figure 1) and discussed four evaluation models in the intertwined space of "open source ecosystem" and "collaboration": Collaboration Development Index Model, Community Service and Support Model, Organizational Activity Model, and Community Activity Model (deployed in OSS-Compass). I used the example of PyTorch vs. TensorFlow to demonstrate the logical relationships between these models.

In this article, we will primarily focus on the intertwining between "open source ecosystem" and "people" that gave rise to two important evaluation models: Contributor Persona Model and Contributor Milestone Model. These two models are the results of collaborative research with Professor Liang Wang and his team of Nanjing University, and I sincerely appreciate the efforts they have put into this.

In my previous article, "Reflections on the Evaluation and Measurement of Open Source Ecosystem (1) - Evolution and Trends", I summarized three main directions for evaluating and measuring open source communities: open source software, open source projects, and open source ecosystems. However, for the purpose of technical insight work, this is just the first step in a long journey.

As an engineer, I hope to see the emergence of a practical and feasible evaluation system with the following characteristics: the ability to help open source communities identify specific problems, assist people in discovering valuable open source communities, and predict industry trends.

In summary, this evaluation system should be practical, providing tangible value rather than being just an elusive idea.

In recent years, I have been engaged in work related to the evaluation of open source ecosystems. This series of articles aims to summarize my work experience, and organize the thoughts for upcoming work. Meanwhile, I'm looking forward to exchanging ideas with everyone.

Through the observation of the academic and open source industry development in the past thirty years, we can find that the evaluation and measurement of the open source community mainly focus on three mainstream directions: open source software, open source projects, and open source ecosystems. The emergence of these directions is closely related to the era, and with the development of open source, their boundaries gradually blur, showing a trend of mutual inclusion. Due to the different focuses of various markets and user concerns, they develop somewhat independently.

Jierui Zhang，Ying Li，Liang Wang，Xianping Tao

Department of Computer Science and Technology, Nanjing University, State Key Laboratory for Novel Software Technology OSS Compass

Abstract​

In this study, we study the community evolution behavior in developer social networks around open source software projects, in response to the limitation of traditional community evolution analysis techniques being biased towards qualitative rather than quantitative analysis. We propose a set of community split, shrink, merge, and expand indices based on information entropy to measure the evolutionary behavior of open source developer social networks. Empirical studies demonstrate that these indices effectively characterize the evolution of open source communities, by achieving a 94.1% accuracy in drawing conclusions consistent with existing qualitative work through simple rules. Furthermore, additional research indicates a significant correlation between the proposed community evolution indices and the productivity of open source projects, represented by the number of commits. The information entropy-based measurement indices presented in this study provide quantitative support for understanding and analyzing the evolutionary behavior of open source communities.

Zheng Liu，Xiaolan Zu，Xingyu Luo，Zihang Wang，Jierui zhang，Yehui wang， Liang Wang, Xianping Tao Department of Computer Science and Technology, State Key Laboratory for Novel Software Technology, Nanjing University Huawei Technologies Co, Ltd. OSS Compass

Abstract​

This study proposes a method to predict the future activeness of open source projects using OSS Compass indicators. It employs a feature-based time series classification approach, extracting statistical features from OSS Compass indicator series and using machine learning algorithms for prediction. The method demonstrated an accuracy of nearly 90% in cross-validation on a dataset of around 600 projects and about 80% accuracy on a larger set of over 10,000 repositories, indicating its practical applicability. The results partially reflect the future health status of open source projects, demonstrating the effectiveness of the OSS Compass indicator system in measuring the health of open source software. The method could provide valuable insights for users, developers, investors, and managers of open source software.

Liang Wang$^{1,2}$，Zhiwen Zheng$^{1,2}$，Xiangchen Wu$^{1,2}$，Baihui Sang$^{1,2}$，Jierui Zhang$^{1,2}$，Xianping Tao$^{1,2}$

$^1$Department of Computer Science and Technology, Nanjing University, State Key Laboratory for Novel Software Technology $^2$OSS Compass

Abstract​

This research focuses on project forks on open-source software (OSS) platforms, revolving around measuring and understanding the diversity of forks in open-source software projects. The paper accomplishes this by constructing a novel fork entropy based on Rao's second entropy and assessing this diversity based on modifications to project files. Empirical studies indicate a significant correlation between fork entropy in open-source projects and various key outcomes, including external productivity of the project (measured by the number of contributions from external contributors), acceptance rate of external contributors' pull requests, and the reported number of bugs. Additionally, we observe significant interactions between fork entropy and other factors, such as the quantity of forks. These findings suggest that fork entropy, as an effective metric, not only enhances the current available measurements of forks in open-source software projects but also deepens our understanding of the development process based on forked repositories. It holds the potential to support further research and applications.