User:Shawndouglas/sandbox/sublevel4

Abstract

Free Software and Open Source Software projects have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process we have initiated a project to generate a library of in-depth reviews of Free Software and Open Source Software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we will examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we will synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure will be executed against a selected class of applications.

Introduction

There is much confusion regarding Free Software and Open Source Software and many people use these terms interchangeably, however, to some the connotations associated with the terms is highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with Open Source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).

When discussing Free Software, we are not explicitly discussing software for which no fee is charged, rather we are referring to free in terms of liberty. To quote the Free Software Foundation (FSF):

A program is free software if the program's users have the four essential freedoms:

• The freedom to run the program as you wish, for any purpose (freedom 0).
• The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

This does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also 'free software' or 'freeware' that is given away at no charge, or gratis, but without the source code, this would not be considered Free Software under the FSF definition.

The Open Source Initiative (OSI), originally formed to promote Free Software, which they referred to as Open Source Software (OSS) to make it sound more business friendly. The OSI defines Open Source Software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines:

• Free redistribution
• Source code Included
• Must allow derived works
• Must preserve the integrity of the authors source code
• License must not discriminate against persons or groups
• License must not discriminate against fields of endeavor
• Distribution of licenses
• License must not be specific to a production
• License must not restrict other software
• License must be technology neutral

Open Source Software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.

In an attempt to placate both groups when discussing the same software application, the term Free/Open Source Software (F/OSS) was developed. Since the term Free was still tending to confuse some people, the term libre, which connotes freedom, was added resulting in the term Free/Libre Open Source Software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all Free Software fits the Open Source Software definition, while not all Open Source Software fits the Free Software definition. However, any Open Source Software that is not also Free Software is the exception, rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.

The reality is that since both groups trace their history back to the same origins, the practical differences between an application being Free Software or Open Source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in Open Source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us. In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With in excess of 300,000 FLOSS applications currently registered in SourceForge.net and over 10 million repositories on GitHub, there are generally multiple options accessible for any class of application, be it a Laboratory Information Management System (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an Open Source application and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers with no indication of the vitality or independent evaluation of the project.

What is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While once they've made a tentative selection, the organization will need to perform their own testing to confirm that the selected application meets their specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.

Illustration 1.: This diagram, originally by Chao-Kuei and updated by several others since,
explains the different categories of software. It's available as a Scalable Vector Graphic and
as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2
or later, or the Creative Commons Attribution-Share Alike v2.0 or later

The primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end it we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.

You can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, commercial is referring to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior quality, to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS is commercial software.

The final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.

Results

Literature review

A search of the literature returns thousands of papers related to Open Source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between Open Source projects is mentioned in a number of these papers and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large scale support in the industry. Stol and Babar have published a framework for comparing evaluation methods targeting Open Source software and include a comparison of 20 of them. They noted that web sites that simply consisted of a suggestion list for selecting an Open Source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, and other factors involved.

No. Name Year Orig Method 1 Capgemini Open Source Maturity Model 2003 I Yes 2 Evaluation Framework for Open Source Software 2004 R No 3 A Model for Comparative Assessment of Open Source Products 2004 R Yes 4 Navica Open Source Maturity Model 2004 I Yes 5 Woods and Guliani's OSMM 2005 I No 6 Open Business Readiness Rating (OpenBRR) 2005 R/I Yes 7 Atos Origin Method for Qualification and Selection of Open Source Software (QSOS) 2006 I Yes 8 Evaluation Criteria for Free/Open Source Software Products 2006 R No 9 A Quality Model for OSS Selection 2007 R No 10 Selection Process of Open Source Software 2007 R Yes 11 Observatory for Innovation and Technological transfer on Open Source software (OITOS) 2007 R Yes 12 Framework for OS Critical Systems Evaluation (FOCSE) 2007 R No 13 Balanced Scorecards for OSS 2007 R No 14 Open Business Quality Rating (OpenBQR) 2007 R Yes 15 Evaluating OSS through Prototyping 2007 R Yes 16 A Comprehensive Approach for Assessing Open Source Projects 2008 R No 17 Software Quality Observatory for Open Source Software (SQO-OSS) 2008 R Yes 18 An operational approach for selecting open source components in a software development project 2008 R No 19 QualiPSo trustworthiness model 2008 R No 20 OpenSource Maturity Model (OMM) 2009 R No

Table 1.: Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar. The selection
procedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described
process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)
or just a list of evaluation criteria (No).

Extensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques. Wasserman and Pal have also published a paper under the title of Evaluating Open Source Software, which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework. Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques. Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.

Since the publication of Stol's and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani, et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology. In their Transferring FAME paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.

Their stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. As specified ”The goals of FAME methodology are to aid the choice of high-quality F/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.” They further state that “The main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.”

This paper breaks the FAME methodology into four activities:

1. Identify the constraints and risks of the projects
2. Identify user requirements and rank
3. Identify and rank all key objectives of the project
4. Generate a priority framework to allow comparison of needs and features

Their paper includes a formula for generating a score from the information collected. The evaluated system with the highest 'major score', Pjtot, indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.

Pani, et al. also describe a FAMEtool to assist in this data gathering and evaluation. However a general web search, as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype, regarding FAME, the FAMEtool, or feedback on its use.

During this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol's and Barbar's table. Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.

Another paper, published by Haaland and Groven also compared a number of Open Source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used. Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a “User view on quality” and QualOSS as a second generation tool with a “business point of view”. An additional variation is that OpenBRR is primarily a manual tool while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first or second generation tools did a better job of evaluation, to me they answered the following even more important, but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation, as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.

Hauge, et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods. From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the 'best fit' application and fall back on what they refer to as a 'first fit'. This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications, are going to be different, as each user's needs are different.

Ayalal, et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies. While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounded as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.

In addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005, and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released. In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that "the current plan is that membership will not be open to all." He quotes Murugan Pal saying "membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system." However, for some reason, at least some in the group "expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community."

While the original Business Readiness Rating web site still exists, it is currently little more than a static web page. It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking "What happened to OpenBRR (Business Readiness Rating for Open Source)?"

It appears that at some point, any development activity regarding OpenBRR was morphed into OSSpal. However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their 'All Projects' tab provides a list of over 30 Open Source projects, but the majority simply show 'No votes yet' under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.

Evaluation and selection recommendations

At this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting FLOSS applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.

In TechRepublic, de Silva recommends 10 questions to ask when selecting a FLOSS application. While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.

1. Are the open source license terms compatible with my business requirements?
2. What is the strength of the community?
3. How well is the product adopted by users?
4. Can I get a warranty or commercial support if I need it?
5. What quality assurance processes exist?
6. How good is the documentation?
7. How easily can the system be customized to my exact requirements?
8. How is this project governed and how easily can I influence the road map?
9. Will the product scale to my enterprise's requirements?
10. Are there regular security patches?

Similarly, in InfoWorld Phipps lists seven questions you should have answered before even starting to select a software package. His list of questions, pulled directly from his article are:

1. Am I granted copyright permission?
2. Am I free to use my chosen business model?
3. Am I unlikely to suffer patent attack?
4. Am I free to compete with other community members?
5. Am I free to contribute my improvements?
6. Am I treated as a development peer?
7. Am I inclusive of all people and skills?

This list of questions shows a moderately different point of view, as it is not just about someone selecting an Open Source system, but looking to be involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate Open Source software into their projects. The list of criteria pulled directly from his blog includes:

1. Does it do what I need it to do?
2. How much more do I need it to do?
3. Documentation
4. Easy to review source code
5. Popularity
6. Tests and specs
7. Licensing
8. Community

Metcalfe of OSS Watch lists his top tips as:

1. Reputation
2. Ongoing effort
3. Standards and interoperability
4. Support (Community)
5. Support (Commercial)
6. Version
7. Version 1.0
8. Documentation
9. Skill setting
10. Project Development Development Model
11. License

In his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an Open Source application:

• Check licensing
• Check code quality
• Test setup time
• Verify extensibility
• Check for separation of concerns
• Check for last updated date
• Check for dependence on outdated toolkits/frameworks

Perhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's How to Evaluate Open Source Software / Free Software (OSS/FS) Programs. The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:

• System functionality
• System cost – direct and in-direct
• Popularity of application, i.e. its market share for that type of application
• Varieties of product support available
• Maintenance of application, i.e, is development still taking place
• Reliability of application
• Performance of application
• Scalability of application
• Usability of application
• Security of application
• Adaptability/customizability of application
• Interoperability of application
• Licensing and other legal issues

While a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying show a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.

With a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations, as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.

Wheeler defines four basic steps to this evaluation process, as listed below:

• Identify candidate applications
• Read existing product reviews
• Compare attributes of these applications to your needs
• Analyze the applications best matching your needs in more depth

Wheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of Open Source applications and examine the output of the protocol.

Identify needs

Realistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the User Requirements Specification (URS). This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document and including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed, When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.

Notes

This article has not officially been published in a journal. However, this presentation is faithful to the original paper, with only a few minor changes to presentation. This article is being made available for the first time under the Creative Commons Attribution-ShareAlike 4.0 International license, the same license used on this wiki.