1. I.
Introduction apid advancements in the area of Information Technology (IT) have enabled the software development organizations to break traditional blocks of building software and explore new methods. The software development approach has significantly changed in the past few years. Modern software systems are becoming increasingly complex and large resulting in high development cost and maintenance. This forced researchers in software engineering field to think about the necessity of designing new methodologies and paradigms to take head on these challenges. Thus the most important component technology was introduced, which advocates development of software applications by creating components and assembling them.
Many outstanding work have been published in the area of COTS evaluation and selection for the development of software using component based technology. This review paper is dedicated to the COTS evaluation and selection frameworks devised by different authors in the past. Best Practices of COTS selection in literature and in industry is an important review work carried out by Land et al [32].The present article looks into the research papers with a view to understand the framework given by different authors for COTS evaluation and selection. An attempt is made to explain few latest frameworks in a nutshell. The review work has been classified into two parts, namely: (a) COTS evaluation (b) COTS selection. The first part is dedicated to the COTS evaluation process. The second part of this paper discusses and reviews some of the optimization models of COTS selection proposed in the literature. Papers are discussed in the chronological order, enabling the readers to get an overview of the past models to the latest trends. For instant glimpses, the references are listed and summarized into a tabular form in each of the part. It is strongly believed that this work will give a quick insight for the future work concerned with COTS evaluation and selection. The following section of the article briefly describes software development using CBSS approach.
The rest of the paper is organized as follows. In Section 2, we discuss the software development using Component based software approach. Section 3 is divided into two sub sections, 3.1 gives the brief overview of the COTS evaluation approaches and subsection 3.2 presents the COTS selection approaches proposed in the literature. Each sub section contains a table that list all the papers in chronological order. Finally in Section 4, we furnish our concluding remarks.
Development of Component based software system advocates building software by selecting reliable, reusable and robust software components and assembling them within appropriate software architectures. This idea can be used to improve the productivity and quality of the software products. These components are usually pre-packaged solutions, known as Commercial-off-the-Shelf (COTS) software products. COTS are pieces of software that can be reused by software projects to build new systems [19,47]. These software products are developed and supported by outside suppliers, also called software vendors in the software market. Generally, COTS software products have the ability to reduce time and cost of software development [48]. Moreover, they enable software buyers to acquire software made up of components, which have been tested many times by other users, hence ensuring improved software quality [3]. For systems that depend on COTS products, the evaluation and selection of appropriate products is essential to the success of the entire system. Yet many organizations struggle during the evaluation and selection process. The selection of COTS components is a major challenge to software developers due to multiplicity of similar COTS products available in the market with varying characteristics and quality differences. Moreover, COTS selection is a complex decision-making problem that is characterized by uncertainty, complexity, multiple stakeholders, multiple objectives [48]. Many outstanding works have been published in the area of COTS evaluation and selection.
In the year 2008, Cortellessa et al, introduced an optimization framework for build-or-buy decision for building a component based software system. In his work, he extended the idea of software development by not only assembling COTS components but also inhouse built components. A common issue in building software architecture is whether to build software components in house or buy them. Build means developing sub-systems from the scratch. After this we integrate these sub-systems to form complete software. Buy means purchasing the sub-systems from the market. The subsystems which are purchased are known as commercial-off-the shelf (COTS) components and this decision is called build vs. buy decision. The decision of whether to build or buy the system does not only depend on the relative price of the alternatives. The complexity is an important criterion to be considered. We should buy COTS components when there is a demand for short delivery time and a small quantity of the product is desired. Build decision is preferable when technology is easily available and cheaper, also sometimes there are existing components that can be reused by modifying them to adjust to the present requirements.
2. III.
3. Analysis of Cots Evaluation and Selection Approaches
The review work has been classified into two parts, namely: (a) COTS evaluation: (b) COTS selection. The first part is dedicated to the COTS evaluation process. The second part of this paper discusses and reviews some of the optimization models of COTS selection proposed in the literature.
4. a) COTS Evaluation Approaches
There are three strategies given in the literature which can be applied to evaluate the COTS products: (a) Progressive Filtering Strategy (b) Keystone Strategy (c) Puzzle Assembly Strategy Progressive Filtering Strategy begins with large number of COTS software candidates in the list, and then each potential COTS software candidate is met with by a set of discriminating criteria which are defined through successive iteration of COTS software estimation cycle [29,32]. COTS software that does not satisfy these evaluation criteria is progressively removed from the COTS software candidates' list in each cycle of estimating. This strategy is done iteratively until the fitness of COTS software candidates are identified and retained in the list. Selecting one or more of COTS software can then be done from the list for integrating in the application [2].
In Keystone Strategy, the COTS software candidates are estimated against a key characteristic [29]. So the key characteristics (e.g. vendor location, type of technology) are identified at the beginning of this strategy, then the searching for COTS software will be based on satisfying this keystone characteristic. This strategy is applied at the beginning stages of the evaluation in order to permit quick removal of the large number of COTS candidates that do not satisfy the keystone characteristic [32].
The idea of Puzzle Assembly Strategy is taken from collecting pieces of a puzzle [39]. This strategy assumes that when selecting the COTS software we must consider the fitting of the COTS software with other components on the system [29,2]. In other words, COTS software that can be considered as fitness in isolation might be not acceptable when assembled with other components in the system. Therefore, in this strategy, choosing COTS software must be done by considering the other components requirements in the puzzle.
Mohmad et al in [39] argues that more than one strategy from the above can be used with the same project. For example, the keystone can be used at the beginning of the project to eliminate the largest possible number of COTS candidates, and then the progressive filtering can be used later on.
Oberndorf also proposed that more than one of the strategies above can be employed in the same project [47]. For example, a developer might use keystone identification first and then progressive filtering later.
Multiple criteria have to be considered during the selection of components for software development. A balance between technical characteristics, financial issues and application requirements is required. Many authors have proposed different methods of selection of COTS components for development of CBSS. One of the first proposals was given by Kontio et al in [26] they proposed the OTSO (Off-The-Shelf Option) approach for COTS selection. The authors developed a method that addresses the selection process of packaged, reusable off-the-shelf software. The OTSO approach supports the search, evaluation and selection of software components.
In 1996, Kontio published several follow-up papers to elaborate OTSO [27]. An approach called PRISM (Portable, Reusable, Integrated, Software Modules) was proposed by Lichota et al, [34]. In their approach a generic component architecture was proposed that can be used during COTS evaluation process. However, it was not until 1998 that another important milestone was reached with the Procurement-Oriented Requirements Engineering approach (PORE) [37]. The importance of PORE is that it proposed a requirements engineering process for COTS-based development. PORE suggested that requirements should be elicited and analyzed at the same time when the COTS products are evaluated.
The STACE (Social-Technical Approach to COTS Evaluation) approach [32] emphasized the importance of non-technical issues, e.g., social, human, and organizational characteristics, during the evaluation process.
Ochs et al [42] proposed the COTS acquisition process (CAP) which highlighted the concept of a "tailorable evaluation process". The approach suggested that the evaluation process should be tailored based on the available effort for project and it relied on experts' knowledge to tailor the process.
In 2001, a project was initiated by Chung et al, [11,12,13,14] to describe a COTS -Aware Requirements Engineering (CARE) Process. CARE uses a flexible set of requirements based on different agents' knowledge. For the same, CARE proposes a method to define relevant agents as well as the system goals and requirements. The PECA (Plan, Establish, Collect and Analyze) approach was proposed by Dorda et al [17] from Software Engineering Institute (SEI) describes a detailed tailorable COTS selection process and gives guidelines which the experts can use to tailor the process. In 2002 [35] proposed the Balanced Reused Model (BAREMO) approach. This approach explains in detail how a decision can be made based on Analytical Hierarchy Process (AHP) [43]. The Combined Selection (CS) approach [9] is used to select multiple COTS products that all together satisfy the requirements. This approach performs its activities at two levels: local and global. The global level addresses the overall process of the combined selection, fires individual selection processes for each area and tries to find the best overall combination of products. The local level use existing COTS evaluation and selection techniques e.g. OTSO [26] or PORE [37] to select individual COTS that are combined at the global level.
The approach by Erol and Ferrell, [18] is an evaluation approach that supports selecting a COTS product from a finite set of products based on more than one objective and a set of quantitative (e.g cost) and qualitative (e.g. linguistic variables) data. The approach uses fuzzy QFD (Quality Function Deployment) [1] to collect and quantify the qualitative data. Then goal programming is used to get near optimal solutions to the decision maker.
5. DesCOTS
(Description, Evaluation and Selection of COTS components) approach was presented by Grau et al, [20]. DesCOTS system includes a set of tools that can be used to evaluate the COTS products based on quality models.
Shyur in [45] in his work models the COTS evaluation problem as Multi-criteria Decision Making (MCDM) problem. He proposes a five phase COTS selection model, combining the technique of ANP (Analytic Network Process) and modified TOPOSIS (technique for order performance by similarity to idea solution). ANP is used to determine the weights of multiple evaluation criteria. The modified TOPOSIS is used to rank competing products in terms of their overall performance.
In 2007, the MiHOS (Mismatch-Handling aware COTS selection) approach was developed [39]. The approach focuses on handling the mismatches between COTS candidates and the requirements. MiHOS uses techniques such as linear programming to identify near optimal solutions. In 2007, an interactive decision support approach for multi-objective COTS selection was addressed by [40]. Authors have introduced a twophase decision support approach for selection of COTS products.
Couts et al [16] have shown that the evaluation and selection of COTS software is performed using adhoc manners.
Tarawneh et al [46] proposed a framework to support and improve the COTS software evaluation and selection processes in industry. To achieve this objective the authors have shown that specific objectives have to be addressed:
6. 1.
Identify the processes which support COTS software evaluation and selection.
2. Determine the criteria or requirements which are important for successful evaluation and selection process. 3. Propose methods and techniques to address the mismatch between COTS features and customer requirements.
4. Develop a repository to manage information from previous selection cases that support the decision making process.
Mead N. R. in [38] developed Software Quality Requirements Engineering for Acquisition (A-SQUARE) methodology for eliciting and prioritizing security requirements as part of the acquisition process. The author in her report, evaluated the effectiveness of the A -SQUARE method by applying it to a COTS product for the advanced metering infrastructure of a smart grid.
7. Global Journal of C omp uter S cience and T echnology
8. b) Optimal COTS Selection Approaches
Several optimization models have been proposed in the literature for the optimal selection of the software components for the development of safe and reliable software systems. The models use basic information on components reliability and cost and allow the trade-off between two factors. Ashrafi and Berman, [4] presented two models which address the tradeoff between reliability and cost. The model is applied to large software packages that consist of several programs. The models can be used as decision support tools for organizations that are in the process of purchasing of variety of computer programs in order to meet the needs of the users, e.g. operations people need software packages to perform functions such as scheduling, inventory control and purchase orders. While the main consideration is to attain high average reliability for the software package, management has to consider both the relative importance of each program in terms of the frequency of the usage of their corresponding function. Programs can be purchased from software development companies. Several programs are usually available for each function. Each program has a known market cost and an estimated reliability. It can be noted here that the assumption of using the ready programs available in the market to make the software package implies the use of COTS (Commercial-off-the-shelf) program. Hence the models are applicable to only those software packages that are designed using COTS products. The authors have formulated two types of models one which does not consider redundancy for performing each function and the other which maintains redundancy under budget limits. Considering the concept of COTS in a software development and the availability of mathematical models to access module reliability, it is possible to have information on module reliability and cost. In their previous work [5] authors used optimization models to determine the redundancy level of a software package consisting of several independent functions where each function is performed by a program with known reliability and cost. In this work, however, they broke down this approach one step further and deals with software systems consisting of one or more programs where each program consists of series of modules, which upon sequential execution will perform a function. Four models are presented, each applicable to a different software system structure.
The optimization models discussed in the previous sections don't consider any of the fault tolerance schemes such as recovery block or NVP. They merely consider the programs consisting of set of modules, which on sequential execution perform the function. Berman and Kumar,[7] studied the problem of optimum selection of component for the recovery blocks for the first time. The author presented optimization models for a fault tolerant software system. Specifically they have formulated optimization problems for two types of recovery blocks namely -Independent and Consensus recovery block schemes.
The optimal component selection problem addressed by Kapur et al, [25] considers software built by assembling COTS component performing multiple functions. Each function is performed by calling a set of modules. Modules can be assembled in a recovery block scheme to provide the fault tolerance. Again for each alternative version multiple choices are available from the supplier with distinct reliability and cost. The version for any alternative choice having higher reliability has higher cost. Two models are formulated for weighted maximization of system reliability, weights being decided with respect to access frequency of functions within the available budget. Each module is comprised with a set of COTS alternative that are available in the market.
Cortellessa et al, [15] introduce a framework that helps developers to decide whether buying or building components for certain software architecture. Once built software architecture, each component can be either bought, or probably adapted to the new software system, or it can be developed in-house. This is a "build-or-buy" decision that affects the software cost as well as the ability of the system to meet its requirements.
Gupta et al, [22] in their work have formulated fuzzy multi-objective optimization model for selection of COTS components for development of a modular software system. The hierarchy structure of the software consists of three programs, four modules and eleven COTS products. Some specific functions of each program can call upon a series of modules, and several alternative COTS products are available for each module. Different weights are assigned to different modules using an AHP technique. The issue of compatibility amongst the COTS products is also discussed.
Kwong et al, [31] have addressed an optimization concept of selection of software components using intra-modular coupling density (ICD) and functional objective along with few system constraints of ICD and functionality without redundancy for a CBSS development.
Jha et al, [24] formulated multi-criteria problem for minimizing the overall cost and maximizing the system reliability by using fuzzy multi objective optimization model for selecting the best COTS software product among alternatives of each module for modular software system.
Jha et al, [23] formulated a multi-objective problem with cost minimization and reliability maximization as the two objectives with an upper bound on cost and lower bound on reliability. This model was formulated for Recovery Block and Consensus Recovery Block fault tolerant software. Author used goal programming approach for solving the problem.
Kumar et al, [28] discussed an effective approach to formulate multi-objective problem with cost minimization and reliability maximization as the two objectives with an upper bound on cost and lower bound on reliability. The author used goal programming approach for solving the problem based on Consensus Recovery Block Scheme.
Bali et al, [6] proposed optimization models for optimal component selection for a fault-tolerant modular software system under the consensus recovery block scheme. It is necessary to identify critical modules in the design of a fault-tolerant modular software system and also to develop a system with a built in redundancy for critical modules. During the planning phase of software development, it is necessary that modules are categorized and identified based on their reusability and criticality to run the application. In order to achieve this, a constraint on criticality of modules can be used to achieve the effective redundancy for all critical modules and at least one effective alternative for non-critical modules. An attempt has been made in this paper to review and critically analyze the COTS evaluation and selection process. Reviewed papers are categorized into two themes: evaluation and selection. We have tried to explore the COTS evaluation and selection practices and compare the most significant approaches. The objective of this study is to identify currently used decision making practices of COTS evaluation and selection. In future the authors would like to propose a framework for COTS selection which is relevant to today's era of software development approach. Moreover, the models proposed in the literature have not talked about build-or-buy strategy in depth and issue of criticality in specific.
9. Global
10. Global
| Year 2015 |
| Volume XV Issue VI Version I |
| ( ) |
| Ref. No. | Year | Authors | Objective Function | Approach | |
| 1. | 4. | 1992 | Ashrafi & Berman | Reliability | Integer Programming Problem |
| 2. | 5. | 1993 | Ashrafi & Berman | Reliability | Integer Programming Problem |
| 3. | 7. | 1999 | Berman & Kumar | Reliability | Integer Programming Problem |
| 4. | 25. | 2003 | Kapur et al | Reliability | Integer Programming Problem |
| 5. | 15. | 2008 | Cortellessa et al | Cost | Integer Programming Problem |
| Fuzzy Multi Objective Optimization | |||||
| 6. | 22. | 2009 | Gupta et al | Quality, Cost | Model |
| ICD and Functionality without | |||||
| 7. | 31. | 2010 | Kwong et al | Intra-modular coupling density | Redundancy |
| 8. | 24. | 2010 | Jha et al | Reliability, Cost | Multi Objective Optimization Model |
| Fuzzy Multi Objective Optimization | |||||
| 9. | 23. | 2011 | Jha et al | Reliability, Cost | Model |
| 10. | 28. | 2012 | Kumar et al | Reliability, Cost | Goal Programming |
| Cost, Development Efforts, | |||||
| Execution Time, Reliability and | |||||
| 11. | 21. | 2012 | Gupta et al | Quality | Fuzzy Interactive Approach |
| Fuzzy Multi Objective Optimization | |||||
| 12. | 6. | 2014 | Bali et al | Reliability, Cost | Model |