# I. Introduction apid advancements in the area of Information Technology (IT) have enabled the software development organizations to break traditional blocks of building software and explore new methods. The software development approach has significantly changed in the past few years. Modern software systems are becoming increasingly complex and large resulting in high development cost and maintenance. This forced researchers in software engineering field to think about the necessity of designing new methodologies and paradigms to take head on these challenges. Thus the most important component technology was introduced, which advocates development of software applications by creating components and assembling them. Many outstanding work have been published in the area of COTS evaluation and selection for the development of software using component based technology. This review paper is dedicated to the COTS evaluation and selection frameworks devised by different authors in the past. Best Practices of COTS selection in literature and in industry is an important review work carried out by Land et al [32].The present article looks into the research papers with a view to understand the framework given by different authors for COTS evaluation and selection. An attempt is made to explain few latest frameworks in a nutshell. The review work has been classified into two parts, namely: (a) COTS evaluation (b) COTS selection. The first part is dedicated to the COTS evaluation process. The second part of this paper discusses and reviews some of the optimization models of COTS selection proposed in the literature. Papers are discussed in the chronological order, enabling the readers to get an overview of the past models to the latest trends. For instant glimpses, the references are listed and summarized into a tabular form in each of the part. It is strongly believed that this work will give a quick insight for the future work concerned with COTS evaluation and selection. The following section of the article briefly describes software development using CBSS approach. The rest of the paper is organized as follows. In Section 2, we discuss the software development using Component based software approach. Section 3 is divided into two sub sections, 3.1 gives the brief overview of the COTS evaluation approaches and subsection 3.2 presents the COTS selection approaches proposed in the literature. Each sub section contains a table that list all the papers in chronological order. Finally in Section 4, we furnish our concluding remarks. Development of Component based software system advocates building software by selecting reliable, reusable and robust software components and assembling them within appropriate software architectures. This idea can be used to improve the productivity and quality of the software products. These components are usually pre-packaged solutions, known as Commercial-off-the-Shelf (COTS) software products. COTS are pieces of software that can be reused by software projects to build new systems [19,47]. These software products are developed and supported by outside suppliers, also called software vendors in the software market. Generally, COTS software products have the ability to reduce time and cost of software development [48]. Moreover, they enable software buyers to acquire software made up of components, which have been tested many times by other users, hence ensuring improved software quality [3]. For systems that depend on COTS products, the evaluation and selection of appropriate products is essential to the success of the entire system. Yet many organizations struggle during the evaluation and selection process. The selection of COTS components is a major challenge to software developers due to multiplicity of similar COTS products available in the market with varying characteristics and quality differences. Moreover, COTS selection is a complex decision-making problem that is characterized by uncertainty, complexity, multiple stakeholders, multiple objectives [48]. Many outstanding works have been published in the area of COTS evaluation and selection. In the year 2008, Cortellessa et al, introduced an optimization framework for build-or-buy decision for building a component based software system. In his work, he extended the idea of software development by not only assembling COTS components but also inhouse built components. A common issue in building software architecture is whether to build software components in house or buy them. Build means developing sub-systems from the scratch. After this we integrate these sub-systems to form complete software. Buy means purchasing the sub-systems from the market. The subsystems which are purchased are known as commercial-off-the shelf (COTS) components and this decision is called build vs. buy decision. The decision of whether to build or buy the system does not only depend on the relative price of the alternatives. The complexity is an important criterion to be considered. We should buy COTS components when there is a demand for short delivery time and a small quantity of the product is desired. Build decision is preferable when technology is easily available and cheaper, also sometimes there are existing components that can be reused by modifying them to adjust to the present requirements. # III. # Analysis of Cots Evaluation and Selection Approaches The review work has been classified into two parts, namely: (a) COTS evaluation: (b) COTS selection. The first part is dedicated to the COTS evaluation process. The second part of this paper discusses and reviews some of the optimization models of COTS selection proposed in the literature. # a) COTS Evaluation Approaches There are three strategies given in the literature which can be applied to evaluate the COTS products: (a) Progressive Filtering Strategy (b) Keystone Strategy (c) Puzzle Assembly Strategy Progressive Filtering Strategy begins with large number of COTS software candidates in the list, and then each potential COTS software candidate is met with by a set of discriminating criteria which are defined through successive iteration of COTS software estimation cycle [29,32]. COTS software that does not satisfy these evaluation criteria is progressively removed from the COTS software candidates' list in each cycle of estimating. This strategy is done iteratively until the fitness of COTS software candidates are identified and retained in the list. Selecting one or more of COTS software can then be done from the list for integrating in the application [2]. In Keystone Strategy, the COTS software candidates are estimated against a key characteristic [29]. So the key characteristics (e.g. vendor location, type of technology) are identified at the beginning of this strategy, then the searching for COTS software will be based on satisfying this keystone characteristic. This strategy is applied at the beginning stages of the evaluation in order to permit quick removal of the large number of COTS candidates that do not satisfy the keystone characteristic [32]. The idea of Puzzle Assembly Strategy is taken from collecting pieces of a puzzle [39]. This strategy assumes that when selecting the COTS software we must consider the fitting of the COTS software with other components on the system [29,2]. In other words, COTS software that can be considered as fitness in isolation might be not acceptable when assembled with other components in the system. Therefore, in this strategy, choosing COTS software must be done by considering the other components requirements in the puzzle. Mohmad et al in [39] argues that more than one strategy from the above can be used with the same project. For example, the keystone can be used at the beginning of the project to eliminate the largest possible number of COTS candidates, and then the progressive filtering can be used later on. Oberndorf also proposed that more than one of the strategies above can be employed in the same project [47]. For example, a developer might use keystone identification first and then progressive filtering later. Multiple criteria have to be considered during the selection of components for software development. A balance between technical characteristics, financial issues and application requirements is required. Many authors have proposed different methods of selection of COTS components for development of CBSS. One of the first proposals was given by Kontio et al in [26] they proposed the OTSO (Off-The-Shelf Option) approach for COTS selection. The authors developed a method that addresses the selection process of packaged, reusable off-the-shelf software. The OTSO approach supports the search, evaluation and selection of software components. In 1996, Kontio published several follow-up papers to elaborate OTSO [27]. An approach called PRISM (Portable, Reusable, Integrated, Software Modules) was proposed by Lichota et al, [34]. In their approach a generic component architecture was proposed that can be used during COTS evaluation process. However, it was not until 1998 that another important milestone was reached with the Procurement-Oriented Requirements Engineering approach (PORE) [37]. The importance of PORE is that it proposed a requirements engineering process for COTS-based development. PORE suggested that requirements should be elicited and analyzed at the same time when the COTS products are evaluated. The STACE (Social-Technical Approach to COTS Evaluation) approach [32] emphasized the importance of non-technical issues, e.g., social, human, and organizational characteristics, during the evaluation process. Ochs et al [42] proposed the COTS acquisition process (CAP) which highlighted the concept of a "tailorable evaluation process". The approach suggested that the evaluation process should be tailored based on the available effort for project and it relied on experts' knowledge to tailor the process. In 2001, a project was initiated by Chung et al, [11,12,13,14] to describe a COTS -Aware Requirements Engineering (CARE) Process. CARE uses a flexible set of requirements based on different agents' knowledge. For the same, CARE proposes a method to define relevant agents as well as the system goals and requirements. The PECA (Plan, Establish, Collect and Analyze) approach was proposed by Dorda et al [17] from Software Engineering Institute (SEI) describes a detailed tailorable COTS selection process and gives guidelines which the experts can use to tailor the process. In 2002 [35] proposed the Balanced Reused Model (BAREMO) approach. This approach explains in detail how a decision can be made based on Analytical Hierarchy Process (AHP) [43]. The Combined Selection (CS) approach [9] is used to select multiple COTS products that all together satisfy the requirements. This approach performs its activities at two levels: local and global. The global level addresses the overall process of the combined selection, fires individual selection processes for each area and tries to find the best overall combination of products. The local level use existing COTS evaluation and selection techniques e.g. OTSO [26] or PORE [37] to select individual COTS that are combined at the global level. The approach by Erol and Ferrell, [18] is an evaluation approach that supports selecting a COTS product from a finite set of products based on more than one objective and a set of quantitative (e.g cost) and qualitative (e.g. linguistic variables) data. The approach uses fuzzy QFD (Quality Function Deployment) [1] to collect and quantify the qualitative data. Then goal programming is used to get near optimal solutions to the decision maker. # DesCOTS (Description, Evaluation and Selection of COTS components) approach was presented by Grau et al, [20]. DesCOTS system includes a set of tools that can be used to evaluate the COTS products based on quality models. Shyur in [45] in his work models the COTS evaluation problem as Multi-criteria Decision Making (MCDM) problem. He proposes a five phase COTS selection model, combining the technique of ANP (Analytic Network Process) and modified TOPOSIS (technique for order performance by similarity to idea solution). ANP is used to determine the weights of multiple evaluation criteria. The modified TOPOSIS is used to rank competing products in terms of their overall performance. In 2007, the MiHOS (Mismatch-Handling aware COTS selection) approach was developed [39]. The approach focuses on handling the mismatches between COTS candidates and the requirements. MiHOS uses techniques such as linear programming to identify near optimal solutions. In 2007, an interactive decision support approach for multi-objective COTS selection was addressed by [40]. Authors have introduced a twophase decision support approach for selection of COTS products. Couts et al [16] have shown that the evaluation and selection of COTS software is performed using adhoc manners. Tarawneh et al [46] proposed a framework to support and improve the COTS software evaluation and selection processes in industry. To achieve this objective the authors have shown that specific objectives have to be addressed: # 1. Identify the processes which support COTS software evaluation and selection. 2. Determine the criteria or requirements which are important for successful evaluation and selection process. 3. Propose methods and techniques to address the mismatch between COTS features and customer requirements. 4. Develop a repository to manage information from previous selection cases that support the decision making process. Mead N. R. in [38] developed Software Quality Requirements Engineering for Acquisition (A-SQUARE) methodology for eliciting and prioritizing security requirements as part of the acquisition process. The author in her report, evaluated the effectiveness of the A -SQUARE method by applying it to a COTS product for the advanced metering infrastructure of a smart grid. # Global Journal of C omp uter S cience and T echnology # b) Optimal COTS Selection Approaches Several optimization models have been proposed in the literature for the optimal selection of the software components for the development of safe and reliable software systems. The models use basic information on components reliability and cost and allow the trade-off between two factors. Ashrafi and Berman, [4] presented two models which address the tradeoff between reliability and cost. The model is applied to large software packages that consist of several programs. The models can be used as decision support tools for organizations that are in the process of purchasing of variety of computer programs in order to meet the needs of the users, e.g. operations people need software packages to perform functions such as scheduling, inventory control and purchase orders. While the main consideration is to attain high average reliability for the software package, management has to consider both the relative importance of each program in terms of the frequency of the usage of their corresponding function. Programs can be purchased from software development companies. Several programs are usually available for each function. Each program has a known market cost and an estimated reliability. It can be noted here that the assumption of using the ready programs available in the market to make the software package implies the use of COTS (Commercial-off-the-shelf) program. Hence the models are applicable to only those software packages that are designed using COTS products. The authors have formulated two types of models one which does not consider redundancy for performing each function and the other which maintains redundancy under budget limits. Considering the concept of COTS in a software development and the availability of mathematical models to access module reliability, it is possible to have information on module reliability and cost. In their previous work [5] authors used optimization models to determine the redundancy level of a software package consisting of several independent functions where each function is performed by a program with known reliability and cost. In this work, however, they broke down this approach one step further and deals with software systems consisting of one or more programs where each program consists of series of modules, which upon sequential execution will perform a function. Four models are presented, each applicable to a different software system structure. The optimization models discussed in the previous sections don't consider any of the fault tolerance schemes such as recovery block or NVP. They merely consider the programs consisting of set of modules, which on sequential execution perform the function. Berman and Kumar,[7] studied the problem of optimum selection of component for the recovery blocks for the first time. The author presented optimization models for a fault tolerant software system. Specifically they have formulated optimization problems for two types of recovery blocks namely -Independent and Consensus recovery block schemes. The optimal component selection problem addressed by Kapur et al, [25] considers software built by assembling COTS component performing multiple functions. Each function is performed by calling a set of modules. Modules can be assembled in a recovery block scheme to provide the fault tolerance. Again for each alternative version multiple choices are available from the supplier with distinct reliability and cost. The version for any alternative choice having higher reliability has higher cost. Two models are formulated for weighted maximization of system reliability, weights being decided with respect to access frequency of functions within the available budget. Each module is comprised with a set of COTS alternative that are available in the market. Cortellessa et al, [15] introduce a framework that helps developers to decide whether buying or building components for certain software architecture. Once built software architecture, each component can be either bought, or probably adapted to the new software system, or it can be developed in-house. This is a "build-or-buy" decision that affects the software cost as well as the ability of the system to meet its requirements. Gupta et al, [22] in their work have formulated fuzzy multi-objective optimization model for selection of COTS components for development of a modular software system. The hierarchy structure of the software consists of three programs, four modules and eleven COTS products. Some specific functions of each program can call upon a series of modules, and several alternative COTS products are available for each module. Different weights are assigned to different modules using an AHP technique. The issue of compatibility amongst the COTS products is also discussed. Kwong et al, [31] have addressed an optimization concept of selection of software components using intra-modular coupling density (ICD) and functional objective along with few system constraints of ICD and functionality without redundancy for a CBSS development. Jha et al, [24] formulated multi-criteria problem for minimizing the overall cost and maximizing the system reliability by using fuzzy multi objective optimization model for selecting the best COTS software product among alternatives of each module for modular software system. Jha et al, [23] formulated a multi-objective problem with cost minimization and reliability maximization as the two objectives with an upper bound on cost and lower bound on reliability. This model was formulated for Recovery Block and Consensus Recovery Block fault tolerant software. Author used goal programming approach for solving the problem. Kumar et al, [28] discussed an effective approach to formulate multi-objective problem with cost minimization and reliability maximization as the two objectives with an upper bound on cost and lower bound on reliability. The author used goal programming approach for solving the problem based on Consensus Recovery Block Scheme. Bali et al, [6] proposed optimization models for optimal component selection for a fault-tolerant modular software system under the consensus recovery block scheme. It is necessary to identify critical modules in the design of a fault-tolerant modular software system and also to develop a system with a built in redundancy for critical modules. During the planning phase of software development, it is necessary that modules are categorized and identified based on their reusability and criticality to run the application. In order to achieve this, a constraint on criticality of modules can be used to achieve the effective redundancy for all critical modules and at least one effective alternative for non-critical modules. An attempt has been made in this paper to review and critically analyze the COTS evaluation and selection process. Reviewed papers are categorized into two themes: evaluation and selection. We have tried to explore the COTS evaluation and selection practices and compare the most significant approaches. The objective of this study is to identify currently used decision making practices of COTS evaluation and selection. In future the authors would like to propose a framework for COTS selection which is relevant to today's era of software development approach. Moreover, the models proposed in the literature have not talked about build-or-buy strategy in depth and issue of criticality in specific. # Global # Global ![Journal of C omp uter S cience and T echnology Volume XV Issue VI Version I Year 2015 ( ) C](image-2.png "") 1Year 2015Volume XV Issue VI Version I( )C 2Ref. No.YearAuthorsObjective FunctionApproach1.4.1992Ashrafi & BermanReliabilityInteger Programming Problem2.5.1993Ashrafi & BermanReliabilityInteger Programming Problem3.7.1999Berman & KumarReliabilityInteger Programming Problem4.25.2003Kapur et alReliabilityInteger Programming Problem5.15.2008Cortellessa et alCostInteger Programming ProblemFuzzy Multi Objective Optimization6.22.2009Gupta et alQuality, CostModelICD and Functionality without7.31.2010Kwong et alIntra-modular coupling densityRedundancy8.24.2010Jha et alReliability, CostMulti Objective Optimization ModelFuzzy Multi Objective Optimization9.23.2011Jha et alReliability, CostModel10.28.2012Kumar et alReliability, CostGoal ProgrammingCost, Development Efforts,Execution Time, Reliability and11.21.2012Gupta et alQualityFuzzy Interactive ApproachFuzzy Multi Objective Optimization12.6.2014Bali et alReliability, CostModel © 2015 Global Journals Inc. (US) * Quality Function Deployment: Integrating Customer Requirements into Product Design YAkao 1999 Productivity Press Cambridge, MA * CRE: A Systematic Method for COTS Components Selection CAlves JCastro XV Brazilian Symposium on Software Engineering (SBES) Ed: Rio de Janeiro, Brazil 2001 * Investigating Conflicts in COTS Decision Making CAlves AFinkelstein International Journal of Software Engineering and Knowledge Engineering 13 5 2003 * Optimal design of large software systems considering reliability and cost AAshrafi OBerman IEEE Transactions on Reliability 41 2 1992 * Optimal design of large software systems using Nversion programming AAshrafi OBerman MCutler IEEE Transactions on Reliability 43 2 1993 * Fuzzy Multi-Objective Build-or-Buy Approach for Component Selection of Fault Tolerant Software System under Consensus Recovery Block Scheme with Mandatory Redundancy in Critical Modules SBali PCJha UDKumar HPham International Journal of Artificial Intelligence and Soft Computing, Inder-science 4 2/3 2014 * Optimization models for recovery block schemes OBerman UDKumar European Journal of Operational Research 115 1999 * A Method for Compatible COTS Component Selection JBhuta BBoehm ICCBSS 3412 2005 Springer LNCS * Combined Selection of COTS Components XBurgues CEstay XFranch JAPastor CQuecr ICCBSS'02 Orlando, Florida 2002 * Filtering COTS Components Through an Improvement-Based Process ACechich MPiattini ICCBSS 3412 2005 Springer LNCS * Matching, Ranking and Selecting COTS components: A COTS-Aware Requirements Engineering Approach Chung KCooper MPEC'04 Scotland, UK 2004b * LChung KCooper TR UTDCS-23-01 A COTS -Aware Requirements Engineering (CARE) Process: Defining System Level Agents, Goals and Requirements Dallas 2001 Department of Computer Science, The University of Texas * Knowledgebased COTS-aware Requirements Engineering Approach LChung KCooper SEKE'02 Ischia, Italy 2002 * Defining goals in a COTS -Aware Requirements Engineering Approach LChung KCooper Systems Engineering Journal 7 1 2004a * An Optimization Framework for Build-or-Buy Decisions in Software Architecture VCortellessa FMarinelli PPotena Journal of Computers and Operational Research 35 2008 * Integrating COTS software: Lessons from a Large Healthcare Organisation CCouts PGerdes IT professional 12 2010 * A Process for COTS Software Product Evaluation SCDorda JDean EMorris POberndorf ICCBSS 2255 2002 Springer LNCS * A Methodology for Selection Problems with Multiple, Conflicting Objectives and both Qualitative and Quantitative Criteria IErol WGFerrell International Journal of Production Economics 86 3 2003 * Achieving Quality Requirements with Reused Software Components: Challenges to Successful Reuse DGFiresmith MPEC'05 St. Louis, Missouri 2005 * DesCOTS: A Software System for Selecting COTS Components GGrau JPCarvallo XFranch CQuer EUROMICRO'04 Rennes, France 2004 * COTS Selection using Fuzzy Interactive Approach PGupta MKMehlawat SVerma Optimization Letters 6 2012 * A Hybrid Approach for Selecting Optimal COTS Products PGupta MKMehlawat GMittal SVerma ICCSA 2009, Part I OGervasi Springer-Verlag 2009 5592 * Fuzzy Approach for Selecting Optimal COTS based Software Products under Consensus Recovery Block Scheme PCJha SBali PKKapur BVICAM's International Journal of Information Technology (BIJIT) No. 0973-5658 3 1 2011 * Optimal Component Selection of COTS Based Software System under Consensus Recovery Block Scheme Incorporating Execution Time PCJha PKKapur SBali UDKumar International Journal of Reliability, Quality and Safety Engineering (IJRQSE) 17 3 2010 * Optimal Reliability allocation Problem for a Modular Software System PKKapur AKBardhan PCJha OPSEARCH, Journal of Operational Research Society of India 40 2 2003 * OTSO: A Systematic Process for Reusable Software Component Selection JKontio CS -TR -3478 1995 Maryland University of Maryland * Defining Factors, Goals and Criteria for Reusable Component Evaluation JKontio GCaldiera VRBasili CASCON'96 Toronto, Ontario, Canada IBM Press 1996 * Optimal Component Selection Problem for Cots Based Software System under Consensus Recovery Block Scheme: A Goal Programming Approach DKumar PCJha PKKapur UDKumar International Journal of Computer Applications 47 4 2012 * STACE: Social Technical Approach to COTS Software Evaluation DKunda Lecture Notes in Computer Science 2003 * Applying Social-Technical Approach For COTS Selection DKunda LBrooks Proceedings of the 4th UKAIS Conference the 4th UKAIS ConferenceYork, McGraw Hill 1999 * Optimization of Software Components Selection for Component-based Software System Development CKKwong LFMu JFTang XGLuo Journal of Computers and Industrial Engineering 58 2010 Elsevier * COTS Selection Best Practices in Literature and in Industry RLand LBlankers MChaudron ICrnkovic Proceedings of 10th International Conference on software Reuse 10th International Conference on software ReuseBeijing, China 2008 * A Formal Process for Evaluating COTS Software Products PKLawlis KEMark DAThomas TCourtheyn IEEE Computer 34 5 2001 * PRISM Product Examination Process for Component Based Development RWLichota RLVesprini BSwanson Proceedings Fifth International Symposium on Assessment of Software Tools and Technologies Fifth International Symposium on Assessment of Software Tools and Technologies IEEE 1997 * BAREMO: How to Choose the Appropriate Software Component using the Analytical Hierarchy Process TALozano PAGomez SEKE'02 Italy 2002 * The Impact of Acceptance Tests on Analyzing Component-Based Systems Specifications: An Experimental Evaluation SMahmood 10th IEEE International Conference on Computer and Information Technology CIT 2010. 2010 * Acquiring COTS Software Selection Requirements NAMaiden CNcube IEEE Software 15 2 1998 * An Evaluation of A-SQUARE for COTS Acquisition NRMead 2014 SEI Carnegie Mellon University Technical report * Decision Support for Handling Mismatches between COTS Products and System Requirements AMohamed GRuhe AEberlein ICCBSS'07 Banff, Canada 2007 * Interactive Decision Support for Multi-objective COTS Selection TNeubauer CStummer Proceedings of the 49 th Hawaii International Conference on System Sciences the 49 th Hawaii International Conference on System Sciences 2007 * Workshop on COTS Based Systems PAOberndorf LBrownsword EMorris CSledge CMU/SEI-97-SR-019 1997 SEI Institute ; CMU special report * A COTS Acquisition Process: Definition and Application Experience MOchs DPfahl GDChrobok BKNothhelfer ESCOM- SCOPE'00 2000 Munich, Germany * The analytical hierarchy process TLSaaty 1990 McGraw-Hill New York * Evaluating COTS Components Using Gap Analysis JSheng BWang 10.1109/ICYCS.2008.472 The 9th International Conference for Young Computer Scientists IEEE 2008 * COTS Evaluation using Modified TOPOSIS and ANP HJShyur Journal of Applied Mathematics and Computation 177 2006 * Evaluation and Selection COTS Software Process: The State of Art FTarawneh FBaharom JHYahaya FAhmad international journal on New Computer Architecture and Their Applications (IJNCAA) 1 2 2011 * MRVigder WMGentleman JDean COTS Software Integration: State of the Art 1996 39198 * Towards Providing Decision Support for COTS Selection TWanyama BFar The proceedings of the Canadian Conference on Electrical and Computer Engineering (CCECE 2005) Saskatoon Saskatchewan, Canada 2005. May 1-4