# Introduction ireless network is a novel technology involving the deployment of hundreds of low-cost, microhardware, and resource-limited sensor nodes. Wireless technologies are becoming increasingly ubiquitous in modern networks; however, this new technology comes with its own set of challenges. Wireless networks are inherently 'open' and viewable by all network scanners. There are no physical barriers between data sent through the air. As such, it is relatively easy to intercept data packets in a wireless network. The biggest concern with wireless network is its security, for some time wireless has had very poor, if any, security on a wide-open medium. Wireless Intrusion Detection System (WIDS) is a new solution to help Author ? : Rupinder Singh, Department of Computer Science, Khalsa College, Amritsar, Punjab, India. E-mail : rupi_singh76@yahoo.com Author ? : Dr. Jatinder Singh, Principal, Golden College of Engg. & Tech., Gurdaspur, Punjab, India. E-mail : bal_jatinder@rediffmail.com combat this problem. An Intrusion Detection System (IDS) is a device or software application that monitors network and/or system activities for malicious activities or policy violations and produces reports to a management station (Wikipedia, 2012). A wireless IDS performs this exclusively for the wireless network. This system monitors traffic on network looking for and logging threats and alerting personnel to respond. Lord Kelvin said "If you cannot measure it, you cannot improve it". This fact also applies to wireless network security issues. An activity cannot be managed if it cannot be measured, this is a widely accepted management principle and security falls under this rubric. Metrics can be an effective tool for detecting the capability of a wireless IDS. Metrics can help in raising the level of security awareness within the network. Security metrics that are related to wireless network are hard to generate because the discipline itself is still in the early stages of development. There is not yet a common vocabulary and not many documented best practices to follow [4]. In this paper we provide a performance metrics scorecard based approach to evaluate intrusion detection systems that are currently popular for wireless network in the commercial sector. We describe a testing methodology we developed to evaluate Wireless IDS by assigning score to various performance metrics concern with wireless IDS. The approach followed in this paper do not compare wireless IDS against each other, but against a set of performance metrics concern with wireless IDS. The generalized approach of this paper will allow systems with any wireless requirements to tailor evaluation of ID technologies to their specific needs. Since evaluation is against a static set of performance metrics the evaluation may be extended for other metrics like logistical metrics, architectural metrics, quality metrics etc. The standard approach of comparison used in this paper also gives us scientific repeatability. # II. # Snort, airdefense guard and kismet wireless ids In order to explain performance metrics scorecard based evaluation approach to wireless IDS, Snort works by implementing a detection engine that allows registering, warning, and responding to attacks previously defined. Snort is available under GPL (General Public License) and runs under Windows and GNU/Linux. It is among the most widely used, has a number of predefined signatures and continuously updated. Snort can be configured in three modes namely sniffer, packet logger, and network intrusion detection. In addition to all of these basic Snort features, Snort can be set up to send real-time alerts. This provides with the ability to receive alerts in real time, rather than having to continuously monitor Snort system. Snort is like a vacuum that takes packets and allows doing different things. Center piece of testing and evaluating wireless IDS will be a "scorecard" containing the set of performance metrics and their definitions. Each metric can have low (+), average (++), or high (+++) score, where higher scores will be interpreted as more favorable ratings. The performance metrics used are general characteristics that are relevant to the design of wireless IDS. The method used for observing each performance metric value can be either analysis (source code analysis) or open source material (such as specifications, white papers or reviews provided by vendors or users). We use open source material to analyze each performance metrics for wireless IDS. We examine publicly available research papers, reports, product documentation, published conference material (proceedings) and other material available for public review. b) Performance Metrics for Wireless IDS Performance metrics are used to measure the ability of a Wireless IDS to perform a particular task and to fit within the performance constraints. These metrics measure and evaluate the parameters that impact the performance of the wireless IDS [15]. The metrics defined in this area are shown in Table 1. Table 1 In this section of the paper we will apply above mentioned approach to popular wireless IDS Snortwireless, AirDefense Guard, and Kismet. We choose these three for evaluation as they are most widely used and have different ways of working. Below with table 2 we describe how scores to performance metrics related to these three wireless IDS are assigned. Performance metric False Positive Ratio can be assigned score depending on the following criteria: The weighted average of False Positive and False Negative ratios. Induced Traffic Latency It measures the delay in the arrival of packets at the target network in the presence and absence of a wireless IDS. # Stress Handling and Point of Breakdown The point of breakdown is defined as the level of network or host traffic that results in a shutdown or malfunction of IDS. # IDS Throughput This metric defines the level of traffic up to which the IDS performs without dropping any packet. # Depth of System's Detection Capability It is defined as the number of attack signature patterns and/or behavior models known to it. # Reliability of Attack Detection It is defined as the ratio of false positives to total alarms raised. # Possibility of Attack It is defined as the ratio of false negatives to true negatives. # Consistency It is defined as the variations in the performance of a wireless IDS. # Error Reporting and Recovery The ability of a wireless IDS to correctly report and recover. # Firewall Interaction The ability of a wireless IDS to interact with the Firewall systems. # User Friendliness The ability of a wireless IDS to configure according to user's environment. # Router Interaction Degree of interaction of a wireless IDS with the router. # Compromise Analysis It is the ability to report the extent of damage and compromise due to intrusions. # Simple Network Management Protocol (SNMP) Interaction Ability of the wireless IDS to send an SNMP trap to one or more network devices in response to a detected attack. # Timeliness Average/maximal time between an intrusion's occurrence and its being reported. Table 2 : Scorecard for Snort, AirDefense Guard and Kismet wireless IDS Snort-Wireless is the most advanced Open Source Wireless IDS. It uses the sequence number analysis technique to detect false frame attacks. In [19] authors tested the effectiveness of the Snort-Wireless with the used data applying the purposed analysis technique. It is not capable of identifying the malicious packets as the threshold-based technique used by Snort-Wireless is prone to false negatives. Table 3 provides the results produced by authors. AirDefense Guard wireless IDS produces very low false negative ratio as it has ability to detect 200+ attacks and policy violations. Kismet has less attack definitions and produces average false negative ratio. Performance metric Cumulative False Alarm Rate can be assigned score depending on the following criteria: Inside the packet processing function, Snort performs several tasks. First, it calls into libpcap using the pcap_dispatch function to process any waiting packets. For each packet that is available, libpcap calls the Pcap Process Packet function, which handles the actual packet processing. This function resets several per-packet counters, collects some statistics about the packet, and calls Process Packet. The Process Packet function handles all of the details of decoding the packet, printing the packet to the screen and either directly calling the packet logging functions or calling into the pre-processors. If no packets are available, Snort performs basic housekeeping chores such as checking for pending signals. In order to perform all this functions, Snort-Wireless IDS delays the arrival of packets at the target network. Air Defense has the most detailed available wireless forensic database in the industry. It has more than 300 wireless statistics per device per minute logged and has instant analysis using the forensic wizard [20]. The point of breakdown is defined as the level of network or host traffic that results in a shutdown or malfunction of IDS. Air Defense. Kismet IDS identifies networks by collecting passively packets The detection engine is the time-critical part of Snort wireless. Depending upon how powerful user machine is and how many rules have been defined, it may take different amounts of time to respond to different packets. If traffic on the network is too high when Snort wireless is working in NIDS mode, it may drop some packets and may not get a true real-time response. The load on the detection engine of snort wireless depends upon the following factors: ? Number of rules ? Power of the machine on which Snort is running ? Speed of internal bus used in the Snort machine ? Load on the network Motorola Air Defense utilizes its 24x7, real-time monitoring of the 802.11a/b/g networks for most accurate intrusion detection of known as well as unknown attacks and does not easily breaks down. Kismet is able to handle stress up to some extent. Performance metric IDS Throughput can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS regularly drops packets. ? Average Score (++): Wireless IDS rarely drops packets. ? High Score (+++): Wireless IDS can perform without dropping any packet. When Snort wireless is working in Inline mode, it works like an Ethernet bridge, that is, in order to monitor a network segment, it has to be inserted transparently with two bridged NICs. With this setup, any packet can flow through the bridge from a network card to the other, unless it matches the drop rules; in that case, the switch opens and blocks the packet. So, Snort wireless drops packet only when it matches the drop rule specified by the user. Studies have shown that Air Defense rarely drops packets. Kismet processes data rate as supported by access point and drops more packets than others. Performance metric Depth of System's Detection Capability can be assigned score depending on the following criteria: Snort wireless maintains a rule set in order to have the latest detection capabilities. Sourcefire Vulnerability Research Team (VRT) Rules are the official rules of snort wireless. One of the best features of Snort is its rule engine and language. Snort's rule engine provides an extensive language that enables user to write their own rules, allowing them to extend it to meet the needs of their own network. Motorola Air Defense wireless IDS utilizes 24x7, real-time monitoring of the 802.11a/b/g networks for producing most accurate intrusion detection of known and unknown attacks. It has ability to detect 200+ attacks and policy violations [20]. Kismet has less number of attacks detections as compare to snort wireless and Air Defense guard. Performance metric Reliability of Attack Detection can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS generates high ratio of false positives to total alarms raised. frames. So, Wireless IDS snort generates high ratio of false negatives to true negatives. Air Defense guard has ability to detect 200+ attacks and policy violations and therefore produces less false positives and it generates low ratio of false negatives to true negatives. Kismet has average Possibility of Attack as it has average false negative ratio. Performance metric Consistency can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS has high variations in the performance. ? Average Score (++): Wireless IDS has average variations inthe performance. ? High Score (+++): Wireless IDS has low or no variations in the performance. In [20] author evaluated two open source network based intrusion detection systems. Snort wireless performed well during tests, but did produce false positives and false negatives. Snort is very lightweight and fast but is limited in its ability to scale in bandwidth per instance. Studies show that Air Defense kismet has average variations in the performance. Performance metric Error Reporting and Recovery can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS has low or no ability to correctly report and recover. ? Average Score (++): Wireless IDS has average ability to correctly report and recover. ? High Score (+++): Wireless IDS has high ability to correctly report and recover. Snort wireless generates reports that show what happened during the last day, week or month. -T option of snort wireless is very useful for testing and reporting on the Snort configuration. This option can be used to find any errors in the configuration files. Snort wireless provides tool that gives user a detailed report of actions taken during the update process. SPADE module keeps a record of history data and uses threshold values to report the anomalies. Air Defense guard has flexible alerting and reporting options with integration capabilities into the various Security Information Management (SIM) systems [20]. Error Reporting and Recovery of kismet is poor as compare to snort wireless and Air Defense. Performance metric Firewall Interaction can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS has poor interaction with the Firewall systems. ? Average Score (++): Wireless IDS has average interaction with the Firewall systems. ? High Score (+++): Wireless IDS has excellent interaction with the Firewall systems. Snort Sam is a tool used to make Snort work with most commonly used firewalls. It is used to create a Firewall/IDS combined solution. Firewall can be configured to automatically block offending data and addresses from entering system when intruder activity is detected. It is available from http://www.snortsam.net/ where one can find the latest information. The tool consists of two parts: 1. A Snort output plug-in that is installed on the Snort sensor. 2. An agent that is installed on a machine close to Firewall or Firewall itself. Snort communicates to the agent using the output plug-in in a secure way. Air Defense Guard supports stateful Layer 2 and rolebased firewalls and base security policy on the user, group, location, encryption strength, etc. studies show that kismet also has good Firewall Interaction. Performance metric User Friendliness can be assigned score depending on the following criteria: ? Low Score (+): It is difficult to configure wireless IDS according to user's environment. ? Average Score (++): Wireless IDS can be configured up to some extent according to user's environment. ? High Score (+++): Wireless IDS can be easily configured according to user's environment. In snort wireless a thorough understanding of what snort. conf file is and how to configure it is essential to a successful deployment of Snort wireless as an IDS in user environment. Snort configuration consists of Global configuration (snort. conf), Optional *.rules file(s), and Additional files. Air Defense Guard is very user friendly as it provides location tracking of the devices on a map, and provides minute by minute granular forensic information for any of the device. Kismet only runs under LINUX and does not have easy to use graphical interface. Performance metric Router Interaction can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS has a poor interaction with the router. ? Average Score (++): Wireless IDS has an average interaction with the router. ? High Score (+++): Wireless IDS has excellent interaction with the router. Depending upon the type of router used, snort wireless can be used on a port. Some routers, like Cisco, allow to replicate all ports traffic on one port where snort machine can be attached. These ports are usually referred to as spanning ports. The best place to install Snort wireless is right behind the firewall or router so that all of the Internet traffic is visible to Snort before it enters any router or hub. Air Defense Guard provides nice router interaction. Kismet does not have good interaction with some of the routers like belking54g. Performance metric Compromise Analysis can be assigned score depending on the following criteria: In snort wireless snort SnmpPlugin is used to send snmp alerts to network management systems (NMS). The alerts can be traps or informs. This adds to significant power of the NMS by allowing it to monitor security of the network. It also allows snort wireless sensor to exploit the features that are built into the existing network management systems. Air Defense Guard eliminates many of vulnerabilities impacting the security of the wireless network by providing good interaction with SNMP. Kismet provides various utilities for configuring and monitoring of wireless Access Points under Linux using SNMP protocol. Performance metric Timeliness can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS takes a lot of time to report the occurrence of an intrusion. ? Average Score (++): Wireless IDS takes average time to report the occurrence of an intrusion. ? High Score (+++): Wireless IDS takes a minimal time to report the occurrence of an intrusion. Snort wireless is a packet-based system. The basic life of a packet inside snort starts with packet acquisition. Once the packet is inside snort it is passed into the packet decoder. After decoding, the packet is passed on to the pre-processors for normalization, statistical analysis, and some nonrule-based detection. Once the pre-processors are done with the packet it goes into the detection engine, where it is evaluated against all of the rules that were loaded from the configuration file. Finally, the packet is sent off into the output plug-ins for logging and alerting. So, it takes lot of time for snort wireless to detect an attack. Air Defense Guard takes average time for reporting of intrusion as it has 200+ attacks and policy violations detection capability. Studies show that kismet is slow in detection as compare to snort wireless and Air Defense. # Conclusion and future work Wireless IDS are used in detecting unwanted activities on a wireless network. Performance metrics can be used to measure the performance of a wireless IDS within the performance constraints. These metrics measure and evaluate the parameters that impact the performance of the wireless IDS. This paper provides a performance metrics scorecard based approach that can be used for evaluating a wireless IDS in order to find out how it behaves within performance constraints. In this paper we provide various performance metrics concern with wireless IDS and a scorecard method for evaluation. Evaluation of a wireless IDS is done by assigning scores to various performance metrics. We use our evaluation methodology to test popular wireless IDS Snort, Air Defense Guard, and Kismet. We define commonly used performance metrics that are important to a wireless IDS, but a lot is required to be done to find out more ones like analysis of intruder intent, clarity of reports, effectiveness of generated filters, evidence collection, information sharing, user alerts, program interaction, session recording and playback, threat correlation, trend analysis, extendibility, adaptability, scalability, overhead, and latency. More performance metrics and their definitions can be defined as lessons are learned while evaluating a wireless network. Future work also includes applying the evaluation methodology to other metrics concern with wireless IDS like logistical metrics, architectural metrics, quality metrics etc. 2012![Global Journals Inc. (US) Global Journal of Computer Science and Technology Volume XII Issue XII Version I wireless IDS namely Snort-wireless, Air Defense Guard, and Kismet as these are one of the most popular and works on different technology. a) Snort Wireless IDS Snort wireless is an open source network intrusion detection and prevention system (IDS/IPS) that combines the benefits of signature, protocol, and anomaly-based inspection, and is the most widely deployed IDS/IPS technology worldwide. With millions of downloads Snort has become the de facto standard for IDS/IPS [8]. Snort-wireless allows for custom rules to be created based on framing information from a wireless packet. It also contains rules to attempt to find rogue access points, war drivers, and ad hoc networks.](image-2.png "W © 2012") 3![Snort-wireless alert results (a) during attack with low traffic. (b) during an attack with high traffic[19] Low Score (+): Wireless IDS generate high Cumulative Alarm Rate. ? Average Score (++): Wireless IDS generate average Cumulative False Alarm Rate. ? High Score (+++): Wireless IDS generate low or no As discussed above Snort-wireless and AirDefense Guard produces low false positive rate and high false negative rate, therefore they get average score for the metric Cumulative False Alarm Rate. Kismet produces average false positive and false negative ratios and therefore generates average score for Cumulative False Alarm Rate. Performance metric Induced Traffic Latency can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS presence highly delays the arrival of packets at the target network. ? Average Score (++): Wireless IDS presence delays the arrival of packets at the target network. ? High Score (+++): Wireless IDS presence has no or little delay in the arrival of packets at the target network.](image-3.png "Table 3 :?") 1Year2Volume XII Issue XII Version ID D D D )(Global Journal of Computer Science and Technology© 2012 Global Journals Inc. (US) Performance MetricsDescriptionFalse Positive RatioCumulative FalseAlarm Rate3Volume XII Issue XII Version ID D D D ) E(signature, and anomaly-based inspection and produces low false positive ratio and gets a high score for this metric [8]. Air Defense guard has ability to detect 200+ attacks and policy violations and therefore produces less false positives. Kismet alert PROBENOJOIN can result excessive false positives while channel hopping is done. False positives are also possible in noisy/lossy situations, it is desirable to disable this alert in some installations [12]. Performance metric False Negative Ratio can be assigned score depending on the following criteria: ? Low Score (+): Wireless IDS generate high False Negative Ratio. ? Average Score (++): Wireless IDS generate average False Negative Ratio. ? and detecting standard named networks, detectinghidden networks, and inferring to the presence ofPerformance Metrics networks (non-beaconing) via data traffic. Snort wireless AirDefense Performance metric Stress Handling and Point Kismet Guard of Breakdown can be assigned score depending on the False Positive Ratio +++ +++ ++ following criteria:False Negative ? Low Score (+): Wireless IDS cannot handle stressRatio and easily breakdowns. +++++2012 Year? Average Score (++): Wireless IDS can handle Cumulative False Alarm Rate ++ ++ ++ stress up to some extent before it breakdowns. ? High Score (+++): Wireless IDS can handle stress Induced Traffic Latency ++ +++ ++ Stress Handling at the maximum and can avoid breakdown.4Breakdown and Point of+++++++IDS Throughput++++++++Depth ofSystem's++++++++DetectionCapabilityReliability ofAttack Detection+++++++Possibility ofAttack+++++Consistency++++++Error ReportingD D D D ) Eand Recovery Firewall++++++++(Interaction+++++++++UserFriendliness++++++RouterInteraction++++++++CompromiseAnalysis++++++SNMPInteraction++++++++Timeliness++++++ © 2012 Global Journals Inc. (US)Global Journal of Computer Science and Technology © 2012 Global Journals Inc. (US) * A Metrics Based Approach to Intrusion Detection System Evaluation for Wireless Network RupinderSingh DrJatinderSingh International Journal of Education and Applied Research IJEAR * Jul.-Dec, 2011 1 * A Metrics -Based Approach to Intrusion Detection System Evaluation for Distributed Real-Time Systems GAFink BLChappell TGTurner KFO'donoghue WPDRTS April 2002 * HarrykarFreelance "Harrykar's Techies Blog Snort Ids .. .Ips Beyond May 2009 31 * A comparative analysis of the snort and suricata intrusion-detection systems EugeneAlbin September 2011 * SNORT Users Manual 2.9.0, The Snort Project March 25, 2011 * Design of a Snort-Based Hybrid Intrusion Detection System JGómez CGil NPadilla1 RBaños CJiménez IWANN 2009, Part II LNCS SOmatu 2009 5518 * Intrusion Detection Systems with Snort Advanced IDS Techniques Using Snort, Apache, MySQL, PHP, and ACID Rafeeq Ur Rehman Prentice Hall * On The Feasibility of Utilizing Security Metrics in Software Intensive Systems ReijoSavola IJCSNS International Journal of Computer Science and Network Security 10 1 January 2010 * Wireless Intrusion Detection System SnehalBoob PriyankaJadhav International Journal of Computer Applications 5 8 August 2010 * A Performance Analysis of Snort and Suricata Network Intrusion Detection and Prevention Engines JDavid MBenjamin ICDS 2011 : The Fifth International Conference on Digital Society 2011 ISBN * Performance analysis Toolset for wireless intrusion detection systems SamerFayssal 2010 International Conference on High Performance Computing and Simulation (HPCS) Caen, France, ISBN * A Testbed for Quantitative Assesment of Intrusion Detection Systems Using Fuzzy Logic GautamSingaraju LawrenceTeo YuliangZheng1 IWIA'04) 0-7695- 2117-7/04 Proceedings of The Second IEEE International Information Assurance Workshop The Second IEEE International Information Assurance Workshop * Wireless Intrusion Detection Systems FatbardhVeseli * Beacon Frame Spoofing Attack Detection in AsierMart´?nez_ UrkoZurutuzayz RobertoUribeetxeberriay MiguelFern´andezy JesusLizarragay AinhoaSernay I?nakiV´elezy IEEE 802.11 * PCI Wireless Compliance Demystified Best Practices for Retail * A Logistic Metrics Scorecard Based Approach to Intrusion Detection System Evaluation for Wireless Network RupinderSingh DrJatinderSingh International Journal of Computer Networks and Wireless Communications (IJCNWC) 2 3 * June 2012 Print * An Architectural Metrics Scorecard Based Approach to Intrusion BDetection System Evaluation for Wireless Network RupinderSingh DrJatinderSingh Global Journal of Computer Science and Technology (GJCST) 12 June 2012 Issue 11 Version 1.0