Ranking Security of IoT-Based Smart Home Consumer Devices

Manufacturers of smart home consumer devices like home theatres, music players, voice-based assistants, smart lighting, and security cameras have widely adopted the Internet of Things (IoT). These devices pose a significant security risk to consumers because the devices are exposed to mobile applications and cloud-based services with known security vulnerabilities. Most current home consumer devices provide little or no information about the level of security they afford. Since most consumers are not tech-savvy, it is currently difficult for a consumer to make an informed decision about which consumer device model (e.g., smart television model) has the best security. Hence, consumers need an objective security ranking of each type (e.g., security cameras) of home consumer devices. This paper proposes a novel methodology to systematically build such security rankings for home consumer devices. The proposed methodology can be applied by utilizing data from any security assessment study. The paper discusses previous efforts in applying Analytic Hierarchy Process (AHP) to rank security risks in general. The paper also presents a systematic survey of security vulnerabilities of smart home consumer devices when viewed from an IoT lens. Using the proposed methodology, a case study, employing an AHP model for ranking commonly used home consumer devices including home theatres, security cameras, smart lighting, smart speakers, video surveillance, smart switches, home automation systems, home security systems, smart routers, wireless doorbell cameras, and home audio systems, was developed. Relative security rankings for each type of consumer device were derived from the AHP model. According to the AHP model, network security was the primary driver of smart home device security with a priority of 0.6893 while application security had the least priority of 0.0591. Critical Vulnerabilities were the most important for device security (priority=0.4397), Man-in-The-Middle attacks for network security (priority=0.2019), exploitable services for cloud security (priority=0.26), and sensitive data for application security (0.7626). The AHP model was internally consistent (Consistency Ratio < 0.1). Sensitivity analysis showed that the AHP model was robust against pairing assumptions.


I. INTRODUCTION
Smart home automation goes back to at least 1985 [1]. Recently, manufacturers of many smart home automation systems and associated devices like surveillance cameras, home voice assistants (e.g., Alexa), and appliances (e.g., fridge) have embraced the Internet of Things (IoT) [2]. Exposing a smart home and devices to the internet raised security concerns as early as 2006 [3]. Smart home security is about protecting privacy of information embedded in a The associate editor coordinating the review of this manuscript and approving it for publication was Zhenhui Yuan . home environment, preserving confidentiality and integrity of consumer's data, and ensuring 24/7 availability of smart home services [4]. A recent study showed that 40.3% of smart homes worldwide had five or more devices connected to the internet, and that 40.8% of homes had at least one vulnerable device that puts the entire home at risk [5]. Similarly, Williams et al. [6], Notra et al. [7], and Ling et al. [8] demonstrated that security of webcams, televisions, home printers, smart lightbulbs, smart plugs, smart power switches and smoke-alarms could be easily compromised. Celik et al. [9] identified security issues in a number of IoT programming platforms. Software development kits for smart home applications (Apps) also have security issues [10]. Mare et al. [11] exposed security flaws in commercially available consumer smart home hubs. The security challenges to smart homes and smart grid have also been explored in [12]- [15].
Consumers are generally aware of security risks associated with consumer devices and are willing to pay for security labeling of such devices [16]. While consumers tend to trust IoT device manufacturers to protect their privacy, neither do they verify nor are they aware of the various privacy risks posed by these devices [17]. Consequently, security labels for IoT consumer devices clearly indicating security mechanisms (e.g., security updates, access control, encryption), data practices (e.g., whether the data is stored on the device or on the cloud), and additional information (e.g., physical actuation) have been proposed [18]. Simplified consumer security indexes for such security-awareness labels have been proposed to inform consumers [19]. Even if these labels were available, as have been proposed in UK, Netherlands and Singapore, the proposed security labels are not easily interpretable by a typical consumer. For example, it is unreasonable to assume that a layperson can intelligently compare two encryption standards stated on labels of competing electronic music players.
Determining the relative security of a type of consumer device is a complex Multi-Criteria Decision-Making (MCDM) problem because a host of interacting factors based on device hardware, networking, middleware, etc., and types of potential security vulnerabilities contribute towards making this decision [20]. Mardani et al. [21] conducted an extensive survey of techniques for solving MCDM problems and found that Analytic Hierarchy Process (AHP) [22] was the top method for solving MCDM problems. One key advantage of using AHP is that the technique can easily incorporate both numeric and qualitative, or judgment-based inputs, and is flexible to incorporate both types of data. Furthermore, AHP can be applied transparently and easily by conducting pair-wise comparisons against individual criterion. Finally, the AHP also provides a mathematical formula to measure the internal consistency in the how the data is being used to make decisions, and hence providing a measure of the quality of goodness of the decision model. This paper proposes an AHP-based methodology of how to develop a simplified security ranking for various types of consumer devices (e.g., smart televisions). The ranking thus developed can be used by consumers to easily assess the relative security of competing device choices. For example, when selecting which smart television set to buy, a consumer can refer to the relative security rankings of smart television sets available, and make an informed choice. The paper makes the following contributions.
• The paper presents a comprehensive survey of use of AHP to rank security aspects of computer-related systems.
• The paper presents a systematic survey of security vulnerabilities of smart home consumer devices.
• The paper presents a novel methodology for applying AHP that relies on a systematic literature review and on empirical data from security assessment studies.
• The paper presents a case study to build and validate an AHP model to determine the relative importance of key factors that have an impact on security of smart home devices today. To our knowledge this has not been done before.
The rest of the paper is organized as follows. Previous work on using AHP to assess security in a variety of computerrelated domains is presented first. A systematic survey of consumer device security vulnerabilities is presented next. This is followed by an example and a description of a novel AHP methodology utilizing systematic literature review and empirical data from a security study. The methodology is then applied to many consumer devices, and the resulting AHP model is presented and discussed. Paper ends with limitations and conclusions.

II. PREVIOUS WORK A. USING AHP TO ASSESS SECURITY
AHP is a well-known technique for solving MCDM problems [22], [23]. AHP is briefly described below followed by a discussion of previous work in applying AHP to assess security is various computer-related domains.
AHP begins by defining the problem and determining a goal. For example, for this paper the goal was to assess the relative security of a type of smart home consumer device (e.g., Which personal assistant is more secure?). The goal for AHP can be very general like ''assessing cybersecurity,'' or be very specific like ''assessing security of nuclear plants.'' Based on the goal, an AHP hierarchy is developed where levels of the hierarchy represent criteria and sub-criteria. For example, information security criteria like integrity, access control, authentication, availability, etc. are potential toplevel AHP criteria. The next step in AHP requires a pairwise comparison of each criterion and sub-criteria. For example, if confidentiality and availability were the two chosen criteria, then a relative importance of one versus the other needs to be established; an expert could indicate that confidentiality was significantly more important than availability in a specific situation. Based on pairwise comparisons, a comparison matrix is then constructed for each level. Subsequently, the AHP algorithm assigns relative priority to each criterion and sub-criteria in the hierarchy. Higher priority means more contribution towards the goal. Relative priorities of the various criteria can then be used to rank any decision alternatives. For example, Syamsuddin and Hawng [24] used AHP to determine, that for information security, cultural elements had the highest priority, followed by economy, management, and technology. The final step in the AHP methodology is to determine internal consistency of the pairwise comparisons. Previous work in applying AHP to assess security in various computer-related and information technology domains is presented next.
Maček et al. [25] used AHP to assess cyber security risks and used top-level criteria like attacks, vulnerabilities, and penetration testing, etc. AHP comparison relied on expert opinions after being provided with a systematic literature review. The results show that AHP facilitated fine-tuning of the cybersecurity risk assessment procedures. Similarly, Bhol et al. [26] presented a taxonomy of cyber security criteria of vulnerabilities, threats, users, protection mechanism and encounter outcomes. The primary goal was to evaluate cyber security strength. The process of pairwise comparisons was not specified. Zhao et al. [27] proposed a methodology for evaluating system security using the criteria of host security, network security, and vulnerability security. They showed that by using AHP and grey relational analysis theory, it was possible to effectively quantify the comprehensive security of the network while avoiding the subjectivity and one-sidedness of traditional security assessment methods through experimental verification. Sohime et al. [28] used AHP to rank the relative importance of various cyber security skills required in the job market. The criteria used were soft skills (e.g., analytical skills), technical skills (e.g., ability to identify potential risks) and certifications (e.g., related security technical/management certifications).
In the information security domain, Zaburko and Szulżyk-Cieplak [29] used AHP to evaluate the risk of information loss among employees. The criteria used were human dependent (e.g., procedural violation), technical (e.g., hardware failure) and random (e.g., consumption wear). Using expert opinions for comparisons, more information was found to be lost based more on human factors than others. Similarly, Bodin et al. [30] used AHP with criteria of confidentiality, data integrity and availability, and emphasized the utility of AHP to assist and organize the ideas of an organization's chief information security officer (CISO).
In the IoT domain, Wang et al. [31] used AHP to determine the security of identity resolutions based on two primary criteria of trust and user experience. For trust, sub-criteria included historical trust, leakage rate, and malicious resolution rate. For user experience, average resolution delay, resolution conscience, and integrity were used as sub-criteria. All AHP comparisons were based on expert opinions. Similarly, Siboni et al. [32] used AHP to determine the relative security of IoT devices. The AHP model was implemented using a device-centric method that considered both device-specific and domain-related features. The criteria used were known vulnerabilities (e.g., software, hardware, and firmware), sensor capabilities (e.g., movement and position, environmental, multimedia, connectivity, and health monitoring), and the operational context (e.g., mobility, time, and location). Varma and Chandra [33] used Fuzzy AHP (FAHP) to assess security of fog-IoT systems. The primary criteria included authentication, access control, intrusion detection, trust and integrity, and the sub-criteria included legitimacy, identification, rapid response, accountability, and credibility. The comparisons were based on expert opinions. Ogundoyin and Kamil [34] used AHP to assess the level of trust in fog computing and sub-criteria included latency and reliability. They used quality of service, quality of security as the two primary criteria. Expert opinions were used for pairwise comparisons. Wang et al. [35] used AHP and another MCDM technique called Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) to rank the security of IoT devices in the healthcare environment. The thirteen criteria included confidentiality, authentication, access control, and integrity, etc. Expert opinion along with the Delphi technique [36] was used for doing comparisons. Ly et al. [37] used fuzzy set theory and AHP to build a rule-based decision support mechanism to evaluate enterprise IoT security and used the criteria of connectivity, telepresence, intelligence, security, and value. Expert opinions were used for comparisons. The tangible variables (e.g., security, value, and connectivity) were found to be more essential for security than the intangible factors (e.g., telepresence and intelligence). Zhang et al. [38] evaluated security of IoT systems using FAHP as early as 2011. They used perceptual, transport, application, and cloud security as the primary criteria. Perceptual layer included sub-criteria like intelligent node security and node's information control certificates, etc. Similarly, the transport security criteria included the sub-criteria of network security, risks of Internet Protocol version 6 (IPV6), etc. The application security criteria included role identification efficiency, normal working hours, and software disaster control capability, etc. Finally, the cloud security criteria included sub-criteria like cloud computing platform security, user access control capability, information application security, etc. Expert judgments were used for comparisons. Security concerns related to perceptual layer were found to be the most important in 2011.
In the web applications domain, Kumar et al. [39] used FAHP-TOPSIS to assess usable-security. They used criteria of security and usability where security included the sub-criteria of confidentiality, integrity, accountability, authentication, and durability, while usability included appropriateness recognizability, operability, user error protection, user interface aesthetics, and accessibility. Expert opinion was used for pairwise comparisons. Agrawal et al. [40] used FAHP and Fuzzy TOPSIS for assessing the sustainable security of web applications. Expert opinions were used with the criteria of confidentiality, integrity, availability, and durability. The evaluation was based on two case studies and six projects. Lai et al. [41] used AHP to assess security threats to websites. They used the two criteria of accidental threat (e.g., hardware/software failure, ineffective management, operational error) and malicious threat (e.g., physical attack, malicious code attack, network attack, ultra vires or abuse, information leakage).
In applications domain Alharbi et al. [42] used AHP and TOPSIS to provide rankings for security of a healthcare applications. Criteria included integrity, access control, confidentiality, and authentication. Expert opinions were used for doing comparisons. Kumar et al. [43] used Hesitant FAHP-TOPSIS approach to assess usability-security. They used security and usability as the top-level criteria. The subcriteria were confidentiality, accountability, authentication, and durability. For usability, they used the sub-criteria of appropriateness recognizability, operability, error-protection and comprehensibility, and user-interface aesthetics. Expert practitioners were asked to do the pairwise comparisons. Kim et al. [44] used AHP to examine cyber-attack taxonomy in Nuclear Power Plants. The primary criteria were divided into attacker related variables with sub-criteria like attack skill and intensity, and target related variables that included the sub-criteria of physical access, logical access, and attack surface. Questionnaires and expert views were used to determine relative significance. Attack skill and physical access, logical access, and attack surface were found to be the most important criteria. Phudphad et al. [45] used AHP to assess the impact of security aspects of Human Resource Information Systems (HRIS) on the work climate. They used confidentiality, integrity, non-repudiation, privacy, and availability as the key criteria. Expert opinion was used for comparisons and the results suggest that the most crucial factor was confidentiality, followed by non-repudiation and privacy. Zhang et al. [46] proposed a three-layer AHP evaluation model for E-Commerce security. Primary criteria were technical criterion with sub-criteria of network and system security, environmental criterion with sub-criteria of legal and cultural security, and managerial criterion with sub-criteria of personnel and equipment security. Experts were used for comparisons, and the Dempster-Shafer (DS) theory of evidence was applied. The model was shown to be capable of handling both qualitative and quantitative data. Syamsuddin and Hawng [24] utilized AHP to assist banking decisionmakers in analysing information on security areas such as management, technology, economy, and culture. The AHP model was derived from questionnaire responses and expert evaluations. According to the findings, the top priority in terms of information security was cultural elements, followed by economy, management, and technology.
In the cloud domain, Tariq et al. [47] used FAHP to prioritize and select the most appropriate collection of information security controls to meet the organization's information security requirements for cloud and sensor networks. Criteria like effectiveness, risk, budgetary constraints, exploitation, maintenance, and mitigation time were used. Expert opinions were used for comparison. The use of FAHP resulted in a more efficient and cost-effective evaluation and assessment of information security controls within an organization, allowing the most appropriate one to be selected based on the International Organization for Standardization and the International Electrotechnical Commission (ISO/IEC). Ruo-Xin et al. [48] used AHP to determine Cloud Security. The primary criteria were technical requirements and administrative requirements. The technical requirements criterion included the sub-criteria of physical security, network security, host system security, application security, data security, and safety management systems. Administrative criterion included safety management institution, safety management, system construction management, system operational management, and service level agreement management. Expert opinions using the Delphi method were used for comparisons. Finally, Taha et al. [49] used AHP to assess and benchmark security provided by a Cloud Service Provider based on its Security Service Level Agreement (sSLA). Compliance, data governance and information security were used at the toplevel criteria. Compliance criterion included the sub-criteria of audit-planning, independent audits and third-party. Data handling and governance policy were the sub-criteria used for governance criterion, and baseline acquirements and policy reviews were used as sub-criteria for information security criterion.
In the networking domain, Li et al. [50] used an improved AHP based on D-S evidence and Gray Theory to assess network security risks. The top-level criteria included assets (e.g., tangible and intangible), access control (e.g., user access management), and communication (e.g., computer network management). Expert opinions were used for pairwise comparisons. The results showed that the proposed technique could potentially increase the reliability of network security risk assessments. Dong et al. [51] used a modified AHP called D-AHP to evaluate security of smart grids. The top-level criteria used included smart terminal, wireless communication channel, password security, application code and the embedded system. Similarly, Yan and Qiao [52] used AHP to assess network security. Top-level criteria used included hardware risk, software risk, and information risks. Sub-criteria for hardware risk criterion were circuit security, network equipment security and computer security. Software security risk criterion included application, database, and operating system security. Information risk criterion included data backup, access control, encryption, and confidentiality strategy. Communication risk criterion included encryption, anti-virus, intrusion detection and firewall. Sub-criteria for organization management risk criterion were security education, management systems, and organization. Physical environment sub-criteria were security power supply, physical equipment protection, physical monitoring, and physical access control. Expert opinion using the Delphi technique was used for pairwise comparisons. Finally, Zhang et al. [53] used a combination of FAHP and variable weight theory to assess wireless network security using the top-level criteria of authenticity, availability, confidentiality, and integrity. A case study suggested that the FAHP variable-weight technique for assessing wireless network security was both efficient and practical.
In summary, as Table 1 shows, many variants of AHP models were developed in a variety of security domains. However, in most cases, the pairwise comparisons were based on expert opinions. In one case, experts were provided with a literature review as background information before seeking their opinions. A word cloud for the top-level criteria from Table 1 is shown in FIGURE 1. The word cloud shows that many previous studies used the traditional security dimensions of confidentiality, integrity, authentication, availability, VOLUME 10, 2022  etc. as the primary top-level criteria. Further, most studies required experts and assumed that experts could meaningfully judge the relative weights of each criterion. A final observation is that AHP criteria were developed based on the goal and the ability to conduct meaningful pairwise comparisons using either experts or some other means. Consequently, the goal and availability of opinions or data to make pairwise comparison dictated the design of the actual AHP hierarchy used.

B. SURVEY OF SMART HOME SECURITY
A typical smart home today contains a variety of consumer devices including surveillance cameras, voice-assistants, thermostats, smart televisions, music streamers, smart lighting, etc. As FIGURE 2 shows, smart homes often utilize heterogenous networks. For example, the three signal symbols in FIGURE 2 refer to different wireless technologies; purple signals refer to ZigBee networking, blue signals refer to Bluetooth networking, and green signals refer to Wi-Fi networking. Some consumer devices may also be connected to a smart home management system using a home area network. Many smart home devices interact with a consumer's mobile phone and use internet gateways to communicate with remotely hosted services offered by various commercial providers [54]. For example, most home security cameras store video on remote servers that can be accessed by consumers anytime anywhere. Similarly, smart assistants like Alexa also leverage cloud-based services. Recently, there is also a trend to move computation from the cloud to the edge devices [55], [56] within smart homes to enact some services locally partially contributing to better network security.
Efforts are also underway to broadly characterize security risks and vulnerabilities of smart home consumer devices [57]. Reference architectures for implementing smart home security at the system level have been proposed as well [58]. There are many ways to conceptualize smart home security [59], [60]. However, this paper uses an IoT lens where smart home security is viewed from the four perspectives of device security, network security, cloud security and application security [20], [38]. Using this lens, consumer devices in a smart home are considered IoT edge devices that can sense, record, and communicate data. A security camera or a voice-based home assistant like Alexa, for example, senses and connects to the outside world through a gateway which could be a home router. The IoT gateways collect data from sensing devices, and transmit the data to cloudhosted servers that, in turn, provide consumer services. For example, when motion is detected by a security camera, a recording of the associated video is optionally saved locally, and also transmitted remotely to the cloud for storage or further analysis. Finally, most smart home devices provide mobile Apps to allow a consumer to configure and interact with the smart home device. For example, in the case of a security camera, a mobile App lets consumers configure the camera, and connect to servers on the cloud to access the recorded videos.
Based on the IoT lens of a smart home, a survey of recent work in smart home security since 2016 was conducted to answer the following four research questions. RQ1: What are the common vulnerabilities of smart home consumer devices when viewed as IoT edge devices?
RQ2: What are the common vulnerabilities of networking when used with smart home devices?
RQ3: What are the common vulnerabilities of cloud when used with smart home devices?
RQ4: What are the common vulnerabilities of applications when used with smart home devices? Table 2 shows the results of searching in the four commonly used digital libraries by using the most frequently used keywords used in the highest cited papers in the area. The keywords used included IoT security, Cybersecurity, Smart Device, Privacy, Information Security, Attack Surface, Communication, Cloud Security, Security of Data, Heterogeneity, Ontology, Home Automation, etc.
The papers were filtered to include only the relevant papers for smart home security. Survey results based on the filtered papers are described below.

1) RQ1: SMART HOME DEVICE SECURITY
IoT edge devices suffer from many vulnerabilities. For example, one vulnerability is eavesdropping where an attacker listens in to the data being transferred to and from a device [61], [62]. The physical device can also be compromised by node capture attacks, replay attacks and sleep deprivation attacks [63], [64]. For example, the Mirai Botnet [65] attack consisted primarily of compromising embedded IoT devices and using these devices for a Distributed Denial-of-Service (DDoS) attack. There are several reasons why devices are susceptible to attacks. Internet connectivity and telepresence are the obvious enablers [66]. There are more particular problems as well like undocumented Secure Shell (SSH) and default passwords [67]. For example, Antonakakis et al. [65] showed that even an unsophisticated dictionary attack could compromise hundreds of thousands of internet-connected devices. Further, device authentication might not be practical for IoT security because securing routing protocols at the network layer may potentially suffer from unacceptable endto-end delays [68]. Legacy authentication mechanisms may VOLUME 10, 2022 also not be suitable IoT devices because many IoT devices are resource-constrained [69]. Astaburuaga et al. [70] analyzed weakness in an embedded Operating System (OS) often utilized in smart home devices and found that pairing mode feature could be easily bypassed which made the OS vulnerable to attacks such as DDoS and takeover. Some devices are also vulnerable by virtue of the hub they connect to [71].
Another common reason for device vulnerability is implicit trust and overprivileged design of the connecting Apps [72]. For example, over 55% of Apps on a popular IoT App store were overprivileged [73]. Over the air update of firmware and Apps also makes these devices vulnerable [74]. For example, Hernandez et al. [75] showed that firmware verification of a commonly used IoT-enabled thermostat could be bypassed, providing the means to completely change the unit's behavior. The compromised thermostat could then act as a beachhead or malicious node to attack other nodes within the local network, and any information stored within the unit was now available to the attacker who no longer needed physical access to the device. Voice is becoming a standard interface for many smart homes. Voice-spoofing has emerged as a recent threat where inaudible voice commands that cannot be understood or heard by the human, but can still be understood by the system, are injected to control the smart home [13].
Some consumer devices are borrowed, rented, gifted, resold, or retired which raises privacy violation concerns for the data stored on these devices [76]. A related issues pointed out by Özkan and Bulkan [77] is that increasingly sub-systems in modern software and devices may have been developed by an ad-hoc team that is no longer available to maintain and fix security risks. Sometimes, the company that developed a sub-system discontinues support for a product, and stops issuing security updates. In a government or enterprise context, such obsolescence security risks can be handled through governance and management level mitigation policies that ensure that either such obsolete products are replaced, or the obsolete parts are isolated [77]. However, the situation is much more complex for consumer devices because even a simple consumer device may contain hardware, firmware and software from different vendors who may not adhere to the same product life-cycle governance policies. Hence, this type of obsolescence security risk remains a big challenge for consumer devices. Consumer devices with longer shelf life like televisions are particularly susceptible to this risk. One possible solution is to periodically subject such devices to rigorous security testing, and to publish the vulnerabilities to warn the consumers. Another option could be introducing legislation to ensure the that the original equipment manufacturers (OEMs) agree to provide some minimum level of extended security support for obsolete consumer devices.
Finally, consumer devices are also susceptible to a variety of attacks at the hardware level [78]. These include architectural and system threats (e.g., secure boot attacks, firmware attacks, etc.), covert and side channels attacks (e.g., timing, electromagnetic channels, etc.), intellectual property theft and counterfeiting threats, and hardware trojans.
Several countermeasures have been proposed for device security. For example, a provenance-based framework called ProvThings by Wang et al. [79] detected errors and malicious activates within deployment, such as weak authentication and misconfiguration. ProvThings was able to provide complete provenance for twenty-six known IoT attacks like side channel, spyware, and backdoor pin code injection. Tian et al. [80] proposed SmartAuth that is implemented by device vendors, where users can specify which third-party applications have permissions, and thus, obviating over privileged Apps. Santoso and Vun [81] proposed public key mutual authentication protocol for devices as a possible solution for authentication vulnerabilities. Han et al. [82] argued that confidentiality, access control and data integrity required a secure trustworthy smart home service in the back end as well. Meng et al. [13] showed that it was possible to use channel state information (CSI) to thwart voice-spoofing in a device-free manner. For hardware attacks, a variety of counter measures including true random number generators (TRNG), physical unclonable functions (PUF), system and architectural protection techniques, trusted execution environments, side channel protection techniques, and intellectual property protection techniques like hardware watermarking and steganography have been proposed [81].
2) RQ2: SMART HOME NETWORK SECURITY Home routers have poor protection against internet-based attacks [83]. Hussain et al. [84] showed that various vulnerabilities like default passwords, infrequent password changes, and the absence of system updates could be reduced by accessing the home automation system using a single network. Lounis and Zulkernine [85] provided a taxonomy of attacks in Wi-Fi, Bluetooth, ZigBee, and Radio Frequency Identification (RFID) infrastructures, as well as a survey of assaults on each network technology. Their findings revealed that most attacks were caused by vulnerabilities in the authentication protocol. This is important because many smart homes utilize wireless heterogenous networks including Bluetooth Low Energy (BLE), ZigBee, Z-Wave, and Transmission Control Protocol / Internet Protocol (TCP/IP) [86]. Alrawi et al. [20] argued that most of the IoT devices depended on insecure protocols and that confidentiality and integrity were missing. For example, some motion sensor and home-surveillance cameras send plain text information which makes it comparatively simple for hackers to deduce when a user is at home based on the motion sensors' state [87]. Even well-known protocols like TCP/IP with Transmission Layer Security (TLS) are not entirely safe [88]. For example, Aviram et al. [89] presented a novel cross-protocol attack on TLS called DROWN which used a server supporting Secure Socket Layer (SSL) v2 as an oracle to decrypt modern TLS connections. Results showed that 26% of Hypertext Transfer Protocol Secure (HTTPS) servers were vulnerable to Man-in-the-Middle (MITM) attack, and that SSL was weak and damaged the TLS ecosystem. Similarly, Apthorpe et al. [90] examined smart home devices and showed that network traffic rate for devices revealed user activities, showing that encryption alone was not sufficient for privacy protection in smart homes. Adrian et al. [91] identified Logjam as a novel flaw of TLS that allows MITM to downgrade connections export grade Diffie-Hellman key exchange. Wi-Fi networks remain a key vulnerability for smart homes. For example, Godwin et al. [92] showed that it was challenging to break into a common voice-based home assistant using the Bluetooth protocol, but the internal Wi-Fi network could be compromised during device setup. The heterogenous nature of networks inside a home also exacerbates the situation. For example, Ho et al. [93] showed how it was possible to have relay attacks against Bluetooth Low Energy (BLE) protocols by serializing the BLE packets and relying on them over IP. Lounis and Zulkernine [94] discussed Bluetooth Low Energy (BLE) security and how the ''Just Works'' pairing option could be used to render a device inoperable. They showed a practical case study of three different Bluetooth smart gadgets. The conclusion was that people should be advised about the risk of purchasing unsecure gadgets and prioritizing convenience over security, privacy, and safety. Oren and Keromytis [95] examined network-level security weaknesses on smart televisions where a number of attacks such as DDoS, authenticated and unauthenticated request forgery, and phishing had taken place. Various types of social engineering attacks can also be used to penetrate the network security of smart homes [96]. Finally, Wood et al. [97] monitored home networks and disclosed multiple vulnerabilities within IoT devices highest of which was due to sharing of sensitive data. VOLUME 10, 2022 Many approaches have been proposed for intrusion detection in smart homes. For example, Gajewski et al. [98] proposed a two-tier intrusion detection mechanism that used machine learning to combine anomaly detection at local level in each home combined with global anomaly detection across homes conducted by the network service provider. Likewise, deep learning approaches to detect IoT device anomalies have been proposed [99]- [102]. Similarly, Pan et al. [103] implemented a context aware intrusion detection framework that could accurately find and classify various kinds of Building Automation and Control Networking protocol (BacNet) attacks.
Better alternatives to TLS for resource-constrained devices have been proposed [104], [105]. Beurdouche et al. [106] proposed a programming approach for protocol implementation that included a systematic testing of unexpected sequences of messages. Peter and Gopal [107] introduced a multilevel smart home network authentication system that offered multiple security features. Huang et al. [108] proposed a security framework called SecIoT which provided important authentication and guaranteed secure communications to support authorized users with risk notification through Fifth Generation (5G) network to operate device-to-device communications at any time. Serror et al. [109] proposed a rule-based approach that automatically complements existing smart home network to provide protection for heterogeneous IoT devices and protocols. Apthorpe et al. [110] evaluated four strategies to protect the home network from threats including blocking traffic, concealing Domain Name System (DNS), and shaping traffic, and showed how traffic shaping on the home network could prevent side-channel snooping. Kim and Keum [111] provided a trusted gateway system architecture that built an IoT trust domain which could safely protect IoT devices from malicious attacks without making any changes to IP-based devices. Finally, Gill et al. [112] proposed a Quality of Service-aware (QoS-aware) resource management technique using fog-assisted cloud computing providing better security for smart homes.

3) RQ3: SMART HOME CLOUD SECURITY
In many instances, consumer data from a smart home device needs to be securely communicated to cloud-based backend services [113], [114]. However, security is sometimes compromised in such transactions [115]. For example, many home surveillance cameras used cloud-based services that had issues with authentication and verification [90], [116]. Cloud-based IoT platforms are also susceptible to security flaws. For example, Surbatovich et al. [117] showed that some IoT recipes on a popular IoT platform could allow attackers to distribute malware and perform Denial of Service (DoS) attack. Platforms for cloud integration can also be compromised, and may expose the OAuth tokens of the user to the public. Analysis of event trigger rules in another popular open-source home automation system showed that 80% of the rules had less triggers than needed, and hence could lead to unexpected security holes that could be exploited [118].
Various countermeasures have been proposed to address cloud-based security for smart homes. For example, Alsadi and Mohan [119] proposed a method to increase secrecy rate by letting the legitimate transmitter find an alternative route to the fusion center in case of an eavesdropper located in between the information passed. Similarly, Tao et al. [120] proposed a new multi-layer architectural cloud model to enable efficient and seamless interactions on heterogeneous devices/services provided by various IoT-based smart home vendors. Another way to empower IoT users who trust their private data to the vendors is Transport Layer Security-Rotate and Release (TLS-RaR) that can be jointly deployed by vendors and users or trusted third parties. Device vendor can also mitigate their exposure by diversifying and subscribing to different cloud providers [121].
Finally, Fernandes et al. [122] proposed using a decentralized framework for trigger-action programmable platforms called Decentralized Trigger-Action Platform (DTAP) that acts as a shim between the IoT cloud platform and the users' local network. In this scenario, broker access to IoT devices was based on transfer tokens (XTokens) where attackers could not misuse the tokens.

4) RQ4: SMART HOME APPLICATION SECURITY
Hu et al. [125] examined mechanisms for testing the security of third-party Apps for smart home assistants. Mobile Apps used to configure or access smart home devices provide a convenient attack surface [124]. It is difficult to maintain security at the application layer because of lack of sufficient protocol security services, incorrect configuration, and resource limitations [125]. For example, Liu et al. [126] showed that it was possible to emulate a commercial edge device using software and then fooling the associated mobile App to uncover home Wi-Fi passphrases, and to trap the user into disclosing personal information. Similarly, Margulies [127] argued that linking garage door openers to the internet network using mobile Apps might easily pose a security threat. Fernandes et al. [128] found that many IoT programming frameworks only support permission-based access control on sensitive data, and hence making it possible for malicious Apps to abuse the permissions and to leak data. Sivaraman et al. [83] demonstrated a smart phone attack on a home network using a doctored smart phone App by scouting for vulnerable IoT devices within the home and then reporting them back to an external entity where they modify the firewall to allow the external entity to directly attack IoT devices. Demetriou et al. [129] proposed HanGuard that allowed the router to enforce access control policies to home area network using mobile phones and IoT devices. Chen et al. [130] suggested that to ensure proper deployment, IoT vendors and developers should follow platform development guidelines and leverage the in-built security features. Finally, Yamauchi et al. [131] proposed a unique intrusion detection mechanism that used Hidden Markov Machines (HMM) to learn the behavior of homeowners. App commands that were not congruent with this behavior were marked as anomalies.  Table 3 shows the key security vulnerabilities for each research question posed in the survey. Table 3 suggests that each component of a smart home device viewed from the IoT lens leads to related but potentially different set of vulnerabilities.

III. AHP METHODOLOGY FOR SMART HOME DEVICE SECURITY
The previous section showed that a variety of task-specific criteria have been used to build AHP hierarchies to assess security in various computer-related domains. In most cases, the generic security criteria of confidentiality, integrity, authentication, availability, etc were used at the top-level in developing these AHP hierarchies. A second common characteristic of most pervious work discussed in the last section was a reliance on expert judgement when comparing alternatives. The previous section also showed that a large number of security vulnerabilities at various levels (e.g., device, network, etc.) have been identified in the literature, and that new vulnerabilities continue to emerge and hence represent a moving target.
The AHP methodology proposed in this paper removes the subjective component of the expert judgements used in previous work by systematically using the extensive literature review on computer security presented earlier in conjunction with current empirical security studies of consumer devices. Hence the proposed methodology relies on the latest empirical data mediated by the collective judgement of experts represented in the research literature as opposed to using individual expert opinions. Further, unlike most previous work, the top-level criteria of the AHP hierarchy in the proposed methodology are not based on generic security concepts, but on the dimensions used by a specific empirical study. Combining literature review systematically with data derived from empirical studies to automatically conduct AHP analysis is unique has not been done before to our knowledge.
This section proposes a novel methodology for applying AHP to create smart home device security rankings. A simplified example of how AHP can be used to rank device security using the IoT lens with the four top-level criteria of device, network, cloud and application security is presented first. The simplified example shows in detail all the steps and the calculations required for each step of the generic AHP methodology. The example is followed by a description of the proposed methodology for applying AHP for smart home security domain. The proposed methodology is then demonstrated by using an existing empirical study, and all the steps and calculations are subsequently described in detail.

A. A SIMPLE EXAMPLE OF APPLYING AHP
This section presents as simple example of using an AHP based on device, network, cloud, and application security criteria. Lower-level sub-criteria are excluded for simplicity.
The first step in AHP is to build a decision hierarchy like the one shown in FIGURE 3. This hierarchy assumes that AHP has the goal of selecting the most secure surveillance camera. This goal can be achieved by evaluating each candidate camera with respect to Device (D), Network (N), Cloud (C) and Application (A) security. The model in FIGURE 3 shows that three alternative camera models (C1-C3) are to be compared.
The second step in applying AHP is to determine the priority or importance of each criterion (e.g., Network Security) in achieving the said goal. This is done by first constructing a pairwise comparison matrix A G for the four criteria. VOLUME 10, 2022 The matrix A G encodes the relative importance of these criteria towards achieving the goal. This example uses arbitrary pairwise comparison numbers. Equation (1) shows an example matrix A G where each a ij ∈A G represents a symmetric pairwise comparison of importance between the i th and j th criterion based on the AHP fundamental scale [26]. For example, a 14 = 9 in (1) shows the assumption that Device (D) security is much more important than the Application (A) security in determining the security of a surveillance camera.

D N C A
w G is a vector representing the relative priority of each criterion with respect to the top goal and is calculated by solving a system of equations given in (2) and (3) for any pairwise comparison matrix A and priority vector w [26] λ max is the maximum eigenvalue of A, and 1 = (1 . . . 1)T. w G calculated using (2) and (3) in (4) shows that according to A G , Device has the highest priority for achieving the goal of determining the most secure camera (i.e., w G D = 0.455).
The internal consistency of an n × n pairwise comparison matrix A is given by (5) where RI n is empirically determined for each n. Matrices with CR(A) < 0.1 are acceptable while those with CR(A) > 0.1 are rejected as being inconsistent [26].
CR A G = 0.069(n = 4) which means that pairwise comparisons in matrix A G are internally consistent.
Once the relative priority of each criterion (e.g., Network) with respect to the goal is determined, the same process is repeated for each criterion by creating a pairwise comparison matrix for each criterion (e.g., Network) with respect to each alternative. For example, A D in (6) shows the pairwise comparison of each of the three camera alternatives with respect to Device (D) security criterion. For example, (6) shows that with respect to device security, camera C1 is less secure than camera C2 (a 12 = 0.5), and more secure than camera C3 (a 13 = 3).

C1 C2 C3
w D in (7) shows that overall priority vector for A D calculated using (2) and (3). Equation (7) shows that camera C2 was the most preferred with respect to device security (w D C2 = 0.528).
Finally, goal level priority and the criteria level priority vectors can be combined into a single priority vector w cameras using (8) [26]. (8) where w N , w C , w A represent the priority vectors with respect to Networking, Cloud and Application layers respectively calculated in a similar fashion as w D . w cameras in (9) shows the relative priority of each camera alternative calculated using (8).
Equation (9) shows that w C1 w C3 w C2 which means that in this example, camera C2 was the best overall choice for a surveillance camera.

B. PROPOSED AHP METHODOLOGY
A methodology of conducting the pairwise comparisons for each level of the AHP model is described next.

1) TOP-LEVEL PAIRWISE COMPARISON
At the top-level, relative importance of various security criteria like Device Security versus Network Security must be determined. The methodology proposed a pairwise comparison scheme based on the literature review for smart home security. Specifically, the number of words discussing issues related to each security criteria were counted for each paper and used as a proxy for relative importance of each security criterion. Table 4 shows an example of how a normalized percentage with respect to each paper's total word count was derived. Each Security column in Table 4 represents the overall importance of each criterion for four sample papers.
An effect size using Rank biserial correlation (r) [133] was used to then calculate the magnitude of difference between  the compared pairs (e.g., Device vs. Network Security column in Table 4). The effect size value (r) was then mapped to the AHP scale [26] by using (10) and (11).
ahp_scale = 8× r + 1 (11) Table 5 shows the resulting pairwise comparisons among the various security criteria based on all the papers reviewed earlier. For example, Table 5 shows that Device security was much less important than Network security (ahp_scale =1/6) and much more important than Cloud or Application security (ahp_scale = 4).

2) LOWER-LEVEL PAIRWISE COMPARISON SCHEME
A second key contribution of this paper is the idea of using empirical data on security assessment of home devices for the low-level AHP comparisons. Empirical security assessment data from a security study conducted by Alrawi et al. [20] was used in this paper. Like most such studies, this study also used pragmatic lower-level criteria for each of the device, network, cloud and application-level security. For example, device security criteria contained the sub-criteria of internet pairing, configuration, upgradability, exposed services, and Common Vulnerability Scoring System (CVSS) [132]. This is different than the general dimensions of security like confidentiality, integrity, etc. being used by many AHP studies in the past as shown in FIGURE 1. Data published by Alrawi et al. [134] based on this study was used to automatically generate pairwise comparisons for the lower level. Since the original study used a ratio scale to represent relative security risk, the ratio scale was first converted to an ordinal scale for AHP. For example, the overall security risk for Application security in the original study was 16. For one consumer device model (e.g., camera model C1), the security risk due to 'sensitive data' with respect to Application security was (7/16), while security risk due to 'programming issues' with respect to Application security was (5/16). This meant that for camera model C1, 'programming issues' was less of a security risk with respect to Application security. Given each such ratio (a/b), (12) was first used to calculate the angle of the vector < a, b > in radians for each reviewed device.
All reviewed devices of a particular type (e.g., security cameras) were then compared in pairs by taking the difference between their respective θs calculated using (12) and subsequently using (13) and (14) to map the difference of θ s to the AHP scale. Table 6 shows an example of the pairwise comparison matrix for Application security which depends on Sensitive Data, Programming Issues and Excess Permissions.

IV. CASE STUDY
Using the methodology described in the previous section, an AHP analysis was conducted using the pairwise comparison matrices as shown in the previous section. A total of 41 devices (e.g., Alexa) within 11 device types including home theatres, security cameras, smart lighting, smart speakers, video surveillance, smart switches, home automation systems, home security systems, smart routers, wireless doorbell cameras, and home audio systems were considered based on security assessment data published by Alrawi et. al [134].
A. THE AHP MODEL FIGURE 4 shows the resulting AHP model based on applying the proposed methodology. The calculated priority or relative weight of each security criterion is shown in parentheses.
As FIGURE 4 shows, at the top level, Network security had the highest priority (0.6893) which implies that network VOLUME 10, 2022 security was by far the most important security criteria for smart home devices. Application (0.059) and Cloud securities (0.0614) each had much lower priority. At the lower level, for Device security, predictably, 'critical vulnerabilities' had the highest priority of 0.4397. Second position was tied between high vulnerabilities and configuration with a priority of 0.1745. Internet pairing, and low vulnerabilities were the least important with a priority of 0.0297 each. Similarly, for Network security, two types of MITM attacks had the highest priority of 0.201. Interestingly, Mobile-to-Device MITM had a low priority of 0.0378. For Cloud security, exploitable services had the highest priority of 0.260. Finally, for Application security, sensitive data had the highest priority of 0.7626, while the least important was excess permissions with a priority of 0.0611.

B. INTERNAL VALIDATION 1) CONSISTENCY RATIOS
It should be emphasized that in this paper, the pairwise comparisons were based on quantitative measures taken directly from a combination of a literature review and on data from an empirical security assessment study. Therefore, it was important to ensure that these automatically derived pairwise comparisons were mathematically consistent. In AHP, internal consistency of judgments of pairwise comparisons is measured by the Consistency Ratio (CR) [135]. The pairwise comparisons are considered unreliable if the CR value is higher than 0.1 and must be revisited [136].
The consistency ratios for all top-level comparisons for the AHP model were mostly within bounds (CR<0.1). The only exception was Application security where the CR value was a bit higher than 0.1 (CR=0.1037) which is acceptable.
Mean CR values for all device types (e.g., surveillance cameras) were well within bounds with the highest mean being 0.0140 for the video surveillance equipment. In more than 50% of the cases, the median CR values were zero, and the standard deviations were small. In summary, it is reasonable to assume that the model was internally consistent at all levels of comparisons.

2) SENSITIVITY ANALYSIS
The goal of sensitivity analysis was to determine how sensitive the ranking outcomes were to the pairwise comparisons. Sensitivity analysis was conducted by varying top-level priorities 10% above and below their respective values and determining the impact on the relative ranking of various devices. Most devices with the topmost ranking within each device type were not sensitive to priority changes. Priority thresholds of 0.01 and 0.05 were used to determine if the rank changed for each device type. For example, if the priority of two alternative devices differed by more than 1% (threshold=0.01), then the rank was considered different based on a threshold of 0.01. For the threshold of 0.05, top rankings changed only twice across all 11 device types. Similarly, for a threshold of 0.01, the ranking changed for 4 device types only. This suggests that although the AHP model was sensitive to some device types, overall, the model was robust with respect to the pairwise comparisons.

3) COMPARISON WITH ORIGINAL SECURITY SCORES
It is instructive to compare the security ranks given to each consumer device with those from the original security scorecards from Alrawi et. al [134]. Table 7 shows an example of original scorecard percentage scores for four video surveillance devices; higher percentages meant better security. For example, Video Surveillance Device 1 (VSD1) was most secure with respect to Device security (86%) as opposed to Network, Cloud or Application security. It is clear from Table 7 that while useful for a researcher, a typical consumer cannot directly interpret this information to determine which device is the most secure. For example, from Table 7, it is not clear whether Video Surveillance Device 2 (VSD2) was better than Video Surveillance Device 4 (VSD4). This is because both devices were similar with respect to Device (93% and 95% respectively) and Network (96% and 93% respectively) security. However, VSD2 was better in Cloud security (84% vs. 63%) while VSD4 was better in Application security (69% vs. 54%). It is not possible for a typical consumer to decide which of the Application or Cloud security is more important for video surveillance devices in making this decision. What the consumers require is a simple ranking as proposed in this paper.
To facilitate a comparison with our approach, percentage scores of the type shown in Table 7 were first normalized using the SoftMax [137] function shown in (15).
In (15), score dl represents the normalized score of a device d (e.g., VSD1) with respect to security criteria l (e.g., Device)and n is the number of alternative devices in the group. The original percentage score as shown in Table 7 is represented by a dl (e.g., a VSD1−Network = 57%). For a particular criterion l (e.g., Network), the normalized scores add up to 1 for the n device alternatives. By definition, score dl ≤ 1 because each device has four such scores, one for each security criterion, (16) was used to calculate the overall rank for each alternative device where i represents each security criterion (e.g., Cloud).
Levenshtein or Edit distance [138] was subsequently used to calculate the distance between the ranks derived from the original score cards and the proposed AHP model. Edit distance of zero means that ranks are the same. For example, the ranks of five smart lights (L1-L5) using the original scorecard [134] were L1 L2 L3 L4 L5 (i.e., L5 was the most secure) while the AHP model's ranks were L1 L3 L2 L5 L4 (i.e., L4 was the most secure), showing that the ranks were different because the edit distance between the two ranks was 4.
Overall, the ranks based on the proposed AHP approach were significantly different than the original scorecard ranks with a total Levenshtein distance of 20 across all device types. Since the lower-level empirical data was common for both the scorecard and the proposed AHP model, this discrepancy between the two approaches can perhaps be explained by the fact that the original scorecard did not explicitly incorporate any top-level assumptions. This speaks to the importance of the more holistic view of the AHP model for a more informed decision.

V. CONCLUSION AND LIMITATIONS
This paper presented a systematic survey, a methodology and a case study to rank the security of home consumer devices. An IoT lens based on the current state-of-the-art research in security of smart home devices was used to propose a novel methodology. The proposed methodology was then used and evaluated in the context of one empirical security assessment study. The key contribution of the methodology is systematically combining the current wisdom behind smart home device security research with empirical on the ground results from security vulnerability studies. The case study showed that even though the AHP model was based on empirical data, remarkably, the resulting AHP model was internally consistent and robust with respect to pairing assumptions and sensitivity analysis. The derived AHP model also showed the importance of various security factors in current home consumer devices in an explicit and quantitative manner. In addition to ranking consumer devices, the AHP model can also be used to inform future research because it incorporates empirical security studies as well. For example, under network security, Third Party DNS and MITM Device-to-Cloud were found to be most significant from a security perspective, and therefore, more research could be directed towards determining and deploying counter measures for these two vulnerabilities.
Although the proposed methodology and approach is general, the proposed methodology has only been applied to one empirical security assessment study. However, the methodology can be easily adapted to any of the many vulnerability assessment studies being conducted today. In addition, as the research focus on smart home device security changes, the current top-level priorities may change as well. For example, network may no longer be the primary weak point in the future. In this case, the proposed methodology can be used to simply recalculate the priorities. Finally, it would also be interesting to compare these device rankings with those based on an AHP built using expert opinions. However, the advantage of the proposed methodology over an expert-based approach is that the methodology can be mostly automated and applied whenever the underlying information changes.

ACKNOWLEDGMENT
This paper represents the opinions of the authors and does not mean to represent the position or opinions of the American University of Sharjah.