Analyzing and Testing Viewability Methods in an Advertising Network

Many of the current online businesses base completely their revenue models in earnings from online advertisement. A problematic fact is that according to recent studies more than half of display ads are not being detected as viewable. The International Advertising Bureau (IAB) has defined a viewable impression as an impression that at least 50% of its pixels are rendered in the viewport during at least one continuous second. Although there is agreement on this definition for measuring viewable impressions in the industry, there is no systematic methodologies on how it should be implemented or the trustworthiness of these methods. In fact, the Media Rating Council (MRC) announced that there are inconsistencies across multiple reports attempting to measure this metric. In order to understand the magnitude of the problem, we conduct an analysis of different methods to track viewable impressions. Then, we test a subset of geometric and strong interaction methods in a webpage registered in the worldwide ad-network ExoClick, which currently serves over 7 billion geo-targeted ads a day to a global network of 65000 web/mobile publisher platforms. We find that the Intersection Observer API is the method that detects more viewable impressions given its robustness towards the technological constraints that face the rest of implementations available. The motivation of this work is to better understand the limitations and advantages of such methods, which can have an impact at a standardisation level in online advertising industry, as well as to provide guidelines for future research based on the lessons learned.


I. INTRODUCTION
Advertisement has been used for many years to encourage consumers to acquire products, branding purposes and even to spread new ideas across society. The new technological era has made advertisement to go through a reimagination process moving from traditional media such as newspapers or billboards to digital media like television, desktop computers and mobile phones. Nowadays, advertisement is quite pervasive, even appearing in online games, social media, blogs and mobile applications [1]. It has also trespassed the boundaries of targeting global populations to a more personalised and efficient approach that is specially tailored to the interests of each individual by using recommendation engines powered by the "big data" era [2]- [4]. Within this heterogeneous context we focus on digital display advertising (shortened as ads from now on), which can be found frequently in websites and apps in the form of banners and other various ad formats.
According to a report of the Internet Advertising Bureau [5], the total expenses in online advertisement in the US during 2019 was 124.6 billion dollars, which represents a 16% more than in 2018. Many of the current online businesses and portals base completely their revenue models in earnings from online advertisement, allowing the end-user to have access to high quality contents or services free of charge [6]. Given also the rapid growth of Internet users around the world, research on online advertising has been evolving from studies very focused on user interaction with the ads to more nuanced approaches motivated by new ad formats or configuration possibilities, such as keyword targeting and the location of the ads within the site [7]. Louisa Ha [8] conducted a substantive review of online advertising research in order to understand the main areas of interest and current trends. In such review, she points out that both academic and industry researchers have different interests when conducting research about online advertising, being the first group focused on the advancement of the theory that can help model the field while the efforts of the latter group are more directed towards developing business-oriented applications, such as for generating more profit or gaining customers. However, there has been scarce research performed in collaboration between academic and industry researchers, despite the potential of combining both viewpoints and expertise areas. The work presented in this article contributes to fill the gap to understand viewability in online advertising, a concept that has been on the spotlight receiving a lot of attention for several years, but has not been completely standardised across online advertising stakeholders. To conduct this research, we collaborate with ExoClick 1 ad network, which is currently serving 7 billion geo-targeted ads a day to a global network of 65000 web/mobile publisher platforms, to perform a case study in their production environment.
The main motivation for spreading viewability metrics across online advertising is that recent studies [9]- [11] have found that more than half of the ads are not detected as viewable. The reasons behind these results are diverse, for example locating ads in a position of the webpage that consumers are unlikely to scroll to, the necessity of specific plugins to display ads or the use of ad blocker software, among others [12]. This has motivated stakeholders to start measuring viewable impressions, a metric which the IAB [13] has defined as an impression that satisfies a percentage of pixels and time requirements within the viewport. In plain words, this metric attempts to measure which impressions could have been consciously seen by the user. However, the Media Rating Council (MRC) released a summary [14] explaining that since there is no consistency across the results reported by different stakeholders when measuring viewable impressions, they do not encourage companies to start using it for monetization purposes just yet.
For this reason, this study aims to shed some light on the effectiveness of the methods that have been reported to be capable of measuring viewable impressions in the literature and the web. To accomplish these objectives, we first conduct a literature review of the viewability methods that have been proposed and we test some of these methods in a real world scenario with inventory from ExoClick's ad network. Our findings can help to inform how to achieve a more standardised online advertising ecosys-tem and new research on measuring viewability. More specifically, the main objectives of this work are: • To survey all the methods claimed to be able to track online advertising viewability and to compare their technical implementations, advantages and disadvantages. • To implement a set of methods that comply with IAB measurement guidelines in a website that contains three banners in different locations and that is registered in ExoClick ad-network, presenting the following results: -An analysis of the results by viewability method and their inter-agreement. -A cross-sectional analysis across the following dimensions: location of the banner, the device, browser and operating system (OS). In previous work [15] we present preliminary results on the variation across dimensions but we did not look exhaustively into the potential causes as well as the connection to the literature as we do in this case study. The rest of the paper is organized as follows: Section II presents the related work on viewability in online advertising and Section III analyzes all reported implementations to measure viewable impressions. In Section IV we describe our methods, by choosing and implementing a set of geometric and strong interaction implementations in a webpage registered in ExoClick. In Section V we perform the analysis and describe the results of the case study, and finally we present the conclusions and future work in Section VI.

II. STATE OF THE ART
In this section, we start by presenting the research performed both by academics and industry practitioners to evaluate the results obtained through ads. Then, we connect these studies with how viewability has emerged as a new metric and present related research that has sought to better understand it and how to implement it.
Given the volume of resources spent daily on online advertising it is essential for all stakeholders to be able to measure the performance and effectiveness of ads. Advertisers invest money to reach out potential customers in order to increase sales and profits, and thus they need to evaluate their return on the investment [16]. This is a challenging task since user perception of ads can be connected to numerous factors such as browsing behaviour (if the user is surfing the Internet aimlessly or not) [17] or the content of the webpage itself [18], among many others. Although there is no standard measure for ad effectiveness, click-through rate (CTR) has been widely used to measure user interest on a product [19], since each click that an ad receives, can be perceived as a user vote of relevance [20]. Pay per click payment method (PPC) was developed in 1998, which basically consists on advertisers paying per each click that their ad receives. Advertisers frequently seek that users purchase their products or services, and publishers want to increase their earnings coming from ads, therefore there has been a lot of research to understand the factors that influence CTR, such as ad features (colours, animations, etc [21], [22]) or locations [23], and also how to predict clicks given a user for adaptation purposes [24]- [26].
However, depending on the business goals, there might be many other actions from the user that can be valuable for the advertiser, such as filling in a form after the click or the installation of an app [27]. This has motivated another payment method called Pay Per Action/Acquisition (PPA) based on how many times the ad can trigger users to perform the specific goal or action desired by the advertiser. However, those actions have to be reported by the advertiser as the action is performed beyond the domain of the publisher, unlike clicks which are measured on the publisher side [28]. Nevertheless, in "IAB best practices for conducting online ad effectiveness" research [29], it was recommended to not longer use CTR as a measure of ad effectiveness. On one hand, not always high CTR is related to ac- VOLUME 4, 2016 quisitions or revenue [23] since after all, a click is just a first step before actually performing the desired action, such as acquiring the product or signing up for the service that the ad is promoting. On the other hand, average CTR value has been decreasing from 2-4% in 1998 to below 1% in 2004 [30]. One possible explanation to this decrease is that users have too much information online and they do not fully focus on what they are reading or watching [31] in fact, based on an analysis performed on the online magazine Slate, users rarely scroll further than halfway of an article [32]. Also, in [33], it has been reported that the probability of a user clicking on an ad grows logarithmically with the number of impressions the user receives. Another explanation to this, is what is called the "banner blindness phenomenon" [34], where users decide to ignore page elements that resemble banners while reading a webpage due to the negative consumers' responsiveness to them. This phenomenon is similar to the "party cocktail effect" defined for the first time as a neurological capacity of our brain to select what auditory stimulus we want to focus on (for example, to hear a conversion in a noisy room) [35]. This phenomenon of selection of sensory stimulus also appears with visual targets [36] and it might explain banner blindness. Thus, researchers have started wondering if users actually notice ads in the first place, even if they are located in a very visible position. Some studies have implemented eyetracking techniques and have measured banner recall on users to see if they look at banners or not (e.g., [18], [37]- [43]). Nevertheless, despite of banner blindness, in a pre-attentive level our subconsciousness sees the ad before deciding if we should pay attention to it or not [44]. In fact, in [45] they show that subliminal messages have a real effect on users behaviour if the message is relevant to the users' interests.
Besides users' interest in ads, another problem is that only half of the impressions displayed are not detected as viewable impressions [11], [12]. In 2014, the IAB defined a viewable im-pression as an ad impression contained in the viewable space of the browser window, on an in-focus browser tab and with a pre-established criteria such as a minimum percent of ad pixels and time that an ad is visible within the viewable space of the browser (post ad render) [13]. Moreover, strong interactions with an ad (e.g., a click) are considered as viewable impressions as well. Since advertisers are interested in promoting or achieving conversions through ads, it is important that those ads are at least viewable to their potential clients. With this idea in mind, a new pricing model was proposed in [46] based on the number of viewable impressions. Based on this pricing model, advertisers would be billed just for those ads that had the chance to be on the viewport of the user, instead of the current pricing models such as cost per mille (CPM) which refers to the price paid for every 1,000 served impressions. There has been research to understand better this measure and its implications in the economics of online advertising [9], [47]. Zhang et al., [48] studied how to measure ad impression viewability from a human-computer interaction perspective; they implemented different measurements of pixels and time exposure on ads and later they asked a subset of users to navigate through sites that contained those ads, and asked them using a questionnaire which ads they saw. Apparently, the best measurement that they found accordingly to what the users reported was to use at least 75% of the pixels in viewport and at least two continuous seconds. Another proposal by C. Wang et al. [49] was to predict the probability of scrolling in a website and to use this estimation for measuring those ads that are guaranteed to be viewable. They have continued that work proposing different models to predict scrolling probability in [50], [51].
However the MRC released a summary [14] explaining that although viewability measure is a strong step forward for the online advertising community, it still needs to evolve to reach a good consensus across the results reported by different advertisers, agencies and publishers. In fact, it has been pointed out as very important to have commonly defined metrics for consistency in reporting and analysis [52] since, without that, it is very difficult to have a baseline under which advertisers and publishers can make business based on fair common grounds. For this reason, in this work we propose to test different methods to track viewable impressions in order to start working towards understanding their limitations and their main advantages. These lessons learned and the case study presented in this manuscript, can help to reach consensus across online advertising stakeholders and stimulate this line of research to keep moving forward.

III. ANALYSIS OF VIEWABILITY METHODS
In this section, we present, to the best of our knowledge, the first analysis of all reported viewability methods found in the academic and practitioner literature. We also perform a comparison of these methods and we study if they can ensure all the IAB conditions for viewability.

A. DESCRIPTION
We group the methods in three main categories, and inside of each one, we describe the different implementations that belong to the category: 1) Geometric. This first category is based on the geometric properties of the ad relative to another element of the site [12].
• Relative position. This method utilises element.getBoundingClientRect JavaScript function to get the smallest rectangle that contains an element with its dimension properties in pixels. By using these coordinates we can estimate the relative position of the element with respect to the window viewport. • Element Intersection. The World Wide Web Consortium (W3C) has developed an API called "Intersection observer" [53] that asynchronously observes changes in the intersection of a targeted element with another element or with the document's viewport. That is to say, it detects when an object is intersecting the target that you have defined.
2) Browser optimisation. This second category relies on the fact that some browsers save resources when certain elements are not on the screen, then we can monitor the browser frame rate in order to know if the ad is being rendered in the viewport or not.
• Flash pixel. By inserting a Flash pixel of 1x1 dimensions on an HTML element, it is possible to monitor the frame rate of the browser and detect when this Flash pixel is within the viewport. • Throttle rendering pipeline. Some browsers have optimisation features for HTML5 content, and thus similarly to the previous Flash pixel implementation, we can infer when an HTML element is visible for a user or not by monitoring the browser frame rate.
3) Strong interactions. The last category is based on strong interactions with the ad, since if there is an interaction with it, that would imply that the user was able to see the ad [13].
• Mouseover. Although IAB does not consider the action of the user being with the cursor over an ad as a strong interaction, if the mouse has been over the ad during a continuous long period of time, it makes common sense to also infer that such impression should have been viewable. • Clicks. As it is defined by the IAB, if a user clicks on an ad we can consider that impression as viewable. VOLUME

B. COMPARISON
This subsection compares the advantages and disadvantages of each method. We have grouped the main attributes of each one in Table 1.
We observer that Relative Position method is the only one that does not support unfriendly iframes (i.e., an iframe hosting a source from a different domain to the site). There is a lot of controversy on the Internet about this practice. The Relative Position, Element Intersection API and mouseover event are the implementations that can satisfy the IAB constraints about ensuring that at least 50% of the ads pixels are on the viewport during at least one continuous second. We consider critical that the method can be implemented in a way that completely satisfies these requirements.
On the other hand, we should also point out the two strong interactions that we have considered, since they have a high confidence (precision) when measuring viewable impressions, although low recall. This means that, once a mouseover or click events are triggered, we can be fairly confident that the user saw the ad, but the absence of these events do not imply at all that the user did not see the ad. This confidence decreases in methods based on geometric techniques, since even if the ad is detected as being on the viewport during a time lapse enough for being able to be viewable for the user, this does not imply that the user saw the ad at the end. Finally, browser optimisation techniques can be considered to have a low confidence, since it is very complex to ensure that the 50% of the ads pixels were on the viewport during that time window.
About the technical requirements, note that most of the methods are dependent of the browser supporting JavaScript. Additionally, the browser optimisation methods are dependant of additional technologies, such as Flash or HTML5. As a summary, all of the methods present some limitations and advantages, so there is not a perfect solution at the moment. Thus, it might be wise to use a combination of several methods to measure viewable impressions.

IV. METHODOLOGY
In this section we explain the setup of our experiment to test the viewability methods. First, we present the selected viewability methods to implement in this website. Later, we go further in detail with the implementations, the website design and data collection.

A. SELECTED VIEWABILITY METHODS
From the existing methods analysed in previous section, we have selected to test in a production online advertising environment the geometric methods and the strong interactions. We have not tested the viewability methods based on browser optimisation due to the following reasons: 1) since we need to comply with the IAB standard to measure viewable impressions, we need to know the exact percentage of pixels rendered in the viewport. On the other hand, 2) the use of Flash to track viewability does not make sense currently since, according to Google, the use of Flash has dropped to 8% for Chrome users and it will be removed completely in Chrome 87 2 . Moreover, after December 2020 it will not be longer supported by Adobe 3 and the HTML5 implementation is working only in selected versions of few browsers. Therefore, we decided not to implement browser optimisation methods for this test as these do not allow to measure the percentage of rendered pixels and they do not seem to scale well across the wide spectrum of browsers available on the Internet. Therefore, the final viewability methods that we test for this case study are:

Viewability methods that we implement and use case examples
• Relative Position. We measure the position of the ad in the viewport and check if 50% of its pixels are on the viewport for one continuous second. • Intersection Observer API. We measure when the API detects half of the ad on the viewport during one continuous second. • Mouseover on an ad during at least one continuous second. • Click on an ad. 3 https://theblog.adobe.com/adobe-flash-update/ In Figure 1 we represent the methods selected in different possible scenarios. In Scenario A, we have an example where all the methods detect that the banners are not viewable because the user did not spend enough time on the site or because technical issues, such as not having JavaScript enabled in the browser. In Scenario B, the user loads the site and then the user clicks on the first banner that the user finds. However, such banner is using an unfriendly iframe and, therefore, the Relative Position implementation is not able to detect whether is viewable or not. In Scenario C, the user scrolls the site until finding the second banner, which calls its attention and the user places the mouse over the ad for a few seconds, but at the end the user leaves the site without clicking on the ad. In this scenario, all the methods but the click event measure the impression of that banner as viewable. The last scenario, we have a user that navigates through all the site without interacting with any ad. Thus, only the geometric methods detect the impression as viewable (if the constraints were met). VOLUME 4, 2016

B. EXPERIMENTAL DESIGN
The purpose of the experiment is to evaluate each viewability method separately and in combination. In order to do so, we want to test them under different scenarios (devices, browsers and operating systems). Moreover, we want to evaluate them also in different banner locations, in order to see if the results are consistent depending of the banner position. We test these implementations using banner ads of 300x250 pixels since a report of the MRC suggest that larger size ads may present some challenges in terms of meeting viewability thresholds [54]. Finally, we want to keep the focus of this study to the comparison between implementations under the same viewability conditions. Thus, the next steps that have been followed are as follows. First, we selected a site registered in ExoClick with three ad zones for banner ads of 300x250 pixels (the most widely used adformat size). The first banner ad zone (Banner 1) is located at the left-top corner, and an user would always be exposed to it when visiting the site. The second banner ad zone (Banner 2) is located a bit below needing some scrolling down to be viewable in any device. Finally, the last one (Banner 3) is at the bottom of the site and it requires the user to scroll down through the entire site to be visible (see Figure 2 for a mock-up of the design of this site). The site is designed to be responsive to the device and its resolution, and therefore the content is properly adjusted to the size of the screen. Then, once the experimente concluded and different campaigns appeared on these ad zones, we analysed the traffic and viewability data of this site to respond the objectives of the research.

C. DATA COLLECTION
After one day of traffic, we collected about a hundred thousand visits in our site and ad zones. In order to filter out noise from this traffic, we remove those visits that are using adblock software, web crawlers, hosting proxies or users that do not support JavaScript code. The output is a broad variety of traffic that can be representative enough of the whole Internet ecosystem, so that we can generate trustworthy analysis that could generalize to other similar case studies.
By analyzing the final data collection, we see that most of the traffic comes from Asia followed by Europe, America and Africa. More-over, the biggest percentage of impressions is from mobile devices, then desktop, and a little percentage from tablets. The most popular operating systems have been Android, followed by Windows and iOS. Finally, Chrome is the browser with more traffic, followed by In-App (native applications that can render web content like Facebook [55]), Firefox, Safari, Samsung Internet and Internet Explorer.

V. RESULTS AND DISCUSSION
This first subsection analyses the results of the reported viewability methods in a production environment in order to evaluate how different are their viewability measurements (Subsection V-A). We perform this analysis both for each method independently and by ensembling all methods together. Also we show how these measurements are dependent on the banner location, and how these locations are affected by the device used to navigate the site and finally, we compare each method under different crosssectional categories such as the browser used when navigating to the site (Subsection V-B).

A. ANALYSIS OF RESULTS BY METHOD
We group the data per user visit funnel (since a user enters the site for the first time until the user leaves the site) and we compare the impressions that were registered as viewable per each method; see Table 2 for an example on how this visit funnel looks like. This knowledge representation can allow us to closer investigate in what scenarios one method detects more viewable impressions than the others, and if there is consensus among the methods. Next, we take a look at the methods individually. In Table 3, we see that the Intersection Observer API has higher percentage of viewable impressions detected compared to the rest of methods (except for Banner 3), followed by Relative Position, mouseover and clicks. This order make sense since the natural funnel of viewability should be first the ad being served to the site, then appearing viewable to the user, afterwards the user moving the mouse over the ad and finally clicking in the ad (see Figure 1). A similar funnel is proposed in [56] where the process starts by creating product awareness, promoting interest to purchase and finishes with the eventual product acquisition. An interesting insight is that the viewability recall decreases as the funnel goes forward, but at the same time the certainty of that ad actually being consciously viewed by the user increases. In other words, the methods do not provide certainty regarding if an ad was seen even if it was viewable, but we are confident that the ad was viewed if the user clicked on it; at the same time, if a user did not put the mouseover or clicked on an ad, that does not necessarily mean that the user did not view it. This interplay between the methods is key to understand the problematic and potential of viewability.

1) Ensemble Method Results
Another possibility is the combination of all these methods together to detect viewable impressions. The main advantage of this approach is that we create an ensemble metric that gathers the strengths of each method in different scenarios, and so it can increase viewability detection. The main disadvantage is that in case we have false positives distributed across methods, we would be taking into account all of them. In order to compute this ensemble method, we apply a boolean OR operation, thus, if any of the methods detect the impression as viewable (v), this ensemble method categorizes it as viewable as well. Finally, we can have a combined percentage of viewable impressions (%V ) as follows: VOLUME 4, 2016 %V = 100 * un u0 (any(v 1 , ..., v n ) == 1) #impressions The output of this metric is also presented in Table 3. As one could expect, the values of the ensemble method %V are always higher than just using any of the methods alone. Nevertheless, this increase might not be due to the combination of all the methods but just a few. Finally, there is a percentage of impressions that is not detected as viewable even if we take all the methods together. This might sound particularly surprising when taking into account that the first banner of the webpage would be already within the viewport when the user access the site. Some reasons for those non-viewable impressions are: the user closing the site before the time requirement defined by IAB is met, the methods being unable to track it due to technical issues (e.g., browser version, user not enabling JavaScript code, etc.) or an inactive tab (among others) [10]. One future research direction would be to disentangle which of those nonviewable impressions were caused due to the user not complying with the IAB requirements (i.e., true non-viewable impressions) compared to the those cause due to technical issues or incompatibilities (which could be both false negatives or true non-viewable impressions). In order to accomplish this, we should expand the information that we log in about the user activity, for example by including the actual time that the user spends within the website or to the maximum percentage of pixels in the viewport for each ad, and not just when it is above the 50%; these indicators plus others can help us understand what is really happening for those non-viewable impressions.

2) Inter-agreement Between Methods
We can better understand the relationship between the different methods by computing their level of inter-agreement. For this purpose, we apply the Cohen's Kappa score [57]. This score is a statistic value that measures the agreement between two categorical items taking into account also the hypothetical probability of agreement occurring by chance. If two metrics are in complete agreement the score should be 1, and if there is no agreement at all, the score should be 0. Results are displayed in Figure 3 and we see that geometric methods are the one with higher ratio of agreement, with a value of 0.5. Given the overall results of inter-agreement, we conclude that all methods here are contributing and have an important role to achieve higher results when detecting viewable impressions.

B. ANALYSIS OF CROSS-SECTIONAL DIMENSIONS 1) Overview by Banner Location
This subsection unpacks the viewability results by the location of the banner. In Table 3, Banner 1, which is located at the top of the site, has almost twice as many viewable impressions detected than Banner 2 and almost four times more than Banner 3, which is at the bottom of the site. As one could expect, this clearly indicates with empirical data that location matters for viewability metrics. This makes sense since in order to see Banner 2 and Banner 3 the user needs to scroll the browser until these banners appear on the screen, whereas for Banner 1, no scrolling is needed with the browser to reach the banner. However, whether an impression was viewable or not, can be strongly affected by not only the location but by other dimensions as well, such as the device the user is using to access the site. For this reason, we calculate the combined percentage of viewable impressions (i.e., %V ) by device type and we show the results in Figure 4. We see that Banner 2 is more viewable in desktop devices than the others. However Banner 3, the one at the bottom of the website, is more viewable for mobile devices. This might indicate that the scroll depth is deeper, taking the user further down the website. We can formulate a couple of hypothesis, 1) that a higher engagement of smartphone users is associated with a deeper scroll depth and 2) that the screen touch actions and mechanics facilitate users to scroll further down the website than the mechanics of interacting with the scrollbar with the mouse.
One interesting step would be to benchmark our viewability results with other studies. Despite a lot of media and companies have discussed about viewability, the results obtained and the implementations used are not reported widely, but a previous report from Meetrics details a number of viewability metrics [10] for different ad types. We specifically use the viewability stat of this report for the billboard banner (which is a type of banner that appears at the top of a website) and compare it with our Banner 1 (also situated at the top of the website). Meetrics report found a 56% viewability for billboard banners, that we see is the same that we have detected in our case for Banner 1 with 57% of viewable impressions. This almost exact match is a good indication of correct technical implementation and perhaps towards finding universal viewability trend that can inform the improvement of the online advertising ecosystem.

2) Overview by Device, Browser and OS
This subsection analyzes the influence and variation on viewability due to the device, browser and OS. The ensemble method shows in Table  3 a percentage of viewable impressions regardless of the banner location (i.e., if any of the three banners was viewable during a visit) of 49.54%. We see that this value is lower than the percentage of viewable impressions of Banner 1. This is due to the influence of Banner 2 and Banner 3, which have less viewable impressions registered than Banner 1 although having the same number of impressions, and therefore it decreases the overall ratio of viewable impressions. However, as explained before, there are other cross-sectional dimension that affect whether an impression was viewable or not.
In our experiment, in order to see Banner 2 or Banner 3, the amount of scrolling needed from one device to another might change depend-  ing on the screen resolution and the interaction with the screen (through a mouse or through the touchscreen). Moreover, the type of browser used or the operating system might also influence the chances of the banners being viewable. In order to better understand these influences, we define a null hypothesis that these variables are not determinant in the prediction of viewability. We run a logistic regression and with a significance alpha threshold of 0.05 (alpha value agreed upon by statisticians and used in several statistical studies [58], [59]). We compute the pvalues, the probability of obtaining a measure equal to or more extreme than the one considering the null hypothesis to be true [60], of each variable and we see that all of them are lower than 0.001.
Since these p-values are lower than our alpha threshold, we can conclude the test to be significant and therefore we can discard the null hypothesis. This points us to further investigate these categorical variables. We decide to look at the average value of viewable impressions by each cross-sectional variable to see the role of each implementation in each scenario (see Figure 5). In previous work [15], we presented preliminary results of this analysis but in this case study we significantly expand on these insights and results.
For example, in Figure 5a we see that Intersection Observer API is the method that has the higher number of viewable impressions among all devices. Also it is interesting how mouseover method displays much lower percentage of viewable impressions for mobile and tablet devices than the rest, probably due to the fact that the mouseover action on an ad is not commonly performed by users because of the touchscreen. Additionally, Figure 5b shows an analogous visualization for all browsers except In-App, the Intersection Observer API detects the highest percentage of viewable impressions followed by the Relative Position. For Safari and Firefox it seems to be the unique one over the average of detections. Lastly, in Figure 5c for the operating systems, we also see that for iOS all methods are very close to the average of viewable impressions and the rest of operating system show a more natural distribution respect the viewability funnel. Note that both Linux and Mac OS were filtered out due to a traffic share a bit below 1%.

VI. CONCLUSIONS AND FUTURE WORK
The advertising ecosystem is a complex one that involves many stakeholders, such as advertisers, publishers, ad networks, affiliate networks or product owners. Despite all of them have different goals and objectives, the ecosystem agrees that more transparent and objective metrics to effectively measure which ads are reaching the public are necessary, and over the last years, viewability has become one of the most promising efforts in this direction. However, there has not been consensus in reaching common viewability metrics, and the efforts have been distributed across private industry researchers and practitioners.
In this paper we conduct the first academic survey of all the viewability implementation methods reported by academic publications and practitioners within the ad industry in order to test a subset of them in a real production environment. Specifically, we use a website registered in ExoClick ad-network and we analyze the number of viewable impressions detected by each viewability method by banner and across other cross-sectional variables. From such analysis we report the following main results: • We have implemented two geometric viewability methods, one based on the Intersection Observer API, and another one based on the Relative Position concept.
Our results indicate that the Intersection Observer API has detected as viewable a higher percentage of impressions, and given that this is a more mature API, it can be an optional choice for those trying to implement viewability metrics for the first time.
• We have presented a viewability funnel as follows: 1) the impression is served to the user 2) the impression becomes viewable by the user 3) the user might trigger a mouseover event 4) the user might click on the impression. We have observed how the presence of these events get narrower along the funnel, our results based on the combined ensemble metric show a 56% viewability for the first banner; this value is very similar to the one reported in 2019 by a Meetrics report [10]. This funnel can become a strong key performance indicator for the future online advertising ecosystem. • We have explored variations in the viewability funnel across the banners. Given the location of each banner, we have found drastic differences across them, with the first banner receiving many more viewable impressions, followed by Banner 2 with less than the half of Banner 1, and lastly Banner 3 with half of viewable impressions than Banner 2. Therefore, we see a strong influence of banner location in viewability metrics. One interesting difference was found when exploring variations across devices where we found higher viewability metrics in Banner 3 in the case of smartphone devices than for desktop and tablet. We hypothesised about the effect of the touch screen in the interaction mechanics as a potential cause.
Our viewability results are in agreement with others reported in the literature, indicating that a high number of the impressions are not detected as viewable. Future work should aim to unearth the potential reasons of why a high percentage of the impressions do not even become viewable to the user. As for open research issues, we aim to conduct case studies using humans to annotate a ground-truth dataset to detect false positives and negatives in the technical viewability methods, to collect richer data samples with more metrics that can serve as a deeper evaluation, broader case studies implementing these metrics across multiple sites or the development of more robust methods that can work well across the diversity of the World Wide Web. Additionally, we should also analyze how these measures and findings translate to other ad formats, such as in-video ads or native ads, in order to find which viewability patterns are universal vs. those that are format-dependent. Finally, future research should also aim to study the intersection of human attention and cognition theories with the aesthetic features of the ads, as this can hold promising new grounds towards understanding viewability. We expect that this study can motivate more stakeholders involved in the online advertising ecosystem to work towards the standardisation of viewability metrics in the ad industry, as these can have a very important role in their financial stability, policy guidelines and revenue models, that can then have a direct influence on the quality that Internet users will experience.