By Topic

Software Engineering, IEEE Transactions on

Issue 3 • Date March 2014

Filter Results

Displaying Results 1 - 6 of 6
  • An Observe-Model-Exercise* Paradigm to Test Event-Driven Systems with Undetermined Input Spaces

    Page(s): 216 - 234
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (5256 KB) |  | HTML iconHTML  

    System testing of software applications with a graphical-user interface (GUI) front-end requires that sequences of GUI events, that sample the application's input space, be generated and executed as test cases on the GUI. However, the context-sensitive behavior of the GUI of most of today's non-trivial software applications makes it practically impossible to fully determine the software's input space. Consequently, GUI testers-both automated and manual-working with undetermined input spaces are, in some sense, blindly navigating the GUI, unknowingly missing allowable event sequences, and failing to realize that the GUI implementation may allow the execution of some disallowed sequences. In this paper, we develop a new paradigm for GUI testing, one that we call Observe-Model-Exercise* (OME*) to tackle the challenges of testing context-sensitive GUIs with undetermined input spaces. Starting with an incomplete model of the GUI's input space, a set of coverage elements to test, and test cases, OME* iteratively observes the existence of new events during execution of the test cases, expands the model of the GUI's input space, computes new coverage elements, and obtains new test cases to exercise the new elements. Our experiment with 8 open-source software subjects, more than 500,000 test cases running for almost 1,100 machine-days, shows that OME* is able to expand the test space on average by 464.11 percent; it detected 34 faults that had never been detected before. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Governing Software Process Improvementsin Globally Distributed Product Development

    Page(s): 235 - 250
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1175 KB) |  | HTML iconHTML  

    Continuous software process improvement (SPI) practices have been extensively prescribed to improve performance of software projects. However, SPI implementation mechanisms have received little scholarly attention, especially in the context of distributed software product development. We took an action research approach to study the SPI journey of a large multinational enterprise that adopted a distributed product development strategy. We describe the interventions and action research cycles enacted over a period of five years in collaboration with the firm, which resulted in a custom SPI framework that catered to both the social and technical needs of the firm's distributed teams. Institutionalizing the process maturity framework got stalled initially because the SPI initiatives were perceived by product line managers as a mechanism for exercising wider controls by the firm's top management. The implementation mechanism was subsequently altered to co-opt product line managers, which contributed to a wider adoption of the SPI framework. Insights that emerge from our analysis of the firm's SPI journey pertain to the integration of the technical and social views of software development, preserving process diversity through the use of a multi-tiered, non-blueprint approach to SPI, the linkage between key process areas and project control modes, and the role of SPI in aiding organizational learning. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees

    Page(s): 251 - 265
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1454 KB) |  | HTML iconHTML  

    Modern software systems are increasingly configurable. While this has many benefits, it also makes some software engineering tasks,such as software testing, much harder. This is because, in theory,unique errors could be hiding in any configuration, and, therefore,every configuration may need to undergo expensive testing. As this is generally infeasible, developers need cost-effective technique for selecting which specific configurations they will test. One popular selection approach is combinatorial interaction testing (CIT), where the developer selects a strength t and then computes a covering array (a set of configurations) in which all t-way combinations of configuration option settings appear at least once. In prior work, we demonstrated several limitations of the CIT approach. In particular, we found that a given system's effective configuration space - the minimal set of configurations needed to achieve a specific goal - could comprise only a tiny subset of the system's full configuration space. We also found that effective configuration space may not be well approximated by t-way covering arrays. Based on these insights we have developed an algorithm called interaction tree discovery (iTree). iTree is an iterative learning algorithm that efficiently searches for a small set of configurations that closely approximates a system's effective configuration space. On each iteration iTree tests the system on a small sample of carefully chosen configurations, monitors the system's behaviors, and then applies machine learning techniques to discover which combinations of option settings are potentially responsible for any newly observed behaviors. This information is used in the next iteration to pick a new sample of configurations that are likely to reveal further new behaviors. In prior work, we presented an initial version of iTree and performed an initial evaluation with promising results. This paper presents an improved iTree algorithm in greater detail. The - ey improvements are based on our use of composite proto-interactions - a construct that improves iTree's ability to correctly learn key configuration option combinations, which in turn significantly improves iTree's running time, without sacrificing effectiveness. Finally, the paper presents a detailed evaluation of the improved iTree algorithm by comparing the coverage it achieves versus that of covering arrays and randomly generated configuration sets, including a significantly expanded scalability evaluation with the ~ 1M-LOC MySQL. Our results strongly suggest that the improved iTree algorithm is highly scalable and can identify a high-coverage test set of configurations more effectively than existing methods. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Magiclock: Scalable Detection of Potential Deadlocks in Large-Scale Multithreaded Programs

    Page(s): 266 - 281
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (4254 KB) |  | HTML iconHTML  

    We present Magiclock, a novel potential deadlock detection technique by analyzing execution traces (containing no deadlock occurrence) of large-scale multithreaded programs. Magiclock iteratively eliminates removable lock dependencies before potential deadlock localization. It divides lock dependencies into thread specific partitions, consolidates equivalent lock dependencies, and searches over the set of lock dependency chains without the need to examine any duplicated permutations of the same lock dependency chains. We validate Magiclock through a suite of real-world, large-scale multithreaded programs. The experimental results show that Magiclock is significantly more scalable and efficient than existing dynamic detectors in analyzing and detecting potential deadlocks in execution traces of large-scale multithreaded programs. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Variability in Software Systems—A Systematic Literature Review

    Page(s): 282 - 306
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (13628 KB) |  | HTML iconHTML  

    Context: Variability (i.e., the ability of software systems or artifacts to be adjusted for different contexts) became a key property of many systems. Objective: We analyze existing research on variability in software systems. We investigate variability handling in major software engineering phases (e.g., requirements engineering, architecting). Method: We performed a systematic literature review. A manual search covered 13 premium software engineering journals and 18 premium conferences, resulting in 15,430 papers searched and 196 papers considered for analysis. To improve reliability and to increase reproducibility, we complemented the manual search with a targeted automated search. Results: Software quality attributes have not received much attention in the context of variability. Variability is studied in all software engineering phases, but testing is underrepresented. Data to motivate the applicability of current approaches are often insufficient; research designs are vaguely described. Conclusions: Based on our findings we propose dimensions of variability in software engineering. This empirically grounded classification provides a step towards a unifying, integrated perspective of variability in software systems, spanning across disparate or loosely coupled research themes in the software engineering community. Finally, we provide recommendations to bridge the gap between research and practice and point to opportunities for future research. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems

    Page(s): 307 - 323
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (1284 KB) |  | HTML iconHTML  

    How do you test a program when only a single user, with no expertise in software testing, is able to determine if the program is performing correctly? Such programs are common today in the form of machine-learned classifiers. We consider the problem of testing this common kind of machine-generated program when the only oracle is an end user: e.g., only you can determine if your email is properly filed. We present test selection methods that provide very good failure rates even for small test suites, and show that these methods work in both large-scale random experiments using a “gold standard” and in studies with real users. Our methods are inexpensive and largely algorithm-independent. Key to our methods is an exploitation of properties of classifiers that is not possible in traditional software testing. Our results suggest that it is plausible for time-pressured end users to interactively detect failures-even very hard-to-find failures-without wading through a large number of successful (and thus less useful) tests. We additionally show that some methods are able to find the arguably most difficult-to-detect faults of classifiers: cases where machine learning algorithms have high confidence in an incorrect result. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.

Aims & Scope

The IEEE Transactions on Software Engineering is interested in well-defined theoretical results and empirical studies that have potential impact on the construction, analysis, or management of software. The scope of this Transactions ranges from the mechanisms through the development of principles to the application of those principles to specific environments. Specific topic areas include: a) development and maintenance methods and models, e.g., techniques and principles for the specification, design, and implementation of software systems, including notations and process models; b) assessment methods, e.g., software tests and validation, reliability models, test and diagnosis procedures, software redundancy and design for error control, and the measurements and evaluation of various aspects of the process and product; c) software project management, e.g., productivity factors, cost models, schedule and organizational issues, standards; d) tools and environments, e.g., specific tools, integrated tool environments including the associated architectures, databases, and parallel and distributed processing issues; e) system issues, e.g., hardware-software trade-off; and f) state-of-the-art surveys that provide a synthesis and comprehensive review of the historical development of one particular area of interest.

Full Aims & Scope

Meet Our Editors

Editor-in-Chief
Matthew B. Dwyer
Dept. Computer Science and Engineering
256 Avery Hall
University of Nebraska-Lincoln
Lincoln, NE 68588-0115 USA
tseeicdwyer@computer.org