By Topic

High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium on

Date 24-27 July 2005

Filter Results

Displaying Results 1 - 25 of 79
  • Adaptive data block scheduling for parallel TCP streams

    Publication Year: 2005 , Page(s): 265 - 275
    Cited by:  Papers (6)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (603 KB) |  | HTML iconHTML  

    Applications that use parallel TCP streams to increase throughput must multiplex and demultiplex data blocks over a set of TCP streams transmitting on one or more network paths. When applications use the obvious round robin scheduling algorithm for multiplexing data blocks, differences in transmission rate between individual TCP streams can lead to significant data block reordering. This forces the demultiplexing receiver to buffer out-of-order data blocks, consuming memory and potentially causing the receiving application to stall. This paper describes a new adaptive weighted scheduling approach for multiplexing data blocks over a set of parallel TCP streams. Our new scheduling approach, compared with the scheduling approached used by GridFTP, reduces reordering of data blocks between individual TCP streams, maintains the aggregate throughput gains of parallel TCP, consumes less receiver memory for buffering out-of-order packets, and delivers smoother application goodput. We demonstrate the improved characteristics of our new scheduling approach using data transmission experiments over real and emulated wide-area networks. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Cost-based scheduling for data-intensive applications on global grids

    Publication Year: 2005 , Page(s): 310 - 311
    Cited by:  Patents (3)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (121 KB) |  | HTML iconHTML  

    We present an algorithm for scheduling distributed data intensive bag-of-task applications on data grids that have costs associated with requesting, transferring and processing datasets. We evaluate the algorithm on a data grid testbed and present the results. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • State and events for Web services: a comparison of five WS-resource framework and WS-notification implementations

    Publication Year: 2005 , Page(s): 3 - 13
    Cited by:  Papers (16)  |  Patents (7)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (865 KB) |  | HTML iconHTML  

    The Web services resource framework defines conventions for managing state in distributed systems based on Web services, and WS-notification defines topic-based publish/subscribe mechanisms. We analyze five independent and quite different implementations of these specifications from the perspectives of architecture, functionality, standards compliance, performance, and interoperability. We identify both commonalities among the different systems (e.g., similar dispatching and SOAP processing mechanisms) and differences (e.g., security, programming models, and performance). Our results provide insights into effective implementation approaches. Our results may also provide application developers, system architects, and deployers with guidance in identifying the right implementation for their requirements and in determining how best to use that implementation and what to expect with regard to performance and interoperability. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Design and implementation tradeoffs for wide-area resource discovery

    Publication Year: 2005 , Page(s): 113 - 124
    Cited by:  Papers (31)  |  Patents (3)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (726 KB) |  | HTML iconHTML  

    This paper describes the design and implementation of SWORD, a scalable resource discovery service for wide-area distributed systems. In contrast to previous systems, SWORD allows users to describe desired resources as a topology of interconnected groups with required intragroup, intergroup, and per-node characteristics, along with the utility that the application derives from various ranges of values of those characteristics. This design gives users the flexibility to find geographically distributed resources for applications that are sensitive to both node and network characteristics, and allows the system to rank acceptable configurations based on their quality for that application. We explore a variety of architectures to deliver SWORD's functionality in a scalable and highly-available manner. A 1000-node ModelNet evaluation using a workload of measurements collected from PlanetLab shows that an architecture based on 4-node server cluster sites at network peering facilities outperforms a decentralized DHT-based resource discovery infrastructure for all but the smallest number of sites. While such a centralized architecture shows significant promise, we find that our decentralized implementation, both in emulation and running continuously on over 200 PlanetLab nodes, performs well while benefiting from the DHT's self-healing properties. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Design and evaluation of a new and effective fairness scheme for multicasting in Internet-scale distributed systems

    Publication Year: 2005 , Page(s): 285 - 286
    Cited by:  Papers (1)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (114 KB) |  | HTML iconHTML  

    This paper argues that simply applying a multiple-tree scheme does not provide sufficient fairness for applications in an Internet-scale distributed system, in terms of performance. Motivated from the observation of the tax and donation systems in our society, we believe that a better way to define fairness, for performance's sake, is to factor in nodes' proportional contributions because it provides the opportunity to support many simultaneous multicasting sessions. This paper then presents a protocol, called FairOM (Fair Overlay Multicast), to enforce proportional contribution among peers in an Internet-scale distributed system. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Scheduling strategies for mapping application workflows onto the grid

    Publication Year: 2005 , Page(s): 125 - 134
    Cited by:  Papers (37)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (579 KB) |  | HTML iconHTML  

    In this work, we describe new strategies for scheduling and executing workflow applications on grid resources using the GrADS [Ken Kennedy et al., 2002] infrastructure. Workflow scheduling is based on heuristic scheduling strategies that use application component performance models. The workflow is executed using a novel strategy to bind and launch the application onto heterogeneous resources. We apply these strategies in the context of executing EMAN, a bio-imaging workflow application, on the grid. The results of our experiments show that our strategy of performance model based, in-advance heuristic workflow scheduling results in 1.5 to 2.2 times better makespan than other existing scheduling strategies. This strategy also achieves optimal load balance across the different grid sites for this application. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Techniques for tuning workflows in cluster environments

    Publication Year: 2005 , Page(s): 303 - 305
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (128 KB) |  | HTML iconHTML  

    An important class of parallel processing jobs on clusters today are workflow-based applications that process large amounts of data in parallel. Traditional cluster performance tools are designed for tightly coupled parallel jobs, and not as effective for this type of application. We describe how the NetLogger Toolkit methodology is more appropriate for this class of cluster computing, and describe our new automatic workflow anomaly detection component. We also describe how this methodology is being used by the Nearby Supernova Factory (SNfactory) project at Lawrence Berkeley National Laboratory. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Interest-aware information dissemination in small-world communities

    Publication Year: 2005 , Page(s): 167 - 175
    Cited by:  Papers (11)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (539 KB) |  | HTML iconHTML  

    Information dissemination is a fundamental and frequently occurring problem in large, dynamic, distributed systems. We propose a novel approach to this problem, interest-aware information dissemination, that takes advantage of small-world usage patterns in data-sharing communities. These small-world characteristics suggest that users naturally form groups of common interest. We propose algorithms for identifying these groups dynamically, without a need for explicit classification of topics or declaration of user interests. These algorithms use information about the data consumed by users to identify, via online computation, groups with similar interests. As a proof of concept, we apply this methodology to the problem of locating files in large user communities. Using real-world traces from a scientific community and from a peer-to-peer system, we show that proactive information dissemination within groups of common interest can reduce the search load by up to 70%. In addition, this approach naturally supports the efficient discovery of collections of files, a requirement specific to scientific data analysis tasks. We hypothesize that our algorithms can find numerous other uses in distributed systems, such as reputation management. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Cluster delegation: high-performance, fault-tolerant data sharing in NFS

    Publication Year: 2005 , Page(s): 100 - 109
    Cited by:  Papers (4)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (582 KB) |  | HTML iconHTML  

    We present cluster delegation, an enhancement to the NFSv4 file system, that improves both performance and recoverability in computing clusters. Cluster delegation allows data sharing among clients by extending the NFSv4 delegation model so that multiple clients manage a single file without interacting with the server. Based on cluster delegation, we implement a fast commit primitive, cooperative caching, and the ability to recover the uncommitted updates of a failed computer. Cluster delegation supports both read and write operations in the cooperative cache, while preserving the consistency guarantees of NFSv4. We have implemented cluster delegation by modifying the Linux NFSv4 client and show that it improves client performance and reduces server load by more than half. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Genetic algorithm based automatic data partitioning scheme for HPF

    Publication Year: 2005 , Page(s): 289 - 290
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (86 KB) |  | HTML iconHTML  

    The performance of a parallel program depends largely on its data partitions. So a good data partitioning scheme is the need of the time. However it is very difficult to arrive at a good solution as the number of possible data partitions for a given real life program is exponential in the size of the program. We present a heuristic technique for automatic data partitioning for HPF. Our approach is based on genetic algorithms and is very simple, yet very efficient to quickly find appropriate data partitions even for large programs with large number of alternatives for data distribution. It makes use of both static as well as dynamic data distribution with the main aim of reducing the overall execution time of the entire program. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Enabling self-management of component-based high-performance scientific applications

    Publication Year: 2005 , Page(s): 59 - 68
    Cited by:  Papers (2)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (721 KB) |  | HTML iconHTML  

    Emerging high-performance parallel/distributed scientific applications and environments are increasingly large, dynamic and complex. As a result, it requires programming systems that enable the applications to detect and dynamically respond to changing requirements, state and execution context by adapting their computational behaviors and interactions. In this paper, we present such a programming system that extends the common component architecture to enable self-management of component-based scientific applications. The programming system separates and categorizes operational requirements of scientific applications, and allows them to be specified and enforced at runtime through reconfiguration, optimization and healing of individual components and the application. Two scientific simulations are used to illustrate the system and its self-managing behaviors. A performance evaluation is also presented. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Calana: a general-purpose agent-based grid scheduler

    Publication Year: 2005 , Page(s): 279 - 280
    Cited by:  Papers (1)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (128 KB) |  | HTML iconHTML  

    Grid resource allocation is a complex task usually solved by systems relying on a centralized information system. In order to create a lightweight scheduling system, we investigated the potential of auctions for resource allocation. Each resource provider runs an agent bidding on the execution of software with respect to local restrictions. This way, the information system becomes obsolete. In addition, each provider can implement different bidding strategies in order to reflect his preferences. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Distributed streaming query planner in Calder system

    Publication Year: 2005 , Page(s): 316 - 317
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (124 KB) |  | HTML iconHTML  

    The contribution of this work has two folds. First, we extend the current query planners' cost metric space by introducing network bandwidth cost, query deployment cost and query re-using cost; second, we develop a suite of algorithms for re-using existing query fragments under different scenarios. One of the most important reusable queries is called structure-sharable query. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The resource oriented authorization manager (ROAM)

    Publication Year: 2005 , Page(s): 308 - 309
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (126 KB) |  | HTML iconHTML  

    The resource oriented authorization manager (ROAM) was created to provide a simple but flexible authorization system for the National Fusion Grid (FusionGrid). This system builds on and extends previous community efforts by both responding to access authorization requests and by providing a Web interface/or resource management. ROAM works with the globus resource allocation manager (GRAM), and is general enough to be used by other virtual organizations that use Globus middleware or X.509/TLS authentication schemes to secure a grid of distributed resources. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Identity boxing: secure user-level containment for the grid

    Publication Year: 2005 , Page(s): 299 - 300
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (121 KB) |  | HTML iconHTML  

    Today, a public key infrastructure allows grid users to be identified with strong cryptographic credentials and and a descriptive, globally-unique name such as /O=UnivNowhere/CN=Fred. This powerful security infrastructure allows users to perform a single login and then access a variety of remote resources on the grid without further authentication steps. However, once connected to a specific system, a user's grid credentials must somehow be mapped to a local namespace. This creates a significant burden upon the administrator of each site to manage a continuously-changing user list. Large systems have worked around this by employing the old insecure standby of shared user accounts. A single user may be known by a different account name at every single site that he or she accesses, in addition to a variety of identity names given by certificate authorities. In order to access a resource, the user may need to have a local account generated. In order to share resources, each user must know the local identities of users that he/she wishes to share with. To solve these problems, we introduce the technique of identity boxing. An identity box is a well-defined execution space in which all processes and resources are associated with an external identity that need not have any relationship to the set of local accounts. That is, within an identity box, a program runs with an explicit grid identity string rather than with a simple integer UID. As a program executes, all access controls are performed using the high level name rather than the low-level account information. A single Unix account may be used to securely manage several identity boxes simultaneously, thus eliminating the need to services to run as root merely to change identities. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • 411 on scalable password service

    Publication Year: 2005 , Page(s): 211 - 221
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (596 KB) |  | HTML iconHTML  

    In this paper we present 411, a password distribution system for high performance environments that provides security and scalability. We show that existing solutions such as NIS and Kerberos do not provide sufficient performance in large, tightly coupled systems such as computational clusters. Unlike existing single-signon services, the 411 design removes the need for communication during password lookup by using aggressive replication techniques. We demonstrate the use of shared keys to efficiently protect user information, and the careful management of system wide consistency and fault tolerance. A theoretical analysis of the behavior of 411 is matched with quantitative evidence of its performance and suitability to a clustered environment. We further show the system effectively responds to stress by simulating 50% message loss on a 60-node cluster. This protocol is currently used worldwide in hundreds of Rocks-based production systems to provide password and login information service. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Assessment and enhancement of meta-schedulers for multi-site job sharing

    Publication Year: 2005 , Page(s): 144 - 153
    Cited by:  Papers (4)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (531 KB) |  | HTML iconHTML  

    Meta-schedulers such as the LSF MultiCluster scheduler and the Moab meta-scheduler (Silver) are being deployed on clusters interconnected over the grid. A primary goal of such meta-schedulers is to share jobs amongst the individual local sites, balancing the load and improving utilization and turnaround time. This work focuses on current methodologies used in implemented meta-schedulers, such as LSF MultiCluster and Silver. The benefits and disadvantages of both centralized and delegated modes are evaluated. Focusing on the negative impact of meta-schedulers to jobs originating from lightly loaded sites, enhancements are proposed and evaluated via trace-driven simulation. It is shown that it is feasible to reduce the detrimental impact on lightly loaded sites while maintaining excellent overall performance. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Recording and using provenance in a protein compressibility experiment

    Publication Year: 2005 , Page(s): 201 - 208
    Cited by:  Papers (13)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (498 KB) |  | HTML iconHTML  

    Very large scale computations are now becoming routinely used as a methodology to undertake scientific research. In this context, 'provenance systems' are regarded as the equivalent of the scientist's logbook for in silico experimentation: provenance captures the documentation of the process that led to some result. Using a protein compressibility analysis application, we derive a set of generic use cases for a provenance system. In order to support these, we address the following fundamental questions: what is provenance? How to record it? What is the performance impact for grid execution? What is the performance of reasoning? In doing so, we define a technology-independent notion of provenance that captures interactions between components, internal component information and grouping of interactions, so as to allow us to analyze and reason about the execution of scientific processes. In order to support persistent provenance in heterogeneous applications, we introduce a separate provenance store, in which provenance documentation can be stored, archived and queried independently of the technology used to run the application. Through a series of practical tests, we evaluate the performance impact of such a provenance system. In summary, we demonstrate that provenance recording overhead of our prototype system remains under 10% of execution time, and we show that the recorded information successfully supports our use cases in a performant manner. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • A framework for efficient inconsistency detection in a grid and Internet-scale distributed environment

    Publication Year: 2005 , Page(s): 318 - 319
    Cited by:  Papers (1)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (106 KB) |  | HTML iconHTML  

    In this paper, we argue that a broad range of Internet-scale distributed applications can benefit from an underlying low-cost consistency detection framework - an alternative to inconsistency avoidance that can detect inconsistency among nodes sharing data or services in a timely manner. After introducing a framework of inconsistency detection, this paper presents the design and evaluation of a two-layer inconsistency detection module. The proposed two-layer inconsistency detection module is evaluated by both analysis and simulations. The results show that this framework can significantly reduce the time to detect inconsistency among nodes without adding much maintenance cost. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • GWiQ-P: an efficient decentralized grid-wide quota enforcement protocol

    Publication Year: 2005 , Page(s): 222 - 232
    Cited by:  Papers (1)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (597 KB) |  | HTML iconHTML  

    Mega grids span several continents and may consist of millions of nodes and billions of tasks executing at any point in time. This setup calls for scalable and highly available resource utilization control that adapts itself to dynamic changes in the grid environment as they occur. In this paper, we address the problem of enforcing upper bounds on the consumption of grid resources. We propose a grid-wide quota enforcement system, called GWiQ-P. GWiQ-P is light-weight, and in practice is infinitely scalable, satisfying concurrently any number of resource demands, all within the limits of a global quota assigned to each user. GWiQ-P adapts to dynamic changes in the grid as they occur, improving future performance by means of improved locality. This improved performance does not impair the system's ability to respond to current requests, tolerate failures, or maintain the allotted quota levels. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Supporting application-tailored grid file system sessions with WSRF-based services

    Publication Year: 2005 , Page(s): 24 - 33
    Cited by:  Papers (4)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (623 KB) |  | HTML iconHTML  

    This paper presents novel service-based grid data management middleware that leverages standards defined by WSRF specifications to create and manage dynamic grid file system sessions. A unique aspect of the service is that the sessions it creates can be customized to address application data transfer needs. Application-tailored configurations enable selection of both performance-related features (block-based partial file transfers and/or whole-file transfers, cache parameters and consistency models) and reliability features (file system copy-on-write checkpointing to aid recovery of client-side failures; replication, autonomous failure detection and data access redirection for server-side failures). These enhancements, in addition to cross-domain user identity mapping and encrypted communication, are implemented via user level proxies managed by the service, requiring no changes to existing kernels. Sessions established using the service is mounted as distributed file systems and can be used transparently by unmodified binary applications. The paper analyzes the use of the service to support virtual machine based grid systems and workflow execution, and also reports on the performance and reliability of service managed wide-area file system sessions with experiments based on scientific applications (NanoMOS/Matlab, CHID, GAUSS and SPECseis). View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • HMPI - hybrid MPI

    Publication Year: 2005 , Page(s): 306 - 307
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (118 KB) |  | HTML iconHTML  

    This paper presents the HMPI, a runtime system to integrate several MPI implementations, used to develop high performance applications that must run both in nodes with several operating systems and clusters of clusters infrastructures. HMPI has two different approaches to achieve this integration, using connection pools and HMPI daemons. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • The impact of Web service integration on grid performance

    Publication Year: 2005 , Page(s): 14 - 23
    Cited by:  Papers (5)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (603 KB) |  | HTML iconHTML  

    The past few years have seen an increasingly tight link between grid computing and Web services, with the latest standards defining a grid computing architecture as a set of services built using Web services standards and protocols. However, the reputation of these technologies (SOAP, XML, WSDL, HTTP) is that they are heavyweight and slow, something that is potentially a concern given the current and anticipated application mix for high performance grid architectures. This paper reports the results of a performance evaluation carried out on Globus 3.9.4, a reference implementation of the new GGF standards that are built on the Web services resource framework (WSRF). The evaluation approach combines low interference measurement (black box) techniques with more sophisticated sampling-based profiling (gray box) techniques. The results indicate possible opportunities for optimization, as well as provide useful input for the designers of grid services. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Lerna: an active storage framework for flexible data access and management

    Publication Year: 2005 , Page(s): 176 - 187
    Cited by:  Papers (3)
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (763 KB) |  | HTML iconHTML  

    In the present paper, we examine the problem of supporting application-specific computation within a network file server. Our objectives are (i) to introduce an easy to use yet powerful architecture for executing both custom-developed and legacy applications close to the stored data, (ii) to investigate the performance improvement that we get from data proximity in I/O-intensive processing, and (in) to exploit the I/O-traffic information available within the file server for more effective resource management. One main difference from previous active storage research is our emphasis on the expressive power and usability of the network server interface. We describe an extensible active storage framework that we built in order to demonstrate the feasibility of the proposed system design. We show that accessing large datasets over a wide-area network through a regular file system can penalize the system performance, unless application computation is moved close to the stored data. Our conclusions are substantiated through experimentation with a popular multilayer map warehouse application. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.
  • Distributed application management using Plush

    Publication Year: 2005 , Page(s): 281 - 282
    Save to Project icon | Request Permissions | Click to expandQuick Abstract | PDF file iconPDF (138 KB) |  | HTML iconHTML  

    Recent computing trends have shown an increase in the demand for large-scale, distributed, federated computing environments. Two of the more popular environments that have emerged are the grid and PlanetLab. At a high level, these systems are similar in many ways; both are comprised of a set of heterogeneous interconnected machines that allows secure resource sharing for a variety of different users and applications. However, at a lower level, the systems are very distinct in the sense that they were designed to solve different types of problems, and therefore have fundamental differences that make it difficult to develop and deploy applications on both platforms. As a result, application designers and researchers create software that runs on either the grid or PlanetLab, but not both. We propose to solve this problem by describing a common abstraction for both PlanetLab and grid applications. Further, we present Plush - a tool that implements the distributed application abstraction by providing a pluggable and extensible infrastructure allowing users to customize their environment for running experiments on both PlanetLab and the grid. View full abstract»

    Full text access may be available. Click article title to sign in or learn about subscription options.