Efficient user-space protocol implementations with QoS guarantees using real-time upcalls

Two important requirements for protocol implementations to be able to provide quality of service (QoS) guarantees within the endsystem are: (1) efficient processor scheduling for application and protocol processing and (2) efficient mechanisms for data movement. Scheduling is needed to guarantee that the application and protocol tasks involved in processing each stream execute in a timely manner and obtain their required share of the CPU. We have designed and implemented an operating system (OS) mechanism called the real-time upcall (RTU) to provide such guarantees to applications. The RTU mechanism provides a simple real-time concurrency model and has minimal overheads for concurrency control and context switching compared to thread-based approaches. To demonstrate its efficacy, we have built RTU-based transmission control protocol (TCP) and user datagram protocol (UDP) protocol implementations that combine high efficiency with guaranteed performance. For efficient data movement, we have implemented a number of techniques such as: (1) direct movement of data between the application and the network adapter; (2) batching of input-output (I/O) operations to reduce context switches; and (3) header-data splitting at the receiver to keep bulk data page aligned. Our RTU-based user-space TCP/Internet protocol (TCP/IP) implementation provides bandwidth guarantees for bulk data connections even with real-time and "best-effort" load competing for CPU on the endsystem. Maximum achievable throughput is higher than the NetBSD kernel implementation due to efficient data movement. Sporadic and small messages with low delay requirements are also supported using reactive RTUs that are scheduled with very low delay. We believe that ours is the first solution that combines good data path performance with application-level bandwidth and delay guarantees for standard protocols and OSs.