Optical Switch in the Middle (OSM) architecture for DCNs with Hadoop adaptations

Optical switching technologies offer a cost-and power-efficient approach for handling the DataCenter Network (DCN) oversubscription problem. We propose a hybrid DCN architecture named Optical Switch in the Middle (OSM), which offers increased flexibility (when compared to prior hybrid architectures) for supporting multiple simultaneous high-speed TOR-to-TOR paths through an Optical Circuit Switch (OCS) and a core-level Electrical Packet Switch (EPS). A multilayer SDN controller supports advanced-reservation scheduling of optical circuits, and the integration of storage in the core EPS increases the usage rate of optical circuits. To effectively use the OSM architecture, we propose four modifications to Hadoop, and illustrate the potential of this architecture for achieving higher compute-resource utilization while simultaneously offering users shorter job completion times.