Demo: Metamorphosis — A Consensus Protocol for Distributed Message Brokers

Metamorphosis is a consensus protocol derived from Raft with addition of a witness replica role. In this role, a replica does not store a full copy of data under normal mode of operation. The selection of a replica into this role is dynamic. It only occurs when the set of full replicas fails to complete a write fully within a timeout. Data replication using a consensus protocol with witnesses is attractive for building distributed message brokers. Message brokers are widely used to collect data over geographically separate locations. To achieve fault tolerance, replication creates redundant data copies. Metamorphosis reduces costs associated with redundancy by only storing complete data on a subset of replicas. Metamorphosis achieves near 1/3 cost reduction in operating distributed data ingestion, when compared to Apache Kafka, a popular message broker.