Dynamic Byzantine Broadcast in Asynchronous Message-Passing Systems

The reconfiguration problem is considered a key challenge in distributed systems, especially in dynamic asynchronous message-passing systems. To keep the data reliability and availability in long-lived systems, any protocols should support reconfigurations, to dynamically add resources, or remove old and slow machines with newer faster ones. Previous results in reconfigurations either rely on consensus, or study the problem restricted to crash failures only. However, it is difficult to argue that real-world systems experience crash failures only. In this paper, we study the dynamic reconfiguration problem in fully asynchronous message-passing systems with Byzantine faults. We first specify dynamic Byzantine broadcast, and then specify a clean and explicit liveness condition. We show that dynamic Byzantine broadcast is solvable by presenting a dynamic Byzantine consistent broadcast algorithm and a dynamic Byzantine reliable broadcast algorithm.


I. INTRODUCTION
Replicating service state is a common technique to design a reliable distributed system. However, in long-lived systems, a typical fault-tolerant protocol may be inadequate, because the possibility of losing more than a minority of participants unavoidably grows with time. In addition, the need to react to changes in application workloads requires the system to dynamically add resources or replace old and slow machines with newer and faster ones. Real-world distributed systems need to be dynamic, i.e., they need to update the active members of a system over time. Such a churn support can be realized by providing interfaces to reconfiguration operations, i.e., add or remove participants.
It is already a challenge to design fault-tolerant distributed services that provide strong availability and consistency. Further requirements on supporting updating participants dynamically complicate any system design.
The first challenge in reconfiguration problems is safety when concurrent reconfiguration requests are submitted to the system. Consider an example from [1] where we build The associate editor coordinating the review of this manuscript and approving it for publication was Daniel Grosu . a fault-tolerant solution using four nodes p 1 , p 2 , p 3 , and p 4 . As shown in Figure 1, a user adds a new node p 5 to the system through p 1 (i.e. by issuing a adding command at node p 1 ). The up-to-date system configuration is stored at a majority of nodes of the current configuration, e.g., at nodes p 1 , p 4 and p 5 . At the same time, the removal of p 4 is initiated by a reconfiguration operation at p 3 . If p 2 and p 3 do not know that node p 5 should be added to the system, they consider themselves the majority of the latest configuration in the system. When these two concurrent reconfigurations occur, the system diverges to two different configurations. Early work on system dynamics could violate safety in such cases [2], [3], [4]. This problem has been solved in many subsequent works by establishing a certain update order [5], [6], [7]. This is equivalent to achieving an agreement in the system or assuming an external, replicated configuration manager [8], [9]. Second, the system's liveness is in danger if arbitrary reconfigurations are allowed. Consider an example from [10]: Three nodes, p 1 , p 2 , and p 3 are in a fault-tolerant system. The latest system state is stored with a majority of nodes of the current configuration, e.g. p 1 and p 2 . If a user removes node p 1 , node p 2 becomes absolutely vital, and cannot crash anymore. Otherwise, the remaining nodes cannot keep the correct system state. This problem will be even more severe if we introduce Byzantine failures to the system. Therefore, it is necessary to regularize a general characterization of tolerable failures for protecting a system's liveness.
As we know, it has been proved that achieving agreement is impossible in a fully asynchronous message-passing distributed system in the existence of even one faulty node [11]. Recent works [10], [12], [13], [14], [15] focus on dynamic distributed systems without consensus. Most of previous reconfiguration results in asynchronous systems without consensus assume that nodes always behave correctly. However, in reality, malicious nodes may arbitrarily attack the system. Thus, the system must be robust against Byzantine nodes.
In this paper, we study the dynamic problem in asynchronous systems with Byzantine faults. We are given an asynchronous message-passing distributed system composed of a set = {p, q, . . . } of interconnected nodes. We want to design protocols that provide interfaces to reconfiguration operations with existence of Byzantine nodes.
In summary, this paper features in following contributions.
• We define a dynamic Byzantine broadcast problem including an explicit liveness condition, which is independent of a particular solution to the problem.
• We solve the dynamic Byzantine problem by designing a dynamic Byzantine consistent broadcast protocol and a dynamic Byzantine reliable broadcast protocol in a completely asynchronous message-passing system.
We follow the framework of dynamic reconfiguration without consensus in the crash-prone model and adjust it to Byzantine failures. Our algorithms realize consensus-less reconfiguration operations with Byzantine failures.
Compared to other dynamic broadcast protocols [16], [17], [18], [19], [20], our results do not require a completely ordered sequence of configurations, or equivalent agreement among nodes. Moreover, our design can be implemented in fully asynchronous systems. Also, our protocols are highly modular, and not restricted to any specific broadcast primitives. In this paper, we present two different broadcast protocols and discuss how to extend our protocols to further models.
The rest of this paper is organized as follows: In the next section, we give an overview of related work.
Section 3 defines the model that we study in this paper. In Section 4, we discuss the dynamic problem with Byzantine failures. Section 5 introduces the weak snapshot object, which provides fundamental functions for constructing our broadcast protocols. In Section 6, we present two dynamic Byzantine broadcast protocols. We conclude this paper in Section 7.

II. RELATED WORK
In this section, we first review several existing solutions retrospectively as solving a dynamic problem. Then, we move to three specific topics that consider broadcast services with dynamism.
The majority of related work on reconfiguration considers read/write storage. RAMBO [5], [8] was the first to implement a dynamic atomic register with asynchronous read/write operations. The main idea of the protocol is to use consensus to agree on reconfigurations. It supports operations to check all available configurations in the system and write to the latest one. Other works [9], [21], [22] use the same idea to design dynamic atomic registers with Byzantine failures.
Other approaches provide a fault-tolerant emulation of arbitrary data types with the State Machine Replication (SMR) [23], [24]. The first method is to achieve an agreement on a sequence of operations applied to the data. SMR was implemented in Paxos [23], which allows nodes to change the system configuration by keeping the configuration as part of the state stored in the state machine. Another method is to introduce a secondary configuration manager to arrange configuration changes. The manager engages this information into the replication protocol. Several practical systems [19], [25], [26], [27] use this method to achieve dynamism.
Another study [28] considers the situation that the opportunity of losing more than a minority of participants unavoidably grows. They design a failure detector to deal with the number of failures right from the beginning. However, their model does not allow adding more participants to the system. Unlike other previous studies, DynaStore [10] was the first to the solve dynamic reconfiguration problem in asynchronous message-passing systems without consensus. It uses a weak snapshot abstraction to formulate a digraph containing information about the changes of system configurations. Even though it does not solve the agreement problem in asynchronous systems, it still ensures the safety by traversing the digraph of the system configuration. Followed by that, SmartMerge [14] and Parsimonious SpSn [13] use very similar ideas to solve the dynamic problem. We also employ the similar idea to construct a digraph of system configurations with the weaker snapshot abstraction in this paper. Moreover, we extend this concept for Byzantine fault tolerance.
Dynamic Broadcast protocols have been studied even earlier than the reconfigurable read/write storage. In 1987, a protocol [17] was proposed for failure-free scenarios. Several works [1], [22], [24], [27] extend this idea for fault tolerance, VOLUME 10, 2022 where failure-prone nodes are replaced by fault-tolerant disjoint groups.
Configuration-oriented group communication systems maintain a dynamic configuration of active participants. They provide broadcast and atomic multicast services within the member of a configuration by keeping agreement on a sequence of configurations [29].
Other dynamic broadcast protocols [16], [30] do not solve the reconfiguration problem. Instead, they seek to allow replicas to dynamically change the broadcast groups they subscribe to, while the membership of the system remains constant.
Guettaoui et al. [31] addressed a similar problem as we study in this paper. Compared to algorithms that we propose, the previous work only provides interfaces for broadcasting a single message. However, the broadcast abstractions [32] for Byzantine failures in a static configuration always support arbitrary number of messages, which is a practical regiment in reality. Our protocols support broadcasting arbitrary number of messages in dynamic configurations, which can be considered as broadcast channels in the system.
Moreover, our protocols are not restricted to any specific broadcast abstractions, it can be easily adjusted to other Byzantine broadcast primitives in distributed applications. In this paper, we present a consistent broadcast and discuss how to modify it for other broadcast abstractions.

III. DISTRIBUTED SYSTEM MODEL
Consider a distributed system composed of a set = {p, q, . . . } of interconnected nodes. Some nodes in the system are Byzantine. They may behave arbitrarily, i.e., generate fake messages, drop messages, or even corrupt messages.
Any nodes pair u and v can send messages to each other directly. The transmission mechanism is point-to-point, i.e., nodes send a message to at most one neighbor at a time. The communication channels between nodes are authenticated. Nodes can recognize who is the sender of the message when they receive a message. Messages cannot be modified by any third parties if messages are delivered via the authenticated channel between nodes.
The system we study in this paper is asynchronous. In asynchronous distributed systems, the message delay from one node to another is without finite upper bound, i.e. messages may be delayed for arbitrary time periods. However, messages will eventually be delivered. Nodes only take actions when they are activated by events, such as messages arriving.
We assume that the total number of nodes are unbounded and possibly infinite. However, not every node remains active in the system throughout the life of the system. At the beginning of execution time, there is a set of available nodes, Init, which can be accessed by users. Other nodes are initially inactive. Active set Init is known to every node. Nodes can be activated when joining the system or be removed from the system during execution. Once they have been removed from the system, they cannot become active again. Algorithms we propose in this paper use operations, functions and upon clauses. Operations are invoked by users. Functions are invoked by operations, upon clauses, or other functions. When a node p receives messages from other nodes, it stores these messages in a local buffer. The upon clauses are internal actions enabled when some conditions hold. In the face of concurrency, at most one operation and one upon clause can be performed simultaneously. Note that operations and upon clauses might execute concurrently.

IV. PROBLEM DEFINITION
The goal of our work is to implement a dynamic broadcast service which tolerates byzantine failures. The broadcast service is deployed on a collection of nodes that interact with asynchronous message-passing.

A. BYZANTINE DYNAMIC BASICS
We give a formal definition of a system configuration below, which comes from [10]. An example of a system configuration w is presented in Figure 2.
Definition 1 (Configuration): A configuration w represents sets of active nodes and removed nodes. It holds two sets: The first set is a set of active nodes, w.active. Nodes in w.active are either active at the beginning of the system or are activated during the execution but not removed from the system. The second set is a set of removed nodes w.removed, which are nodes removed from the system. The size of w is the sum of the size of w.active and the size of w.removed.
Users can change the system configuration by reconfiguration operations. A set of configuration changes cng contains two subsets. Subset cng.join contains nodes which are activated for joining the system and subset cng.remove contains nodes which will be removed from the system.
We define an addition operation between a configuration w and a set of changes cng. The result of w = w + cng is a configuration. w .active = w.active ∪ cng.join \ cng.remove and w .removed = w.removed ∪ cng.remove. An example of such computation is shown in Figure 3.
Once a node is excluded from the system, it can no longer be included in the system with the same identity. If there exists some change sets cng 1 , cng 2 , . . . that validate the equation w + cng 1 + cng 2 + . . . = w , then we have w ⊆ w .

B. LIVENESS CONDITIONS
Before we specify the liveness conditions for the dynamic reconfiguration problem with Byzantine failures, we first indicate some preliminary definitions for this problem.
At any time t in the execution, we define P(t) be the set of pending changes at time t such that a reconfig(cng) was invoked but has not completed. We define B(t) as the set of nodes that are Byzantine (including crashed nodes) by time t.
It is impossible to achieve agreement in an asynchronous message passing system with Byzantine failures [11]. Therefore, the configurations known by each node may be different during the execution. Let us denote CurConfig p (t) as the latest configuration that node p keeps at time t. A configuration AvaiConfig is available at time t if and only if there exists an active node p such that CurConfig p (t) ⊆ w.
The following liveness conditions hold during the whole execution time of the system: at any time t, for any available configuration AvaiConfig in the system at time t, fewer than |AvaiConfig.active| 3 nodes out of set P(t).join ∪ AvaiConfig.active are in set B(t) ∪ P(t).remove.

C. DYNAMIC BROADCAST WITH BYZANTINE FAULTS
The dynamic broadcast is a communication abstraction that helps with highly available and scalable system applications. Dynamic broadcasts support two communication primitives, broadcast(m) and deliver(m), and a reconfiguration primitive, reconfig(cng). In this paper, we realize dynamic Byzantine broadcast primitives, which allow nodes to broadcast an arbitrary number of messages by invoking broadcast(m) multiple times and enable communications beyond point-topoint in the system.
Users call broadcast p (m) to broadcast a message m in the system. When a message m is delivered at node p, node p will get an indication such that a message m with label l broadcast by node p is delivered. Label l can be an arbitrary bit string. All messages delivered for a particular sender are distinctively labeled.
In addition to broadcast and deliver primitives, the service exposes an interface, i.e., reconfig p (cng), for invoking reconfigurations. Operation reconfig p (cng) allows an active node p to add new nodes to the system or remove nodes from the system, where cng is a set of nodes changing.
Only nodes that are members of the current system configuration should be allowed to initiate actions. Therefore, the broadcast p , reconfig p operations at node p are initially disabled, until an enable event occurs at p. Operations are no longer accessible at node p if halt p event is invoked. After terminated by the halt p event, node p can still deliver messages that were broadcast before it left.
We assume that Byzantine nodes cannot arbitrarily remove or add nodes. Otherwise, the system will not be valid anymore if infinite Byzantine nodes are added to the system. The correct reconfiguration behavior can be regulated outside our protocols. We say node p is active in the system if and only if it is added to the system correctly, i.e., it starts with a correct configuration.
We study two dynamic Byzantine broadcast protocols in this paper: dynamic Byzantine consistent broadcast and dynamic Byzantine reliable broadcast. Both of them support multiple messages broadcast. We point out the properties of these two broadcasts as follows.
The dynamic Byzantine consistent broadcast has four properties [33]: Validity: If a correct node (active node that follows the algorithm protocol) p broadcasts a message m, then every correct node q which joins the system before and does not leave until after the broadcast of m eventually delivers m.
No duplication: If a correct node p broadcasts a message m with label l, then every other correct node delivers at most one message with label l from p.
Integrity: If a correct node q delivers a message m from sender p and p is correct, then m was previously broadcast by p.
Consistency: If a correct node q delivers a message m with label l from sender p, and another correct node s delivers a message m with label l from sender p, then m = m .
The dynamic Byzantine reliable broadcast has four properties [33]: Validity, No duplication, Integrity: same properties as the dynamic Byzantine consistent broadcast.
Agreement: If a correct node q delivers a message m with label l from sender p in configuration w, then every other correct node in configuration w delivers a message m with label l from sender p eventually. VOLUME 10, 2022

V. THE WEAK SNAPSHOT ABSTRACTION
Before discussing broadcast protocols, we first introduce a weak snapshot object [10] S, which provides fundamental functions for constructing broadcast protocols.
A weak snapshot object ws is accessible by a set P of nodes. It supports two operations for every node p ∈ P: update p (cng) and scan p (). Operation update p (cng) has an input value cng. The operation returns a Boolean value that indicates whether the update operation success. Operation scan p () returns a set of values that have been successfully updated by update p () previously. For each weak snapshot object, the nodes set P is static. Operations scan p () and update p () have following properties: integrity: Let o be a scan p () operation that returns C. Then, for each cng ∈ C, an update(cng) operation is invoked by some node q prior to the completion of o.
validity: Let o be a scan p () operation that is invoked after the completion of an update(cng) operation, and that returns C. Then, C = ∅. monotonicity of scans: Let o be a scan p () operation that returns C and let o be a scan q () operation that returns C . If o is invoked after the completion of o. Then, C ⊆ C .
non-empty intersection: There exists a change set cng such that for every scan() operation that returns C = ∅, it holds that cng ∈ C.
termination: If some majority M of nodes in P do not crash, then every scan p () and update p (cng) invoked by any node p ∈ M eventually completes.
Generally speaking, these properties require that all scan operations with non-empty returns observe the ''first'' update operation.
Every configuration w in the system has a weak snapshot object ws(w), which can be accessed by nodes in w.active. The algorithm uses an array of single-writer multireader atomic registers as Mem[w] of size |w.active| for each weak snapshot object. We use the Byzantine Quorum with Listeners algorithm [34] to implement such Byzantine atomic registers in asynchronous message-passing systems.
Operation collect p () reads updates which are stored in Mem [w]. Node p invokes update p (w, cng) to write a set of changes cng to the Mem[w] [p] in ws(w), which is considered as reconfigurations to configuration w. In update p (w, cng), the algorithm first reads all updates that have been written in Mem[w] by invoking collect(w). If it has not found any existing updates, it writes cng in Mem[w] [p]. Change set cng will be observed by following collect q (w) functions for any q ∈ w.active. Node p learns updates proposed by other nodes to w by invoking scan p (w). It calls collect(w) to read all previous updates to w. If no update has been written in Mem[w], it returns an empty set ∅. Otherwise, it calls collect(w) again and returns all previous updates.
The relationship between configurations in a system can be represented by a digraph G = (V , E). Each vertex in G indicates a system configuration. There is a directed edge pointing from vertex v w to vertex v w if and only if the end if 8: end function 9: function scan p (w) 10: C ← collect(w) 11: if C = ∅ then 12: return ∅ 13: end if 14: C ← collect(w) 15: return C 16: end function 17: function collect p (w) 18: for all q ∈ w.active do 20: 21: if c =⊥ then 22: 23: end if 24: end for 25: return C 26: end function following relationship holds for the configuration w and the configuration w , i.e., w = w + scan p (w) for a node p.
Intuitively, we use these weak snapshots as follows: when a node p wants to change the configuration w, it invokes update p (w, cng) to propose a set of changes cng; it learns the changes of configuration proposed by other nodes to configuration w by invoking scan p (w), which returns a set of changes. Compare to atomic snapshot objects [35], we do not require atomicity, i.e., all operations can be ordered in a sequential execution.
Integrity and termination properties have been guaranteed by the correctness of Byzantine atomic registers [34]. Aguilera et al. [10] explained in detail why Algorithm 1 preserves validity, monotonicity and non-empty intersection properties.

VI. DYNAMIC BYZANTINE BROADCAST
This section describes in detail the dynamic Byzantine broadcast protocols. We first introduce the consistent broadcast from Section 6.1 to Section 6.4. Then we present the reliable broadcast in Section 6.5.
Key components of our consistent broadcast protocol are shown in Algorithm 3, which contain function BroadcastInConfig (lines 48-56) and upon clauses (lines 73-86) for broadcasting and delivering messages in a given configuration. We realize the broadcast service in a given configuration with the authenticated echo broadcast CurConfig p , initially Init, latest configuration found by p in the system 5: ReceivedM p , initially ∅, the set of ECHO messages received by node p 6: ValidatedM p , initially ∅, the set of messages validated by a quorum of a configuration, waiting for final delivery by node p 7: DeliveredM p , initially ∅, the set of messages delivered by node p 8: end local variables protocol [36], [37]. The algorithm has two phases, broadcastphase and check-phase. When there are no reconfigurations, all nodes stay in the same configuration. Operation broadcast executes a check-phase to find configuration updates in the system and enters the broadcast-phase after the check-phase. When a node receives new messages from other nodes, it runs a check-phase to update its current configuration.
To allow reconfiguration, system members store information about configuration changes in weak snapshot objects. Members of a configuration are able to change the configuration by updating a change set in the weak snapshot object. We allow concurrent reconfiguration operations with any broadcast operations and deliver events. Node p delivers all messages that broadcast between its appearance in the system and its departure. Furthermore, once a reconfiguration operation is completed, future broadcast can be carried out in the new configuration. The key challenge is to ensure that no message delivers linger behind in the old configuration, while updates are made to the new configuration. We use the following strategies to make sure the correctness of broadcast services in dynamic systems.
• In our dynamic protocol, nodes do not directly deliver messages when it is confirmed by a quorum of the system. The procedure is modified so that nodes first accept all messages that they receive in any configurations. Then nodes deliver messages according to the configuration they have observed. Messages are successfully delivered at node p if they are broadcast in a configuration that p has visited. If a new message is sent to a node, the node starts the check-phase to detect new configurations in the system.
• The broadcast-phase works in the latest configuration found by the check-phase. First, nodes broadcast the message in the latest configuration during the broadcastphase, and then nodes read the reconfiguration information. If nodes find a new configuration, the protocol restarts the check-phase.
• The reconfig operation starts with a check-phase and then writes information about the new configuration to the quorum of the old configuration. These procedures will be iterated until no new configurations are found.
Function BroadcastInConfig is the core of a broadcast phase, which implements the basic functionality of the broadcast-phase. It first broadcasts the message in the current configuration (line 50-52), and then finds previous update of the current configuration by scanning its weak snapshot object in line 54. To deliver a message, nodes run function ReceiveMsg when they receive messages from other nodes (line 84) or visit a new configuration (line 26).
Let us discuss a simple example where only one reconfiguration request, i.e., RC, is called, from w 1 to w 2 . Consider a broadcast operation, i.e., B, runs a broadcast-phase with message m. There are two possible cases with respect to B. One is that it does not observe any updates of configurations. This means that B's execution of broadcast-phase is in w 1 and all correct nodes will receive m and deliver it successfully. Otherwise, B observes RC in its check-phase and broadcasts m in w 2 . After receiving messages from other nodes, correct nodes start check-phase to detect new configuration in the system. Because RC is already found by B, it will be also found by other check-phases which start after B's checkphase. Message m will be delivered in w 2 at correct nodes eventually.
In the example above, we ignored the correctness of the dynamic broadcast if several nodes concurrently propose changes to w 1 . Therefore, the rest of our algorithm aims to handle the complexity that incurs due to multiple reconfiguration requests. In Section 6.1, we introduce the local state of nodes. Section 6.2 presents the pseudo-code of the dynamic Byzantine consistence broadcast. We discuss the notion of established configurations in Section 6.3. The analysis and proof of the correctness are presented in Section 6.4. Finally, we further introduce the dynamic Byzantine reliable broadcast in Section 6.5.

A. STATE OF NODES
In this subsection, we discuss the local state of nodes. Each node stores several variables in local memory. These variables are shown in Algorithm 2. Note that all actions to local variables are atomic. Local variables at node p are denoted with subscript p.
All configurations which have been visited by node p are stored in ConfigList p , which is initialed as an empty set. The latest configuration found by node p is kept in CurConfig p , which is initialed as Init, i.e., the primary system configuration. MsgNum p is the number of messages that broadcast by node p. VOLUME 10, 2022 Algorithm 3 Code for Node p, Dynamic Byzantine Consistent Broadcast 1: initially: 2: if p ∈ Init.active then 3: enable operations 4: end if 5: end initially: 6: operation broadcast p (m) 7: newConfig ← Traverse p (∅, m) 8: MsgNum p ← MsgNum p + 1 9: end operation 10: operation reconfig p (cng) 11: CurConfig p ← Traverse p (cng, ⊥) 12: for all q ∈ cng.join do 13: send (Join, newConfig, p) 14: end for 15: return newConfig 16 if w = desiredConfig then 28: update p (w, desiredConfig \ w) 29: end if 30: ChangeSets ← scan p (w) 31: if ChangeSets = ∅ then 32: Front ← Front \ {w} 33: for all c ∈ Changesets do 34: desiredConfig ← desiredConfig + c Node p delivers a message in four steps. First, it sends ECHO messages to other nodes when it receives a SEND message from the sender. Second, it puts messages in ReceivedM p whenever it receives ECHO messages from others. A mes-sage m is considered to be correct in configuration w if a quorum of w.active confirms its correctness with an ECHO message. These messages are stored it in ValidatedM p with its broadcast configuration w in the third step. Until w is visited by p, messages in ValidatedM p with configuration w are successfully delivered and enter in DeliveredM p .

B. DYNAMIC BYZANTINE CONSISTENT BROADCAST
This subsection describes the algorithm for the dynamic Byzantine consistent broadcast, which is shown in Algorithm 3. Operations and functions at node p are denoted with subscript p. There are two operations that can be invoked directly by users at node p, broadcast p (m) and reconfig p (cng).
Users call broadcast p (m) to broadcast a message m in the latest configuration and propose a set of configuration changes cng to the system by invoking reconfig p (cng). We introduce these two operations respectively in this subsection. Then, we discuss how nodes deliver messages. The main logic function Traverse p is introduced in the last part of this subsection.

1) BROADCASTING MESSAGES
Operation broadcast p (m) invokes Traverse p (∅, m) to find the latest configuration w in the system and broadcast message m, where ∅ implies that this execution does not change the system configuration. After broadcasting, variable MsgNum p increases by 1 for further broadcasting.

2) RECONFIGURATIONS
Similar to a broadcast p (m) operation, a reconfig p (cng) also invokes Traverse p (cng, ⊥) to find the latest configuration w, and apply a change set cng to the system configuration, where ⊥ implies that this execution does not broadcast any messages.
Node p sends the new configuration to all joining nodes in line 12-14. A new node successfully joins the system if it receives a correct configuration.

3) DELIVERING MESSAGES
Message delivery is a passive procedure in the protocol. It is triggered when nodes receive messages from other nodes. We implement an authenticated echo broadcast algorithm in a certain configuration. After receiving a message m with a SEND tag broadcast in configuration w by node s (line 50-52), node p broadcasts m with ECHO tag to other nodes in configuration w (line 74-76). When node p receives more than 2·|w.active| 3 ECHO messages containing the same message m broadcast by node s, in the same configuration m, with the same message number MsgNum, it validates m and store it in set ValidatedM p (line 80-83). Node p starts to find the latest configuration in the system whenever it receives ECHO messages from other nodes by invoking Traverse p (∅, ⊥) (line 84), where ∅ implies that this execution does not change the system configuration and ⊥ implies that this execution does not broadcast any messages.
Message m received by p in configuration w is not delivered immediately because w could be a fake configuration and message m may not be broadcast in the real system. Node p delivers message m if it has visited the configuration w where m was broadcast (line 57-72). Node p checks all validated messages by calling function ReceiveMsg p () when it visits a new configuration (line 26) or it receives an ECHO message (line 84).

4) TRAVERSING THE GRAPH OF CONFIGS
Weak snapshots form all configurations into a digraph, where configurations are vertices in the graph. If there is an update q (w, cng) happening with return true during the execution by node q ∈ w.active where w = w + cng, cng = ∅, we have a direct edge from vertex w to vertex w in the digraph. Our algorithm requires that a removed node cannot join the system again, thus the size of w is always larger than w and the graph or configurations is acyclic.
Node p gets the latest configuration information through the function Traverse p (), which is invoked by all operations or when node p receives messages from other nodes. This function traverses the digraph of configurations from its current configuration CurConfig. Then, the digraph is visited for collecting configuration changes which are previously applied to the system. All changes are gathered in the set desiredConfig. After finding all changes, desiredConfig is included in the digraph if it does not exist. A weak snapshot object of desiredConfig is connected to the graph by function update p (). Other nodes in the system can find desiredConfig in subsequent searches.
We implement a well-known Dijsktra algorithm [38] to traverse the graph, with a modification such that the traversal also reforms the digraph, which is similar to the traversal algorithm in [10]. Function Traverse p () keeps a set of configurations Front, which contains discovered configurations waiting for further visit. Set Front is initialed as CurConfig p (line 19). Each iteration, Traverse p visits the nearest configuration from CurConfig p in Front (line 21), i.e., the configuration which has the minimal size in the set, and remove this vertex from Front (line 33) after visiting. If more than one configuration has the same size, Traverse p () first visits the one with the minimal size of removed subset.
When Traverse p () visits a configuration w (line 20-46), it first checks if p is still in the system and then delivers messages that p has stored in ValidatedM p by invoking ReceuveMsg p (). It is possible that w does not include all known changes found by the traversal, i.e., w = desiredConfig. In this case, p calls update p () to build an edge from w to desiredConfig in the digraph (line [28][29][30]. This update can be failed because another node has already built an edge outgoing from w. If so, Traverse p () will visit the successor node of w and update it again.
Once an update has been applied to the weak snapshot of w successfully (line 29), latter scan p (w) (line 31) will return the update. A non-empty changes set is returning if there are edges outgoing from w. For every change set c ∈ ChangeSets returned by scan p (w), there is an edge from w to w + c. Thus, we add w + c to Front for furthering visit When there is only one configuration in Front, desiredConfig is visited by Traverse p () because it has the maximum size among all configurations visited by Traverse p () in the execution. This configuration contains all changes observed during the traversal. Moreover, if w = desiredConfig, the condition in line 32 is always satisfied. Therefore, BroadcastInConfig(w, m) is invoked when there is only one configuration in Front, i.e., desiredConfig. For the same reason, when Traverse p finishes, we have Front = {desiredConfig}. This configuration is assigned to CurConfig p in line 43 and returned by Traverse p .
We consider the example that we discussed in Introduction. Operations reconfig p (cng 1 ) and reconfig q (cng 2 ) are executed concurrently, where cng 1 = {+p 5 } and cng 2 {−p 4 }. There are two possible cases resulted from such simultaneous operation. In the first case, either cng 1 or cng 2 are first updated succefully with a update(w, cng) function. Suppose it is cng 1 . The other update function observes cng 1 and will not write cng 2 in the weak snapshot object of w. Function Traverse q () invoked by reconfig q (cng 2 ) renews its desiredConfig from w+cng 2 to w+cng 2 +cng 1 , visits configuration (w+cng 1 ) in the next iteration and writes cng 2 in ws(w+cng 1 ). Traverse p () will find w + cng 2 + cng 1 from w + cng 1 in future executions.
In the second case, both cng 1 and cng 2 are successfully written into the weak snapshot object of w. Benefited from the validity property of the weak snapshot object, scan p (w) and scan q (w) both return non-empty sets C p and C q . The non-empty intersection property of the weak snapshot object ensures C p ∩ C q = ∅, i.e., either C p ∩ C q = cng 1 , C p ∩ C q = cng 2 or C p ∩ C q = cng 1 ∪ cng 2 . Suppose it is cng 1 . Then Traverse q () discovers both configuration w + cng 1 and configuration w + cng 2 . It will visit them in the following iterations and build edges from configuration w + cng 1 and configuration w + cng 2 to configuration w + cng 1 + cng 2 . Traverse p () will find w + cng 2 + cng 1 in future executions. If C p ∩ C q = cng 1 ∪ cng 2 , both Traverse p () and Traverse q () will find w + cng 1 + cng 2 in this execution. The digraph of this example is shown in Figure 4.

C. SEQUENCE OF ESTABLISHED CONFIGURATIONS
The difficulty of the reconfiguration problem is that different nodes might keep different configurations if several reconfiguration operations are invoked concurrently. In our algorithm, when multiple nodes invoke update(w) at the same time, concurrent scan(w) might see different outgoing edges from w.
Fortunately, the non-empty intersection property of weak snapshot objects ensures that nodes will never work on different branches of the digraph. Actually, at least one outgoing edge is returned by all scan(w) functions and the destination of this edge will be visited by all traverse() functions. This fact enables us to define a totally ordered subset of configurations, i.e., established configuration, as follows: Definition 2 (SequenceofEstablishedConfigurations [10]): The unique sequence of established configurations ε is constructed as follows: • the first configuration in ε is the initial configuration Init; • if w is in ε, then the next configuration after w in ε is w = w+cng, where cng is an element chosen arbitrarily from the intersection of all sets C = ∅ returned by some scan(w) operations in the execution. The first digraph traverse in the system starts from CurConfig p = Init, which is an established configuration by definition. At each iteration, Traverse p () visits a configuration w, removes it from Front and adds its children in Front. If w is an established configuration, then one of its children is also an established configuration, which is included in Front. If w is not established, an established configuration is still in Front. Thus, at least one established configuration in Front in each iteration. At the end of the execution of Traverse p , there is only one configuration desiredConfig in Front. Therefore, configuration desiredConfig is also an established configuration. We conclude that desiredConfig which is assigned to CurConfig p in line 43 and returned from Traverse p is established at the first digraph traverse. By induction, Traverse p () always starts with an established configuration and returns an established configuration. Configuration CurConfig p is also always established.
Function BroadcastInConfig p (line 48-56) is performed in an established configuration as well. Moreover, we prove that each Traverse p () visits every established configuration in ε between the starting configuration CurConfig p and the returned configuration desiredConfig. Thus, intuitively, by visiting each configuration during a traversal, it is guaranteed to never broadcast on different branches on the digraph and all correct messages can be delivered eventually.
Proof: As we discussed above, the first digraph traversal stars from an established configuration CurConfig p = Init. Therefore, by induction, configuration desiredConfig returned from Traverse p in line 44 is always established and CurConfig p is always an established configuration.
We claim that if scan p (w) returns ChangeSets = ∅ at some iterations of the loop in lines 20-46, then w = desiredConfig and Front = {desiredConfig}.
If w is not equal to desiredConfig, then there must be a function update p (w, desiredConfig \ w) which completes before scan p (w) starts. This is impossible because of the validity property of weak snapshot objects.
Moreover, if Front = {desiredConfig}, there is another w ∈ Front. We either have |w| < w , or w .removed < |w.removed|. Because w = desiredConfig = w + c for some change set c, these inequalities are impossible to hold.
Since during the iteration, scan(w) returns ∅, Front does not change from the beginning of the iteration. Any configuration w passed to BroadcastInConfig p (w, * ) is established.
Lemma 2: Let T be an execution of Traverse p () and initConfig be the value of CurConfig p when node p starts this execution. Let desiredConfig be the value of CurConfig p when this execution terminates. Then Traverse p () visits all configurations from the sequence of established configurations between initConfig and desiredConfig.
Proof: When T begins, Front = {initConfig}. Let estConfig = initConfig. Consider some iterations during T visit w = estConfig. This happens only if estConfig is removed from Front and some scan(estConfig) during T returns a non empty ChangeSets.
After removing estConfig from Front, for every c ∈ ChangeSets returned by scan(estConfig), (estConfig + c) is added to Front. By the non-empty intersection property of weak snapshot objects and the definition of the sequence of established configurations, the configuration after estConfig in ε is added to Front and will be visited by Traverse p in future iterations.
Arguments above hold for every established configuration. When BroadcastInConfig(w, * ) is invoked, scan(w) completes in line 31 and returns ∅, which ends the proof.

D. CORRECTNESS
In this subsection, we show that our algorithm satisfies all properties of the dynamic Byzantine consistent broadcast. We indicate the correctness of the algorithm respectively in Lemma 3, Lemma 4, Lemma 5 and Lemma 6.
Lemma 3: Our algorithm satisfies the validity property. Proof: If node q joins the system before node p broadcasts the message m, then p will enter an established configuration w where q ∈ w.active ∪ w.removed. Because q does not leave the system when p broadcasts m, q / ∈ w.removed. When node q joins the system, it starts with an established configuration which is already included in the digraph. So this configuration must be visited by node p when it broadcast m. By lemma 2, q will also visit w and deliver m eventually.
Lemma 4: Our algorithm satisfies the no duplication property.
Proof: Because node p only broadcasts m in a certain configuration w and updates MsgNum p for each broadcasting. The MsgNum and configuration form a unique label for each message. In addition, the no duplication property of the authenticated echo broadcast algorithm ensures that there is no duplication in broadcasting of messages in a certain configuration.
Therefore, every correct node only delivers at most one message with label l from sender p.

Lemma 5: Our algorithm satisfies the integrity property.
Proof: The integrity property of the authenticated echo broadcast algorithm ensures that if node q validates a message in a correct configuration w from sender p, it must previously broadcast by sender p in configuration w.
According to Lemma 1, a correct node p only broadcasts messages in established configuration. Hence, a correct node q delivers a message m in an established configuration after p broadcasting it.
Our assumption of the number of Byzantine nodes and the correctness of reconfiguration operations ensure that every correct node will never visit a fake configuration. Therefore, any validated messages in fake configurations will never be delivered.

Lemma 6: Our algorithm satisfies the consistency property.
Proof: The integrity property of the authenticated echo broadcast algorithm directly implies that if correct node q delivers a message m with label l from sender p in configuration w and correct node q delivers a message m with label l from sender p in configuration w, then m = m .
The label in our protocol contains the information of the system configuration. Therefore, if two delivered messages have the same label, then they are delivered in the same configuration. The consistency property holds intuitively.
Theorem 1: Algorithm 3 realizes a dynamic Byzantine consistent broadcast.
Proof: Because Algorithm 3 satisfies validity, no duplication, integrity, and consistency properties, it realizes a dynamic Byzantine consistent broadcast.

E. DYNAMIC BYZANTINE RELIABLE BROADCAST
In this subsection, we present the dynamic Byzantine reliable broadcast protocol. Only the differences between the consistent broadcast and the reliable broadcast are listed in Algorithm 4.  15: for all t ∈ w.active \ s do 16: send (READY, p, s, MsgNum, m, w) to t 17: end for 18: end if 19: if #t > 2·|w.active| Traverse p (∅, ⊥)

23:
ReceiveMsg p () 24: end upon As we discussed in previous sections, our protocol is able to be adjusted into any broadcast abstractions. Here we show how to build the reliable broadcast based on our protocols.
Compare to the consistent broadcast, the basic broadcast algorithm is changed from the authenticated echo broadcast algorithm to the authenticated double-echo broadcast algorithm [39]. The algorithm is called authenticated double-echo broadcast because it has two ECHO steps. In the first ECHO step, node p sends message m with ECHO tag to other nodes when it accepts the message m with SEND tag from node s. In the second step, if node p receives message m with ECHO tag, it stores it in ReceivedM p (line 5). If more than 2·|w.active| 3 copies of the message m with ECHO tag from other nodes in configuration w are stored in ReceivedM p , node p sends message m with READY tag to other nodes in configuration w (line [6][7][8][9][10][11].
We introduce the second echo procedure in the broadcast algorithm. Nodes broadcast messages with READY tag to response the first echo. If node p receives a message with READY tag, it stores it in ReadyM p (line 14). If more than |w.active| 3 copies of the message m with READY tag from other nodes in configuration w are stored in ReceivedM p , node p sends message m with READY tag to other nodes in configuration w (line [15][16][17][18][19][20]. If more than 2·|w.active| 3 copies of the message m with READY tag from other nodes in configuration w are stored in ReceivedM p , node p validates message m in configuration w and stores it in ValidatedM p (line 21-24), for further delivery.
With the same argument as in Section 6.4, we can prove that the algorithm satisfies the validity, no duplication, and integrity properties of the dynamic Byzantine reliable broadcast.
Generally speaking, if a Byzantine broadcast algorithm works in a static configuration of nodes, it can be directly adjusted to a dynamic Byzantine broadcast algorithm within our protocols.
Lemma 7: Our algorithm satisfies the agreement property. Proof: The totality property [33] of the authenticated echo broadcast algorithm directly implies that if correct node q delivers a message m with label l from sender p in configuration w, then every correct node eventually delivers a message m with label l from sender p in configuration w.
Correct nodes are stimulated to discover the updated configuration when it receives a message in such a configuration. Our assumption of the number of Byzantine nodes and the correctness of reconfiguration operations ensure that every correct node will eventually visit all established configurations in the system. When a configuration w is visited by node p, it will deliver all messages it validates in w. The agreement property holds intuitively.
Theorem 2: Algorithm 3 and Algorithm 4 realize a dynamic Byzantine reliable broadcast.
Proof: Because Algorithm 3 satisfies the validity, no duplication, integrity, and consistency properties, and Algorithm 4 satisfies the agreement property. With the modification in Algorithm 4, our protocol realizes a dynamic Byzantine reliable broadcast.

VII. CONCLUSION
In this paper, we studied the dynamic broadcast problem in asynchronous systems with Byzantine faults. Previous reconfiguration results in asynchronous systems restricted to crash only. They assumed nodes behave correctly during the execution. However, in real distributed systems, some malicious nodes would like to attack the system. It is essential to ensure that the robustness of system is still guaranteed even some nodes are Byzantine.