CAPER: A Cross-Application Permissioned Blockchain

Mohammad Javad Amiri Divyakant Agrawal Amr El AbbadiDepartment of Computer Science, University of California Santa Barbara

Santa Barbara, California{amiri, agrawal, amr}@cs.ucsb.edu

ABSTRACTDespite recent intensive research, existing blockchain sys-tems do not adequately address all the characteristics of dis-tributed applications. In particular, distributed applicationscollaborate with each other following service level agree-ments (SLAs) to provide different services. While collab-oration between applications, e.g., cross-application trans-actions, should be visible to all applications, the internaldata of each application, e.g, internal transactions, mightbe confidential. In this paper, we introduce CAPER, a per-missioned blockchain system to support both internal andcross-application transactions of collaborating distributedapplications. In CAPER, the blockchain ledger is formedas a directed acyclic graph where each application accessesand maintains only its own view of the ledger including itsinternal and all cross-application transactions. CAPER alsointroduces three consensus protocols to globally order cross-application transactions between applications with differentinternal consensus protocols. The experimental results re-veal the efficiency of CAPER in terms of performance andscalability.

PVLDB Reference Format:Mohammad Javad Amiri, Divyakant Agrawal, Amr El Abbadi.CAPER: A Cross-Application Permissioned Blockchain. PVLDB,12(11): 1385-1398, 2019.DOI: https://doi.org/10.14778/3342263.3342275

1. INTRODUCTIONBlockchain, originally devised for the Bitcoin cryptocur-

rency [40], is a distributed data structure for recording trans-actions maintained by nodes without a central authority[17]. In a blockchain, nodes agree on their shared statesacross a large network of untrusted participants. The uniquefeatures of blockchain such as transparency, provenance, faulttolerance, and authenticity are used by many systems to de-ploy a wide range of distributed applications such as supplychain management [31] and healthcare [12] in a permissionedsettings. Unlike permissionless settings, e.g., Bitcoin [40],

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Forany use beyond those covered by this license, obtain permission by emailinginfo@vldb.org. Copyright is held by the owner/author(s). Publication rightslicensed to the VLDB Endowment.Proceedings of the VLDB Endowment, Vol. 12, No. 11ISSN 2150-8097.DOI: https://doi.org/10.14778/3342263.3342275

where the network is public, and anyone can participatewithout a specific identity, a permissioned blockchain con-sists of a set of known, identified nodes that might not fullytrust each other.To support distributed applications, different character-

istics of such applications need to be addressed by permis-sioned blockchain systems. Distributed applications requirehigh performance in terms of throughput and latency, e.g.,a financial application needs to process tens of thousandsof requests every second with very low latency. Distributedapplications also collaborate with each other to provide dif-ferent services. Collaborations are defined in service levelagreements (SLAs) which are agreed upon by all involvedapplications. SLAs can be written as self executing com-puter programs, called smart contracts [3]. The collabora-tion is realized by means of cross-application transactionsthat are visible to every application. During the executionof cross-application transactions, agreement on the sharedstate of the collaborating applications is needed withouttrusting a central authority or any particular participant.While cross-application collaborations and the involved dataare visible to every application, the internal data of each ap-plication, i.e., the application logic, internal transactions,and their data, might be confidential. Hence, it is desir-able to restrict access to such data. Although cryptographictechniques can be used to achieve confidentiality, the highoverhead of such techniques makes them impractical [10].Existing permissioned blockchains mostly suffer from per-

formance issues in terms of throughput and latency becauseof the sequential execution of transactions on all nodes, andalso confidentiality issues since a single blockchain ledgerwith all transactions is maintained at every node.Hyperledger Fabric [10], on the other hand, improves per-

formance by executing the transactions of different applic-ations that are deployed on the same channel in paralleland addresses the confidentiality leaks by restricting accessto the blockchain state (using private data collections) andsmart contracts which include the application logic. How-ever, the blockchain ledger is still maintained by every node.To provide confidentiality, different applications could have

independent disjoint blockchains. However, to support col-laboration between applications on distinct blockchains, tech-niques such as atomic cross-chain swap [27] and Interledgerprotocol [49] are needed to exchange (transfer) assets or in-formation between the blockchains (Interoperability). Suchtechniques are often costly, complex, and mainly designedfor permissionless blockchains. Techniques that support col-laborating applications on a single blockchain either do not

1385

support internal transactions of applications [51] (results indata integration issues), or suffer from confidentiality is-sues since the entire ledger is visible to all applications,e.g., single-channel Fabric [10], or require a trusted chan-nel among participants, e.g., multi-channel Fabric [11].In this paper, we present CAPER: a permissioned block-

chain system that supports both internal and cross-applicationtransactions of collaborating distributed applications. InCAPER, each application orders and executes its internaltransactions locally while cross-application transactions arepublic and visible to every node. In addition, the blockchainledger of CAPER is a directed acyclic graph that includesthe internal transactions of every application and all cross-application transactions. Nonetheless, for the sake of con-fidentiality, the blockchain ledger is not maintained by anynode. In fact, each application maintains its own local viewof the ledger including its internal and all cross-applicationtransactions. Since ordering cross-application transactionsrequires global agreement among all applications, CAPERintroduces different consensus protocols to globally ordercross-application transactions.Lack of trust is an important problem in collaboration

between applications. Lack of trust has two main origins:first, a physical node (server) might fail, thus behave mali-ciously, and second, an application could behave maliciouslyin the communication with other applications for its bene-fits. To address both types of behavior, CAPER distin-guishes between trust at the node level and trust at the ap-plication level, e.g., while the nodes of an application mightbehave non-maliciously within the application, the applica-tion (as a collection of nodes) might still behave maliciouslyin communication with other applications.A key objective of this paper is to demonstrate how in-

ternal and cross-application transactions of a set of collabor-ating distributed applications which do not trust each othercan be processed efficiently by a blockchain system whileboth confidentiality constraints of internal transactions andvisibility constraints of cross-application transactions aremet. The contributions of this paper are three-fold:

• Introducing Blockchain views where each applicationmaintains only its own view of the ledger including itsinternal and all cross-application transactions.

• CAPER, a permissioned blockchain that supports col-laborating distributed applications. CAPER supportsboth internal and cross-application transactions.

• Three different consensus protocols for globally order-ing cross-application transactions among applicationswith different local consensus protocols.

The rest of this paper is organized as follows. Section 2briefly describes the limitations of current blockchain sys-tems and motivates the problem. The CAPER model andarchitecture are introduced in Section 3 and Section 4. Sec-tion 5 and Section 6 present consensus in CAPER. Section 7presents a performance evaluation of CAPER. Section 8 dis-cusses related work, and Section 9 concludes the paper.

2. BACKGROUND AND MOTIVATIONA blockchain is a distributed data structure for record-

ing transactions, maintained by nodes without a central au-thority [17]. Existing blockchain systems can be divided

into two main categories: permissionless blockchains, e.g,Bitcoin [40], and permissioned blockchains, e.g., Tender-mint [32].

2.1 Limitations of Permissioned BlockchainBlockchains have unique features, such as transparency,

provenance, fault tolerance, and authenticity, which appealto a wide range of distributed applications, e.g., supply chainmanagement [31] and healthcare [12]. However blockchainsystems suffer from some limitations. In the following, wemainly focus on permissioned blockchains and discuss threeof their significant limitations: poor performance, lack ofconfidentiality, and inefficient cross-application transactionsupport. Note that some of these limitations exist in per-missionless blockchain systems as well.Existing blockchains mostly utilize an order-execute archi-

tecture where nodes agree on a total order of the blocks oftransactions using a consensus protocol and then the trans-actions are executed in the same order on all nodes sequen-tially. The sequential execution of transactions, however,reduces the blockchain performance in terms of throughputand latency. To address this issue, Hyperledger Fabric [10]presents a new architecture for permissioned blockchains byswitching the order of the execution and ordering phases.In Fabric, the transactions of different applications are firstexecuted in parallel and then an ordering service establishesconsensus on a total order of requests.While Fabric reduces the latency of the system by execut-

ing transactions in parallel, the sequential construction ofblocks could still be a performance bottleneck when the sys-tem is deployed on a single channel or if channels use thesame ordering service. In Bitcoin [40], (single-channel) Fab-ric [10], and many other permissioned and permissionlessblockchains, blocks cannot be constructed simultaneouslyand only a single chain is allowed on the entire network.In contrast to this linear structure, some other blockchains,e.g., Iota [44], and Hashgraph [14], are structured as a direc-ted acyclic graph (DAG). The DAG structure can increasethe throughput of transactions by exploiting parallel con-struction resulting in the parallel execution of transactionsin different blocks. In such a structure, the blocks (transac-tions) that are independent of each other can be constructedand added to the ledger simultaneously.Batching transactions into blocks is another reason for the

low performance of blockchains. Blocks were originally cre-ated, first, to amortize the cost of cryptography, e.g., solvingproof-of-work, and second, to make data transfers more ef-ficient in a large geo-distributed setting. However, Stream-Chain [28] demonstrates that in permissioned blockchains,since proof-of-work is not required, batching transactionsinto blocks decreases performance.Besides the performance issues, confidentiality of data is

required in many permissioned blockchains. A blockchainmight need to restrict access to smart contracts (which in-clude the logic of applications), blockchain ledger, and trans-action data. While cryptographic techniques can be usedto achieve confidentiality, the considerable overhead of suchtechniques makes them impractical [10]. Fabric ensures theconfidentiality of data using Private Data Collections [7].Private Data Collections manage confidential data that twoor more applications on a single channel want to keep privatefrom other applications on that channel. The confidential-ity of smart contracts in Fabric is also ensured by storing

1386

Figure 1: A Supply Chain Scenario

the smart contracts of different applications on different setsof nodes, called endorsers. The endorsers of an applicationare responsible for executing the transactions of the applic-ation independent of the other applications. However, theblockchain ledger is still maintained by the agents of everyapplication within a channel.In addition to performance and confidentiality issues, many

applications need to collaborate with each other to providedifferent services. Distributed applications are often de-signed and implemented in different blockchain systems, eachof which processes and stores data independently [29]. Inthis case, inter-application collaboration could be performedas an atomic cross-chain swap [27] or using the Interledgerprotocol [49] where two applications (parties) exchange (trans-fer) assets or information across their blockchains. Atomiccross-chain transactions are also addressed between permis-sioned blockchains (channels) in Fabric by either assumingthe existence of a trusted channel among the participantsor using an atomic commit protocol [11]. In general, sup-porting cross-application transactions is an expensive andchallenging task. Nonetheless, for collaborating applicationsin a single permissioned blockchain, since the participantsof cross-application transactions are known beforehand andthe transactions follow service level agreements, we mightbe able to find a solution that preserves both confidentialityand performance.

2.2 A Motivating Example: Supply ChainWe now consider collaborative workflows as a use-case to

illustrate the aforementioned limitations. In a collaborativeworkflow, different parties need to communicate across or-ganizations to provide services. However, the lack of trustbetween parties is problematic.Figure 1 shows a Supply Chain scenario (reported in [51]).

The workflow involves five participants (applications): Sup-plier, Manufacturer, Bulk Buyer, Carrier, and Middlemanwhere each of the participants might have multiple trus-ted or untrusted nodes that perform different internal tasks(transactions). For example, the production process in theManufacturer involves Financial, Marketing, and Purchas-ing departments and includes different internal tasks suchas assembly, painting, drying, testing, and packaging. Parti-cipants also need to communicate with each other to providedifferent services. The Bulk Buyer communicates with theManufacturer to place an order, the Manufacturer places anorder for materials via a Middleman, the Middleman for-wards the order to a Supplier and arranges transportationby a Carrier. Once the materials are acquired, the Supplierinforms the Carrier and the Carrier picks them up and deliv-ers them to the Manufacturer. As soon as the goods becomeavailable, the Manufacturer delivers them to the Bulk Buyer.These collaborations are defined in service level agreementswhich are agreed upon by all participants.

Now the Manufacturer might receive the materials laterthan agreed upon or might receive something different fromwhat they agreed on. In this case, the Supplier might ar-gue that this is exactly what was ordered by the Middlemanwhile the Middleman would blame the Supplier. The situ-ation is complicated for the Carrier since the Manufacturermight refuse to accept the delivery. The Carrier is noweligible for compensation from either the Supplier or theMiddleman depending on who is responsible for the fault.To tackle such an issue, permissioned blockchain systems

can be used among all the different participants to ensureagreement on the shared state of the collaborating partieswithout trusting a central authority or any particular parti-cipant [51]. The blockchain basically monitors the executionof the collaborative process and checks conformance betweenthe process execution and SLAs. Any blockchain-based solu-tion has to address the following concerns.First, the blockchain system should support both cross-

application and internal transactions. For example, in theSupply Chain scenario, in addition to cross-application trans-actions, the Manufacturer might want to use the blockchainfor its internal transactions, e.g., calculating materials de-mand or testing the product, to benefit from the the uniquefeatures of blockchain. Second, in contrast to the cross-application transactions which are public and can be ac-cessed by all participants, the internal transactions of eachapplication and their data should only be accessed by thenodes of the application to preserve confidentiality, e.g., theinternal transactions of the Manufacturer show its internalprocess for producing a product which the Manufacturermight intend to keep as a secret. Third, the solution has toaddress the performance aspect as well.One possible solution is to implement all applications within

a single blockchain where all the transactions are maintainedin a single blockchain ledger which is replicated among allthe nodes in the blockchain. This solution handles the cross-application transactions efficiently because every node ac-cesses all the data. However, since the ledger is replicatedamong all the nodes and every transaction is visible to allapplications, the confidentiality of data is not preserved.Another solution is to implement each application on a

separate blockchain. In that way, participants can performtheir internal transactions in parallel resulting in higher per-formance and since their data is maintained on differentblockchains, the confidentiality of data is preserved. Toperform communication between different applications, oneapproach is to use cross-chain swap operation, which, as dis-cussed earlier, is expensive. An alternative approach is touse a new blockchain to maintain public transactions. How-ever, since public transactions use data which is providedby internal transactions, e.g., to place an order, the Man-ufacturer needs to calculate demand internally, and thesetwo types of transactions are stored in different blockchains,data integration becomes an expensive and challenging task.In this paper, we present a new approach that not only

addresses the performance and confidentiality issues, butalso handles cross-application transactions efficiently.

3. THE CAPER MODELIn this section, the CAPER model is introduced. We first

present distributed applications and the blockchain ledger,and then show how the distributed applications are deployedin the blockchain.

1387

Figure 2: (a): A blockchain ledger consisting of Fourapplications, (b), (c), (d), and (e): The views of theblockchain from different applications

3.1 Distributed ApplicationsCAPER is a blockchain system designed to support a set

of collaborating distributed applications which might nottrust each other. Each application maintains two sets ofprivate and public records. The private records of an ap-plication are accessible only to the application whereas thepublic records are replicated on all applications.CAPER supports internal and cross-application transac-

tions. Internal transactions are performed within an ap-plication following the logic of the application, e.g., in theSupply Chain scenario, the Manufacturer calculates mater-ials demand internally. Internal transactions of an applic-ation can read and write its private records, however theycan only read (and not write) the public records. Cross-application (public) transactions, on the other hand, involvemultiple applications and are visible to all applications, e.g.,the Manufacturer places an order for materials via a Middle-man. Cross-application transactions follow the service levelagreements (SLAs) between the involved applications. SLAspresent the flow of communication between the applicationsand indicate different aspects of the services, e.g., quality,availability, and responsibilities, that should be provided bydifferent applications. For example, in the Supply Chainscenario, the Carrier is responsible for delivering requestedmaterials to the Manufacturer in two business days from thedate it is informed by the Supplier. Public records can onlybe updated via Cross-application transactions.

3.2 Blockchain LedgerThe blockchain ledger is an append-only data structure

recording transactions in the form of a hash chain where eachblock contains a batch of transactions. In a permissionedblockchain, as discussed in Section 2, batching transactionsinto blocks decreases performance (Since proof-of-work isnot required [28]). Thus, in CAPER, each block consistsof a single transaction. The blockchain ledger in CAPERconsists of both internal transactions of all applications andall cross-application transactions in the system. To supportboth types of transactions, we generalize the notion of ablockchain ledger from a linear chain to a directed acyclicgraph (DAG) where nodes of the graph are transactions andedges enforce the order of transactions.Within an application, since transactions have access to

the same datastore, a total order between the transactionsthat are initiated by the application is enforced to ensureconsistency. To present the total order of transactions inthe blockchain ledger, transactions are chained together, i.e.,

each transaction includes the cryptographic hash of the pre-vious transaction. In addition, since cross-application trans-actions update data which is replicated on all the applica-tions, to ensure consistency, cross-application transactionsare totally ordered as well. Furthermore, internal trans-actions of applications might use data that is provided bycross-application transactions, e.g., the Manufacturer cal-culates materials demand based on the place-order cross-application transaction of the Bulk Buyer. To show suchdata dependencies, an internal transaction includes the cryp-tographic hash of a cross-application transaction in the ledger.In summary, the blockchain ledger has three properties:

(1) There is a total order between all transactions (internalas well as cross-application) that are initiated by an applic-ation, (2) There is a total order between cross-applicationtransactions, and (3) An internal transaction of an applica-tion might include the cryptographic hash of a cross-applicationtransaction (that is initiated by another application).In addition to internal and cross-application transactions,

a unique initialization transaction (block), called genesistransaction, is considered for the blockchain. Function H(.)also denotes the cryptographic hash function. For simplicity,to show that transaction t includes H(t′) (i.e. t is orderedimmediately after t′ as explained in properties 1 and 2 orhas data dependency to t′ as explained in property 3) weinclude an edge from t to t′ in the DAG representation ofthe blockchain ledger.Fig. 2(a) shows a CAPER blockchain ledger consisting of

four applications α1, α2, α3, and α4. In this figure, λ is thegenesis transaction. Internal and cross-application transac-tions of each application are also specified. For example,t11, t13, t14, and t15 are the internal transactions of applic-ation α1, and t12,1, t23,2, and t34,3 are the cross-applicationtransactions initiated by α1, α2, and α3 respectively. Notethat each cross-application transaction is labeled with ti,jwhere i indicates the order of the transaction among thetransactions that are initiated by its initiator applicationand j presents the order of the transaction among all cross-application transactions. As can be seen, transactions (bothinternal and cross-application) that are initiated by an ap-plication are chained together (property 1), e.g. t31, t32,t33, t34,3, and t35. In addition, cross-application transac-tions are chained together (property 2), i.e., t12,1, t23,2, andt34,3. Finally, an internal transaction might include the hashof a cross-application transaction that shows the data de-pendency of the internal transaction to the cross-applicationtransaction (property 3), e.g., internal transactions t22 of α2has edge to cross-application transaction t12,1.Note that, since an edge from transaction t to transaction

t′ indicates that t occurs after t′ (t includes the hash of t′,thus t′ has to be appended to the ledger earlier), it is easyto show that the resulting graph is acyclic.In contrast to the cross-application transactions that are

visible to and maintained by all applications, the internaltransactions of an application present confidential data aboutthe application, e.g., its business logic. For example, in theSupply Chain scenario, the internal transactions of the Man-ufacturer show its internal process for producing a productwhich the Manufacturer might intend to keep as a secret.The presented blockchain ledger, however, is at odds withconfidentiality because every application has access to everytransaction. For the sake of confidentiality, we want to pro-hibit an application from observing the internal transactions

1388

of other applications. To achieve this, in CAPER, the en-tire blockchain ledger is not maintained by any application.In fact, each application only maintains its own view of theblockchain ledger that includes its internal transactions andall the cross-application transactions. The blockchain ledgeris indeed the union of all these physical views.Fig. 2(b)-(e) show the views of the blockchain ledger for

applications α1, α2, α3, and α4 respectively where each ap-plication maintains only the part of the ledger consisting ofits internal and all the cross-application transactions.

3.3 Application Deployment on a BlockchainEach application in CAPER, in addition to the datastore

and its view of the blockchain ledger, maintains a “privatesmart contract” to implement the application logic, and a“public smart contract” to implement the logic of cross-application transactions.A smart contract, as exemplified by Ethereum [3], is a

computer program that self-executes once it is establishedand deployed. Smart contracts are similar to database trig-gers where the logic of the contract is triggered to be ex-ecuted once some conditions or terms are met. It has theadvantages of being updated in real-time, ensuring accurateexecution, and requiring little human intervention. A smartcontract can also handle automatic conditional paymentsfrom escrow. When a payment function is triggered, thesmart contract automatically checks the defined conditions,and transfers the money according to the defined rules [51].Each application in CAPER has its own private smart

contract which includes the logic of the internal transactions.In addition, a public smart contract is written to include thelogic of cross-application transactions which is determinedby the service level agreements between the applications.As discussed in Section 2, to execute a collaborative pro-cess, participants agree on SLAs which are then writtenin the public smart contracts. The public smart contractruns on every application to check if the cross-applicationtransactions are conforming to the SLAs, and enforce theconditions defined in the cross-application transactions. Incontrast to most existing blockchains where every smart con-tract runs on all nodes, which is at odds with confidential-ity, in CAPER, each private smart contract runs only onits application. To support cross-application transactions,however, the public smart contract runs on every applica-tion. Furthermore, both private and public smart contractscan be written in domain-specific languages, e.g., Solidity,to ensure the deterministic execution of transactions.In the Supply Chain scenario in Figure 1, each of the five

involving applications, i.e., Supplier, Manufacturer, BulkBuyer, Carrier, and Middleman, executes its internal trans-actions following the application logic, which is implemen-ted in its private smart contract. Once a cross-applicationtransaction is requested, e.g., the Bulk Buyer places an or-der with the Manufacturer, every application executes andappends the transaction to its view of the ledger. To executethe cross-application transactions, the public smart contractis used which includes the SLAs (that are agreed upon byall applications). Hence, if the requested transaction doesnot conform to the SLAs, it will be detected. For each cross-application transaction the SLA defines several conditions tocheck. SLA also includes actions for non-conforming trans-actions, e.g., if the Carrier delivers the materials later thanagreed upon, it will be penalized as specified in the SLA or

if the delivered materials are something different from whatthey agreed on, the transaction will be aborted.

4. THE CAPER ARCHITECTURECAPER consists of a set of nodes in an asynchronous

distributed system where each application runs on a (non-empty) disjoint subset of nodes called the agents of the ap-plication. We use N and A to denote the set of nodes andapplications. In addition, Nα indicates the set of agents ofapplication α ∈ A where for each pair of applications α1 andα2 in A, Nα1 ∩Nα2 = ∅.Nodes are connected by point-to-point bi-directional com-

munication channels. Network channels are pairwise au-thenticated, which guarantees that a malicious node cannotforge a message from a correct node, i.e., if node i receivesa message m in the incoming link from node j, then node jmust have sent message m to i beforehand.Furthermore, messages may contain public-key signatures

and message digests [19]. A message digest is a numericrepresentation of the contents of a message produced bycollision-resistant hash functions. Message digests are usedto detect changes and alterations to any part of the message.We denote a messagem signed by replica r as 〈m〉σr and thedigest of a message m by D(m). For signature verification,we assume that all machines have access to the public keysof all other machines.Nodes in CAPER might crash, behave maliciously, or be

reliable. In addition, we assume that applications do nottrust each other, thus we model application failures as Byz-antine failures. In fact, we define two levels of behavior inthe system. First, at the node level, each agent might bea crash-only, a Byzantine, or a reliable node. In a crashfailure model, nodes operate at arbitrary speed, may fail bystopping, and may restart. Whereas, in a Byzantine failuremodel, faulty nodes may exhibit arbitrary, potentially ma-licious, behavior. A reliable node, on the other hand, neverfails. Second, at the application level, an application (as agroup of agents) might behave maliciously. Note that thesetwo levels of behavior are independent of each other. Thus,even if the agents of an application are crash-only nodes, theapplication might still behave maliciously.Ordering the transactions within each application needs

consensus among the agents of the application. To estab-lish consensus, asynchronous fault-tolerant protocols can beused. Fault-tolerant protocols use the state machine rep-lication algorithm [34] where nodes agree on an ordering ofincoming requests. The algorithm has to satisfy two mainproperties, (1) safety: all correct nodes receive the same re-quests in the same order, and (2) liveness: all correct clientrequests are eventually ordered. Fischer et al. [24] show thatin an asynchronous system, where nodes can fail, consensushas no solution that is both safe and live. Based on thatimpossibility result, in most fault-tolerant protocols, safetyis satisfied without any synchrony assumption, however, asynchrony assumption is considered to ensure liveness.Crash fault-tolerant protocols guarantee safety in an asyn-

chronous network using 2f+1 nodes to overcome the sim-ultaneous failure of any f nodes while in Byzantine fault-tolerant protocols, 3f+1 nodes are usually needed to providethe safety property in the presence of f malicious nodes.We now briefly introduce Paxos [35] and PBFT [19] as two

well-known crash and Byzantine fault-tolerant protocols.

1389

In Paxos [35], which guarantees safety in an asynchronousnetwork using 2f+1 nodes, upon receiving a request from aclient, the primary (assuming it is already elected) initiatesa consensus protocol among the agents of the applicationsby multicasting an accept message including the transac-tion. Once an agent receives an accept message, it sends anaccepted message to the primary. The primary waits for faccepted messages from different agents (plus itself becomesf + 1), multicasts a commit message to all the agents, andsends a reply to the client. Upon receiving a commit messagefrom the primary, each agent executes the transaction.In the presence of malicious nodes, PBFT [19] can be used.

In PBFT, which guarantees safety in an asynchronous net-work using 3f+1 nodes, during a normal case execution, aclient sends a request to the primary node, and the primarymulticasts a pre-prepare message to all agents. Then, duringthe prepare and commit phases, all agents communicate toeach other to reach agreement and send responses back tothe client. Note that in Paxos only the primary sends thereply message to the client whereas in PBFT every agentsends the reply message and the client waits for f+1 match-ing reply messages before accepting the result.Paxos and PBFT can be used to establish consensus among

the agents of an application to order internal transactions.To establish consensus for cross-application transactions, sinceapplications may not trust each other, a Byzantine fault-tolerant protocol among applications is needed. To provideboth safety and liveness for consensus at the applicationlevel, we assume that at most b |A|−1

3 c applications mightbe malicious. As a result, to commit a cross-applicationtransaction, by a similar argument as in PBFT [19], atleast two-thirds (b 2|A|

3 c + 1) of the applications includingthe initiator application of each cross-application transac-tion must agree on the order of the transaction. We needagreement from agents of the initiator application to ensurethat the cross-application transaction is consistent with theinternal transactions of the initiator application. The con-formance between a cross-application transaction and theSLAs is checked by every application during the executionof the transaction, thus, if the initiator application initiatesa transaction that does not conform to the SLAs, it will bedetected by other applications during the execution.

5. LOCAL CONSENSUS IN CAPERIn this section, we show how internal transactions are

ordered and executed in CAPER. CAPER employs localconsensus within an application to order transactions wherethe agents of an application, independent of other nodes inthe network, agree on the order of transactions.The local consensus protocols in CAPER are pluggable.

Depending on the failure model of nodes (agents), the ap-plication can use a crash fault-tolerant protocol, e.g., Paxos[35], or a Byzantine fault-tolerant protocol, e.g., PBFT [18],as the local consensus protocol. The application might noteven use a consensus protocol and rely on a single non-faultyreliable node to order the transactions. The number of re-quired agents is also determined by the protocol and themaximum number of simultaneous failures in the network.The local consensus protocol to order the internal trans-

actions of an application is initiated by one of the agents,called the primary. The normal case operation for CAPERto execute an internal transaction proceeds as follows. A

client c requests an internal transaction tx for an applica-tion by sending a message 〈REQUEST, tx, τc, c〉σc to the agentp of the application it believes to be the primary. Here, τc isthe client’s timestamp and the entire message is signed withsignature σc. The timestamps of clients are used to totallyorder the requests of each client and to ensure exactly-oncesemantics for the execution of client requests.When the primary p receives a request from a client, it

first checks the signature to ensure it is valid, and then ini-tiates a local consensus algorithm by multicasting a mes-sage, e.g., accept message in Paxos or pre-prepare message inPBFT, including the requested transaction to other agents.To provide a total order between transactions, the primaryalso includes H(t) in the message where H(.) denotes thecryptographic hash function and t is the previous transac-tion that is ordered by the application. If the transactionhas a data dependency to a cross-application transaction (asdiscussed in Section 3.2), the primary includes the crypto-graphic hash of the cross-application transaction as well.The agents then establish agreement on a total order of

transactions using the utilized consensus protocol, executethe transaction, and append it to the blockchain ledger.Finally, either the primary or every agent node (depend-ing on the local consensus protocol) sends a reply message〈REPLY, τc, u〉σo to client c where tsc is the timestamp of thecorresponding request and u is the execution result.

6. GLOBAL CONSENSUS IN CAPERIn this section, we show how cross-application transac-

tions are ordered and executed in CAPER using global con-sensus among applications. We introduce three ordering ap-proaches for achieving global consensus in CAPER.Ordering transactions using a disjoint set of nodes was

introduced by Hyperledger [10] to enhance the scalability ofthe system and to add flexibility for implementing the con-sensus protocol, i.e., different protocols can be used to estab-lish consensus. Similarly, in the first approach of CAPER,a disjoint set of nodes, called orderers, which are not theagents of any application, are used to globally order cross-application transactions. The global consensus protocol amongorderers is pluggable and depending on the specifications ofthe system, a crash, a Byzantine, or any other fault-tolerantprotocol can be used.In the absence of such orderer nodes, and in the second

approach, CAPER relies on the applications to order cross-application transactions in a hierarchical way. To distin-guish between trust at the node level and trust at the ap-plication level, agreement is established in two levels: a locallevel among the agents of each application, and a global levelamong the applications of the system.Although the second approach does not require a set of

orderers to order transactions, the hierarchical nature of thealgorithm, which needs local consensus within each applic-ation for each step of global ordering, makes the protocolexpensive. In the third approach, similar to the second ap-proach, the agents of all applications order cross-applicationtransactions, however, agreement is established in one level.In all three approaches, when agreement is achieved, the

agents execute the transaction and append it to their localviews of the ledger. Note that if the transaction does notfollow the service level agreements, which are implementedin the public smart contract, it will be detected during theexecution of the transaction (as discussed in Section 3.3).

1390

Algorithm 1 Global Consensus using Orderers1: init():2: r := node_id3: O := the set of orderer nodes4: p := the primary agent of the initiator application5: o := the primary agent of the orderers

6: upon receiving m=〈REQUEST, tx, τc, c〉σc and (r == p):7: if m is valid then8: initiate local consensus9: if agreement is achieved then10: send 〈〈ORDER, hL, d, r〉σr ,m〉 to o . either primary or every

agent r of the initiator application

11: upon receiving valid 〈〈ORDER, hL, d, r〉σr ,m〉 from the sufficientnumber of agents and node is the primary orderer o

12: initiate a global consensus among orderers O13: if agreement is achieved then14: multicast 〈SYNC, hL, hG, d, o〉σo ,m〉 . o or every orderer

15: upon receiving 〈SYNC, hL, hG, d, o〉σo ,m〉 from the primary or-derer (or a sufficient number of orderers)

16: execute and append the transaction to the ledger

6.1 Global Consensus using a Separate Set ofOrderers

Using a separate set of nodes to order transactions addsflexibility to the system by allowing the global consensusimplementation to be tailored to the trust assumption of aparticular deployment. In addition, since the orderers aredecoupled from the agents that execute transactions andmaintain the blockchain ledger, using orderers enhances thescalability of the system.In the first approach, CAPER orders the cross-application

transactions using a disjoint set of orderers O where for eachapplication α in A, O∩Nα = ∅. As discussed earlier, a cross-application transaction is ordered locally among the trans-actions that are initiated by the initiator application andglobally among all cross-application transactions, thus bothlocal and global orderings are needed. Since orderers are notinvolved in the local consensus and application agents do notparticipate in the global consensus, these two orderings areseparated from each other. As a result, cross-applicationtransactions are first, similar to the internal transactions,ordered locally within each application using the applica-tion consensus protocol and then ordered globally among allcross-application transactions using orderers. Note that thecross-application transactions are ordered first locally andthen globally to prevent the case where the agents of theinitiator application do not agree on the local order of atransaction that has already been ordered globally. Onceorderers agree on the global order of the transaction, theymulticast the transaction to all the applications, thus, everyagent of every application executes and appends the trans-action to its ledger.The normal case operation of global consensus using or-

derers to execute a cross-application transaction is presentedin Algorithm 1. Although not explicitly mentioned, everysent and received message is logged by the nodes. As in-dicated in lines 1-5 of the algorithm, nodes p and o are theprimary agent of the initiator application and the primaryorderer respectively and O is the set of orderers.As shown in lines 6-10 of the algorithm, when primary

agent p receives a valid request 〈REQUEST, tx, τc, c〉σc from anauthorized client c (with timestamp τc) to execute a cross-application transaction tx, it initiates the local consensus

Figure 3: Global Consensus using a Set of Orderers

protocol to establish agreement on the order of the cross-application transaction among all transactions that are ini-tiated by the application. Once agreement is achieved, de-pending on the local consensus protocol, either the primaryor every agent r of the initiator application sends a signedorder message 〈〈ORDER, hL, d, r〉σr ,m〉 including the client’srequest message m to the primary orderer node where d ism’s digest and hL = H(t) where t is the previous transactionthat is ordered by the application. Note that hL is includedin the order messages to let orderers know that the agents ofthe application agree on the local order of the transaction.When the order message is sent to the primary orderer,

the primary of the application waits for the correspondingreply (sync) message from the orderers before initiating anyother transactions. In addition, when local agreement is es-tablished, nodes wait for global agreement before appendingthe transaction to the blockchain ledger.As shown in lines 11-14, once primary orderer o receives

a sufficient number (determined by the local consensus pro-tocol of the initiator application, i.e., 1 for crash fault-tolerantand f + 1 for Byzantine fault-tolerant) of valid matching or-der messages (with valid signature, matching hL, and match-ing message digest), primary orderer o initiates the (global)consensus protocol by multicasting the transaction to otherorderers to establish a total order on cross-application trans-actions. Once the orderers reach agreement on the order ofthe transaction, depending on the global consensus protocol,either the primary or every orderer node o multicasts a syncmessage 〈SYNC, hL, hG, d, o〉σo ,m〉 to all the agent of everyapplication where d is the digest of m, hL is copied fromorder message of the initiator application, and hG = H(t)such that t is the previous cross-application transaction.Upon receiving a sync message, the agents of each ap-

plication log the message. Once an agent receives a suf-ficient number of matching sync messages (determined bythe utilized global consensus protocol), the agent executesthe transaction and appends the transaction to its block-chain ledger (as can be seen in lines 15 and 16). Note thatthe agents of the initiator application consider both hL andhG hashes to append the transaction to the ledger whilethe agents of the other applications only consider the hashof the previous cross-application transaction (hG). Finally,depending on the utilized local consensus protocol, eitherthe primary or every agent node r of the initiator applica-tion sends a reply message 〈REPLY, τc, u〉σr to client c wheretsc is the timestamp of the corresponding request and u isthe result of executing the request.The flow of cross-application transactions using a set of

orderers in a blockchain consisting of four applications α1,α2, α3, and α4 can be seen in Figure 3. Here application α1initiates the cross-application transaction. In addition, o1,

1391

Algorithm 2 Hierarchical Global Consensus1: init():2: r := node_id3: α := the application that initiates the consensus4: P := the set of primary agents of all applications5: p := the primary of α

6: upon receiving transaction m and (r == p)7: if m is valid then8: multicast 〈〈PROPOSE, hL, hG, d〉σp ,m〉 to P

9: if (r == p) OR (upon receiving 〈〈PROPOSE, hL, hG, d〉σp ,m〉 frominitiator primary p and r ∈ P )

10: initiate local consensus11: if agreement is achieved then12: multicast 〈ACCEPT, hL, hG, d, r〉σr to P . r or all agents

13: upon receiving matching 〈ACCEPT, hL, hG, d, q〉σq from two-thirds of the applications including α and r ∈ P

14: initiate a local consensus15: if agreement is achieved then16: multicast 〈COMMIT, hL, hG, d, r〉σr . either r or all agents

17: upon receiving matching 〈COMMIT, hL, hG, d, r〉σr from two-thirds of the applications including α

18: execute and append the transaction to the ledger

o2, and o3 are the orderer nodes. Upon receiving a cross-application transaction from a client, the primary node n1of α1 validates the transaction and similar to internal trans-actions, initiates a local consensus algorithm to order thetransaction within the application. Once the transaction isinternally ordered, an order message is sent to the primaryorderer node. The orderers use a global consensus protocol,e.g., a crash fault-tolerant protocol with f = 1 in Figure 3,to agree on the global order of the transaction and then sincehere the orderers are crash-only nodes, the primary orderer(node o1) multicasts sync messages including the transactionto every agent of every application. Each agent then valid-ates the transaction, executes the transaction, and appendsit to the ledger.Safety and Liveness. Since the global ordering protocolis pluggable, its safety and liveness are implied due to Paxos[35] and PBFT [19]. The order of cross-application trans-actions on different applications is unique because cross-application transactions are ordered sequentially by orderersand the agents of every application follows the order that isprovided by orderers. Note that if the agents of an applica-tion do not follow the provided order, the application mightnot be able to initiate cross-application transactions in thefuture (since the initiated transactions might not conformto SLAs). To ensure a total order between transactions thatare initiated by the same application, once the primary ofan application sends a cross-application transaction to beordered by the orderers, the primary stops initiating anyother transactions and waits for the reply from the orderers.

6.2 Hierarchical Global ConsensusWhile using a separate set of orderer nodes makes the

agreement routine simple and modular, it comes with anextra cost of adding orderers to the system. In the absenceof such orderer nodes, reaching consensus on the order of thecross-application transactions needs the participation of allapplications. To distinguish between trust at the node leveland trust at the application level, CAPER uses an asyn-chronous Byzantine fault-tolerant protocol for the globalconsensus where for each cross-application transaction and

Figure 4: Hierarchical Global Consensus

in each phase of the global consensus, every application runsits local consensus protocol between its agents to internallydecide on the application vote in that phase.In addition, since the agents of the initiator application

participate in the global consensus, in contrast to the firstapproach, the local and global orderings are merged to-gether. However for each step of the global ordering theprotocol ensures that the initiator application agrees withthe ordering. Hence the transaction is ordered correctlywith respect to the transactions that are initiated by theinitiator application.Furthermore, in each step of the global ordering and within

each application, once agreement is established, dependingon the utilized local consensus protocol, either the primary,e.g., in Paxos, or every agent, e.g., in PBFT, sends the voteto other applications.Algorithm 2 presents the hierarchical global consensus.

Same as before, every sent and received message is loggedby the agents. As presented in lines 1-5 of the algorithm, Pis the set of primary agents of all applications and p is theprimary agent of the initiator application α.Once the primary agent p of the initiator application re-

ceives a valid cross-application transaction, as indicated inlines 6-8, it multicasts a signed 〈〈PROPOSE, hL, hG, d〉σp ,m〉message to the primary agents of every application whered is the digest of m, hL = H(t) such that t is the pre-vious transaction that is initiated by the application, andhG = H(t′) such that t′ is the previous cross-applicationtransaction. Note that hash hL is only used by the agentsof the initiator application to ensure that the new trans-action is ordered correctly with respect to the transactionsthat are initiated by the application. The agents of the otherapplications ignore hL.As can be seen in lines 9-12, upon receiving a propose mes-

sage from an application, the primary agent of every applic-ation checks the signature, hash hG, and message digest toensure the message is valid. Then, every primary agent (in-cluding the primary of the initiator application) internallyinitiates the local consensus protocol to establish agreementon the order of the requested transaction. If agreement isachieved, depending on the utilized local consensus protocol,either the primary or every agent multicasts an accept mes-sage to the primary agents of all other applications. Notethat local consensus is needed to ensures that non-faultyagents agree with the received propose message. Hence, ifagreement achieved, they just log the messages and do notappend the transaction to their copies of the ledger.

1392

Algorithm 3 One-Level Global Consensus1: init():2: r := node_id3: α := the application that initiates the consensus4: p := the primary agent of α

5: upon receiving transaction m and (r == p)6: if m is valid then7: broadcast 〈〈PROPOSE, hL, hG, d〉σp ,m〉 to every agents

8: upon receiving 〈〈PROPOSE, hL, hG, d〉σp ,m〉 from primary p9: if the message is valid then10: broadcast 〈ACCEPT, hL, hG, d, r〉σr

11: upon receiving matching 〈ACCEPT, hL, hG, d, r〉σr from local-majority of two-thirds of the applications including α

12: if the message is valid then13: broadcast 〈COMMIT, hL, hG, d, r〉σr

14: upon receiving matching 〈COMMIT, hL, hG, d, r〉σr from local-majority of two-thirds of the applications including α

15: execute and append the transaction to the ledger

As shown in lines 13-16, each application waits for validaccept messages from two-thirds of the applications includ-ing the initiator application which are matched with theaccept message that is sent by the application. Note that avalid accept message from initiator application is needed toensure that the transaction is consistent with the transac-tions that are initiated by that application. Upon receivinga sufficient number of accept messages, the primary agentof each application initiates the local consensus protocol toestablish agreement on the received accept messages. Onceagreement is achieved, the application (either the primaryor every agent) multicasts a commit message to every agentof all other applications.The propose and accept phases of the global consensus,

similar to pre-prepare and prepare phases of PBFT [19],guarantee that non-malicious applications agree on a totalorder for the transactions. Indeed, they ensure that no forkhappens in the blockchain, i.e., it is not possible to havetwo different transactions with the same hash hG. This istrue because at least two-thirds (b 2|A|

3 c + 1) of the applic-ations agreed with the order of each transactions, thus anytwo quorums of applications intersect in at least b |A|−1

3 c+ 1applications. Since at most b |A|−1

3 c applications might bemalicious, there is at least one non-malicious application inthe intersection of any two quorums.Finally, in lines 17 and 18, similar to the previous phase,

if an application receives matching commit messages fromtwo-thirds of the applications including the initiator one thatmatch the application’s commit message, its agents executethe transaction and append it to their ledgers.Figure 4 shows the hierarchical consensus with four ap-

plications (similar to Figure 3) where applications α1 andα3 use a crash fault-tolerant (CFT) protocol and applica-tion α2 uses a Byzantine-fault-tolerant (BFT) protocol astheir local consensus protocol. Here, the primary of applic-ation α1 initiates the consensus.As an optimization, for a system with a high percentage of

cross-application transactions and to prevent the initiationof concurrent cross-application transactions, the primarynode of one of the applications can be designated as a superprimary where every application sends its cross-applicationtransaction to the super primary and the super primary ini-tiates the protocol.

Figure 5: One-level Global Consensus

Safety and Liveness. In the hierarchical consensus, asdiscussed earlier, since at least b 2|A|

3 c + 1 of the applica-tions must agree with the order of a transaction and at mostb |A|−1

3 c applications might be malicious, safety is ensured.Indeed, if two or more concurrent transactions are initiated,at most one of them collects the required number of mes-sages (two-thirds of the applications) , i.e., it is not possiblefor more than one of them to be ordered with the same hashhG. If none of the concurrent transactions collects enoughvotes, all initiator applications try to send their transactionsagain. In such a situation and to ensure liveness, CAPERassigns a timer to each transaction and delays the transac-tions to prevent concurrent re-initiation of the transactions.

6.3 One-Level Global ConsensusWhile hierarchical consensus eliminates the need for hav-

ing an extra set of orderer nodes and also distinguishesbetween trust at the node level and trust at the applicationlevel, it requires an expensive two-level consensus protocolwhere each step of the global consensus needs the entire localconsensus protocol to be run within each application. In thissection, we introduce a one-level global consensus protocolwhere for each cross-application transaction, the agents ofall applications participate to achieve consensus on the orderof the transaction.Since the number of agents of each application depends

on the utilized consensus protocol within the application,the required number of matching replies to ensure that themajority of agents of an application agree on the order ofthe transaction is different from application to application.Therefore, we define local-majority as the required number ofmatching messages from the agents of an application. If theagents of an application are crash-only nodes, local-majorityfor the application is equal to f + 1 (from the total 2f + 1agents), and if the agents of an application might behavemaliciously, local-majority for the application is equal to2f + 1 (from the total 3f + 1 agents). For an applicationthat has only a single reliable agent, local-majority is one.Algorithm 3 presents the one-level global consensus. Vari-

able p indicates the primary agent of the initiator applica-tion α. As shown in lines 5-7, the primary of the initiatorapplication broadcasts a signed propose message includingthe transaction, the hash of the previous transaction that isinitiated by the application (hL), and the hash of the previ-ous cross-application transaction (hG) to the agents of everyapplication. Same as before, hL is only used by the agentsof the initiator application.

1393

0 10 20 30 40

10

20

30

40

Throughput [ktrans/sec]

Late

ncy

[ms]

CAPERFabric

(a) 0% Cross-application

0 6 12 18 24 30

200

400

600

Throughput [ktrans/sec]

Late

ncy

[ms]

OrderersHierarchical

One-levelFabric

(b) 20% Cross-application

0 3 6 9 12

1

2

3

Throughput [Ktrans/sec]

Late

ncy

[s]

OrderersHierarchical

One-levelFabric

(c) 80% Cross-application

0 2 4 6 8

1

2

3

Throughput [Ktrans/sec]

Late

ncy

[s]

OrderersHierarchical

One-levelFabric

(d) 100% cross-application

Figure 6: Throughput/Latency Measurement by Increasing the percentage of Cross-application Transactions

Once an agent receives a propose message, it checks thesignature, message digest, and hash hG to ensure the mes-sage is valid. If the agent belongs to the initiator application,it also checks hash hL. Once the message is validated, theagent broadcasts an accept message to every agent of everyapplication, as indicated in lines 8-10.As presented in lines 11-13, upon receiving valid accept

messages from the local-majority of two-thirds of the ap-plications including the initiator application that match theaccept message which is sent by the agent, each agent broad-casts a commit message to every agent of every application.The propose and accept phases of the algorithm, similar topre-prepare and prepare phases of PBFT [19], guaranteethat non-faulty agents agree on a order for the transactions.Finally, as shown in lines 14 and 15, once an agent re-

ceives valid commit messages from local-majority of two-thirds of the applications including the initiator applicationthat matches its commit message, the agent considers thetransaction as committed, thus, executes the transactionand appends the transaction to the ledger.Figure 5 shows the one-level consensus in CAPER for four

applications with different failure modes.

Safety and Liveness. To ensure safety in one-level con-sensus, in each of the accept and commit phases, matchingmessages from the local majority of two-thirds of the ap-plication is required. Local majority of each application isneeded to ensure that any two quorums intersect in at leastone non-faulty node within the application and two-thirds ofthe applications is needed to ensure that any two quorumsintersect in at least one non-malicious application (since atmost b |A|−1

3 c applications might be malicious). To ensureliveness, similar to the hierarchical consensus, CAPER as-signs timers to delay concurrent transactions and also as anoptimization uses a super primary for systems with a highpercentage of cross-application transactions.

7. EXPERIMENTAL EVALUATIONSIn this section, we conduct several experiments to evalu-

ate CAPER. As explained earlier, CAPER consists of a setof collaborating distributed applications where each applic-ation maintains its data in a datastore consisting of privateand public records. The private records of the datastore,which are replicated across the agents of application, in-clude the data of internal transactions. The public records,on the other hand, are replicated across every agent of everyapplication and include the data of cross-application trans-actions. For the purpose of this evaluation, applications are

implemented as simple accounting applications where clientscan initiate transactions to transfer assets from one or moreof their accounts to other accounts.In addition to CAPER, we also implemented a permis-

sioned blockchain system specifically designed in the execute-order-validate architecture introduced by Fabric [10] wherethe transactions (internal as well as cross-application) of dif-ferent applications are executed by the agents (endorsers)of their applications in parallel, ordered by a separate setof orderers, and then validated by every agent of every ap-plication. Each block also consists of a single transaction(as in [28]). Note that in the case of Fabric, all internal aswell as cross-application transactions are ordered by order-ers and all the applications maintain the same blockchainledger (i.e., the confidentiality of data is not preserved).The experiments are conducted on the Amazon EC2 plat-

form. Each VM is Compute Optimized c4.2xlarge instancewith 8 vCPUs and 15GB RAM, Intel Xeon E5-2666 v3 pro-cessor clocked at 3.50 GHz. In each experiment, we increasethe total number of transactions per second from 100 to100000 (by increasing the number of clients running on asingle VM) and measure the end-to-end throughput (x-axis)and latency (y-axis) of the system. The load is equally dis-tributed among the applications.

7.1 Workloads with Cross-Application trans-actions

In the first set of experiments, we measure the perform-ance of CAPER for workloads with different percentage ofcross-application transactions, i.e., 0%, 20%, 80%, and 100%.We consider four applications where each application hasthree agents and uses a Paxos protocol with f = 1 to estab-lish consensus on its internal transactions. To process cross-application transactions we implement all three approaches,using a set of orderers (we refer to this approach as or-derers), hierarchical, and one-level, which are explained inSection 6. Orderers are implemented using a typical Kafkaorderer setup with 3 ZooKeeper nodes, 4 Kafka brokers and3 orderers (similar to the ordering service of Fabric [10]).The results are shown in Figure 6(a)-(d).When all transactions are internal (Figure 6(a)), each ap-

plication processes its transactions independent of other ap-plications. In such a situation, CAPER is able to processupto 36000 transactions (9000 transactions per application)with very low latency (∼10 ms). Here, since there is no de-pendency between the transactions of different applicationsand the blockchain views of applications are constructed inparallel, the throughput of the entire system increases lin-

1394

early by increasing the number of applications. With thesame latency as CAPER (∼10 ms), Fabric processes 3000transactions. Fabric is able to process upto 7000 transac-tions in total with 40 ms latency, however, the end-to-endthroughput is saturated beyond 7000 transactions. In Fab-ric, adding more applications only increases the number ofparallel execution threads and since every transaction of allapplications is ordered by the same set of orderers, the per-formance of Fabric is not significantly improved. Note thatsince all transactions are internal, if Fabric uses differentchannels for different applications, it can linearly scale asthe number of applications increases. However, even withthat improvement, CAPER still provides higher throughput(∼29% higher in its peak throughput).In the second set of experiments, the workload is changed

to include 20% cross-application transactions which are equallyinitiated by different applications. As can be seen in Fig-ure 6(b), when CAPER is not heavily loaded, the one-levelconsensus approach has better performance since it involvesless number of communication phases. As can be seen, theone-level approach processes ∼16000 transactions with 90ms latency whereas the hierarchical and orderers approacheshave latencies of 220 and 150 ms latency respectively in or-der to process the same number of transactions.Once CAPER becomes heavily loaded, cross-application

transactions might be initiated in parallel. As a result, thelatencies of the hierarchical and one-level approaches aredramatically increased (as discussed in Section 6). With400 ms latency, using orderers, CAPER is able to process30000 transactions whereas the one-level and hierarchicalapproaches process 18000 and 19000 transactions (resp.).The performance of Fabric, however, is not affected by

increasing the percentage of cross-application transactions(since all transactions are ordered by the same set of or-derers) and Fabric still processes upto 7000 transactions intotal with 40 ms latency.Increasing the percentage of cross-application transactions

to 80% (Figure 6(c)) decreases the performance of all threeapproaches. In this case, using orderers, CAPER is able toprocess∼10000 transaction with sub-second latency whereasthe hierarchical and one-level approaches process upto 6000transactions with the same latency. This is expected be-cause in the hierarchical and one-level approaches when thesystem is heavily loaded, different nodes receive concurrenttransactions in different orders.When all transactions are cross-application, the one-level

consensus can only process ∼4000 transactions per secondwith 800 ms latency whereas using the orderers, CAPER isable to process ∼9000 transactions with the same latency.Fabric, same as before, is able to process upto 7000 transac-tions in total with 40 ms latency which is better than bothhierarchical and one-level approaches.As mentioned before, since Fabric orders all internal as

well as cross-application transactions by the same set of or-derers, the performance of Fabric is not affected by increas-ing the percentage of cross-application transactions. As aresult, in workloads with 80% and 100% cross-applicationtransactions, Fabric performs better than all other approachesin terms of latency. In fact, for cross-application transac-tions and to achieve consensus, CAPER uses either mul-tiple rounds of consensus (in the orderers and hierarchicalapproaches) or a Byzantine fault-tolerant protocol with alarge number of participants (in the one-level approach).

0 6 12 18 24 30 36

0.15

0.3

0.45

0.6

Throughput [ktrans/sec]

Late

ncy

[s]

OrderersHierarchical

One-levelFabric

(a) 4 Applications

0 15 30 45 60

0.25

0.5

0.75

1

Throughput [ktrans/sec]

Late

ncy

[s]

OrderersHierarchical

One-levelFabric

(b) 8 Applications

Figure 7: Performance with 4 and 8 Applications

This is in contrast to Fabric that relies on a single crashfault-tolerant protocol (with only three nodes) to order thetransactions which results in a lower latency. However, Fab-ric does not ensure confidentiality of data in this settings.Note that even with 100% cross-application transactions,

the throughput of CAPER using the orderers approach isslightly higher than Fabric (9% higher in its peak through-put) because the throughput of Fabric is affected by con-flicting transactions, i.e., transactions that access the samerecords, due to its execute-order-validate architecture [9] (inthe experiments, ∼10% of the transactions are conflicting).

7.2 Performance with Multiple ApplicationsIn the next set of experiments, we measure the perform-

ance of CAPER in two deployments with 4 and 8 applic-ations where each application has three agents and uses aPaxos protocol with f = 1 to establish consensus on its in-ternal transactions. For each deployment, we consider work-loads with 90% internal and 10% cross-application transac-tions (the typical settings in partitioned databases [50] [48]).For the deployment with four applications (Figure 7(a)),

the performance of CAPER is close to the scenario in Fig-ure 6(b) where in the workloads with less than∼20000 trans-actions per second, the one-level consensus has better per-formance and beyond that, using orderers is more beneficial.Increasing the number of applications (Figure 7(b)), how-ever, results in higher latency for the same throughput inboth one-level and hierarchical consensus due to the increas-ing number of nodes which increases the chance of conflictbetween concurrent transactions. Note that since 90% ofthe transactions are internal, increasing the number of ap-plications improves the overall throughput of CAPER near-linearly (using orderers, CAPER processes upto ∼32000 and∼59000 transactions per second with four and eight applic-ations (respectively) with 30 ms latency). As mentionedearlier, in Fabric, increasing the number of applications doesnot significantly improve the performance.

8. RELATED WORKMost existing permissioned blockchain systems follow the

order-execute architecture and differ mainly in their order-ing routines. The ordering protocol of Tendermint [32] isdifferent from the original PBFT in two ways. First, onlya subset of nodes participate in the consensus protocol andsecond, the leader is changed after the construction of everyblock (leader rotation). Quorum [20], as an Ethereum-based[3] permissioned blockchain, introduces a consensus protocolbased on Raft [41], a well-known crash fault-tolerant pro-tocol. Quorum, similar to CAPER, supports public andprivate transactions, however, in Quorum, both public and

1395

private transactions are ordered using the same consensusprotocol which results in lower throughput in comparison toCAPER. Quorum also uses cryptography techniques to en-sure confidentiality of private transactions. Chain Core [1],Multichain [26], Hyperledger Iroha [5], and Corda [2] aresome other prominent permissioned blockchains that utilizeorder-execute architecture.Fabric [10], as a permissioned blockchain, introduces the

execute-order-validate architecture and leverages parallel-ism by executing the transactions of different applicationssimultaneously. Fabric presents modular design, pluggablefault-tolerant protocol, policy-based endorsement, and non-deterministic execution for the first time in the context ofpermissioned blockchains. In a recent release, Fabric alsoutilizes the Raft protocol [41] for its ordering service where aleader node is elected (per channel) and replicates messagesto the followers. Raft mainly helps organizations to havetheir own ordering nodes, participating in the ordering ser-vice, which leads to a more decentralized system. CAPERutilizes some of the Fabric properties such as modular designand pluggable protocol. In addition, and in contrast tosingle-channel Fabric, CAPER constructs blocks simultan-eously and ensures the confidentiality of both transactiondata and ledger, whereas Fabric ensures only transactiondata confidentiality using Private Data Collections [7].Applications deployed on the same or different blockchains

need to communicate with each other in order to exchangeassets or information. Atomic cross-chain swaps [27] areused for trading assets on two unrelated blockchains. Atomicswaps use hash-lock and time-lock mechanisms to either per-form all or none of a cryptographically linked set of transac-tions. Interledger protocols (ILPV [49]) which are presentedby the World Wide Web Consortium (W3C) use a general-ization of atomic swaps and enable secure transfers betweentwo blockchain ledgers using escrow transactions. Since theredemption of an escrow transaction needs fulfillment of allthe terms of an agreement, the transfer is atomic. Lightningnetwork [39] [43] also generalizes atomic swap to transferassets between two different clients via a network of micro-payment channels. Blocknet [22], BTC [16], Xclaim [53],POA Bridge [6] (designed specifically for Ethereum), Wan-chain [8], and Fusion [4] are some other blockchain systemsthat allow users to transfer assets between two chains.Fabric also addresses atomic cross-chain swap between

permissioned blockchains that are deployed on different chan-nels by either assuming the existence of a trusted channelamong the participants or using an atomic commit pro-tocol [11] [10]. CAPER is different from Fabric in two ways:First, cross-chain communications follow the service levelagreements and are visible to everyone, and second, CAPERdoes not need a trusted channel among the participants.Using sidechain is proposed in [13] to transfer assets from

a main blockchain to the sidechain(s) and execute sometransactions in the sidechain(s). Sidechains can reduce con-firmation time, support more functionality than the mainblockchain, and reduce the transaction cost. In sidechains,a set of known nodes, called functionaries, are responsiblefor moving the assets back from the sidechain to the mainchain. Liquid [23], Plasma [42], Sidechains [25], and RSK[37] are some other blockchain systems that use sidechains.Polkadot [52] and Cosmos [33] also construct a main chainwhich is used by a set of (side) blockchains, i.e., parachainsin Polkadot and zones in Cosmos, to exchange value or in-

formation. Both Polkadot and Cosmos rely on Byzantineconsensus protocols in both sender and receiver sides.Our work is also related to blockchain systems with Direc-

ted acyclic graph structure. Byteball [21] and Iota [44] aretwo DAG structured permissionless blockchains. In Byte-ball, a set of privileged users, called witnesses, determines atotal order on the DAG to prevent double spending, whereas,in Iota [44], the number of descendant transactions is usedto commit a transaction and abort the other one. In Iota,the blockchain, called Tangle, grows in more than one dir-ection. Indeed, once a user issues a transaction, the usermust pick two existing transactions and approve them (res-ults in adding edges from the new transaction to the existingones). The user will also solve a small PoW puzzle similar toBitcoin. Hashgraph [14] is another DAG structured permis-sioned blockchain that combines a voting algorithm with agossip protocol to achieve consensus among nodes. In Hash-graph nodes submit transactions (events) and gossip abouttransactions by randomly choosing other nodes (neighbors).Each transaction in Hashgraph includes the hash of the pre-vious transactions of both sender and receiver. This processcontinues until convergence when all nodes become awareof all transactions. Hashgraph, in contrast to CAPER, doesnot distinguish between internal and cross-application trans-actions which results in lower performance as well as con-fidentiality issues. Vegvisir [30], Ghost [46], Inclusive pro-tocol [38], DagCoin [36], Phantom [47], Spectre [45], andMeshCash [15] are some other DAG structured blockchainsystems. In CAPER, in contrast to all these blockchains,internal transactions of different applications are added in-dependent of each others and only cross-application transac-tions need a global consensus. As a result, first, the doublespending problem never occurs, and second, internal trans-actions can be processed simultaneously, which results inlower latency and higher throughput.

9. CONCLUSIONIn this paper, we proposed CAPER, a permissioned block-

chain system that supports both internal and cross-applicationtransactions of collaborating distributed applications. CAPERtargets both performance and confidentiality aspects of block-chain systems. To achieve better performance, CAPER or-ders and executes internal transactions of different applica-tions simultaneously. In addition, to achieve confidentiality,the blockchain ledger is not maintained by any node andeach application maintains its own local view of the ledgerincluding its internal and all cross-application transactions.CAPER also distinguishes between trust at the node leveland trust at the application level and allows an applicationto behave maliciously for its benefit while its nodes are non-malicious. Furthermore, CAPER introduces three consensusprotocols to globally order cross-application transactions:using a separate set of orderers, hierarchical consensus, andone-level consensus. Our experiments show that for lightlyloaded applications one-level consensus shows better per-formance whereas using a set of orderers is more beneficialfor heavily loaded applications. In the absence of extra re-sources for orderers, the hierarchical approach can providebetter performance in heavily loaded applications.

AcknowledgementThis work is funded by NSF grants CNS-1703560 and CNS-1815733.

1396

10. REFERENCES[1] Chain. http://chain.com.[2] Corda. https://github.com/corda/corda.[3] Ethereum blockchain app platform.

https://www.ethereum.org. 2017.[4] Fusion whitepaper: An inclusive cryptofinance

platform based on blockchain.https://docs.wixstatic.com/ugd/76b9ac_be5c61ff0e3048b3a21456223d542687.pdf.

[5] Hyperledger iroha.https://github.com/hyperledger/iroha.

[6] Poa bridge.https://github.com/poanetwork/token-bridge.

[7] Private data collections: A high-level overview.https://www.hyperledger.org/blog/2018/10/23/private-data-collections-a-high-level-overview.

[8] Wanchain: Building super financial markets for thenew digital economy.https://www.wanchain.org/files/Wanchain-Whitepaper-EN-version.pdf.

[9] M. J. Amiri, D. Agrawal, and A. E. Abbadi.Parblockchain: Leveraging transaction parallelism inpermissioned blockchain systems. In 2019 IEEE 39thInternational Conference on Distributed ComputingSystems (ICDCS). IEEE, 2019.

[10] E. Androulaki, A. Barger, V. Bortnikov, C. Cachin,K. Christidis, A. De Caro, D. Enyeart, C. Ferris,G. Laventman, Y. Manevich, et al. Hyperledger fabric:a distributed operating system for permissionedblockchains. In Proceedings of the Thirteenth EuroSysConference, page 30. ACM, 2018.

[11] E. Androulaki, C. Cachin, A. De Caro, andE. Kokoris-Kogias. Channels: Horizontal scaling andconfidentiality on permissioned blockchains. InEuropean Symposium on Research in ComputerSecurity, pages 111–131. Springer, 2018.

[12] A. Azaria, A. Ekblaw, T. Vieira, and A. Lippman.Medrec: Using blockchain for medical data access andpermission management. In International Conferenceon Open and Big Data (OBD), pages 25–30. IEEE,2016.

[13] A. Back, M. Corallo, L. Dashjr, M. Friedenbach,G. Maxwell, A. Miller, A. Poelstra, J. Timón, andP. Wuille. Enabling blockchain innovations withpegged sidechains. 2014.

[14] L. Baird. The swirlds hashgraph consensus algorithm:Fair, fast, byzantine fault tolerance. Swirlds, Inc.Technical Report SWIRLDS-TR-2016, 1, 2016.

[15] I. Bentov, P. Hubácek, T. Moran, and A. Nadler.Tortoise and hares consensus: the meshcashframework for incentive-compatible, scalablecryptocurrencies. IACR Cryptology ePrint Archive,2017:300, 2017.

[16] V. Buterin. Chain interoperability. R3 ResearchPaper, 2016.

[17] C. Cachin and M. Vukolić. Blockchain consensusprotocols in the wild. In 31 International Symposiumon Distributed Computing, DISC, pages 1–16, 2017.

[18] M. Castro and B. Liskov. Practical byzantine faulttolerance and proactive recovery. ACM Transactionson Computer Systems (TOCS), 20(4):398–461, 2002.

[19] M. Castro, B. Liskov, et al. Practical byzantine faulttolerance. In OSDI, volume 99, pages 173–186, 1999.

[20] J. M. Chase. Quorum white paper, 2016.[21] A. Churyumov. Byteball: A decentralized system for

storage and transfer of value. 2016.[22] A. Culwick and D. Metcalf. The blocknet design

specification, 2018.[23] J. Dilley, A. Poelstra, J. Wilkins, M. Piekarska,

B. Gorlick, and M. Friedenbach. Strong federations:An interoperable blockchain solution to centralizedthird-party risks. arXiv preprint arXiv:1612.05491,2016.

[24] M. J. Fischer, N. A. Lynch, and M. S. Paterson.Impossibility of distributed consensus with one faultyprocess. Journal of the ACM, 32(2):374–382, 1985.

[25] A. Garoffolo and R. Viglione. Sidechains: Decoupledconsensus between chains. arXiv preprintarXiv:1812.05441, 2018.

[26] G. Greenspan. Multichain private blockchain-whitepaper. URl: http://www. multichain.com/download/MultiChain-White-Paper. pdf, 2015.

[27] M. Herlihy. Atomic cross-chain swaps. In ACMSymposium on Principles of Distributed Computing(PODC), pages 245–254. ACM, 2018.

[28] Z. István, A. Sorniotti, and M. Vukolić. Streamchain:Do blockchains need blocks? In Proceedings of the 2ndWorkshop on Scalable and Resilient Infrastructures forDistributed Ledgers, pages 1–6. ACM, 2018.

[29] H. Jin, X. Dai, and J. Xiao. Towards a novelarchitecture for enabling interoperability amongstmultiple blockchains. In 2018 IEEE 38th InternationalConference on Distributed Computing Systems(ICDCS), pages 1203–1211. IEEE, 2018.

[30] K. Karlsson, W. Jiang, S. Wicker, D. Adams, E. Ma,R. van Renesse, and H. Weatherspoon. Vegvisir: Apartition-tolerant blockchain for the internet-of-things.In IEEE 38th International Conference on DistributedComputing Systems (ICDCS), pages 1150–1158.IEEE, 2018.

[31] K. Korpela, J. Hallikas, and T. Dahlberg. Digitalsupply chain transformation toward blockchainintegration. In proceedings of the 50th Hawaiiinternational conference on system sciences, 2017.

[32] J. Kwon. Tendermint: Consensus without mining.Draft v. 0.6, fall, 2014.

[33] J. Kwon and E. Buchman. Cosmos: A network ofdistributed ledgers. URL https://cosmos.network/whitepaper, 2016.

[34] L. Lamport. Time, clocks, and the ordering of eventsin a distributed system. Communications of the ACM,21(7):558–565, 1978.

[35] L. Lamport et al. Paxos made simple. ACM SigactNews, 32(4):18–25, 2001.

[36] S. D. Lerner. Dagcoin: a cryptocurrency withoutblocks, 2015.

[37] S. D. Lerner. Rootstock: Bitcoin powered smartcontracts, 2015.

[38] Y. Lewenberg, Y. Sompolinsky, and A. Zohar.Inclusive block chain protocols. In InternationalConference on Financial Cryptography and DataSecurity, pages 528–547. Springer, 2015.

[39] A. Miller, I. Bentov, R. Kumaresan, C. Cordi, and

1397

P. McCorry. Sprites and state channels: Paymentnetworks that go faster than lightning. arXiv preprintarXiv:1702.05812, 2017.

[40] S. Nakamoto. Bitcoin: A peer-to-peer electronic cashsystem. 2008.

[41] D. Ongaro and J. K. Ousterhout. In search of anunderstandable consensus algorithm. In USENIXAnnual Technical Conference, pages 305–319, 2014.

[42] J. Poon and V. Buterin. Plasma: Scalable autonomoussmart contracts. White paper, 2017.

[43] J. Poon and T. Dryja. The bitcoin lightning network:Scalable off-chain instant payments. 2016.

[44] S. Popov. The tangle. cit. on, page 131, 2016.[45] Y. Sompolinsky, Y. Lewenberg, and A. Zohar.

Spectre: A fast and scalable cryptocurrency protocol.IACR Cryptology ePrint Archive, 2016:1159, 2016.

[46] Y. Sompolinsky and A. Zohar. Accelerating bitcoin’stransaction processing. Fast Money Grows on Trees,Not Chains, 2013.

[47] Y. Sompolinsky and A. Zohar. Phantom: A scalableblockdag protocol, 2018.

[48] R. Taft, E. Mansour, M. Serafini, J. Duggan, A. J.

Elmore, A. Aboulnaga, A. Pavlo, and M. Stonebraker.E-store: Fine-grained elastic partitioning fordistributed transaction processing systems. PVLDB,8(3):245–256, 2014.

[49] S. Thomas and E. Schwartz. A protocol for interledgerpayments. 2015.

[50] A. Thomson, T. Diamond, S.-C. Weng, K. Ren,P. Shao, and D. J. Abadi. Calvin: fast distributedtransactions for partitioned database systems. InProceedings of the ACM SIGMOD InternationalConference on Management of Data, pages 1–12.ACM, 2012.

[51] I. Weber, X. Xu, R. Riveret, G. Governatori,A. Ponomarev, and J. Mendling. Untrusted businessprocess monitoring and execution using blockchain. InInternational Conference on Business ProcessManagement, pages 329–347. Springer, 2016.

[52] G. Wood. Polkadot: Vision for a heterogeneousmulti-chain framework. White Paper, 2016.

[53] A. Zamyatin, D. Harz, J. Lind, P. Panayiotou,A. Gervais, and W. J. Knottenbelt. Xclaim: Aframework for blockchain interoperability. 2019.

1398