Zilliqa Technical Update 14 September — Version 5.0.0 is here!

Greetings everyone,

We are happy to announce that our upgrade of the Zilliqa network to Version 5.0.0 is now complete. The upgrade, which has taken roughly three months, will significantly improve smart contracts execution and transaction processing in our blockchain.

Here’s what you need to know about the upgrade:

1. Efficient state accesses for Scilla: Prior to the upgrade, smart contract execution in the blockchain involved invoking the Scilla interpreter with a message and the current state of the contract account as arguments. This contract state, provided as an input JSON file to the interpreter, could grow to be in the order of megabytes, when the contract used maps that had hundreds of thousands of entries. The interpreter, after processing the message would then produce an output state JSON, again the size being in the order of megabytes. These file i/o overheads were independent of the actual transition being invoked (and the code executed). So even if the transition involved just adding one more entry to a Scilla map field, our i/o involved the entire map (with hundreds of thousands of entries).

The upgrade solves this problem of passing around large state JSON files, and instead establishes a communication channel between the core blockchain and the Scilla interpreter, with protocols to fetch and update parts of the state as is necessary. This communication channel, in the form of inter-process-communication (IPC) via UNIX sockets between the two processes (core blockchain and Scilla interpreter) has now enabled us to efficiently execute contracts whose states can grow to be large, in the order of megabytes.

To support efficient state accesses for Scilla and to be able to serve fetch and update queries (of state data) efficiently, our storage schema for the backend database had to be redesigned. We next explain this in detail. We are also planning a dedicated blog article to explain the design details of both the communication protocol as well as the storage schema changes.

2. Contract map storage: To enable efficient access of state data during a smart contract execution, we have now redesigned the storage schema in our backend leveldb database. The central idea behind this is that the storage backend is now aware of map fields in Scilla, including nested maps.

We take advantage of leveldb’s sorted keys (and fast random access to keys) to store Scilla map fields directly as leveldb entries. Previously, a JSON encoding of the entire state (and in particular the map) was stored. This forced us to fetch the entire map from disk, when all that was required was a single key.

While this sounds simple enough, the design gets more intricate because Scilla allows fetching and updating entire maps and submaps (which can be inherently inefficient, but still allowed). Our design to handle this involves encoding Scilla map keys into leveldb keys such that direct accesses are supported efficiently, while accessing entire map values are supported (but are inefficient). The Scilla checker, of course, warns against such inefficient accesses, and the gas charged will be linear in the size of the map in such cases.

3. Scilla checker gas mechanics: In the recent bug bounty program, we were reported a potential DOS attack on the scilla checker. The type checker performs type-substitutions when checking polymorphic Scilla code. The attack demonstrated that these substitutions can grow the size of a term’s inferred type exponentially wrt. the term size resulting in an out-of-memory crash. The vulnerability is inherent to type checking System-F (on which Scilla is based). Here’s a good discussion on the subject: http://lambda-the-ultimate.org/node/5524

Previously, scilla-checker’s cost was statically determined to the size of the contract, but not on the semantics of the contract itself. However, with this vulnerability, we have now integrated gas mechanics into the type-checker so that such attacks are prevented. The details of the costing itself will be updated in the documentation that describes the gas costs for contract execution.

4. DS reputation: One of the long-standing challenges within the DS committee has been how to deal with nodes that, for one reason or another, do not actively participate in the consensus protocol. These Byzantine nodes have a fixed tenure of 18 epochs (over 24 hours) and can jeopardize the resiliency of the network when enough of them make it into the committee. With the introduction of the DS reputation feature, these nodes can be removed much sooner if they don’t meet a minimum performance threshold. This makes DS membership more meritocratic by ensuring that only the higher-performing nodes make up the committee. Credit for this feature goes to our community contributor and wave four grant program awardee Jeremy Heng.

5. Server API updates: Developers can now take advantage of some useful additions to our API library. GetTotalCoinSupply returns the current total Zils in circulation. GetSmartContractSubState allows a client to query a specific variable (or an index in a map) and get the state for it.

We’ve also updated some of our existing APIs. GetTransaction now returns the code and data fields. GetSmartContractState output has been reformatted to enable more efficient access of field variables and map keys by representing these as JSON associative objects instead of JSON arrays. This also makes automatic comparisons of states represented as JSONs simpler. With this change, we no longer specify the types of values in the JSON; they must be obtained and inferred from the output of the Scilla checker.

We will be updating our API documentation (https://apidocs.zilliqa.com) soon to reflect these new changes.

6. Protobuf 3: Since the mainnet launch, Zilliqa’s messaging and data storage format have been based on Google Protocol Buffers (Protobuf) version 2, the version available on the Ubuntu OS compatible with our code. We have finally modified our build infrastructure to integrate Protobuf 3. The move was critical to implementing our Scilla IPC feature, which relies on the map data type that exists only in Protobuf 3. Moreover, this upgrade finally opens up the possibility of leveraging other open-source libraries that run on Protobuf 3, and we’ll gradually introduce these in future improvements to the core.

7. Lookup and seed node resilience: Every core upgrade has included one or more improvements to lookup and seed nodes, and this upgrade is no exception. Lookup nodes play a critical part in the mainnet, from dispatching transactions to archiving incremental data that can be used by other nodes to synchronize with the network. As full nodes, lookups must store all available data relevant to the blockchain.

With the ever-increasing size of the blockchain data that has been mined on the mainnet, it gradually becomes difficult for an offline lookup node to both download all this data and finish its recovery sequence before the mainnet has progressed to a new transaction epoch.

In the past, this would have meant that the lookup remains out-of-sync and would need to perform the recovery sequence again. Now, lookup nodes will be able to dynamically resynchronize with the rest of the network once it has completed the initial recovery operation.

Similarly, Zilliqa and community-hosted seed nodes, which are also full nodes, now have the ability in version 5.0.0 to fetch missing data (namely, microblocks and transaction bodies) from lookup nodes. This provides yet another guarantee of data availability while also making the nodes more autonomous and self-managing.

For further information, connect with us on one of our social channels:

• Discourse Forum
• Telegram
• Twitter
• Reddit
• Github
• Discord