Collections

Collections is a cosmos-sdk package that allows developers to manage their module’s state in an easy and efficient way.

Motivation

In the ever-evolving world of blockchain development, module developers using the Cosmos SDK frequently grapple with the intricate task of manually implementing storage functionalities in their modules. These responsibilities encompass a wide range of tasks, such as defining key-to-byte and value-to-byte formats, creating secondary indexes, devising query methods for external access, establishing local methods for storage writing, and handling genesis imports and exports. Additionally, they must also ensure thorough testing of all these components.

Being entrenched in these cumbersome processes not only prevents developers from concentrating on the most crucial aspect of their work—writing efficient business logic—but also presents a myriad of problems. For instance, developers often struggle with the complexity of key-to-byte formats, which can be error-prone and require careful consideration, for example when formatting time-to-byte or when avoiding namespace collisions in secondary indexes.

The lack of standardization further complicates matters for clients, particularly when it comes to obtaining proofs for objects present in the state of blockchain applications built on the Cosmos SDK. As a result, clients are compelled to maintain extensive lists of object paths to gather these essential proofs, which can become increasingly challenging as the ecosystem grows.

Given the context of the Cosmos SDK and the critical role it plays in blockchain development, the motivation for addressing these issues is evident. By streamlining and simplifying storage functionalities, we can empower developers to focus on what truly matters—developing robust business logic for their blockchain applications—while simultaneously enhancing the overall user experience for clients. Furthermore, by fostering a more standardized approach to state management, we can significantly reduce the complexity associated with proofs, and state access in general, ultimately leading to a more efficient and effective blockchain ecosystem.

History

Looking back at the history of Collections, it’s worth noting that its initial development took place in Nibiru. The driving force behind its creation was the discovery of a bug in the funding calculation process. This bug turned out to be a result of the way funding state was being iterated; specifically, when attempting to retrieve all the funding associated with the “BTC-USD” pair, the implemented iteration function inadvertently returned funding information for other pairs as well, due to its flawed implementation.

But the issues didn’t stop there; another bug was discovered, this time in relation to the use of reverse iterators. This problem can be traced back to inadequate documentation and the general complexities surrounding the handling of elaborate keys, such as DB joint keys, in a raw key-value storage system. It became apparent that addressing these issues required a comprehensive solution, which ultimately led to the development of Collections.

The history of Collections serves as a testament to the importance of creating robust and efficient solutions to overcome the challenges presented by complex key-value state layouts.

Tracing the history of Collections further, we can see its journey from Nibiru to the Cosmos SDK. After identifying the issues and developing Collections to address them, the solution was proposed for integration into the Cosmos SDK through an ADR (Architecture Decision Record). This proposal aimed to bring the improvements from Collections to a broader audience and benefit the entire Cosmos SDK ecosystem.

Recently, the implementation of Collections in the Cosmos SDK was finalized, albeit in an unstable form. As the integration process continues, we are working to refine and stabilize the implementation, ensuring that the valuable lessons learned from Collections’ history contribute to the ongoing evolution and enhancement.

Why ORM didn’t settle it

ORM offers a wealth of functionality designed to address the specific issues discussed earlier. However, it is not without its drawbacks. One major concern is the necessity for migrations, which can be a significant problem as they require code updates, compel nodes of a blockchain to undergo updates, and may cause downtime as state is migrated. This can have substantial implications for the performance and stability of a blockchain. Furthermore, it relies on the newest protobuf Golang API, while the SDK still predominantly uses gogoproto. As a result, integrating this solution into a module forces developers to juggle two different Golang frameworks representing the same API objects. Additionally, it has a steep learning curve and does not support all types used in the state, as it only allows the use of protobuf types.

CosmWasm

The Collections API draws inspiration from cw-storage-plus, which has proven to be an effective tool for handling storage in CosmWasm contracts. It boasts simplicity, eliminates the need for extra tooling, and effortlessly accommodates complex storage structures such as indexes and snapshots. Furthermore, its API is straightforward and explicit, making it a more accessible option for developers.

Architecture

Collections leverages Golang generics to transform raw bytes key-value (KV) storage into a typed one. This enhancement in type safety simplifies data handling while also allowing for greater flexibility. The abstractions in Collections do not tie you to any specific form of encoding, as it can work with raw Golang types just as seamlessly as with protobuf.

Furthermore, you have the freedom to encode your protobuf types using different encoding formats, provided that there’s a collection.ValueEncoder capable of handling the encoding and decoding of the type to and from bytes. This flexibility in encoding allows you to migrate your code to use Collections without the need for state migrations, minimizing the impact on your existing blockchain infrastructure.

Collections also features built-in support for composite keys, streamlining the iteration process for them, such as prefixing over only the first part of the key. This makes iterations more straightforward and provides a more expressive and user-friendly iteration API compared to raw key-value storage.

Lastly, the IndexedMap type, which is part of Collections, enables developers to build complex state layouts with indexes. This added capability allows for more efficient management and querying of data within the blockchain ecosystem.

Docs

If this article got you interested, you can find the full documentation here.