CockroachDB Launches Advanced Key-Value Store Pebble
Pebble is a key-value store developed by CockroachDB that is backed on LevelDB/RocksDB with special focus on performance and internal usage. The file formats from RocksDB are carried over to Pebble, along with a few extensions, including range deletion tombstones, table-level bloom filters, and MANIFEST format updates. Pebble particularly targets the use case and feature set required by CockroachDB and does not incorporate every functionality in RocksDB. RocksDB features that are included in Pebble: Block-based tables Checkpoints Indexed batches Iterator options (lower/upper bound, table filter) Level-based compaction Manual compaction Merge operator Prefix bloom filters Prefix iteration Range deletion tombstones Reverse iteration SSTable ingestion Single delete Snapshots Table-level bloom filters RocksDB features that are not included in Pebble: Backups Column families Delete files in range FIFO compaction style Forward iterator/tailing iterator Hash table format Memtable bloom filter Persistent cache Pin iterator key/value Plain table format SSTable ingest-behind Sub-compactions Transactions Universal compaction style But Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature that Pebble doesn’t support. In CockroachDB v20.1 (published in May 2020), Pebble was introduced as an alternate storage engine to RocksDB and was utilised in production then. In CockroachDB v20.2, Pebble became the default storage engine (released November 2020). Users of CockroachDB are using Pebble in production on a large scale. Advantages of Pebble Faster reversal iteration using the skiplist of the memtable’s backward links. Better concurrency is achieved using a quicker commit pipeline. Indexed batch iterations that are seamlessly combined. The batch’s mutations theoretically occupy a different memtable level. Smaller, easier-to-understand code base. RocksDB Compatibility Pebble aims for RocksDB 6.2.1 forward compatibility (the latest version of RocksDB used by CockroachDB). Forward compatibility allows Pebble to use a database created by RocksDB. When employing its FormatMostCompatible format, Pebble offers bidirectional compatibility with RocksDB (a DB generated by Pebble may be used by RocksDB). New format major versions are gated behind new functionality that is incompatible with earlier versions. In general, Pebble only offers compatibility with the portion of features and settings that CockroachDB deploys. RocksDB functionality and settings cover too much ground to test and document all the incompatibilities thoroughly. Pebble is based on the incomplete Go version of LevelDB.
Pebble has faster reversal iteration using the skiplist of the memtable’s backward links and better concurrency with a quicker commit pipelin
Pebble has faster reversal iteration using the skiplist of the memtable's backward links and better concurrency with a quicker commit pipelin
["AI (Artificial Intelligence)", "Data Science", "Deep Learning", "Google", "Machine Learning"]
["Go", "programming_languages:R", "AI", "ML", "Machine Learning", "programming_languages:Go", "RAG", "CuPy", "Aim", "Google", "Deep Learning", "Data Science", "R", "AI (Artificial Intelligence)"]
["AI", "ML", "Aim", "CuPy", "RAG", "R", "Go", "programming_languages:R", "programming_languages:Go"]
Shritama (she/her) is a technology journalist at AIM who is passionate to explore generative AI with a special focus on big techs, database, healthcare, DE&amp;I, hiring in tech and more.