Introduction

The Datomic information model has the following benefits:

• Trivial read scaling: because databases are immutable, any number of processes can have copies of the data to serve read loads. Readers do not need to wait for or coordinate with any other process.
• The database within your application: application processes have their own copy of the query engine and database in local memory. This enables low-latency data access and supports access patterns that would be impractical with server roundtrips.
• Auditability: the total ordering of transactions and the fact that nothing is ever changed or removed means that you know and can demonstrate every step in the history of your data. The fact that transactions are themselves entities also offers an ideal place to store domain-specific provenance metadata.
• History: since every datom encodes its time "t", databases can be filtered to include only data as of any specific point in the past. Existing queries can be used against such “as-of” databases without modification.
• Powerful query: the universal relation of datoms is well suited for datalog+recursion, which has expressive power equivalent to the relational algebra behind SQL.
• Impedance-free data modeling: if you think of your domain entities as associative collections of facts, then your logical model translates directly to Datomic’s information model. This stands in contrast to solutions like the join tables required in SQL.
• Flexibility: any entity can possess any attribute, which is a great fit for real-world data where exceptions are the rule. Unlike traditional databases, there is no need for NULL - datoms either exist or they don’t.
• Efficiency across multiple read access patterns: Datomic data is indexed in many ways, including:
  • By entity for “row” access (EAVT)
  • By attribute and value for key/value lookup (AVET)
  • By attribute for “column” access (AEVT)
  • By (entity id) value for hierarchical or graph navigation (VAET)

There’s no such thing as a free lunch. Datomic’s information model has the following tradeoffs:

• No per-db write scaling: since transactions are totally serialized, only one transaction can occur at a time in a given database.
• Data constraint: not suitable for high churn, non transactional data like observability telemetry or logs.
• No structural “types”: Datomic’s universal schema means that all entities have the same universal type. While this is very flexible, it provides no declarative mechanism (e.g. tables) for structural requirements such as “a person must have a name”. Where needed, such structural requirements are enforced by transaction-time specifications.

In addition to these inherent tradeoffs, Datomic's current implementations do not have value types suitable for storing large documents, images, audio, or video. It is common practice in Datomic to store such data in a key/value store such as S3 and then store pointers to that data in Datomic. Datomic’s combination of benefits and tradeoffs makes it well suited for data of record in various domains including finance, law, the sciences, manufacturing, retail, wholesale, healthcare, and supply chain logistics. It is also well-suited for configuration data and local application state.

Datomic is designed and implemented by the creators and maintainers of the Clojure programming language, and the two share design sensibilities. Having familiarity with Clojure is a great help in understanding the Datomic architecture, in particular:

• Datomic, like Clojure, encourages functional programming with dynamic typing as a way to create robust and flexible systems.
• A Datomic database value is a persistent data structure, similar to Clojure’s collections.
• A Datomic transaction is a functional, atomic update to a database value, similar to using swap! on a Clojure atom.
• Datomic entities are open, similar to Clojure maps, instead of being based on a fixed set of slots as is typical in object-oriented programming and SQL tables. Please refer to the “just use maps” section (3.4.2) in the history of Clojure paper for further details.
• The Datomic team shares the Clojure team’s commitment to stability. As Datomic evolves, we seek growth instead of breaking change and semantic versioning, as described in Rich Hickey’s Spec-ulation talk (transcript). You should be able to use Datomic in your programs for decades with minimal to no modification.

A Datomic database is logically a set of datoms. To make this structure practical for use, Datomic provides leverage via indexes and query capabilities that use those indexes.

Datomic maintains four different indexes designed to support different access patterns. Each index is implemented as an immutable, persistent tree with a wide branch factor (typically thousands of items). In storage, each node of an index tree is a separate segment, similar to pages in a traditional relational database management system (RDBMS).

Datomic’s query and pull APIs use appropriate indexes transparently. The segments of an index are immutable and can be cached anywhere, for example: inside an application’s process-local object cache, in a memory-backed cache like memcached, or an SSD backed cache such as Valcache. When a query requires a segment, it first checks the object cache. On a miss, it then populates the object cache from cache/storage. This has the following implications:

• Databases can exceed the size of application memory, with most of the database available through tiers of least-recently-used (LRU) caches.
• Application processes can access a portion of the database at memory speed without requiring client/server interaction. If an application process has sufficient memory for its typical working set, then its query performance will be comparable to that of an in-memory database.
• Each distinct process has its own object cache, which automatically caches the working set for that process without requiring user configuration. Consequently, a single system can efficiently serve very different load patterns by simply directing the requests for each load pattern to a different set of instances.

Datomic uses an efficient algorithm to construct indexes, achieving time complexity that is sublinear in the overall size of the database. In addition, all Datomic processes cache the most recent transactions in memory, so that indexes need not be rebuilt continually and can be rebuilt occasionally in the background.

Datomic’s transaction log is an index sorted by time t. Each time t represents a single transaction. There is no ordering within a transaction, which is a set of datoms. Different Datomic editions employ different data structures for the log, all of which are indelible, chronological, transactional, and tangible:

• Indelible: The log accumulates new information and never removes existing information. Where update-in-place databases would delete, Datomic instead adds a new retraction.
• Chronological: The log contains the complete history of the database, in chronological order.
• Transactional: Datomic writes are always ACID transactions, serialized in a total order.
• Tangible: The log in Datomic is not just an implementation detail; it is an integral part of Datomic's information model. You can query the log using the log API.