Transactions

All writes to Datomic databases are protected by ACID transactions. Transactions are submitted to the system's Transactor component, which processes them serially and reflects changes out to all connected peers. This document describes transactions in detail.

Topics covered in this page include:

Transaction structure

Datomic represents transaction requests as data structures. This is a significant difference from SQL database, where requests are submitted as strings. Using data instead of strings makes it easier to build requests programmatically.

A transaction is simply a list of lists and/or maps, each of which is a statement in the transaction.

List forms

Each list a transaction contains represents either the addition or retraction of a specific fact about an entity, attribute, and value; or the invocation of a data function, as shown below.

[:db/add entity-id attribute value]

[:db/retract entity-id attribute value]
[data-fn args*]

Map forms

Each map a transaction contains is equivalent to a set of one or more :db/add operations. The map may include a :db/id key identifying the entity data that the map refers to. It may additionally include any number of attribute, value pairs.

{:db/id entity-id
 attribute value
 attribute value
 ... }

Internally, the map structure gets transformed to the list structure. Each attribute/value pair becomes a :db/add list. If you did not provide an entity-id, Datomic will generate a temporary entity-id for you.

[:db/add entity-id attribute value]
[:db/add entity-id attribute value]
...

The map structure is supported as a convenience when adding data. As a further convenience, the attribute keys in the map may be either keywords or strings.

Identifying entities

There are three ways to sepcify an entity id:

  • a temporary id for a new entity being added to the database
  • an existing id for an entity that's already in the database
  • an identifier for an entity that's already in the database

They are described in the sections below.

Temporary ids

When you are adding data to a new entity, you identify it using a temporary id. Temporary ids get resolved to actual entity ids when a transaction is processed.

Creating tempids

The simplest kind of temporary id is a string that follows these rules:

  • a temporary id cannot begin with a colon (:) character
  • the temporary id "datomic.tx" always identifies the current transaction
  • other strings beginning with "datomic" are reserved for future use by Datomic

In the transaction data below, the temporary id "jdoe" is used to indicate that the two datoms are both about the same entity:

[[:db/add "jdoe" :person/first "Jan"]
 [:db/add "jdoe" :person/last "Doe"]]

Temporary id resolution

When a transaction containing temporary ids is processed, each unique temporary id is mapped to an actual entity id. If a given temporary id is used more than once in a given transaction, all instances are mapped to the same actual entity id.

In general, unique temporary ids are mapped to new entity ids. However, there is one exception. When you add a fact about a new entity with a temporary id, and one of the attributes you specify is defined as :db/unique :db.unique/identity, the system will map your temporary id to an existing entity if one exists with the same attribute and value (update) or will make a new entity if one does not exist (insert). All further adds in the transaction that apply to that same temporary id are applied to the "upserted" entity.

Existing entity ids

To add, modify or retract data about existing entities in a transaction, you must know the entity id. You can retrieve a specific entity id by querying the database for an external key (see Schema for information about external keys).

For example, this query retrieves the id of an existing entity based on an email address.

[:find ?e :in $ ?email :where [?e :person/email ?email]]

If the entity id returned by the query is 17592186046416, the following transaction data will set the entity's customer status:

{:db/id 17592186046416
 :customer/status :active}

If the entity in question has a unique identifier, you can specify the entity id by using a lookup ref. Rather than querying the database, you can provide the unique attribute, value pair corresponding to the entity you want to assert or retract a fact for. Note that a lookup ref specified in a transaction will be resolved by the transactor.

{:db/id [:customer/email "joe@example.com"]
 :customer/status :active}

Entity identifiers

The system defines a special attribute, :db/ident, that can be used to assign an keyword identifier to a given entity. If an entity has a :db/ident attribute, its value can be used in place of the entity's id.

This mechanism is what allows you to refer to attributes, partitions, types, etc., by specifying keywords.

You can also use :db/ident to define entities representing enumerated values that can then be referred to by name (as described in Schema).

In the example below, the entity is specified by the ident :person/name:

[:db/add :person/name :db/doc "A person's full name"]

Building transactions

This section explains how to build transactions to add, modify and retract facts. Each example shows a single transaction, but you can combine adding and retracting in a single transaction if desired.

Adding data to a new entity

To add data to a new entity, build a transaction using :db/add implicitly with the map structure (or explicitly with the list structure), a temporary id, and the attributes and values being added.

This example builds a transaction that creates a new entity with two attributes, :person/name and :person/email.

[{:person/name "Bob"
  :person/email "bob@example.com"}]

This code constructs the same transaction programmatically.

tx = Util.list(Util.map(":person/name", "Bob", ":person/email", "bob@example.com"));

Note that there is no requirement about which attributes are added to which entities, this is left entirely up to your application. This provides a great deal of flexibility as your system evolves.

Adding data to an existing entity

To add data to an existing entity, build a transaction using :db/add implicitly with the map structure (or explicitly with the list structure), an existing entity id or entity identifier, and the attributes and values being added.

This example queries for an existing entity id and uses it in a new transaction to change the value of the entity's :person/name attribute to "Robert".

bob_id = Peer.query(
  "[:find ?e . :in $ ?email :where [?e :person/email ?email]]",
  conn.db(),
  "bob@example.com");

tx = Util.list(Util.map(":db/id", bob_id, ":person/name", "Robert"));

The following example uses a lookup ref to perform the same transaction.

tx = Util.list(Util.map(":db/id", Util.list(":person/email", "bob@example.com"),
                        ":person/name", "Robert"));

Adding entity references

Attributes of reference type allow entities to refer to other entities. When a transaction adds an attribute of reference type to an entity, it must specify an entity id as the attribute's value. The entity id specified for the value may be a temporary id (if the entity being referred to is being created by the same transaction) or a real entity id (if the entity being referred to already exists in the database).

This example shows the literal representation of a transaction that creates two new entities, people named "Bob" and "Alice". Each entity has a reference to the other, connected using temporary ids, "bobid" and "aliceid", respectively. All instances of a given temporary id within a transaction will resolve to a single entity id.

[{:db/id "bobid"
  :person/name "Bob"
  :person/spouse "aliceid"}
 {:db/id "aliceid"
  :person/name "Alice"
  :person/spouse "bobid"}]

This example shows constructing the same transaction from code.

tx = Util.List(
       Util.map(":db/id", "bobid",
                ":person/name", "Bob",
                ":person/spouse", "aliceid"),
       Util.map(":db/id", "aliceid",
                ":person/name", "Alice",
                ":person/spouse", "bobid"));

Cardinality many transactions

You can transact multiple values for a :db.cardinality/many attribute at one time using a list. The following example transacts a person named "Bob" with multiple aliases:

[{:db/id #db/id[:db.part/user]
  :person/name "Bob"
  :person/email "bob@example.com"
  :person/aliases ["Robert" "Bert" "Bobby" "Curly"]}]

Nested maps in transactions

Often, a group of related entities are created or modified in the same transaction. Nested maps allow you to specify these related entities together in a single map. If an entity map contains a nested map as a value for a reference attribute, Datomic will expand the nested map into its own entity. Nested map expansion is governed by two rules:

  • If the nested map does not include a :db/id, Datomic will assign a :db/id automatically, using the same partition as the :db/id of the outer entity.
  • Either the reference to the nested map must be a component attribute, or the nested map must include a unique attribute. This constraint prevents the accidental creation of easily-orphaned entities that have no identity or relation to other entities.

As an example, the following data uses nested maps to specify two line items belonging to an order:

[{:db/id order-id
  :order/lineItems [{:lineItem/product chocolate
                     :lineItem/quantity 1}
                    {:lineItem/product whisky
                     :lineItem/quantity 2}]}]

Notice that

  • The two line items do not need to specify a :db/id, and will automatically get ids in the same partition as the order-id entity.
  • Since the line items do not have unique ids, you can infer that :order/lineItems must be a component attribute. (This is sensible given the domain. Line items have no independent existence outside orders.)

Nested maps are often much more convenient than their equivalent flat expansions. The data below shows the same information order and line items as three independent entity maps.

[{:db/id order-id
  :order/lineItems [item-1-id, item-2-id]}
 {:db/id item-1-id
  :lineItem/product chocolate
  :lineItem/quantity 1}
 {:db/id item-2-id
  :lineItem/product whisky
  :lineItem/quantity 2}]

In addition to being more verbose, this form requires additional work to explicitly manage item-1-id and item-2-id and their connections to the containing order.

Retracting data

To retract data from an existing entity, build a transaction using :db/retract, an existing entity id or entity identifier, and the attribute and value being retracted.

This example queries for an existing entity id and uses it in a new transaction to retract the value of the entity's :person/name attribute.

bob_id = Peer.query(
  "[:find ?e . :in $ ?email :where [?e :person/email ?email]]",
  conn.db(),
  "bob@example.com");

tx = Util.list(Util.list(":db/retract", bob_id, ":person/name", "Robert"));

The following example accomplishes the same thing with a lookup ref.

tx = Util.list(Util.list(":db/retract", Util.list(":person/email" "bob@example.com"),
                         ":person/name", "Robert"));

Datomic keeps the values of data over time and allows you to query the value of the database as of a point in time. That means it's possible to recover data even after it has been retracted, simply by querying a database value from the past.

Built-in Transaction Functions

In addition to using the built-in transactions below, you can also write your own transaction functions.

:db.fn/retractEntity

The :db.fn/retractEntity function takes an entity id as an argument. It retracts all the attribute values where the given id is either the entity or value, effectively retracting the entity's own data and any references to the entity as well. Entities that are components of the given entity are also recursively retracted.

The following example transaction data retracts two entities, specifying one of the entities by id, and the other by a lookup ref.

[[:db.fn/retractEntity id-of-jane]
 [:db.fn/retractEntity [:person/email "jdoe@example.com"]]]

:db.fn/cas

The :db.fn/cas (compare-and-swap) function takes four arguments: an entity id, an attribute, an old value, and a new value. The attribute must be :db.cardinality/one. If the entity has the old value for attribute, then the new value will be asserted. Otherwise, the transaction will abort and throw an exception.

You can use nil for the old value to specify that the new value should be asserted only if no value currently exists.

The following example transaction data sets entity 42's :account/balance to 110, if and only if :account/balance is 100 at the time the transaction executes:

[[:db.fn/cas 42 :account/balance 100 110]]

Processing transactions

After a transaction data structure is built, you must submit it to the transactor for processing. The transactor queues transactions and processes them serially. Since the transactor is not processing queries (they are handled in the peers), no locking is required and the processing is very fast.

Submitting transactions

Once a transaction data structure is built, you submit it to the transactor by calling the datomic.Connection.transact or transactAsync method. Both methods return a Future<Map>. Its get method returns a map with information about transaction if the transaction commits and throws an exception if the transaction aborts. In Clojure, you can also use the deref method or @ to get a transaction's result.

txResult = conn.transact(tx).get();

future = conn.transactAsync(tx);

... // do other work

txResult = future.get();

Transaction Timeouts

The transact method is synchronous, it waits until the transaction completes before returning. The transactAsync method is asynchronous, it submits the transaction to the transactor and returns immediately. A call to get on the Future<Map> returned from transactAsync will block until the transaction completes. Application code should use the overload for get that takes a timeout value expressed as a long and a java.util.concurrent.TimeUnit in order to ensure that the calling thread never blocks indefinitely.

When a transaction times out, the peer does not know whether the transaction succeeded, and will need to query a recent value of the database to discover what happened.

Monitoring transactions

Peers can monitor all transactions being processed by the system's transactor. The datomic.Connection.txReportQueue method provides a method to get a queue of transaction notifications.

The queue delivers a report for every transaction submitted while a peer is connected to the database, even those submitted by other peers. The reports are instances of the datomic.TxReport type, which provides access to the value of the database before and after the transaction was applied, and the set of facts added by the transaction.

Once your code gets the transaction report queue, reports will be added to it. It is the responsibility of your peer code to empty the queue. When you are done monitoring the queue, you must remove it by calling the datomic.Connection/removeTxReportQueue method so it doesn't continue to consume memory.

This example shows how to connect to the notification queue, and retrieve a transaction report from it.

queue = conn.txReportQueue();

report = (Map) queue.poll();

The report map contains the following keys, which are static members of the datomic.Connection class:

keyusage
DB_BEFOREdatabase value before the transaction
DB_AFTERdatabase value after the transaction
TX_DATAdatoms produced by the transaction
TEMPIDSused to resolve temporary ids

The TX_DATA member of a transaction report contains the set of datoms created by a transaction (both assertions and retractions). The TX_DATA can be used as an input source for a query. The query below uses the transaction data and the database value after the transaction was applied to show each datom of the transaction.

[:find ?e ?aname ?v ?added
 :in $ [[?e ?a ?v _ ?added]]
 :where
 [?e ?a ?v _ ?added]
 [?a :db/ident ?aname]]

The query expects the DB_AFTER and TX_DATA values of a transaction report as its two input sources, in that order.

See Query for more information on using sets of tuples as input sources in queries.

Reified transactions

When the transactor processes a transaction, it creates a transaction entity to represent it. By default, this entity has one attribute, :db/txInstant, whose value is the instant that the transaction was processed. In addition, every datom in a transaction refers to the transaction entity through the :tx field of datom.

You can add additional attributes to a transaction entity to capture other useful information, such as the purpose of the transaction, the application that executed it, the provenance of the data it added, or the user who caused it to execute, or any other information that might be useful for auditing purposes.

To annotate the current transaction, simply include an add statement (either map or list) that uses a temporary id from the :db.part/tx partition. The transactor will resolve a temporary id from that partition to the actual id of the transaction entity it creates.

Here is an example that annotates a transaction using a :data/src attribute to indicate where the data came from.

tx = Util.list(
       Util.map(":db/id", "datomic.tx",
                ":data/src", "catalog-2_29_2012.xml"),
       Util.map(":product/name", "Marbles"));

txResult = conn.transact(tx).get(); 

You can query for transaction entities with particular attributes and values the same way you would query for any other entity in the system (see Query).

Explicit :db/txInstant

You can set :db/txInstant explicitly, overriding the transactor's clock time. When you do, you must choose a :db/txInstant value that is not older than any existing transaction, and not newer than the transactor's clock time. This capability enables initial imports of existing data that has its own timestamps.

Redundancy Elimination

A datom is redundant with the current value of the database if there is a matching datom that differs only by transaction id. If a transaction would produce redundant datoms, those datoms are filtered out, and do not appear a second time in either the indexes or the transaction log.

Transactions are never entirely redundant, because the transaction's :db/txInstant is always a new fact. Also, the schema attributes :db.install/attribute and :db.alter/attribute trigger side effects, and are never considered redundant.

The Tempid Data Structure

As an alternative to string tempids, peers (but not clients) can choose to make a structural tempid, allowing them to explicitly specify a partition for the entity.

You can make a temporary id by calling the datomic.Peer.tempid method. The first argument to Peer.tempid is the name of the partition where the new entity will reside as an argument. There are three partitions built into Datomic.

PartitionPurpose
:db.part/dbSchema partition, used only for attributes and partition entities
:db.part/txTransaction partition, used only for transaction entities
:db.part/userUser partition, for application entities

The :db.part/db partition should only be used for schema entities, like attributes and partitions, not for application data.

The :db.part/tx partition should only be used for transaction entities, which are created automatically for each committed transaction.

You should use :db.part/user for your application's entities, or you should create one or more partitions of your own, as described in Schemas.

This code gets a new temporary id in the :db.part/user partition.

temp_id = Peer.tempid(":db.part/user");

Each call to tempid produces a unique temporary id.

There is an overloaded version of tempid that takes a negative number as a second argument. This version of tempid creates a temporary id based on the number you pass as input. If you invoke it multiple times with the same partition and negative number, each invocation will return the same temporary id. This can be useful when constructing transactions that add references between entities, as explained below.

In some cases, you may want to store the literal representation of a transaction in a file. The literal form can be read with datomic.Util.read and submitted as a transaction. This is a useful way to store a schema definition, for example.

You can insert temporary ids into the literal representation of a transaction using the following syntax:

#db/id[partition-name value*]

where partition-name is the name of a partition in the system and the value is an optional negative number.

When the literal representation of a transaction is parsed, this syntax is interpreted and a temporary id is generated.

Default Partition

String tempids create entities in the default-partition for the transactor process. You can set the default partition by editing the transactor properties file:

default-partition=:my.namespace/my-partition

If no default partition is specified, the transactor will use the built-in :db.part/user partition.