«

Local Dev and CI with Datomic Local

With Datomic local you can develop and test applications with minimal connectivity and setup. Get the datomic local library, add it to your classpath, and you have full access to the Client API. This allows you to:

Datomic Local is available at no cost and is Apache Licensed.

This document includes everything you need to know to use Datomic Local:

Setup

Datomic local is available on maven. You can include it in project deps with:

com.datomic/local    {:mvn/version "1.0.277"}

Configure Local Storage

By default, datomic stores all databases under a common storage directory. To specify this directory, create a .datomic/local.edn file in your home directory, containing a map with :storage-dir and an absolute path:

{:storage-dir "/full/path/to/where/you/want/data"}

Using Datomic Local

There are two steps to use Datomic Local: add the Datomic Local library to your classpath and create a local client.

To add Datomic Local to your classpath, add a com.datomic/local entry to your deps.edn file.

{com.datomic/local
 {:mvn/version "1.0.277"}}

The Datomic Local dependency is all you need for using Datomic Local.

There are two ways to get a local client:

  • you can explicitly request a local client
  • you can divert requests for Datomic Cloud clients to use local storage for development and testing

To explicitly request a local client, pass a map to d/client with

  • :server-type :datomic-local
  • a :system name,
(require '[datomic.client.api :as d])
(def client (d/client {:server-type :datomic-local
                       :system "dev"}))

If you are using Datomic Local to develop and test a Datomic Cloud application, add the client-cloud dependency to your project. Then, to divert an existing Datomic Cloud system to Datomic Local, call divert-system:

(require '[datomic.local :as dl])
(dl/divert-system {:system "production"})

;; existing requests for Cloud system will be served locally!
(def client (d/client {:system "production"
                       :server-type :ion
                       :region "us-east-1"
                       :endpoint "https://ljfrt3pr18.execute-api.us-east-1.amazonaws.com"}))

You can also use import-cloud to import data from Datomic Cloud to local storage.

If you are new to Datomic, you can now work through the tutorial.

Sample Data

datomic-samples is a set of databases that are used throughout these Datomic docs and the Day of Datomic tutorials. To install the datomic samples on your local computer:

You can connect and use them by setting your Datomic Local system name to "datomic-samples":

(require '[datomic.client.api :as d])
(def client (d/client {:server-type :datomic-local
                       :system "datomic-samples"}))
(d/list-databases client {})
=> ["mbrainz-subset" "solar-system" "social-news" "movies" ...]

Durability

Datomic Local stores data to your local filesystem, in directories under the :system you specify when creating a Datomic Local client.

Each database will store transactions in a directory named <storage-dir>/<system-name>/<database-name>. You can "backup" or "restore" a Datomic Local database simply by copying the database directory.

Memdb

Sometimes durable storage is unnecessary and/or inconvenient. For example, a CI system may have no need of data after the tests run and no access to a file system.

You can force Datomic Local to use a memory-only database by passing :storage-dir :mem to the map you pass when creating a client, as shown in the example below:

(def client (d/client {:server-type :datomic-local
                       :storage-dir :mem
                       :system "ci"}))

API

Most usage of Datomic Local should be via portable Client API calls. Capabilities that are specific to Datomic Local are available through the datomic.local namespace:

All Datomic Local API calls take a single arg-map argument.

(import-cloud arg-map)
(divert-system arg-map)
(release-db arg-map)

divert-system

Diverts subsequent d/client calls for system to local storage. arg-map has the following keys:

Key Value Required
:system the system to divert yes
:storage-dir overrides :storage-dir in ~/.datomic/local.edn  

release-db

Closes the connection to a database and releases the memory used by it. Values of that database can no longer be used. arg-map has the following keys:

Key Value Required
:system system name yes
:db-name database name yes
:storage-dir overrides :storage-dir in ~/.datomic/local.edn  

import-cloud

Import a Datomic Cloud database into a durable Datomic Local database. arg-map has the following keys:

Key Value Required
:source arg map for (Cloud) d/client merged with arg map for d/connect yes
:dest arg map for (Datomic Local) d/client merged with arg map for d/connect yes
:filter filter spec limits datoms to import  

A filter spec has the following keys, all optional:

Key Value
:before t - include only txes before t, exclusive
:since t - include only txes after t, inclusive
:include-attrs map of attr -> filter
:exclude-attrs vector of attrs to be excluded

:include-attrs keys are either fully-qualified attribute names or a namespaced keyword with * as a name (includes all attrs in namespace, e.g. :order/*).

:include-attrs filters are maps with optional :before and :since keys limiting the time range for attributes as per before/since above.

Schema attributes are always included. When a filter spec is present:

  • All other attributes must be included explicitly.
  • Excludes take precedence over includes.

You can re-import to get more recent data if you have not transacted locally. You cannot import to a database that currently has an open connection.

The following example imports a customer database, including

  • all schema
  • all history of customers (except their secrets!)
  • orders since May 2020
(dl/import-cloud
 {:source {:system "customers"
           :db-name "customers"
           :server-type :ion
           :region "us-east-1"
           :endpoint "https://ljfrt3pr18.execute-api.us-east-1.amazonaws.com"}
  :dest {:system "production-imports"
         :server-type :datomic-local
         :db-name "customers-and-recent-orders"}
  :filter {:include-attrs
           {:customer/* {}
            :order/* {:since #inst \"2020-05\"}}
           :exclude-attrs [:customer/secret]}})

Limits

  • Datomic Local is in-process with your application code, and has all the tradeoffs (vs. a server or cluster) that this implies.
  • Datomic Local requires 32 bytes of JVM heap per datom. You should plan your application with this in mind, while also leaving memory for your application's use.
  • Datomic Local relies on OS page caching for performance, so leave some RAM available (i.e. not allocated to JVM heap) for this.
  • Datomic Local limits the total number of datoms in a transaction to 105.
  • Datomic Local limits strings to 4096 characters.
  • Datomic Local uses shared FileChannels. This is very efficient, but is intolerant of Thread.interrupt. If you interrupt Datomic Local I/O, subsequent I/O may trigger a ClosedChannelException. Try to avoid using interrupt as a control mechanism. If you cannot, you can resume by calling release-db and reacquiring a connection.

Datomic Local is best suited for small, single process applications. For larger projects, create a Datomic Cloud system.

Change Log

2024/02/12 - 1.0.277

  • Fix: Correctly deserialize URIs to java.net.URI

2023/12/20 - 1.0.276

  • Fix: Regression introduced in 1.0.267 that could result in a NoClassDefFoundError Exception when an exception is encountered in a query.
  • Fix: Regression introduced in 1.0.267 that could cause a database with blank keyword idents to fail to load.
  • Upgrade: Datomic Local now requires an LTS version of Java 11 or greater.
  • Upgraded core.async to 1.6.681
  • Upgraded commons-codec to 1.16.0
  • Upgraded http-client to 1.0.126
  • Upgraded fressian to 0.6.8

2023/08/16 - 1.0.267

  • Changed name to `local` and released under Apache 2.0
  • `:server-type` updated to `datomic-local`

2022/04/06 - 1.0.243

  • Upgrade Client to 1.0.126

2022/01/10 - 1.0.242

  • Upgrade Client to 1.0.125

2021/09/27 - 1.0.238

  • Enhancement: The datalog engine will now do self-unification within a single clause.

2021/07/13 - 0.9.235

  • Use latest Client
  • Fix anomaly where Attr-spec's for boolean attributes with false values failed

2021/01/20 - 0.9.232

  • Improvement: enable BigInt fressian handler

2020/11/23 - 0.9.229

  • New: Change the scale of a BigDecimal attribute in a transaction.
  • Improvement: Better error messages for import-cloud.
  • Fix: Query correctly treats range functions as functions (not as predicates).

2020/10/21 - 0.9.225

  • New feature Memdb
  • Improvement: Increase the limit on the total number of datoms in a transaction imported with import-cloud to 16 million.
  • Improvement: Increase the limit on the length of strings imported with import-cloud to 1 million characters

2020/09/25 - 0.9.203

  • Provide dev-tools via Cognitect maven repo

2020/07/24 - 0.9.184

  • Improvement: better error message when calling a function that is not in the :allow list of datomic/ion-config.edn

2020/07/21 - 0.9.183

  • Update: use latest Client API

2020/07/17 - 0.9.180

  • Enhancement: Reread the deps-local.edn config file when creating a client.
  • Enhancement: Provide better feedback while loading and importing databases.
  • Enhancement: Update client documentation about connecting to dev-local systems

2020/07/10 - 0.9.172

Initial Release