Local Dev and CI with dev-local
With Datomic dev-local you can develop and test applications with minimal connectivity and setup. Get the dev-local library, add it to your classpath, and you have full access to the Client API. This allows you to:
- Develop and test Datomic Cloud applications without connecting to a server and without changing your application code.
- Create small, single-process Datomic applications and libraries.
Dev-local is available at no cost. Note that dev-local is not redistributable. If you make Datomic apps/libs for others to use they must get a copy of dev-local themselves.
This document includes everything you need to know to use dev-local:
- how to get and configure dev-local
- how to add dev-local to your classpath and create a client
- how to create durable and in-memory databases
- API for using dev-local as a test system for Datomic Cloud
- operational limits
- the changelog
Setup
Datomic dev-local is part of Cognitect dev-tools. Get the latest version of dev-tools and unzip it. Then install dev-local in your local maven repo:
./install
Configure Local Storage
By default, dev-local stores all databases under a common storage
directory. To specify this directory, create a .datomic/dev-local.edn
file in
your home directory, containing a map with :storage-dir
and an
absolute path:
{:storage-dir "/full/path/to/where/you/want/data"}
Maven
When you request dev-tools, your download email also includes instructions for adding the dev-tools maven repository to your project.
Using dev-local
There are two steps to use dev-local: add the dev-local library to your classpath and create a local client.
To add dev-local to your classpath, add a com.datomic/dev-local
entry to your deps.edn file.
{com.datomic/dev-local {:mvn/version "1.0.243"}}
The dev-local dependency is all you need for using dev-local.
There are two ways to get a local client:
- you can explicitly request a local client
- you can divert requests for Datomic Cloud clients to use local storage for development and testing
To explicitly request a local client, pass a map to d/client
with
:server-type :dev-local
- a
:system
name,
(require '[datomic.client.api :as d]) (def client (d/client {:server-type :dev-local :system "dev"}))
If you are using dev-local to develop and test a Datomic Cloud application, add the client-cloud dependency to your project. Then, to divert an existing Datomic Cloud system to dev-local, call divert-system:
(require '[datomic.dev-local :as dl]) (dl/divert-system {:system "production"}) ;; existing requests for Cloud system will be served locally! (def client (d/client {:system "production" :server-type :ion :region "us-east-1" :endpoint "https://ljfrt3pr18.execute-api.us-east-1.amazonaws.com"}))
You can also use import-cloud to import data from Datomic Cloud to local storage.
If you are new to Datomic, you can now work through the tutorial.
Sample Data
datomic-samples
is a set of databases that are used throughout these
Datomic docs and the Day of Datomic tutorials. To install the datomic
samples on your local computer:
- Get dev-local.
- Download the latest (2020-07-10) datomic-samples.zip.
- Unzip the datomic-samples zip into your storage dir.
You can connect and use them by setting your dev-local system name to "datomic-samples":
(require '[datomic.client.api :as d]) (def client (d/client {:server-type :dev-local :system "datomic-samples"})) (d/list-databases client {}) => ["mbrainz-subset" "solar-system" "social-news" "movies" ...]
Durability
dev-local stores data to your local filesystem, in directories under
the :system
you specify when creating a dev-local
client.
Each database will store transactions in a directory named
<storage-dir>/<system-name>/<database-name>
. You can "backup" or
"restore" a dev-local database simply by copying the database
directory.
Memdb
Sometimes durable storage is unnecessary and/or inconvenient. For example, a CI system may have no need of data after the tests run and no access to a file system.
You can force dev-local to use a memory-only database by passing :storage-dir :mem
to the map you pass when creating a client, as shown in the example below:
(def client (d/client {:server-type :dev-local :storage-dir :mem :system "ci"}))
API
Most usage of dev-local should be via portable Client API
calls. Capabilities that are specific to dev-local are available
through the datomic.dev-local
namespace:
- divert-system to develop and test Datomic Cloud applications against dev-local
- release-db to release memory used by a database
- import-cloud to import a Cloud database for dev-local use
All dev-local API calls take a single arg-map argument.
(import-cloud arg-map) (divert-system arg-map) (release-db arg-map)
divert-system
Diverts subsequent d/client calls for system to local storage. arg-map has the following keys:
Key | Value | Required |
---|---|---|
:system | the system to divert | yes |
:storage-dir | overrides :storage-dir in ~/.datomic/dev-local.edn |
release-db
Closes the connection to a database and releases the memory used by it. Values of that database can no longer be used. arg-map has the following keys:
Key | Value | Required |
---|---|---|
:system | system name | yes |
:db-name | database name | yes |
:storage-dir | overrides :storage-dir in ~/.datomic/dev-local.edn |
import-cloud
Import a Datomic Cloud database into dev-local. arg-map has the following keys:
Key | Value | Required |
---|---|---|
:source | arg map for (Cloud) d/client merged with arg map for d/connect | yes |
:dest | arg map for (dev-local) d/client merged with arg map for d/connect | yes |
:filter | filter spec limits datoms to import |
A filter spec has the following keys, all optional:
Key | Value |
---|---|
:before | t - include only txes before t, exclusive |
:since | t - include only txes after t, inclusive |
:include-attrs | map of attr -> filter |
:exclude-attrs | vector of attrs to be excluded |
:include-attrs
keys are either fully-qualified attribute names or a
namespaced keyword with * as a name (includes all attrs in namespace,
e.g. :order/*
).
:include-attrs
filters are maps with optional :before
and :since
keys
limiting the time range for attributes as per before/since above.
Schema attributes are always included. When a filter spec is present:
- All other attributes must be included explicitly.
- Excludes take precedence over includes.
You can re-import to get more recent data if you have not transacted locally. You cannot import to a database that currently has an open connection.
The following example imports a customer database, including
- all schema
- all history of customers (except their secrets!)
- orders since May 2020
(dl/import-cloud {:source {:system "customers" :db-name "customers" :server-type :ion :region "us-east-1" :endpoint "https://ljfrt3pr18.execute-api.us-east-1.amazonaws.com"} :dest {:system "production-imports" :server-type :dev-local :db-name "customers-and-recent-orders"} :filter {:include-attrs {:customer/* {} :order/* {:since #inst \"2020-05\"}} :exclude-attrs [:customer/secret]}})
Limits
- dev-local is in-process with your application code, and has all the tradeoffs (vs. a server or cluster) that this implies.
- dev-local requires 32 bytes of JVM heap per datom. You should plan your application with this in mind, while also leaving memory for your application's use.
- dev-local relies on OS page caching for performance, so leave some RAM available (i.e. not allocated to JVM heap) for this.
- dev-local limits the total number of datoms in a transaction to 105.
- dev-local limits strings to 4096 characters.
dev-local is best suited for small, single process applications. For larger projects, create a Datomic Cloud system.
Change Log
2022/04/06 - 1.0.243
- Upgrade Client to 1.0.126
2022/01/10 - 1.0.242
- Upgrade Client to 1.0.125
2021/09/27 - 1.0.238
- Enhancement: The datalog engine will now do self-unification within a single clause.
2021/07/13 - 0.9.235
- Use latest Client
- Fix anomaly where Attr-spec's for boolean attributes with false values failed
2021/01/20 - 0.9.232
- Improvement: enable BigInt fressian handler
2020/11/23 - 0.9.229
- New: Change the scale of a BigDecimal attribute in a transaction.
- Improvement: Better error messages for import-cloud.
- Fix: Query correctly treats range functions as functions (not as predicates).
2020/10/21 - 0.9.225
- New feature Memdb
- Improvement: Increase the limit on the total number of datoms in a transaction imported with import-cloud to 16 million.
- Improvement: Increase the limit on the length of strings imported with import-cloud to 1 million characters
2020/09/25 - 0.9.203
- Provide dev-tools via Cognitect maven repo
2020/07/24 - 0.9.184
- Improvement: better error message when calling a function that is not in the :allow list of datomic/ion-config.edn
2020/07/21 - 0.9.183
- Update: use latest Client API
2020/07/17 - 0.9.180
- Enhancement: Reread the deps-local.edn config file when creating a client.
- Enhancement: Provide better feedback while loading and importing databases.
- Enhancement: Update client documentation about connecting to dev-local systems
2020/07/10 - 0.9.172
Initial Release