Database Filters

A filter view allows you to produce values of the database filtered by your own domain-specific predicates. A filter can be used to e.g. remove datoms that are incorrect, not applicable at a point in time, or not available for security reasons.

For example, suppose that the 9999 value for :item/count in the example database above turns out to be a mistake. The transaction that made this assertion could be identified, and used to filter out the incorrect value:

(def error-txes
  "Known bad transactions"
  #{13194139534317})
(defn correct?
  [_ datom]
  (not (contains? error-txes (:tx datom))))
(def corrected (d/filter history correct?))

Now, the corrected database value can be passed to all our existing queries, which will never see or consider the incorrect datoms.

Imagine that you want to exclude all values of an attribute entirely from consideration. The following code creates a filter that rejects all datoms about the attribute :user/passwordHash.

(def password-hash-id (d/entid plain-db :user/passwordHash))
(def password-hash-filter (fn [_ ^Datom datom] (not= password-hash-id (.a datom))))
(def filtered-db (d/filter (d/db conn) password-hash-filter))

It is idiomatic to join different filters of the same database. One motivating case is performance: Since filtering is not free, it makes sense to filter only the parts of the query that need filtering. For example, the query below uses an unfiltered database to find entities, and then a filtered version of the same database to limit the attributes visible through the d/entity call:

[:find ?ent
 :in $plain $filtered ?email
 :where
 [$plain ?e :user/email ?email]
 [(datomic.api/entity $filtered ?e) ?ent]]

Filters are applied to an unfiltered database value obtained from db or with. In particular, the combination of with and as-of means "with followed by as-of", regardless of which API call you make first. with plus as-of lets you see a speculative db with recent datoms filtered out, but it does not let you branch the past.

The original database value returned from connection does no filtering. It sees only the present, without having to filter the past. So queries about "now" are as efficient as possible–they do not consider history and pay no penalty for history, no matter how much history is stored in the system.

Filters are applied to database indexes, filtering out data that does not match a predicate. Filters cannot do anything that could not be done directly with indexes, but they do allow you to separate some aspect of a query, reusing query logic across different filtered views.

Consider the following two queries:

• "What happened between 8:00 and 9:00 this morning?"
• "What did entity 42 look like at 8:00 this morning?"

The first query is limited only by time. Such a query can be answered directly from the log, which is a time index. Answering the same question via as-of and since would require a filtering scan of the entire database.

The second query is limited by both entity and by time. Thus it makes sense to augment indexes with a filter, calling a query on an as-of view of the database. The query will use the EAVT index to jump straight to the requested datoms, which are then filtered to exclude the most recent datoms. Because all the time-related work is done by as-of, the query can be time-agnostic and can work with any point-in-time. You can use the same approach in your own queries, writing queries that are agnostic about time (or about some domain value), and using filters to allow reuse of those queries across different domain values.