Datomic Cloud Change Log

• Upgraded Clojure to 1.11.4. See https://clojure.org/news/2024/08/03/clojure-1-11-4

• Feature: tx-stats. See https://docs.datomic.com/reference/tx-stats.html
• Performance: Improve performance in transactions and calls to d/with by prefetching reads. Config option datomic.prefetchConcurrency limits the number of concurrent prefetches. See https://docs.datomic.com/operation/system-properties.html
• Fix: Ensure datoms are virtual and are no longer added to the log.
• Upgraded com.datomic/client-cloud to 1.0.130
• Upgraded com.datomic/ion to 1.0.71

Upgraded com.cognitect/s3-creds to 1.0.31

• Upgraded com.cognitect/transit-clj to 1.0.333

• Updated dep list for Datomic Cloud 1125-9339

Fix: correctly deserialize URIs to java.net.URI

• Enhancement: migrate CloudFormation templates to use AutoScalingGroup LaunchTemplate instead of the deprecated LaunchConfiguration.
• Fix: pull expression could return true when supplying a :default for a boolean attribute.
• Fix: regression introduced in 990-9202 that could cause a database with blank keyword idents to fail to load.
• Performance: replace BufferedInputStream with an unsynchronized stream.
• Upgrade CFT Lambda Runtime to nodejs18.x.
• Upgraded core.async to 1.6.681.
• Upgraded commons-codec to 1.16.0.
• Upgraded aws-java-sdk-bundle to 1.12.564.
• Upgraded memcache-asg-java-client to 1.1.0.36.
• Upgraded http-client to 1.0.126.
• Upgraded transit-clj to 1.0.333.
• Upgraded client-cloud to 1.0.124.
• Upgraded fressian to 0.6.8.
• Upgraded guava to 32.0.1-jre.
• Upgraded jul-to-slf4j to 1.7.36.
• Upgraded log4j-over-slf4j to 1.7.36.
• Upgraded jcl-over-slf4j to 1.7.36.
• Upgraded tools.namespace to 1.4.4.
• Upgrade The AMI is now based on the 2023/10/16 version of the Linux 2023 base AMI.

• Upgraded core.async to 1.6.681.
• Upgraded commons-codec to 1.16.0.
• Upgraded aws-java-sdk-bundle to 1.12.564.
• Upgraded http-client to 1.0.126.
• Upgraded org.eclipse.jetty/jetty-* to 9.4.52.v20230823.
• Upgraded org.msgpack/msgpack to 0.6.12.

Upgraded aws-java-sdk-ssm to 1.12.564.

• Upgraded Clojure to 1.11.1.
• Upgraded AWS Java SDK to 1.12.564.
• Upgraded org.yaml/snakeyaml to 2.2.

• Performance: reduced resources required by indexing.
• Performance: improved EFS IOPS usage.
• Performance: indexing is more aggressive in removing :db/noHistory datoms.
• Performance: improved fressian read.
• Upgrade: NodeJS used in Lambdas upgraded to 16.x.
• Upgrade: Java on compute nodes upgraded to Java 17.
• Upgrade The AMI is now based on the 2023/08/09 version of the Linux 2023 base AMI.
• Enhancement: use latest AWS recommendations for mounting EFS Volumes.
• Enhancement: support all instance types in the i3 family.
• Fix: prevent a problem where in rare situations a node could stop applying transactions.

Datomic cloud AMIs are free of any markup outside the AWS marketplace under the Apache 2.0 license. Check the Datomic Cloud is Free blog post.

• Performance: Datomic now uses caffeine instead of guava for the object cache.
• Performance: improved fressian read.
• Feature: io-stats for transactions and queries. See Io-stats.
• Feature: query-stats. See Query-stats.
• Fix: attribute names that begin with an underscore can be used for forward lookups in entity and pull. Note that naming attributes with a leading underscore is strongly discouraged because it conflicts with the convention for reverse lookup.
• Fix: critical bug preventing transactions in 9188.
• Fix: pull performance regression introduced in 9188.
• Fix: exception when using io-stats in nested contexts, you should no longer see "No implementation of method: :-merge-stats of protocol".
• Upgraded AWS libraries to 1.12.358.

Feature: query-stats for transactions and queries. Check Query-stats.

The 981-9188 release templates have been removed (12/12/2022) per Critical Notice.

This release introduced a problem in Datomic that could eventually halt transactions (but not Reads). All templates have been removed and de-listed from the docs and marketplace. The 990-9202 release contains all features in this release and replaces 981-9188.

Recommendation: if you upgraded to this specific version is to immediately upgrade again to the latest 990-9202.

Fix: requiring `dev-local` no longer disables `ion.cast/initialize-redirect`.

Feature: Io-stats for transactions and queries. Check io-stats.

• Upgrade Clojure to 1.11.1.
• Upgrade AWS SDK to 1.12.132.
• Upgrade com.cognitect/s3-creds to 1.0.27.
• Upgrade com.datomic/memcached-asg-java-client to 1.1.0.33.
• Upgrade transit-clj to 1.0.329.
• Upgrade com.datomic/client to 1.0.126.
• Upgrade com.datomic/client-cloud to 1.0.120.
• Upgrade tools.namespace to 1.2.0.
• Upgrade The AMI is now based on the 3/16/2022 version of Linux 2 base AMI.
• Upgrade NodeJS used in Lambdas to 14.x.
• Add new output IonApiIntegrationId.
• Upgrade Corretto11 to 11.0.14+10.

• Upgrade transit-clj to 1.0.329.
• Upgrade aws-java-sdk-ssm to 1.12.132.

• Upgrade aws-java-sdk-* to 1.12.132.
• Upgrade s3-libs to 1.0.46.

• Upgrade Client to 1.0.126.
• Upgrade s3-creds to 1.0.27.
• Upgrade aws-java-sdk-s3 to 1.12.132.

Resolved an issue where a user may encounter the error:

"WARNING: abs already refers to: #'clojure.core/abs in namespace: cognitect.s3-libs.file, being replaced by: #'cognitect.s3-libs.file/abs"

• Upgraded AWS Java SDK to 1.12.100.
• Upgraded Jetty to 9.4.41.v20210927.
• Upgraded core.async to 1.5.6348.
• Upgraded core.cache to 1.0.225.
• Upgraded core.memoize to 1.0.253.
• Upgraded data.priority-map to 1.1.0.
• Upgraded tools.analyzer to 1.1.0.
• Upgraded tools.analyzer.jvm to 1.2.2.
• Upgraded tools.namespace to 1.1.1.
• Upgraded tools.reader to 1.3.6.
• Upgraded asm to 9.2.
• Upgraded org.slf4j to 1.7.32.
• Upgraded http-client to 1.0.111.
• Upgraded Guava to 31.0.1.
• Upgraded commons-codec to 1.15.
• Upgraded Java 11 to 11.0.13+8.
• Upgrade: The AMI is now based on the 12/01/2021 version of Linux 2 base AMI.

• Upgraded AWS Java SDK to 1.12.100.
• Upgraded Jetty to 9.4.44.v20210927.
• Upgraded core.async to 1.5.648.
• Upgraded core.cache to 1.0.225.
• Upgraded data.priority-map to 1.1.0.
• Upgraded http-client to 1.0.110.

• Upgraded AWS Java SDK to 1.12.100

Upgraded AWS Java SDK to 1.12.100.

Enhance error message when using `:local/root`.

• Fix: ensure that code bucket IAM policy points to the correct bucket if the bucket was deleted and recreated.
• Fix: prevent a problem where in rare situations indexing jobs can fail repeatedly with an ArityException.
• Enhancement: improve handling of throttling errors during CloudFormation stack creation.
• Enhancement: the datalog engine will now do self-unification within a single clause.
• Upgraded AWS Java SDK to 1.12.1.
• Upgraded Clojure to 1.10.3.
• Upgraded jackson-core to 2.12.3.
• Upgraded Jetty to 9.4.41.v20210516.
• Upgraded core.async to 1.3.618.
• Upgraded http-client to 1.0.108.
• Upgraded transit-clj to 1.0.324.
• Upgraded data.json to 2.4.0.
• Upgraded tools.namespace to 1.1.0
• Upgrade: The AMI is now based on the 07/01/2021 version of Linux 2 base AMI.

• Upgraded AWS Java SDK to 1.12.1.
• Upgraded Jetty to 9.4.41.v20210516.
• Upgraded core.async to 1.3.618.
• Upgraded core.cache to 1.0.207.
• Upgraded http-client to 1.0.108.
• Upgraded transit-clj to 1.0.324.

• Upgraded Lambda runtime to Corretto, per AWS policy.
• Upgraded AWS Java SDK to 1.12.1.
• Upgraded data.json to 2.4.0.

• Fix: Nil error thrown when calling cast before calling initialize-redirect.
• Upgraded AWS Java SDK to 1.12.1.
• Upgraded transit-clj to 1.0.324.
• Upgraded data.json to 2.4.0.

• Enhancement: many new EC2 instance sizes.
• Enhancement: lower pricing for all instance sizes.
• Enhancement: API Gateways for client and application internet access.
• Enhancement: run analytics anywhere.
• Upgrade: compute nodes have been upgraded from Java 8 to Java 11.
• Replaced: the client access gateway is no longer available; you can connect directly to the new client API gateway.
• Replaced: the analytics gateway is no longer available; you can run your own analytics node or cluster.
• Replaced: the solo tier is no longer available; the production compute stack now scales all the way down to t3.small.
• Replaced: the socks proxy is no longer available; clients can connect directly to the client API Gateway.
• Replaced: each compute group now uses an ALB instead of an NLB.

This upgrade will cause up to one minute of downtime for each compute group. Make sure to perform this upgrade at a time that minimizes the impact of these disruptions.

In addition, make sure to read the upgrade instructions below for each of the following scenarios that apply to your system today: solo systems, production systems, developer connections, application connections, analytics connections, and web application (ion) connections.

Enhancement: The storage template now sets DDB provisioning to fit within the AWS free tier if your usage is low enough.

Add describe-groups system command

Add support for hmac through API Gateway

• Update dep list for Datomic Cloud 884-9095
• Enhancement: improved integration between ions and AWS Application Load Balancers for the cloud version release.

• Upgrade: AWS Lambda Runtimes moved to NodeJs 12.x.
• Enhancement: new AWS Region - ca-central-1.
• Upgrade Jetty server to version 9.4.36.v20210114.

• Ion-dev
  • Improvement: limit how long to wait for a cluster node to gracefully shut down.
  • Improvement: increase performance of Ion request handler.
  • Fix: suppress noisy AsyncContext Alert caused by Ion timeout.
• Ion
  • Improvement: get-params now allows arbitrary number of parameters.

• Fix: attribute predicates are now applied to assertions only.
• Upgrade to presto 348 for Datomic analytics.

Improvement: make client dynaload thread-safe.

• New: change the scale of a BigDecimal attribute in a transaction.
• Upgrade to presto 346, fixing a bug that prevents analytics gateway from launching.
• Fix: reenable query counts metric in dashboard.
• Fix: query correctly treats range functions as functions (not as predicates).
• New region: eu-north-1.

• New region: ap-south-1.
• Fix: improve storage garbage collection.
• Fix: prevent situation where tx-range sometimes returned one extra transaction prior to a specified instant.

• New feature: cancel.
• New region: `eu-west-2`.
• Fix: prevent race conditions in Valcache that could lead to a failed read.
• Improvement: better entity predicate error messages.
• Improvement: better indexing performance.
• Upgrade: the version of the Presto server running on the access gateway is now 338. This includes an upgrade to Java 11 on the access gateway.
• Upgrade: the AMI is now based on the 6/17/2020 version of Linux 2 base AMI.
• Upgrade: AWS SDK for Java to version 1.11.826.
• Upgrade: core.async to version 1.3.610.

• New feature: cancel.
• Cloud-deps update.

Fix: datomic.api.client.async/client now accepts :dev-local as a :server-type.

New feature: dev-local.

• New feature: qseq
• New feature: index-pull
• New feature: xform
• Enhancement: improve index efficiency.
• Enhancement: improve performance of pull in queries.
• Enhancement: improve internal record keeping of active databases that could lead to spurious error messages.
• Enhancement: base AMI is now sourced from Amazon Linux 2, and includes recent security updates.

Upgrade: As recommended by AWS, we have upgraded our base AMI to Amazon Linux 2. This new AMI includes a breaking change to the Linux Netcat utility that Ions use to deploy new code to a cluster. We encourage all users of Ions to upgrade to version 0.9.265 or later of ion-dev prior to upgrading to this release.

• Enhancement: calculate transitive list of dependencies in `:dependency-conflict` map.
• Update dep list for Cloud 668-8927 release.

• New feature: qseq.
• New feature: index-pull.
• Updated jetty dependencies from 9.4.24.v20191120 to 9.4.27.v20200227.

Enhancement: add `–ssho` option to the gateway restart command to pass ssh configuration options as if using `ssh -o <option>`

• New feature: `[:db/retract eid aid]` will retract all values for an eid/aid combination.
• Critical fix: regression introduced in 569-8835 where `datoms`, `seek-datoms`, and `index-range` return incorrect results if `Iterable.iterator` is called more than once on the returned value.
• Fix: respect `nil` as a limit in query `pull` expression.
• Fix: respect `false` as a default in query `pull` expressions for boolean-valued attributes.
• Fix: allow arbitrary `java.util.List` (not just Clojure vectors) for lookup refs.

Many new features - see docs.

Increase timeout for how long a deployment will wait for dependencies to sync on an instance.

• New region: `sa-east-1`.
• Better tagging of resources created by Datomic.
• Fixed problem using with-db from ions.

This release requires a version of Datomic Cloud 569-8835 or newer.

• Enhancement: datomic-gateway restart now just restarts the process running on the access gateway, instead of restarting the whole instance. This results in faster restart times when making configuration changes. Restarting a bastion-only access gateway has no effect.
• Enhancement: automatically retrieve the host key from the access gateway. There will no longer be a warning about the authenticity of the host.

• Fix: resolve tempids for reference attributes inside tuples.
• Enhancement: performance improvement for range predicates in analytics support.
• Enhancement: SSH public keys automatically handled by access gateway and proxy script.
• Enhancement: AutoScalingGroup details exposed as CloudFormation outputs.
• Enhancement: JDBC metadata support in analytics.
• Upgrade: AWS Lambda Runtimes moved to NodeJs 10.x.

• Better error message for invalid ion-config.edn.
• Update commands to match the preferred approach of installing ion-dev in your user deps.edn.
• Use slf4j-simple to prevent console warnings when invoking tools.

Documentation string updates

Upgrade Clojure tools.deps.alpha to 0.8.584.

• Enhancement: added --ssho to allow any SSH -o option to be passed, removed --no-host-key-checking option.
• --instance-wait option renamed to --wait.

• Fix: prevent exception thrown when handling COUNT(*) clauses in analytics support.
• Fix: do not create redundant `:db.install/attribute` datoms for idempotent schema operations.
• Update transactor Clojure dependency to 1.10.1.
• Enhancement: new AWS Region - ap-northeast-1.

Initial release of CLI tools

• New feature preview: analytics. Check Analytics Concepts.
• Fix: allow Datomic Cloud to launch correctly in accounts that have a large number of existing ASGs.

Enhancement: allow for the inclusion of the target dir in the :paths list.

• New: upgrade from t2 to t3 instances for bastion and query groups.
• Fix: tx-range now returns a map with a :data key, consistent with the docs.
• Fix: correctly handle boolean attributes in composite tuples.
• Fix: prevent erroneous attempts to use Datomic On-Prem's excision feature.

Fixed problem where databases that change an attribute from `:db.cardinality/one` to`:db.cardinality/many` may become unavailable after a process restart.

• New feature: tuples.
• New feature: attribute predicates.
• New feature: entity specs.
• New feature: return maps.
• Fix: `sample` aggregate could hang the query thread.

• New feature: return maps.
• New feature: reverse references in nav (try it in REBL).

Improvement: fixed issue where some older Java libraries could not be loaded in an ion application

The 477-8741 release contains fixes, enhancements, and dependency updates:

• Enhancement: HTTP Direct integration for ions.
• Enhancement: improved integration between ions and AWS Network Load Balancers. This eliminates transient errors during rolling deployments.
• Enhancement: improved integration between ions and AWS Lambda. This also eliminates transient errors.
• Fix: issue where cluster nodes could become unresponsive when serving multiple databases or a burst of `tx-range` queries.
• Fix: nested queries could deadlock the query pool on Solo nodes.
• Fix: increased direct memory on Production nodes to prevent out-of-direct-memory errors.
• Update: version 1.7.26 of org.slf4j libraries.

• New feature: HTTP Direct.
• Enhancement: improved integration between ions and AWS NLBs.

Our Cloud Formation templates use node.js lambdas to create support functions for Cloud Formation. April 30th 2019 is the end-of-life date for AWS Lambda node.js runtime that we currently use.

Effective May 1, 2019, Datomic Cloud CloudFormation templates older than 4/25/2019 will no longer be executed. Datomic systems launched with those templates will continue to run, but any template operations (launching or upgrading CloudFormation stacks) will fail.

All Datomic Cloud users should upgrade to 470-8654.1 at their earliest convenience.

For more information see the AWS runtime support policy.

Critical: Updated to Node.js 8.10 runtime. All Datomic Cloud users should upgrade at their earliest convenience.

If you have created a VPC Endpoint using the provided CloudFormation Template, you will need to delete that CloudFormation Stack before upgrading to this version of Datomic Cloud.

Attempting to upgrade to or past this release without deleting the Stack will result in a failed update with the message: Export <SystemName>-VpcEndpointServiceName cannot be deleted as it is in use by <SystemName>-vpc-endpoint

See the troubleshooting documentation for more information.

The 470-8654 release contains fixes, enhancements, and dependency updates:

• Fix: production nodes now use the documented -Xss values for starting the JVM, preventing stack overflow when compiling complex ion applications.
• Fix: fixed bug where ion connections could fail to become aware of recent transactions in a timely manner.
• Fix: allow bigint values in transaction data.
• Fix: corrected allowlist handling of function calls from datalog rules.
• Enhancement: new AWS Region - ap-southeast-1.
• Enhancement: improved Valcache cleanup algorithm.
• Enhancement: reduce storage overhead of transactions.
• Enhancement: defer the creation of VPC Endpoint Service until specifically needed for cross-VPC client access.
• Update to version 2.9.8 of Jackson.
• Update to version 1.11.479 of the AWS SDK for Java.

The 454-8573 release contains performance and availability enhancements:

• Enhancement: you can configure systems to preload a database before serving requests, eliminating a source of unavailable anomalies.
• Enhancement: ion deployments load active databases before serving requests, eliminating a source of unavailable anomalies.
• Enhancement: improved throughput for transactions initiated by ion applications running the Production Topology.
• Enhancement: improved performance for systems that create and delete many databases (e.g. test systems).
• Enhancement: improvements to logging.

• Enhancement: increase CodeDeploy timeout to 5 minutes.
• Enhancement: include doc strings for user-facing functions in the build artifact.

• Add compatibility with com.cognitect/aws-api.
• Add Datafy.
• Upgraded http-client to 0.1.87.
• Improved error reporting.

Release 441-8505 includes critical updates. All users of the Production Topology and Query Groups should upgrade to 441-8505 immediately. The Solo Topology is not affected.

Critical: fixes a problem that can cause portions of the log to become inaccessible. Please update as soon as possible.

• New feature: query groups.
• New instance type option: i3.xlarge for Production.
• Improved memory settings: more stack space for Solo, more heap space for Production.
• Bugfix: fixed bug that could prevent a Production node from beginning indexing jobs.

• Bugfix: fixed bug that could cause a with database query to go to the wrong node in a production cluster.
• Bugfix: fixed Jetty configuration that could cause a client to prevent JVM from shutting down.
• Upgraded transit-clj to 0.8.313.
• :query-group parameter is no longer required in the client arg-map.

Enhancement: parameters library for working with the AWS systems manager parameter store.

• Bugfix: allow retraction of :db/unique attributes.
• Bugfix: coerce Integer values to long when needed in transactions.
• Enhancement: better memory utilization allows larger query results.
• Enhancement: automatically rollback deployments when an ion application fails to load.
• Added support for ap-southeast-2 (Sydney) region.

• New Feature: Datomic.ion.cast library for monitoring ions.
• Bugfix: fixed race condition in ion code loading that could allow ion invocation before namespace completely loaded.
• Enhancement: warn on dependency conflicts.
• Improvement: prefer shell-friendly symbols instead of strings as arguments to datomic.ion.dev CLI commands.
• Improvement: better error messaging when deploying to the wrong region.
• Improvement: list available deploy groups in push output.
• Improvement: enforce the requirement for :uname when project has a :local/root in deps.edn.

• Enhancement: added sync to client API. Check Client Synchronization.
• Better error message when unable to connect to cluster or proxy.

• Upgraded AWS libs to 1.11.349.
• Upgraded Jackson libs to 2.9.5.
• Fixed cache problem where all d/with databases deriving from a common initial call to d/with-db had the same common value.

• Enhancement: added :server-type :ion.
• Enhancement: ensure recentness of d/db conn.

• Enhancement: Datomic Ions.
• Improvement: Replaced Application Load Balancer with Network Load Balancer. If your applications run in a separate VPC you will need to configure a VPC endpoint.

• Bugfix: doubles and floats allowed in transactions.
• Bugfix: avoid unnecessary ":AdopterSkippedOlder" alert when creating a new database.
• Update: latest Amazon Linux patches.
• Improvement: better error handling in the storage template.
• Improvement: reduce memcached timeout.

Initial release.