Ions Tutorial

This tutorial will show you how to write applications that run entirely inside Datomic Cloud without any additional servers.

With ions you can build

  • Data-driven applications that are invoked via AWS Lambda
  • Web services accessed via AWS API Gateway for use by other applications or single page web apps

In this tutorial, you will build a simple inventory application with functions to add and query inventory, and then expose this application via both Lambda and AWS API Gateway. Along the way, you will learn how to develop, push, deploy and use your functions via:

Tutorial Prerequisites

Before this tutorial, you should

Setup

Clone the sample project

git clone https://github.com/datomic/ion-starter

The ion-starter project contains a complete ion-based application. To begin the tutorial, clone the ions-starter project.

Set application name

Ions are deployed with an application name that must match your compute group. Edit the resources/datomic/ion-config.edn file and set the :app-name to your application name.

Install the ion-dev tools

Make sure you have installed the ion-dev tools.

Develop at the REPL

With ions, you can develop your application interactively at the REPL. In this tutorial, the application code is already written, so we will use the REPL to explore the code.

Configure Connection

The ion-starter project keeps connection arguments in a resource file resources/ion/starter/config.edn, Edit this file to include the connection arguments for your system.

Start the Access Gateway

Follow the access gateway documentation to start your Datomic access gateway with the `client` connection type.

Test Your Connection

The core of the tutorial application lives in the starter.clj namespace. Start a REPL to explore and test these functions.

First, load some namespace aliases, and verify that you can get a client object.

(load-file "siderail/user.repl")
(def client (starter/get-client))

If this call succeeds, continue to the next step. Otherwise, return to Configure Connection.

Test data functions at the REPL

The inventory application exposes two simple data functions: get-schema returns the database schema, and get-items-by-type returns inventory items by type (:dress, :hat, :pants, or :shirt).

Review the data functions, and then test that these functions work at the REPL:

(def conn (starter/get-connection))
@(def db (d/db conn))
(starter/get-schema db)
(starter/get-items-by-type db :shirt '[:inv/sku :inv/color :inv/size])

Test lambda entry points at the REPL

A lambda entry point is just a function with a lambda-compatible signature, receiving JSON input an returning an arbitrary string of output.

The lambda entry points are in a separate lambdas namespace, so that you can easily see the difference between domain functions and lambda data marshaling.

Review the lambda entry points, and then try them from the REPL.

(lambdas/get-schema nil)
(lambdas/get-items-by-type {:input (json/write-str "shirt")})

Test HTTP entry points at the REPL

An HTTP entry point is just a function with an HTTP-compatible signature, receiving input and output maps that describe web requests and responses.

The HTTP entry points are in separate http namespace. Review the HTTP entry points, and then try them from the REPL.

(http/get-items-by-type {:body (s-edn/input-stream :shirt)})

Note that the REPL tests provide only the parts of the request map that are actually needed. This optionality is a benefit over more static systems that might require mocking and stubbing.

Now you are ready to push and deploy you ion to Datomic Cloud.

Push

The push operation creates an application revision in S3 and AWS CodeDeploy that can then be deployed multiple times to separate systems.

Configure entry points

When you push an application revision, the resources/datomic/ion-config.edn file specifies the applications entry points:

  • the :lambda section specifies functions that will be callable via AWS Lambda
  • the :http-direct section specifies functions that will be callable via HTTP
  • the :allow section specifies functions that will be callable from inside Datomic transactions or queries

For this tutorial, the entry points are already specified. Review them before you push.

Commit your changes

By default, push requires a clean git repository, and uses the git SHA to name the application revision in AWS. Such a push is reproducible. To prepare for a reproducible commit, you should git add files that are part of your application, git ignore any extraneous files, and git commit. If you are precisely following the tutorial, you have only changed resource files to set your application name and configure a client.

git add resources
git commit -m 'application name and client config'

To double check that you have a clean commit, you can run git status:

>git status
... details elided ...
nothing to commit, working tree clean

Push a revision

Now you are ready to push an application revision!

clojure -A:ion-dev '{:op :push}'

The push operation will report progress on the console. If it succeeds, it will print an edn map that includes a :deploy-command. Congratulations, you have pushed an app!

If push fails, don't worry. Use the push troubleshooting guide to diagnose and fix any problems you encounter.

Deploy

The deploy operation installs your pushed application on a Datomic compute group.

Let's deploy the revision you just pushed. In the command below, replace $(GROUP) with the name of your compute group, and $(REV) with the :rev output by push.

clojure -A:ion-dev '{:op :deploy :rev $(REV) :group $(GROUP)}'

On success, a deploy will print an edn map that includes a :status-command. Congratulations, your app revision is now deploying!

If deploy fails, don't worry. Use the deploy troubleshooting guide to diagnose and fix any problems.

Monitor Your Deployment

The deploy operation does the minimal work necessary to ensure that your application revision is running on all your compute nodes. This can take anywhere from a few seconds to several minutes. You can monitor deploy's progress with the deploy-status following command, replacing $(EXECUTION_ARN) with the :execution-arn output by deploy:

clojure -A:ion-dev '{:op :deploy-status :execution-arn $(EXECUTION_ARN)}'

This will return the status of the AWS Code Deploy and the overall ion deploy, with "SUCCEEDED" indicating when each process has successfully completed:

{:deploy-status "SUCCEEDED", :code-deploy-status "SUCCEEDED"}

If your deploy-status doesn't reach SUCCEEDED, don't worry. Use the deploy-status troubleshooting guide to diagnose and fix and problems.

Invoke a Lambda

Now you are ready to invoke your lambda entry points from the AWS CLI. First, call get-schema. In the command below, replace GROUP with the name of your compute group:

aws lambda invoke --function-name $(GROUP)-get-schema --payload '' /dev/stdout

On success, this call will print your lambda's response, plus some AWS CLI status information:

(#:db{:id 39,
      :ident :fressian/tag,
      :valueType :db.type/keyword,
      :cardinality :db.cardinality/one,
      :doc
      "Keyword-valued attribute of a value type that specifies the underlying fressian type used for serialization."}
 ;; more schema elided
 #:db{:id 80,
      :ident :inv/count,
      :valueType :db.type/long,
      :cardinality :db.cardinality/one})
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

On your first invocation, this call may take a while due to Lambda cold start. Outside of dev, most lambda invocations are warm, so try calling the function a second time to see typical production performance.

To see a function that takes an argument, try get-items-by-type:

aws lambda invoke --function-name $(GROUP)-get-items-by-type --payload '"shirt"' /dev/stdout

On success, this will return details about shirts:

[[#:inv{:sku "SKU-28", :size :xlarge, :color :green}]
 ;; more shirt details
 [#:inv{:sku "SKU-56", :size :large, :color :yellow}]]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Add an Attribute Predicate

You can use ions to deploy functions for use as attribute predicates, entity predicates, transaction functions, or query functions.

When you deployed the tutorial application, your ion-config.edn file allowed a valid-sku? function suitable for an attribute predicate.

(defn valid-sku?
  [s]
  (boolean (re-matches #"SKU-(\d+)" s)))

You can now test this function at the REPL:

(load-file "siderail/user.repl")

;; try the fn at the REPL
(attrs/valid-sku? "SKU-112")
(attrs/valid-sku? "SKU-1B")

Then you can install valid-sku as a predicate for :inv/sku:

(def conn (starter/get-connection))

(def tx [{:db/ident :inv/sku
          :db.attr/preds 'datomic.ion.starter.attributes/valid-sku?}])
(d/transact conn {:tx-data tx})

After you install the function, you can use a with-db to prove that the predicate is in effect:

(def with-db (d/with-db conn))
(d/with with-db {:tx-data [{:db/id "should-not-work"
                            :inv/sku "not-a-sku"}]})

=> Entity -9223301668109598141 attribute :inv/sku value not-a-sku failed pred datomic.ion.starter.attributes/valid-sku?

API Gateway Web Services

Choose the next step in the tutorial based on your system topology:

  • If you are running the Solo Topology, you can create a lambda proxy, and expose that proxy via the AWS API Gateway.
  • If you are running the Production Topology, you can expose an HTTP direct service via the AWS API Gateway.

Creating a Lambda Proxy Web Service

This tutorial step is for Solo Topology users.

You can use AWS API Gateway lambda proxy integration to expose a Lambda as a web service. Lambda proxies have their own special signature. The ion support library provides a helper function api-gateway/ionize that takes a web entry point, and returns a lambda proxy entry point.

The tutorial application already includes a call to ionize that makes a lambda proxy for get-items-by-type:

(def get-items-by-type-lambda-proxy
  (apigw/ionize get-items-by-type))

If you ran the push and deploy steps for add-item above, then get-items-by-type-lambda-proxy is also deployed and exposed as an entry point. Let's connect it to API Gateway.

Create an API in API Gateway

In this step you will create an API backed by an AWS Lambda proxy.

  • Go to the AWS API Gateway console to create a new AWS API Gateway.
    • If this is the first AWS API Gateway you are creating, choose Get Started, then click OK and choose thew New API radio button.
    • If you have existing AWS API Gateways, choose Create API.
  • Give your API a unique name, e.g. "ion-get-items-by-type". Leave the other default settings.
  • Click Create API in the bottom right.
  • Under the Actions dropdown, choose Create Resource.
  • Check the box "Configure as proxy resource." Other fields will be updated with defaults. Leave these as is.
  • Click Create Resource.

Connect API Gateway to Datomic Cloud

To connect API Gateway to get-items-by-type:

  • Set the Lambda Function to $(GROUP)-get-items-by-type-lambda-proxy and click Save. Note that the autocomplete does not always work correctly on this step, so trust and verify your own spelling.
  • Choose OK to give API Gateway permission to call your lambda.
  • Under your API in the left side of the UI, click on the bottom choice Settings.
  • Choose Add Binary Media Type, and add the */* type, then Save Changes.

Deploy API

NOTE This step of the tutorial will expose an ion on the public internet.

  • Select your API name (if you have not already), and choose Deploy API under the Actions dropdown.
  • Choose New Stage as the Deployment Stage, add the Stage Name "dev", then choose Deploy.

The top of the Stage Editor will show the "Invoke URL" for your deployed app. Append "/datomic" to the URL, and call your web service via curl to make sure everything looks ok:

curl https://$(Invoke URL)/datomic -d :hat

On success, this will return an EDN representation of inventory data for a particular type:

#{{:color :red, :type :hat, :size :medium, :sku "SKU-7"}
  ...
  {:color :blue, :type :hat, :size :small, :sku "SKU-35"}}

On your first invocation, this call may take a while due to Lambda cold start. Outside of dev, most lambda invocations are warm, so try calling curl a second time to see typical production performance.

Congratulations, you have a deployed a web service on the public internet. Don't forget to clean up when you are done with the tutorial.

Creating an HTTP Direct Web Service

HTTP Direct requires that you are running the Production Topology due to the need for an AWS NLB.

Configuration

The ion-config must contain an :http-direct key containing a handler-fn that takes a web request and returns a web response.

:http-direct {:handler-fn datomic.ion.starter/items-by-type}

Create an API Gateway

To create an API Gateway associated with a specific VPCLink:

  • Ensure that the VPC Link that you created in the previous step shows "Status: Available"
  • Go to the AWS API Gateway console
    • If this is the first AWS API Gateway you are creating, choose Get Started, then click OK and choose the New API radio button.
    • If you have existing AWS API Gateways, choose Create API.
  • Give your API a unique name, leave the other default settings.
  • Click Create API in the bottom right.
  • Under the Actions dropdown, choose Create Resource.
  • Check the box "Configure as proxy resource." Other fields will be updated with defaults. Leave these as is.
  • Click Create Resource.
  • Select VPC Link as the "Integration Type".
  • Check the box "Use Proxy Integration".
  • Select the desired VPC Link target for this API Gateway from the dropdown box
  • Enter your http://$(NLB URI):port/{proxy} as the "Endpoint URL". This NLB URI can be found in the Outputs tab of your compute or query group CloudFormation Template entry under the "LoadBalancerHttpDirectEndpoint" key:

    ions-api-gw-proxy.png

  • Click Save

Deploy the API Gateway

NOTE This step of the tutorial will expose an ion on the public internet.

  • Click on your API name, and choose Deploy API under the Actions dropdown.
  • Choose New Stage as the Deployment Stage, add the Stage Name "dev", then choose Deploy.

The top of the Stage Editor will show the "Invoke URL" for your deployed app. Append "/datomic" to the URL, and call your web service via curl to make sure everything looks ok:

curl https://$(Invoke URL)/datomic -d :hat

On success, this will return an EDN representation of inventory data for a particular type:

#{{:color :red, :type :hat, :size :medium, :sku "SKU-7"}
  ...
  {:color :blue, :type :hat, :size :small, :sku "SKU-35"}}

Cleanup (Optional)

The API Gateway is an external connection point, not managed by Datomic. If you created an API Gateway in the previous steps, you can select and delete it in the console.

Conclusion

In this tutorial, you have seen how ions can take ordinary Clojure functions, and install them in Datomic Cloud where they have in-memory access to Datomic data. You have used ions to deliver:

  • a data-driven application via AWS Lambda
  • a web service via API Gateway

There is a lot more to explore! Here are some things to try: