Troubleshooting

Not finding what you need here? Ask on the forum.

Getting Help

The Datomic Forum is an excellent resource for asking questions and getting help with Datomic. In addition to the community members, the Datomic Team actively monitor the Forum.

Troubleshooting CloudFormation Templates

Failures in Nested Templates

If a nested template fails, the root template will fail also, and the console will by default show only the root template's CREATE_FAILED event. To see the important detials in the nested template's events:

  • Click on the Filter popup, and choose "Deleted" images/template-filters.png
  • Select the nested template, and then choose the Events tab to see the cause of failure.

Check for CloudFormation failure

To check your CloudFormation for errors, find your stack in the CloudFormation window and click the checkbox at the start of its row.

If the status of the stack is "CREATE_IN_PROGRESS" the stack is still being created, and you should continue to wait for the stack to start. You can monitor it from the Cloudformation details window. The stack is done when it says "CREATE_COMPLETE."

If the status of the stack is "CREATE_FAILED" there was an error starting the stack. See the other topics on this page for help tracking down the source of the failure.

Production Topology Create Failed

If the production topology fails to create in CloudFormation with a CREATE_FAILED on the AWS::AutoScaling::AutoScalingGroup for the TxAutoScalingGroup resource with the message:

You have requested more instances (2) than your current instance limit of 1 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. Launching EC2 instance failed.

You need to request a limit increase in your allowed number of running i3.large instances.

Starting a compute stack without a storage stack.

Without a storage system, compute will fail immediately and rollback. The stack event message will report the following error, substitute <system-name> for the system name you specified when launching your compute stack.

No export named <system-name>-FileSystemId found. Rollback requested by user.

./images/compute-only-error.png

Running in EC2 Classic

Datomic Cloud is currently only supported in accounts that support only EC2-VPC. If you attempt to run Datomic Cloud in an EC2-Classic, the Cloudformation stack will fail. The stack event message will have an error with a Type of "Custom::ResourceCheck" and a Logical ID of "EnsureEc2Vpc."

See Setting Up for more information.

Troubleshooting Client Errors

Missing Connect Arguments

If you leave out a required argument.

CompilerException clojure.lang.ExceptionInfo: Expected string for :query-group {:cognitect.anomalies/category :cognitect.anomalies/incorrect, :cognitect.anomalies/message "Expected string for :query-group", :datomic.client.impl.shared.validator/got {:server-type :cloud, :region "us-east-1", :system "errortest2", :endpoint "http://entry.errortest2.us-east-1.datomic.net:8182/", :proxy-port 8888}, :datomic.client.impl.shared.validator/op :client, :datomic.client.impl.shared.validator/requirements {:region string, :system string, :query-group string, :endpoint string, :server-type keyword}}, compiling:(form-init8129817634595122502.clj:1:13)

::cognitect.anomalies/forbidden

If you get a forbidden error when using the Client API, this means that your AWS credentials do not grant permission to this functionality.

  • Make sure that you have an IAM policy that authorizes the operation you are trying to perform.
  • Make sure that policy is associated with the identity you are running under.

Connection Failure

The client error:

Exception in thread "main" clojure.lang.ExceptionInfo: Unable to connect to system: 
{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.anomalies/message "SOCKS4 tunnel failed, connection closed", :cognitect.http-client/throwable #error {
 :cause "SOCKS4 tunnel failed, connection closed"
 :via
  [{:type java.io.IOException
    :message "SOCKS4 tunnel failed, connection closed"
    :at [org.eclipse.jetty.client.Socks4Proxy$Socks4ProxyConnection onFillable "Socks4Proxy.java" 166]}] ...

Accompanied by the following message from your SOCKS Proxy:

debug1: channel 2: new [dynamic-tcpip]
channel 2: open failed: administratively prohibited: open failed

Indicates that the client was unable to reach the Datomic system through the proxy. Check your configuration :endpoint carefully and use the Bastion Connection Test to ensure your proxy is configured correctly.

Jetty Dependency Conflict

The following error will occur if your project has a transient dependency on a more recent version of Apache Jetty than the version used by Datomic:

\#error {
 :cause "org.eclipse.jetty.util.thread.Invocable$InvocationType"
  :via
   [{:type java.lang.NoClassDefFoundError
     :message "org/eclipse/jetty/util/thread/Invocable$InvocationType"
     :at [org.eclipse.jetty.io.ManagedSelector <init> "ManagedSelector.java" 79]}
    {:type java.lang.ClassNotFoundException
     :message "org.eclipse.jetty.util.thread.Invocable$InvocationType"
     :at [java.net.URLClassLoader findClass "URLClassLoader.java" 381]}]
 ...}

The conflict can be resolved by excluding the version of Jetty used by Datomic:

[com.datomic/client-cloud "0.8.46"
 :exclusions [org.eclipse.jetty/jetty-client
              org.eclipse.jetty/jetty-http
              org.eclipse.jetty/jetty-util]]

::cognitect.anomalies/busy

If you get a busy response from the Client API, your request rate has temporarily exceeded the capacity of a node, and has already been through an exponential backoff and retry implemented by the client. At this point you have three options:

  • Continue to retry the request at the application level with you own exponential backoff. The Mbrainz Importer example project demonstrates a batch import with retry.
  • Expand the capacity of your system, by upgrading from Solo to Production or increasing the number of instances in your AutoScaling Group.
  • Give up on completing the request.

Troubleshooting Socks Proxy Errors

Unsupported AWS CLI Version

Symptom: The script prints a generic AWS CLI help message before failing:

$ bash datomic-socks-proxy -r eu-central-1 good-system
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument command: Invalid choice, valid choices are:
...

Fix: Update to the latest version of the AWS CLI.

Wrong AWS Creds

$ bash datomic-socks-proxy -r eu-central-1 good-system
Unable to read bastion key, make sure your AWS creds are correct.

Fix: Make sure your AWS creds are connected to a policy that is authorized for your Datomic system.

Wrong System Name

$ bash datomic-socks-proxy -r eu-central-1 not-a-system
Datomic system not-a-system not found, make sure your system name and AWS creds are correct.

Using CloudWatch Logs

To view the CloudWatch Logs for a Datomic system:

  • Open the CloudWatch Logs in the AWS Console
  • Click on the Log Group named "datomic-{System}", where System is your system name

Each EC2 instance will create a separate log stream. Usually you will want to search across all log streams:

  • click the "Search Log Group" button
  • click to the right of the filter box to choose an appropriate time window
  • enter a metric filter pattern to scope your search

A good pattern to start with is "Alert - Alerts". (Datomic includes "Alert" in every event that warrants review by a person, and the "- Alerts" removes redundancy.

The example below demonstrates reviewing a week's worth of alerts. The single event that matches is a transient DynamoDB request failure, so not a problem.

images/search-for-alerts.png