I was showing duct to a friend not versed into Clojure at all, and he told me that he did not understand what was going on. How he was supposed to add redis to the stack, that he saw the duct redis module but didn’t even know where to start from there.
Duct is super awesome, Clojure is too obviously, the litterature on duct is not really thorough on the web, you have some wiki pages on the duct github, some posts on James Reeves’ blog and obviously duct documentation which is really good but can be complex for a Clojure beginner, even if we don’t take into account the fact that you need to understand what ataraxy and a lot of other libraries do.
So I thought I would write a small how-to to get you going with a simple duct project and a redis database, that should explain what’s going on and how things connects.
Duct
As explained in their GitHub project:
Duct is a highly modular framework for building server-side applications in Clojure using data-driven architecture.
It is similar in scope to Arachne, and is based on Integrant. Duct builds applications around an immutable configuration that acts as a structural blueprint. The configuration can be manipulated and queried to produce sophisticated behavior.
Duct comes with a lein generator that will let you get started pretty quickly by adding functionality like API middlewares, ataraxy a specific clojure router, ClojureScript, a PostgreSQL component, and so on.
But for a beginner it can be hard to grasp at the beginning how these modules play together, and how the fact that Duct is a highly modular framework matters.
Duct & Redis
Duct enables access to redis through the use of its duct/database.redis.carmine module.
As you can see from the README of this module it says that it provides a Boundary record that holds connection options for Carmine, and a multimethod :duct.database.redis/carmine that initiates thoses options into the said Boundary.
What are we going to build?
The goal of this article is to show you how to create a simple API that when called on /example will reply a JSON containing information stored in a Redis database.
It will show how to scaffold an example application with the lein duct generator, and how to modify it step by step to:
- add a connection to Redis
- retrieve these information at runtime
- use them to connect to redis and fetch some data
- return them to the client
- make the tests pass by stubbing our Boundary whith
shrubbery - make sure that it runs from the REPL and from
lein run(or in production) - and finally how things connects in this highly modular framework
Installation
We’ll start with lein new duct my-project +api +example +ataraxy and we’ll update the project.clj file to add the dependency to database.redis.carmine and to shrubbery for some testing at the end:
project.clj
(defproject my-project "0.1.0-SNAPSHOT"
...
:dependencies [[org.clojure/clojure "1.9.0"]
[duct/core "0.6.2"]
[duct/module.logging "0.3.1"]
[duct/module.web "0.6.4"]
[duct/module.ataraxy "0.2.0"]
[duct/database.redis.carmine "0.1.0"] ; <--
[com.gearswithingears/shrubbery "0.4.1"]] ; <--
:plugins [[duct/lein-duct "0.10.6"]]
...Then we need to update our dev/resources/dev.edn file to configure the options to access redis:
dev/resources/dev.edn
{:duct.core/environment :development
:duct.core/include ["my-project/config"]
:duct.database.redis/carmine
{:spec {:host "127.0.0.1", :port 6379}}}Finally we need to modify the resources/config.edn so that the generated handler (the +example option) knows about redis:
resources/config.edn
{:duct.core/project-ns my-project
:duct.core/environment :production
:duct.module/logging {}
:duct.module.web/api {}
:duct.module/ataraxy
{[:get "/example"] [:example]}
:my-project.handler/example
{:db #ig/ref :duct.database.redis/carmine} ; <--
}Please take special attention to the :db part. It is the key that we’ll extract from the option in the following part of the tutorial.
Data
Since we want to read data from redis, just use redis-cli to add some data to our redis, so that we’ll be able to read them from our duct application later on.
$ redis-cli sadd persons Rich Stuart James Yogthos
(integer) 4
$ redis-cli smembers persons
1) "Rich"
2) "James"
3) "Stuart"
4) "Yogthos"
We just created a set whose key is named persons in Redis having 4 members.
Tinker
Ok now just run lein repl and then when the repl is ready, type (dev), then (go).
$ lein repl
nREPL server started on port 64229 on host 127.0.0.1 - nrepl://127.0.0.1:64229
REPL-y 0.3.7, nREPL 0.2.13
Clojure 1.9.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_111-b14
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (dev)
:loaded
dev=> (go)
:duct.server.http.jetty/starting-server {:port 3000}
:initiated
dev=>
Now you can hit http://localhost:3000/example and you should get the example created by the lein plugin at the very beginning of this small tutorial.
$ curl http://localhost:3000/example
{"example":"bar"}
Now bear in mind that if you modify the source code of your project, you need to get back in the repl and type (reset) so that theses changes are reflected.
Code
Boundary
Create a new file src/my-project/boundary/person_db.clj with the following content:
src/my-project/boundary/person_db.clj
(ns my-project.boundary.person-db
(:require [duct.database.redis.carmine]
[taoensso.carmine :as car :refer (wcar)]))
(defprotocol PersonDatabase
(list-persons [db]))
(extend-protocol PersonDatabase
duct.database.redis.carmine.Boundary
(list-persons [db]
(let [conn (:conn-opts db)]
(car/wcar conn
(car/smembers "persons"))))) In this file we require the carmine library and the duct.database.redis.carmine module.
We then define a protocol named PersonDatabase which contains a function list-persons which needs a db to operate on.
Finally we extend this protocol using the duct.database.redis.carmine.Boundary and we implement the list-persons function.
If you remember in your dev.edn file we declared:
:duct.database.redis/carmine
{:spec {:host "127.0.0.1", :port 6379}}}Extending the protocol using duct.database.redis.carmine.Boundary will gave us access to the following map {:spec {:host "127.0.0.1", :port 6379}} in which we want to extract the :spec key that Carmine needs.
All there is to do is passing this to car/wcar and a body to execute (here (car/smembers "persons") which will run SMEMBERS persons on redis side).
Note: we extract the spec needed by Carmine from the duct.database.redis.carmine.Boundary using the :conn-opts key. The spec is being embedded in this :conn-opts key in the implementation of duct/database.redis.carmine, you can see it in the duct/database/redis/carmine.clj source file:
(defrecord Boundary [conn-opts])
(defmethod ig/init-key :duct.database.redis/carmine [_ conn-opts]
(->Boundary conn-opts))That’s it for the boundary part.
Handler
All is well, we should now retrieve these information and return them in the handler so that it gets converted into JSON.
Open the src/my-project/handler/example.clj and modify the code like this:
src/my-project/handler/example.clj
(ns my-project.handler.example
(:require [ataraxy.core :as ataraxy]
[ataraxy.response :as response]
[integrant.core :as ig]
[my-project.boundary.person-db :as person-db]))
(defmethod ig/init-key :my-project.handler/example [_ {:keys [db]}]
(fn [{[_] :ataraxy/result}]
[::response/ok {:persons (person-db/list-persons db)}]))We just changed {:example "data"} to {:persons ...} so that we return the JSON structure we wants.
Additionnaly we changed [_ options] to [_ {:keys [db]}] in order to extract the db configuration from the option so that we can pass it to our Boundary.
Ok, go back in your REPL and type (reset).
dev=> (reset)
:reloading (my-project.boundary.person_db my-project.handler.example my-project.handler.example-test my-project.main dev user)
:resumed
dev=>
You can now hit http://localhost:3000/example to verify that it works.
$ curl http://localhost:3000/example
{"persons":["Rich","James","Stuart","Yogthos"]}
Testing
Back in your REPL type (test), you should be greeted with some stacktrace.
dev=> (test)
ERROR in my-project.handler.example-test/smoke-test)
Uncaught exception, not in assertion.
expected: nil
actual:
java.lang.IllegalArgumentException: No implementation of method: :list-persons of protocol: #'my-project.boundary.person-db/PersonDatabase found for class: nil
clojure.core/-cache-protocol-fn core_deftype.clj: 583
my-project.boundary.person-db/eval22226/fn/G person_db.clj: 5
my-project.handler.example/eval22272/fn/fn example.clj: 10
my-project.handler.example-test/fn example_test.clj: 10
clojure.test/test-var/fn test.clj: 716
clojure.test/test-var test.clj: 716
eftest.runner/test-vars/test-var/fn runner.clj: 54
clojure.test/default-fixture test.clj: 686
eftest.runner/test-vars/test-var runner.clj: 50
eftest.runner/wrap-test-with-timer/fn runner.clj: 30
...
clojure.core/apply core.clj: 657
clojure.core/with-bindings* core.clj: 1965 (repeats 2 times)
...
clojure.core/apply core.clj: 661
clojure.core/bound-fn*/fn core.clj: 1995
...
clojure.core/pmap/fn/fn core.clj: 6942
clojure.core/binding-conveyor-fn/fn core.clj: 2022
...
java.util.concurrent.FutureTask.run FutureTask.java: 266
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1142
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 617
java.lang.Thread.run Thread.java: 745
1/1 100% [==================================================] ETA: 00:00
Ran 1 tests in 0,087 seconds
1 assertion, 0 failures, 1 error.
{:test 1, :pass 0, :fail 0, :error 1, :type :summary, :duration 86.557333}
If you open the test/my-project/example_test.clj you will see the following code:
test/my-project/example_test.clj
(ns my-project.handler.example-test
(:require [clojure.test :refer :all]
[integrant.core :as ig]
[ring.mock.request :as mock]
[my-project.handler.example :as example]))
(deftest smoke-test
(testing "example page exists"
(let [handler (ig/init-key :my-project.handler/example {})
response (handler (mock/request :get "/example"))]
(is (= :ataraxy.response/ok (first response)) "response ok"))))You can see that the handler is being created using ig/init-key but the options are empty ({}), this is where we’d like to inject our mocked redis that we’re going to create just now using shrubbery.
Create a new file test/my-project/boundary/person_db_test.clj:
test/my-project/boundary/person_db_test.clj
(ns my-project.boundary.person-db-test
(:require [my-project.boundary.person-db :refer :all]
[shrubbery.core :refer :all]))
(def mock-list-persons ["Foo" "Bar" "Bazz"])
(def mock-redis
(stub PersonDatabase
{:list-persons mock-list-persons})) We stub PersonDatabase by redefining the list-persons function so that it return always 3 members Foo, Bar and Bazz.
Now get back to test/my-project/example_test.clj and require this mock and inject it as configuration option for the handler.
Finally add a new test so that we can check that redis was correctly mocked.
(ns my-project.handler.example-test
(:require [clojure.test :refer :all]
[integrant.core :as ig]
[ring.mock.request :as mock]
[my-project.handler.example :as example]
[my-project.boundary.person-deb-test :refer [mock-redis mock-list-persons]]))
(deftest smoke-test
(testing "example page exists"
(let [handler (ig/init-key :my-project.handler/example {:db mock-redis})
response (handler (mock/request :get "/example"))
[status {:keys [persons]}] response]
(is (= :ataraxy.response/ok status) "status is ok")
(is (= persons mock-list-persons) "redis is mocked"))))Go bak to your REPL and type (reset) then (test):
dev=> (test)
1/1 100% [==================================================] ETA: 00:00
Ran 1 tests in 0,003 seconds
2 assertions, 0 failures, 0 errors.
{:test 1, :pass 2, :fail 0, :error 0, :type :summary, :duration 3.241681}
dev=>
Now you have 1 test with 2 assertions passing.
Congratulations!
RUnning
Quit the REPL and run lein run. What happens ?
$ ~/Dev/tools/lein/lein run
Exception in thread "main" clojure.lang.ExceptionInfo: Missing definitions for refs: :duct.database.redis/carmine {:reason :integrant.core/missing-refs, :config {:duct.handler.static/method-not-allowed {:body {:error :method-not-allowed}}, :duct.logger/timbre {:level :info, :appenders #:duct.logger.timbre{:println #integrant.core.Ref{:key :duct.logger.timbre/println}}}, :duct.module/ataraxy {[:get "/example"] [:example]}, :duct.middleware.web/format {}, :duct.core/handler {:router #integrant.core.Ref{:key :duct.router/ataraxy}, :middleware [#integrant.core.Ref{:key :duct.middleware.web/not-found} #integrant.core.Ref{:key :duct.middleware.web/format} #integrant.core.Ref{:key :duct.middleware.web/defaults} #integrant.core.Ref{:key :duct.middleware.web/log-requests} #integrant.core.Ref{:key :duct.middleware.web/log-errors} #integrant.core.Ref{:key :duct.middleware.web/hide-errors}]}, :duct.router/ataraxy {:routes {[:get "/example"] [:example]}, :handlers {:ataraxy.error/unmatched-path #integrant.core.Ref{:key :duct.handler.static/not-found}, :ataraxy.error/unmatched-method #integrant.core.Ref{:key :duct.handler.static/method-not-allowed}, :ataraxy.error/missing-params #integrant.core.Ref{:key :duct.handler.static/bad-request}, :ataraxy.error/missing-destruct #integrant.core.Ref{:key :duct.handler.static/bad-request}, :ataraxy.error/failed-coercions #integrant.core.Ref{:key :duct.handler.static/bad-request}, :ataraxy.error/failed-spec #integrant.core.Ref{:key :duct.handler.static/bad-request}, :example #integrant.core.Ref{:key :my-project.handler/example}}, :middleware {}}, :duct.middleware.web/log-requests {:logger #integrant.core.Ref{:key :duct/logger}}, :duct.middleware.web/defaults {:params {:urlencoded true, :keywordize true}, :responses {:not-modified-responses true, :absolute-redirects true, :content-types true, :default-charset "utf-8"}}, :duct.server.http/jetty {:port 3000, :handler #integrant.core.Ref{:key :duct.core/handler}, :logger #integrant.core.Ref{:key :duct/logger}}, :duct.handler.static/internal-server-error {:body {:error :internal-server-error}}, :duct.middleware.web/hide-errors {:error-handler #integrant.core.Ref{:key :duct.handler.static/internal-server-error}}, :duct.middleware.web/log-errors {:logger #integrant.core.Ref{:key :duct/logger}}, :my-project.handler/example {:db #integrant.core.Ref{:key :duct.database.redis/carmine}}, :duct.handler.static/not-found {:body {:error :not-found}}, :duct.router/cascading [], :duct.middleware.web/not-found {:error-handler #integrant.core.Ref{:key :duct.handler.static/not-found}}, :duct.core/environment :production, :duct.module/logging {}, :duct.middleware.web/stacktrace {}, :duct.handler.static/bad-request {:body {:error :bad-request}}, :duct.logger.timbre/println {}, :duct.module.web/api {}, :duct.core/project-ns my-project}, :missing-refs (:duct.database.redis/carmine)}, compiling:(/private/var/folders/pn/jcqmh8s53hn_wtkjvlwnsv300000gn/T/form-init2116886874611819468.clj:1:125)
at clojure.lang.Compiler.load(Compiler.java:7526)
at clojure.lang.Compiler.loadFile(Compiler.java:7452)
at clojure.main$load_script.invokeStatic(main.clj:278)
at clojure.main$init_opt.invokeStatic(main.clj:280)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invokeStatic(main.clj:311)
at clojure.main$null_opt.invokeStatic(main.clj:345)
at clojure.main$null_opt.invoke(main.clj:342)
at clojure.main$main.invokeStatic(main.clj:424)
at clojure.main$main.doInvoke(main.clj:387)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:702)
at clojure.main.main(main.java:37)
Well, that was unexpected. The stacktrace can seem a bit obscure, but if you read it carefully it says that it cannot find the information for :duct.database.redis/carmine and this is true!
We only defined it in our dev/src/dev.edn file. We need to define something in our resources/my-project/config.edn.
resources/my-project/config.edn
:duct.database.redis/carmine
{:spec {:host "127.0.0.1", :port 6379}}But that’s not right, I don’t know for sure that in production this is the correct information…
What we want is to use #duct/env which will read the information from the environment variables.
resources/my-project/config.edn
{:duct.core/project-ns my-project
:duct.core/environment :production
:duct.module/logging {}
:duct.module.web/api {}
:duct.module/ataraxy
{[:get "/example"] [:example]}
:duct.database.redis/carmine
{:spec
{:host #duct/env "REDIS_HOST",
:port #duct/env ["REDIS_PORT" Int]}}
:my-project.handler/example
{:db #ig/ref :duct.database.redis/carmine}
}Then run REDIS_HOST=127.0.0.1 REDIS_PORT=6379 ~/Dev/tools/lein/lein run
It should now work correctly :)
If you’re still unsure, set a random port for REDIS_PORT and retry, you should have a stacktrace saying something like clojure.lang.ExceptionInfo: Carmine connection error
What to remember?
Duct serves web requests using duct/module.web, and destructures them using duct/module.ataraxy.
By importing the duct/database.redis.carmine module, Duct gives us a redis Boundary (using Carmine) that we can use to retrieve the redis connection option in order to implement our business Boundary (here PersonDatabase).
It is good practice to stub (mock) your boundaries so that we can use them in unit tests.
Developping using the REPL is cool and let us tinker with the code easily.
Having configuration in production coming from environment variables is good practice, duct has us covered with #duct/env, and you can override it in development in your dev.edn config file.