exploring four datomic superpowers
DESCRIPTION
https://www.youtube.com/watch?v=7lm3K8zVOdY This session will explore four common problems, and the unique and surprising tools Datomic provides to solve them elegantly: HTTP caching - How to generically generate and validate Last-Modified and If-Modified-Since headers Audit trail - how to extend Datomic’s immutable transaction log to include arbitrary audit related metadata Mobile database sync - trivial implementation of an incremental update API for high latency/low bandwidth clients Authorization - easily determine resource ownership, and centrally isolate users from data they are not allowed to see These problems have certainly been solved before using other databases, but Datomic provides features that make the proposed implementations concise, generic, and purely functionalTRANSCRIPT
Lucas Cavalcanti & Edward Wible
Exploring Four Datomic Superpowers
2
Why build a bank from scratch in Brazil using Clojure and Datomic?
3
3
4
1 Audit Trail
5
Customer joins waiting list for a card01 NOV 10:00
event stream
5
Customer joins waiting list for a card01 NOV 10:00
Robot 437aae3 approves R$3K limit01 NOV 11:00
event stream
5
Customer joins waiting list for a card01 NOV 10:00
Robot 437aae3 approves R$3K limit01 NOV 11:00
Mastercard purchase, Starbucks, R$10009 NOV 08:00
event stream
5
Customer joins waiting list for a card01 NOV 10:00
Robot 437aae3 approves R$3K limit01 NOV 11:00
Mastercard purchase, Starbucks, R$10009 NOV 08:00
Support agent increases limit to R$5K15 NOV 15:00
event stream
5
Customer joins waiting list for a card01 NOV 10:00
Robot 437aae3 approves R$3K limit01 NOV 11:00
Mastercard purchase, Starbucks, R$10009 NOV 08:00
Support agent increases limit to R$5K15 NOV 15:00
Customer blocks card15 NOV 17:05
event stream
facts over time
6
01 NOV 10:00
01 NOV 11:00
09 NOV 08:00
15 NOV 15:00
15 NOV 17:05
[<customer> :customer/id #uuid “b2c90…”]
[<account> :account/customer <customer>][<account> :account/limit 3000][<card> :card/account <account>][<card> :card/status :card.status/active]
[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100][<purchase> :purchase/merchant “Starbucks”]
[<account> :account/limit 5000][<account> :account/limit 3000]
[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]
facts over time
6
01 NOV 10:00
01 NOV 11:00
09 NOV 08:00
15 NOV 15:00
15 NOV 17:05
[<customer> :customer/id #uuid “b2c90…”]
[<account> :account/customer <customer>][<account> :account/limit 3000][<card> :card/account <account>][<card> :card/status :card.status/active]
[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100][<purchase> :purchase/merchant “Starbucks”]
[<account> :account/limit 5000][<account> :account/limit 3000]
[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]
entity attribute value
7
7
don’t lose data
7
don’t lose data
what changed
when
7
don’t lose data
what changed
when
as-of since
7
don’t lose data
what changed
when
as-of since
fork merge
8
What was the initial limit for the card?
8
What was the initial limit for the card?
At the time the Starbucks transaction occurred, which fraud triggers would have activated?
8
What was the initial limit for the card?
At the time the Starbucks transaction occurred, which fraud triggers would have activated?
How long did the customer spend on each stage of the acquisition funnel?
8
What was the initial limit for the card?
At the time the Starbucks transaction occurred, which fraud triggers would have activated?
How long did the customer spend on each stage of the acquisition funnel?
How frequently do we see amount changes on Starbucks transactions?
9
What sequence of events resulted in the Starbucks transaction being persisted?
9
What sequence of events resulted in the Starbucks transaction being persisted?
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
9
What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
#uuid “b2c90…”“[email protected]”:kafka-LIMIT-CHANGED:robot-437aae3
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
user#uuid “b2c90…”“[email protected]”:kafka-LIMIT-CHANGED:robot-437aae3
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted?
Why was the customer’s card blocked?
correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
user#uuid “b2c90…”“[email protected]”:kafka-LIMIT-CHANGED:robot-437aae3
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted?
Why was the customer’s card blocked?
correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
:fraud-preventative:recurring-scheduled:late-payment:data-migration
user#uuid “b2c90…”“[email protected]”:kafka-LIMIT-CHANGED:robot-437aae3
Who was responsible for the credit limit increase?
9
What sequence of events resulted in the Starbucks transaction being persisted?
Why was the customer’s card blocked?
correlation-id
iOShttp-inkafka-outkafka-in
iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A
reason:fraud-preventative:recurring-scheduled:late-payment:data-migration
user#uuid “b2c90…”“[email protected]”:kafka-LIMIT-CHANGED:robot-437aae3
(defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked]
]))
10
storing transaction metadata
(defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked]
]))
{:db/id (d/tempid :db.part/tx) :audit/user "[email protected]" :audit/cid "iZb.jnA.9Cd.l9A" :audit/tags :fraud-preventative}
10
storing transaction metadata
11
[<customer> :customer/id #uuid “b2c90…”] 01 NOV 10:00
[<account> :account/customer <customer>][<account> :account/limit 3000M][<card> :card/account <account>][<card> :card/status :card.status/active]
01 NOV 11:00
[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100M][<purchase> :purchase/merchant “Starbucks”]
09 NOV 08:00
[<account> :account/limit 5000M][<account> :account/limit 3000M]15 NOV 15:00
[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]15 NOV 17:05
transactions over time with metadata
11
[<customer> :customer/id #uuid “b2c90…”] 01 NOV 10:00
[<account> :account/customer <customer>][<account> :account/limit 3000M][<card> :card/account <account>][<card> :card/status :card.status/active]
01 NOV 11:00
[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100M][<purchase> :purchase/merchant “Starbucks”]
09 NOV 08:00
[<account> :account/limit 5000M][<account> :account/limit 3000M]15 NOV 15:00
[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]15 NOV 17:05
transactions over time with metadata
12
querying transaction metadata
(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
12
querying transaction metadata
(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
[?card :card/status ?status ?tx]entity attribute value txn
12
querying transaction metadata
(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
12
querying transaction metadata
(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
12
querying transaction metadata
(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
13
2 Authorization
GET /purchases/1337/comments
should we return data?
GET /purchases/1337/comments
200 OK 401 Unauthorized
should we return data?
customer
account
card purchase
payment
comment
GET /purchases/1337/comments
200 OK 401 Unauthorized
should we return data?
customer
account
card purchase
payment
comment
GET /purchases/1337/comments
200 OK 401 Unauthorized
should we return data?
customer
account
card purchase
payment
comment
GET /purchases/1337/comments
200 OK 401 Unauthorized
should we return data?
15
explicit relationship traversal
(defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))
15
explicit relationship traversal
(defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))
generic relationship traversal
16
recursive rule (logical OR)
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
query using rule
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
generic relationship traversal
16
recursive rule (logical OR)
query using rule
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
(owns? customer-id [:purchase/id id] db)
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules))
generic relationship traversal
16
recursive rule (logical OR)
query using rule
(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
(owns? customer-id [:purchase/id id] db)
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules))
17
Recursion rules!
But can we go further?
Can we prevent queries from returning entities owned by
other customers?
all owned entities
18
query binding attribute
all owned entities
18
query binding attribute
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
all owned entities
18
query binding attribute
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
all owned entities
18
query binding attribute
attribute unbound
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
all owned entities
18
query binding attribute
attribute unbound
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
all owned entities
18
query binding attribute
attribute unbound
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
all owned entities
18
query binding attribute
attribute unbound
(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
filtered db (as value)
19
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
filtered db (as value)
19
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
filtered db (as value)
19
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
filtered db (as value)
19
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
filtered db (as value)
19
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
filtered db (as value)
19
filtered db will not contain datoms from other owners!
(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
passing a filtered db is transparent for queries
20
(defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id]))
(all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))
passing a filtered db is transparent for queries
20
(defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id]))
(all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))
21
We’re Mobile!
Bandwidth is costly
22
3 HTTP Cache
23
GET /accounts/1234/purchases200 OKLast-Modified: Fri, 14 Nov 2014 14:28:50 UTC{"purchases": [...]}
http last modified header
23
If-Modified-Since: Fri, 14 Nov 2014 14:28:50 UTCGET /accounts/1234/purchases304 Not Modified
GET /accounts/1234/purchases200 OKLast-Modified: Fri, 14 Nov 2014 14:28:50 UTC{"purchases": [...]}
http last modified header
24
How to keep track of a good last-modified date for a URL?
24
How to keep track of a good last-modified date for a URL?
The last time any owned entity changed!
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
If no customer-owned data changed, 304
25
(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
26
Cache hits are awesome, but what about cache misses?
27
4 Mobile Sync
desired hypermedia from the API
28
GET /accounts/1223/purchases200 OK{"purchases": [...] "_links": { "updates": "https://...?since=2014-11-10T11:12:13Z" }}
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
serve purchases added or modified since our last sync
29
(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))
(updated-purchases account-id db #inst "2014-11-14"))
30
Bonus
31
5 Future DBs
(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))
generating a virtual db
32
(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))
generating a virtual db
32
(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))
generating a virtual db
32
(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))
generating a virtual db
32
(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))
generating a virtual db
32
33
6 Testing
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - locally scoped
34
(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
db testing - vector of datoms
35
(fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))
db testing - vector of datoms
35
(fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))
36
7 Schema Extension
schema is data!
37
(d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])
:nubank/transform :pii
schema is data!
37
(d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])
38
8 Sharding Reads
39
sharding to the transactor
customers
notification
auth
processor
acquisition
accounts
39
sharding to the transactor
customers
notification
auth
processor
acquisition
accounts
customers
1 2 …
39
sharding to the transactor
customers
notification
auth
processor
acquisition
accounts
customers
1 2 …
39
sharding to the transactor
customers
notification
auth
processor
acquisition
accounts
shard-specific peer cache
1
customers
1 2 …
39
sharding to the transactor
customers
notification
auth
processor
acquisition
accounts
shard-specific peer cache
1
inter-shard ACID transactions
2
40
9 Db Aggregation
41
data science aggregation
customers
notification
auth
processor
acquisition
accounts
data science
42
querying multiple databases
(defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))
42
querying multiple databases
(defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))
43
1 Audit Trail
2 Authorization
3 HTTP Cache
4 Mobile Sync
5 Future DBs
6 Testing
7 Schema Extension
8 Sharding Reads
9 Db Aggregation
44
We’re hiring
45
We’re hiring
