hacking mongodb at relateiq, a salesforce company
DESCRIPTION
Learn how RelateIQ takes advantage of MongoDB's tooling and oplog to drive near-realtime features and infrastructure for their customers. In this session we will cover how we exploit Morphia to make at-rest encryption of critical data both DRY and easy. We will also be covering our consumption of MongoDB's unsung hero, the oplog, which drives a significant portion of our back-end computing systems. While both projects are simple enough to be the product of Hackdays (a time-honored tradition at RelateIQ), they've blossomed into critical components of our infrastructure.TRANSCRIPT
Title of slide to go here
RelateIQ @ MongoSF
RelateIQ @ MongoSF
Blog
blog.relateiq.com
RelateIQ Twitter
@relateiq
{
name: “Jón Tómas Grétarsson”,
pretty_description: “Early Engineer”,
email: “[email protected]”,
twitter: @jontomas
}
1. MongoDB at RelateIQ
2. Encryption at Rest
3. Golden Horde (Oplog)
4. Q&A
Agenda
Our stack
● Multiple consumers
● Distributed services
● Batch and stream processing
We speak JSON
● Cross-language compatibility
● Engineer-debuggable
● Easy to test!
MongoDB
● Strong Java support via morphia
o Lifecycle methods!
MongoDB usage at RelateIQ
Lifecycle methods
Golden Horde
At-Rest Encryption
@preLoad
Called before mapping the datastore object to the
entity (POJO); the DBObject is passed as an
argument (you can add/remove/change values)
You have
● The DBObject
● A class definition (reflection!)
You don’t have
● That POJO with all the
convenience methods
@preSave
Called right before DBCollection.save() is called.
Changes made to the entity (POJO) will not be
persisted; the DBObject can be passed as an
argument (you can add/remove/change values)
You have
● The original POJO, including class
definitions and convenience
methods
At-Rest Encryption
SadPojo.java
public class SadPojo {
public String clientId;
public String secret;
};
SadDAO.java
public class SadDAO
extends BasicDAO<SadPojo> {
@Inject
public SadDAO(Datastore ds) {
super(SadPojo.class, ds);
}
};
At-Rest Encryption
Command Line
> db.SadPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "What follows are my evil plans to take over the
world..."
}
SketchyPojo.java
public class SketchyPojo {
public String clientId;
public String secretKey;
public String secret;
@PreSave
public void preSave(DBObject obj) {
obj.put(“secret”, encryptWithSecret(secretKey, secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take One)
At-Rest Encryption (Take One)
Command Line
> db.SketchyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secretKey": "<BASE64-encoded byte string>",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzV
idKs5JIbtvwX0SQqGmKf_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-
BJtybcMiJd-DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
SketchyPojo.java
public class SketchyPojo {
public String clientId;
public String secretKey;
public String secret;
@PreSave
public void preSave(DBObject obj) {
obj.put(“secret”, encryptWithSecret(secretKey, secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take One)
At-Rest Encryption (Take Two)
LessSketchyDAO.java
public class LessSketchyDAO extends BasicDAO<LessSketchyPojo> {
@Inject
public LessSketchyDAO(Datastore ds) {
super(LessSketchyPojo.class, ds);
}
@PreSave
public void preSave(LessSketchyPojo pojo, DBObject obj) {
obj.put(“secret”, encryptWithSecret(pojo.secretKey, pojo.secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take Two)
Command Line
> db.LessSketchyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secretKey": "<BASE64-encoded byte string>",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzV
idKs5JIbtvwX0SQqGmKf_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-
BJtybcMiJd-DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
At-Rest Encryption (Take Two)
LessSketchyDAO.java
public class LessSketchyDAO extends BasicDAO<LessSketchyPojo> {
@Inject
public LessSketchyDAO(Datastore ds) {
super(LessSketchyPojo.class, ds);
}
@PreSave
public void preSave(LessSketchyPojo pojo, DBObject obj) {
obj.put(“secret”, encryptWithSecret(pojo.secretKey, pojo.secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take Three)
HappyPojo.java
public class HappyPojo {
@EncryptionScope public String clientId;
@EncryptAtRest public String secret;
};
At-Rest Encryption (Take Three)
Command Line
> db.HappyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzVidKs5JIbtvwX0SQqGmK
f_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-BJtybcMiJd-
DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
At-Rest Encryption (Take Three)
Command Line
> db.HappyPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "APu5vPONiaLD4-
ykkgG6gG7hlN7reB_XY3rqLGkQ5aBTZBXveUqH_jJwRSjNCMCjrT0rFSe0ZCyrUAxPvjzVidKs5JIbtvwX0SQqGmK
f_853a-L_CkoyCRgJfPwzf81XM1dpGR8Kp9hMNAvkrdbco-
GdZgE6HENRnXcLzmoYTVhUIhwSCi3rX_8w9_Wn_GVLGI3EF-s-
0r7bqElxn0CB2l4W80sjDTlom_0nbZfwF92Xq9cctnnWghgLYaVEWxL3Va2tQL-BJtybcMiJd-
DnxXjHxlRPRB4jtIP8eJMagW7hCh7aDEOC5s0SEiZs5fLRt-dExKgbwxjz"
}
> db.ScopedKeys.find({_id : "Dr. No"})
{
"_id": "Dr. No",
"metadata": "{ < some JSON describing a Keyczar DECRYPT_AND_ENCRYPT keyring > }",
"secrets": [
"0": "{\"aesKeyString\": \"22mM-utDU_LSzhhwus3cuA\",\"mode\":\"CBC\",\"size\":128}",
"1": "{\"aesKeyString\": \"22mM-utDU_LSzhhwus3cuA\",\"mode\":\"CBC\",\"size\":128}"
]
}
At-Rest Encryption (Take Three)
MongoModule.java
public class MongoModule extends AbstractModule {
@Provides
public Morphia providesMorphia(CrypterFactory crypterFactory) {
Morphia morphia = new Morphia();
...
morphia.getMapper().addInterceptor(
new EncryptAtRestInterceptor(crypterFactory));
return morphia;
};
Key Takeaways
1. Don’t Repeat Yourself.
2. Morphia’s lifecycle handling is incredibly useful, when used properly.
3. Don’t Repeat Yourself!
4. Encryption is a scary word, but we can make it as natural as annotating a POJO.
5. Use OSS to DRY your code
https://github.com/relateiq/EncryptAtRestInterceptor
1. Pull requests appreciated!
Encrypt-At-Rest like the boss
The Oplog
Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database for
analytics and data warehousing?
● Better, how about we transform
documents and feed them back into
the system!?
Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
Oplog to the rescue!
● We can rely on the same system
that Mongo uses internally for
tracking and updating repls.
Golden Horde
Golden Horde
Closing Arguments
Golden Horde
Q & A
Golden Horde