sqrrl september webinar: cell-level security
DESCRIPTION
On-demand slides provide a technical overview of the open source, NoSQL database Apache Accumulo. We will discuss how Accumulo was born out of the security and performance needs of the National Security Agency (NSA) and cover the concept of "cell-level security".TRANSCRIPT
sqrrl Secure.'Scale.'Adapt.'
Sqrrl Data, Inc. All Rights Reserved
Sqrrl,&Apache&Accumulo,&and&Cell3Level&Security&
Adam'Fuchs,'CTO'Sqrrl'Data,'Inc.'
September'12,'2013'
2'Sqrrl Data, Inc. All Rights Reserved
• Introduc@on'to'Sqrrl'and'Accumulo'
• Security'In'The'Wild'
• Sqrrl'and'Accumulo'Technology'
• The'DataECentric'Security'Ecosystem'
• How'to'Learn'More'
Outline
3'Sqrrl Data, Inc. All Rights Reserved
• What'is'your'level'of'familiarity'with'Accumulo?'a. Only'heard'of'it'b. Well'read'on'it'
c. Downloaded'and'experimented'with'it'
d. Currently'using'it'
Poll
4'Sqrrl Data, Inc. All Rights Reserved
Who We Are
Google’s(BigTable(Paper(
2006&
NSA(Builds(Accumulo(
2008&
NSA(Open(Sources(Accumulo(
2011&
Sqrrl(Founded(2012&
3rd(Sqrrl(Release(September&2013&
Investors
5'Sqrrl Data, Inc. All Rights Reserved
Security'
Adap@vity'Scalability'
The Value of Sqrrl and Accumulo
6'Sqrrl Data, Inc. All Rights Reserved
• Introduc@on'to'Sqrrl'and'Accumulo'
• Security'In'The'Wild'
• Sqrrl'and'Accumulo'Technology'
• The'DataECentric'Security'Ecosystem'
• How'to'Learn'More'
Outline
7'Sqrrl Data, Inc. All Rights Reserved
Secure Data Lake Concept
NonEsensi@ve'data'Sensi@ve'data'Highly'sensi@ve'data'Highly'sensi@ve'data'Sensi@ve'data'NonEsensi@ve'data'NonEsensi@ve'data'
Real>Time(Apps(
Sqrrl(Enterprise(
8'Sqrrl Data, Inc. All Rights Reserved
Healthcare Security Requirements
PII(
SensiBve(Diagnoses(
Doctor’s(Notes(
9'Sqrrl Data, Inc. All Rights Reserved
• Introduc@on'to'Sqrrl'and'Accumulo'
• Security'In'The'Wild'
• Sqrrl'and'Accumulo'Technology'
• The'DataECentric'Security'Ecosystem'
• How'to'Learn'More'
Outline
10'Sqrrl Data, Inc. All Rights Reserved
Sqrrl Enterprise Built on Apache Accumulo
Sqrrl(Server(
Bulk%Processing%Integra2on%
Exploratory%/%Opera2onal%Apps%
Graph%+%Document%I/O%
Sqrrl&API&over&Apache&ThriC&RPC&(JSON,(Graph,(AggregaBon,(Search,(etc.)(
• Sqrrl(proprietary(• Automated(indexing(• Custom(iterators(• Lucene(integraBon(• Security(extensions( Accumulo&RPC&
(Sorted(Key/Value(I/O)(
Hadoop&RPC&(File(I/O)(
• Open(source((including(Sqrrl(contribuBons)(
• Open(source(or(commercial(distribuBons(
11'Sqrrl Data, Inc. All Rights Reserved
Hadoop(Distributed(File(System((commercial(or(open(source)(
Commodity(Hardware(( Private(Cloud( Public(Cloud(
Sqrrl(AnalyBcs(Sqrrl(Security(
Apache(Accumulo(
Data(Structures(
Languages(
Processing(
Sqrrl(Data(Loaders( Lucene(
Documents((JSON)(
EncrypBon>At>Rest(
EncrypBon>In>MoBon(
Audit(
IdM(IntegraBon( Indexing(Tools(
Policy(&(Labeling(Engines(
Sqrrl&Enterprise&
SQL(Subset(
MapReduce(Connector(
Pig(Connector(
(Sqrrl(Iterators(
User(Interface(D3(Demos(
Flume(
ThriY(API(
Sqrrl(Ingest(
Graphs(
Sqrrl Architecture
12'Sqrrl Data, Inc. All Rights Reserved
An(Accumulo(key(is(a(5>tuple,(consisBng(of:((
" Row:(Controls'Atomicity'" Column(Family:(Controls'Locality''" Column(Qualifier:((Controls'Uniqueness'" Visibility(Label:((Controls'Access'" Timestamp:((Controls'Versioning'
Row( Col.(Fam.( Col.(Qual.( Visibility( Timestamp( Value(
John'Doe' Notes' PCP' PCP_JD' 20120912'Pa@ent'suffers'from'an'acute'…'
John'Doe' Test'Results' Cholesterol' JD|PCP_JD' 20120912' 183'
John'Doe' Test'Results' Mental'Health' JD|PSYCH_JD' 20120801' Pass'
John'Doe' Test'Results' XERay' JD|PHYS_JD' 20120513' 1010110110100…'
Accumulo(Key/Value(Example(
Accumulo Data Format
13'Sqrrl Data, Inc. All Rights Reserved
Accumulo Technology
InEMemory'Map'
Write'Ahead'Log'
(For'Recovery)'
Sorted,'Indexed'File'
Sorted,'Indexed'File'
Sorted,'Indexed'File'
Tablet(Data(Flow(
Reads%Iterator'Tree'
Minor%Compac2
on%
Merging%/%Major%Compac2on%
Iterator'Tree'
Writes% Iterator'Tree'
Scan%
Fate'States'
Tablet'Server'
Tablet'
Tablet'Server'
Tablet'
Tablet'Server'
Tablet'
Applica@on'
Zookeeper'
Zookeeper'
Zookeeper'
Master'
HDFS'
Read/Write%
Store/Replicate%
Assign/Balance%
Delegate%Authority%
Delegate%Authority%
Applica@on'
Applica@on'
14'Sqrrl Data, Inc. All Rights Reserved
Table Design Patterns
Table:(
Row:(
Column(Family:(
Column(Qualifier:(
Value:(
Forward(Index(
<UUID>(
<Type>(
<Field>(
<Term>(
Inverted(Index(
<Term>(
<Type>(+(<Field>(
<UUID>(
<Digest(of(Event>(
Table:(
Row:(
Column(Family:(
Column(Qualifier((Tuples):(
Value:(
Shard(Table(
<ParBBon(ID>(
“Docs”( “Inv.(Index”( “Field(Index”(
<UUID>(
<Value>(
<Term>(
<UUID>(
<Field:Term>(
<UUID>(<Field>(
“Geo”(
<Hash>(
<UUID>(
Event( Term(
Ingest(Process( Query(Process(
Indexed(Event(Table(
Event(Columns(
Index(Columns(
Event(Columns(
Index(Columns(
Event(Columns(
Index(Columns(
Event(Columns(
Index(Columns(
Event(Columns(
Index(Columns(
15'Sqrrl Data, Inc. All Rights Reserved
• Introduc@on'to'Sqrrl'and'Accumulo'
• Security'In'The'Wild'
• Sqrrl'and'Accumulo'Technology'
• The'DataECentric'Security'Ecosystem'
• How'to'Learn'More'
Outline
16'Sqrrl Data, Inc. All Rights Reserved
Data-Centric Security
Row Col Value 1 Name Jones
1 Sales 100
1 Age 28
2 Name Smith
2 Sales 350
2 Age 25
2' Quota' 1000'
Row Col Value 1 Name Anon1
1 Sales 100
2 Name Smith
2 Sales 350
2' Age' 25'
2' Quota' 1000'
User 1 User 2 Sqrrl/(
Accumulo(
DefiniBon:'Data'carries'with'it'informa@on'that'is'required'to'make'policy'decisions'on'its'releasability.'
17'Sqrrl Data, Inc. All Rights Reserved
Security for Transformed Data
Logs/Observa@ons'
Input'
Indexes'
Ques@onEFocused'Datasets'
Transforma@
on'
Simple(Provenance:(Row(+(Column(Security(OK(
Complex(Provenance:(Cell>Level(Security(Needed(
18'Sqrrl Data, Inc. All Rights Reserved
Security
Row( Col.(Fam.( Col.(Qual.( Visibility( Timestamp( Value(
John'Doe' Notes' PCP' PCP_JD' 20120912'Pa@ent'suffers'from'an'acute'…'
John'Doe' Test'Results' Cholesterol' JD|PCP_JD' 20120912' 183'
John'Doe' Test'Results' Mental'Health' JD|PSYCH_JD' 20120801' Pass'
John'Doe' Test'Results' XERay' JD|PHYS_JD' 20120513' 1010110110100…'
Example(Accumulo(Key/Value(Pairs(
Accumulo&is&the&only&NoSQL&database&with&cell3level&access&controls&
19'Sqrrl Data, Inc. All Rights Reserved
• JSON'maps'to'document'and'index'entries,'all'of'which'preserve'the'security'labels'
• Labels'follow'the'document'hierarchy'• The'label'is'part'of'the'name'of'the'field'(affects'uniqueness)'
JSON Document Security Labels
20'Sqrrl Data, Inc. All Rights Reserved
Data-Centric Security Ecosystem
Data( Labeler( Sqrrl(Enterprise(
Apps(
User(Acributes(
Audits(
Policies(
End(Users(
Auth.(Service(
Policy(Engine(
Key(Mgmt(
21'Sqrrl Data, Inc. All Rights Reserved
• Introduc@on'to'Sqrrl'and'Accumulo'
• Security'In'The'Wild'
• Sqrrl'and'Accumulo'Technology'
• The'DataECentric'Security'Ecosystem'
• How'to'Learn'More'
Outline
22'Sqrrl Data, Inc. All Rights Reserved
hip://accumulo.apache.org'
Current'Developer'Base:'• 18'Commiiers'• 39'Addi@onal'Recognized'
Contributors'Contribu@ng'Organiza@ons:'
'Sqrrl,'Koverse,'Basis,'Hortonworks,'Praxis,'NSA,'Texeltek,'Objec@ve'Solu@ons,'Booz'Allen'Hamilton,'SW'Complete,'Endgame,'SRA,'Peterson'Technologies,'Cloudera,'Agile'Technology'Group,'Data'Tac@cs,'Tetra'Concepts,'JHU/APL,'Applied'Technical'Systems,'and'more.'
Accumulo Community Growth
23'Sqrrl Data, Inc. All Rights Reserved
• Download'our'White'Paper'– www.sqrrl.com/whitepaper'
• Watch'a'video'– www.sqrrl.com/downloads#videos'
• Request'a'demo'or'oneEonEone'workshop'– 'www.sqrrl.com/contact'
• Come'meet'us'– Strata'Rx'(Sept'25E27,'Boston)'– Hadoop'World'(Oct'28'–'30,'New'York)'
How To Learn More
24'Sqrrl Data, Inc. All Rights Reserved
Thank you
Thanks(for(acending!(
To(keep(up(to(date(with(Sqrrl,(check(out(or(social(media(sites:(www.twicer.com/sqrrl_inc(
www.linkedin.com/company/sqrrl((