m/db and m/db:x

GT.M: Rock Solid, Lightning Fast

M/DB & M/DB:X

Open Source NoSQL Projectsbased on GT.M

Rob Tweed: M/Gateway Developments Ltd


M/DB & M/DB:X

• What are they?

• Why might you use them?

• What is GT.M?

• Why are they based on GT.M?

• What should you be interested in GT.M?


M/DB

– Free Open Source alternative to SimpleDB– API-compatible with SimpleDB

• Behaves identically• Accessed identically• Just a different endpoint URL

– Designed for:• Alternative Public Cloud to Amazon• Private Cloud• Local use


Why emulate Amazon SimpleDB?

• Great concept for data that fits its model– “spreadsheet-like” data structure– Simple to use– Simple to manage

• Potential for freedom from SimpleDB’s limits• Make SimpleDB model no longer exclusive to

Amazon cloud• Potential for hybrid environment


Why emulate Amazon SimpleDB?

• Great concept for data that fits its model– “spreadsheet-like” data structure– Simple to use– Simple to manage

• Potential for freedom from SimpleDB’s limits• Make SimpleDB model no longer exclusive to

Amazon cloud• Potential for hybrid environment

• Because we can…!


SimpleDB Model

• Domains: individual tables or databases• Items: rows in a table• Attributes: columns in a table• Values: cells (note: an attribute can have multiple values)


Main SimpleDB Operations

• CreateDomain• PutAttributes

• List Domains• GetAttributes• Select

• DeleteDomain• DeleteAttributes


SimpleDB v M/DB LimitsSimpleDB M/DB

No of domains 100 per account Effectively unlimited

Max domain size 10Gb 512Gb – 8Tb(depends on GT.M block-size)

Attributes Text only, 1k max size Dynamic data type,

4k – 32k max(depends on GT.M block-size)

Cost Pay as you go, can get expensive!

Free Open Source

Consistency Eventually Consistent Immediately Consistent

Max query execution time 5 seconds Unlimited

Responses XML over HTTP XML or JSON over HTTP

Web Server(Apache)

SimpleDBClient

m_apache gateway + xinetd

GT.M

M/DBSecurity

M/DBMethods

Ubuntu (or other) Linux

Physical or Virtual Machine

HTTPRequest

XMLResponsecreation

HTTPResponse

M/DBDatabase

m_pythongateway

PythonM/DB APIs


M/DB: Availability

• Free Debian Installer at www.mgateway.com– Installs entire architecture

• GT.M• Apache & m_apache• M/DB

• Ubuntu Enterpr– Private cloud

• Eucalyptus used to emulate Amazon APIs

– M/DB now available in the UEC Image Store• Adds SimpleDB API to UEC’s emulation of Amazon cloud


M/DB Client Support

• Many SimpleDB Clients already support M/DB:– Bolso– Mindscape's SimpleDB Management Tools and

LightSpeed – Boto (Python client)– Others?

• Just requires ability to change the endpoint URL

• M/DB emulates SimpleDB API precisely


M/DB:X


M/DB:X

• Simple Native XML Database– Persistent XML DOM model– Multiple XML documents

• API based on SimpleDB– Uses SimpleDB security– W3C XML DOM API, expressed as:

• HTTP request, XML response• JSON over HTTP

– XPath, also expressed as:• HTTP request, XML response• JSON over HTTP


M/DB:X

• Creating DOMs:– Parse an XML document

• Non-validating parser

– JSON input• Converted to corresponding XML

– Build programmatically using DOM API methods, eg• createDocument• createElement• createTextNode• appendChild• insertBefore


M/DB:X

• Modifying DOMs:– programmatically using DOM API methods, eg

• createElement• createTextNode• appendChild• insertBefore• setAttribute• removeChild• removeAttribute• copyNode


M/DB:X

• Searching DOMs:– programmatically using DOM API methods, eg

• getElementById• getElementsByTagName• getAttribute• getFirstChild• getNextSibling

– XPath• Individual DOM• Across multiple DOMs


M/DB:X

• Deleting DOMs:– removeDocument– Otherwise a DOM will persist indefinitely


M/DB:X

• Identifying DOMs:– Each DOM has a unique:

• DocumentName• DocumentOID

• Identifying Nodes:– Each node in a DOM has:

• nodeType• nodeOID (unique across all DOMs)


M/DB:X

• Typical API call (XML)– Request:

db=mdbx &Action=SetAttribute &AttributeName=width &AttributeValue=100px &ElementOID=1-3 &MDBAccessKeyId=[your access key id] &Signature=[computed signature] &SignatureMethod=HmacSHA256 &SignatureVersion=2 &Timestamp=2009-06-14T13%3A19%3A41 &Version=2009-04-15

– Response:<?xml version='1.0'?> <SetAttributeResponse xmlns="http://mdbx.mgateway.com/doc/2009-06-05/" />


M/DB:X

• Typical API call (JSON)– Request:

db=mdbx &

Action=SetAttribute &

JSON={AttributeName:”width”,AttributeValue:”100px”, ElementOID:”1-3”} &

MDBAccessKeyId=[your access key id] &

Signature=[computed signature] &

SignatureMethod=HmacSHA256 &

SignatureVersion=2 &

Timestamp=2009-06-14T13%3A19%3A41 &

Version=2009-04-15

– Response:{ok:true}


M/DB:X Hybrid JSON/XML Database

• JSON input:{animals:{dog:[{name:"Rufus",breed:"labrador"},{name:"Marty",breed:"whippet"}], cat:

{name:"Matilda"}}}

• XML representation in DOM<animals>

<dog><name>Rufus</name><breed>labrador</breed>

</dog><dog>

<name>Marty</name> <breed>whippet</breed>

</dog><cat>

<name>Matilda</name></cat>

</animals>

• Output as:– XML– JSON


M/DB:X: Use cases

• XML meets SimpleDB and CouchDB!• Lightweight XML “scratchpad”• Simple, easy to use XML DOM• Perfect for storage/manipulation of

hierarchical data• If you get XML documents in and send out

XML documents, why convert to/from some database format?– Store as XML


M/DB:X Client Support

• None available to my knowledge

• XML API based on SimpleDB model + SimpleDB security– Should be simple to adapt existing SimpleDB

clients


M/DB:X availability

• Bundled with M/DB– Install M/DB and you also have M/DB:X


M/DB and M/DB:X – Underlying technology

• GT.M


What is GT.M?

• Open source database

• Runs on Linux

• Schema-free, key-pair database

• Schema implicit in application

• Persistent dynamic sparse arrays– Alphanumeric subscripts

^employee(“london”,”mgt”,1234)=“Rob Tweed”


M/DB and M/DB:X – Why GT.M?

• Why GT.M was the ideal technology:– Flexibility of its database structure:

• Very simple to represent SimpleDB model and its indices

• Very simple to represent XML DOM• No pre-declared schema• Build/modify “on the fly”

– No pre-declared array size– Completely dynamic– Automatic garbage collection



• Why GT.M was the ideal technology:– Free Open Source– Very high performance database– Highly scalable:

• from netbook to replicated network of high-end servers

– Rock-solid reliability• Its pedigree is in banking & healthcare systems since mid-

1980s• It runs the world’s largest core banking system

– >17 million accounts



• So GT.M is no toy!– It’s a serious, well tried and tested enterprise-

grade database

• M/DB and M/DB:X can make use of GT.M’s system management facilities:– Backup– Journalling– Replication (multi-tier)


Why you should check out GT.M?

• “Swiss Army Knife of Databases”• Underlying persistent sparse array model:

– Can be used to represent many other logical database structures:

• Tables• Hierarchicial• XML• Relational• Object

• Own built-in language, directly linked to database• Or can use most other languages, eg:

– Python– Java– Ruby, etc


Why you should check out GT.M?

• Speed of development:– M/DB took 1 week to write and release– M/DB:X about the same

• Documentation was the most time-consuming task!

• Open source– Can redistribute freely on Linux– No limitations imposed– Full version, full functionality


Find out more

• M/DB & M/DB:X:– http://www.mgateway.com

• GT.M: – http://www.fis-gtm.com

http://www.mgateway.com/

http://www.fis-gtm.com/

m/db and m/db:x

Technology