amazon s3 storage engine plugin for mysql

A Storage Engine for Amazon S3

How to store a hundred billion BLOBs without buying a single disk.

Who is this guy?

http://fallenpegasus.com/code

Storage Engine

Protocol Translator

TraditionalStorage Engines

use the local disk

Network Storage Engines

Federated

ODBC

HTTP

MemCacheD

and...

S3

What is S3?

Petabytes?!

This can't be free

Why do I want to use it?

“An empty disk costs the same as a full one.”

Can I move myexisting database over?

Translating S3 to MySQL• AWS Account → CREATE SERVER

• S3 Bucket → Table

• S3 Item → Row

• S3 Item Key → VARCHAR PRIMARY KEY

• S3 Item Contents → BLOB or VARCHAR

CREATE SERVER 'MyAWSAcct'

FOREIGN DATA WRAPPER 'AWS'

OPTIONS

(USER 'aws id string',

PASSWORD 'aws secret string');

CREATE TABLE 'bierce' (

'word' VARCHAR(255) NOT NULL PRIMARY KEY,

'defn' BLOB)

CHARSET=utf-8

ENGINE=AWSS3

CONNECTION='awss3 DevilDictionary $server MyAWSAcct';

SELECT defn FROM bierce WHERE word='WIT';

SELECT defn FROM bierce WHERE word='WIT';

WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.

INSERT INTO bierce (word, defn) VALUES

('AUTHOR',

'One noted for confusing bitterness with humor.');

DELETE FROM bierce WHERE word='AUTHOR';

So, what's it good for?

“If you build it,they will come.”

• Saving EC2 work

• SQL CMS for S3 virtual hosting

• Huge list of persistent primary keys

• Big slow BLOBs, to join against fast local tables.

• “The Image Server Problem”

• ... and more

Transactions?

What do you mean,“No temporal guarantees”?!

No WHERE clausemeans

“MAKE MONEY FAST”for Amazon

The Future(watch me wave my hands)

• Code Improvement

• S3 & HTTP Metadata

• Multiple Data Columns

• Information Schema

• Security & Authentication

• Transfer & Storage Compression

• Streaming BLOB

• More Storage Engines

Code Improvement

S3 & HTTPMetadata

Multiple Data Columns

Sharing a solution withHTTP and MemCacheD

Engines

Information Schema

Security &Authentication

Transfer & StorageCompression

(new slide)Streaming BLOBs

like PBXT

More storage engines

More storage engines• “A Storage Engine for AWS SQS”

• There are more AWS services rolling out

• Replication distribution via S3

• Cluster persistent storage via S3

• ... and the EC2 holy grail:

“A Generic SchemaStorage Engine for S3”

Back to the present...

“Where do I get it?”

“Where do I get it?”

• MySQL 5.1

• GPL

• http://fallenpegasus.com/code

“You never write,you never call...”

• Suggestions

• Use Cases

• Bug Reports

• Patches

• Feature Requests

• Support Requests

• Gigs & Proposals

mark@fallenpegasus.com

Questions?