amazon s3 storage engine plugin for mysql
TRANSCRIPT
A Storage Engine for Amazon S3
How to store a hundred billion BLOBs without buying a single disk.
Who is this guy?
http://fallenpegasus.com/code
Storage Engine
Protocol Translator
TraditionalStorage Engines
use the local disk
Network Storage Engines
Federated
ODBC
HTTP
MemCacheD
and...
S3
What is S3?
Petabytes?!
This can't be free
Why do I want to use it?
“An empty disk costs the same as a full one.”
Can I move myexisting database over?
Translating S3 to MySQL• AWS Account → CREATE SERVER
• S3 Bucket → Table
• S3 Item → Row
• S3 Item Key → VARCHAR PRIMARY KEY
• S3 Item Contents → BLOB or VARCHAR
CREATE SERVER 'MyAWSAcct'
FOREIGN DATA WRAPPER 'AWS'
OPTIONS
(USER 'aws id string',
PASSWORD 'aws secret string');
CREATE TABLE 'bierce' (
'word' VARCHAR(255) NOT NULL PRIMARY KEY,
'defn' BLOB)
CHARSET=utf-8
ENGINE=AWSS3
CONNECTION='awss3 DevilDictionary $server MyAWSAcct';
SELECT defn FROM bierce WHERE word='WIT';
SELECT defn FROM bierce WHERE word='WIT';
WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.
INSERT INTO bierce (word, defn) VALUES
('AUTHOR',
'One noted for confusing bitterness with humor.');
DELETE FROM bierce WHERE word='AUTHOR';
So, what's it good for?
“If you build it,they will come.”
• Saving EC2 work
• SQL CMS for S3 virtual hosting
• Huge list of persistent primary keys
• Big slow BLOBs, to join against fast local tables.
• “The Image Server Problem”
• ... and more
Transactions?
What do you mean,“No temporal guarantees”?!
No WHERE clausemeans
“MAKE MONEY FAST”for Amazon
The Future(watch me wave my hands)
• Code Improvement
• S3 & HTTP Metadata
• Multiple Data Columns
• Information Schema
• Security & Authentication
• Transfer & Storage Compression
• Streaming BLOB
• More Storage Engines
Code Improvement
S3 & HTTPMetadata
Multiple Data Columns
Sharing a solution withHTTP and MemCacheD
Engines
Information Schema
Security &Authentication
Transfer & StorageCompression
(new slide)Streaming BLOBs
like PBXT
More storage engines
More storage engines• “A Storage Engine for AWS SQS”
• There are more AWS services rolling out
• Replication distribution via S3
• Cluster persistent storage via S3
• ... and the EC2 holy grail:
“A Generic SchemaStorage Engine for S3”
Back to the present...
“Where do I get it?”
“Where do I get it?”
• MySQL 5.1
• GPL
• http://fallenpegasus.com/code
“You never write,you never call...”
• Suggestions
• Use Cases
• Bug Reports
• Patches
• Feature Requests
• Support Requests
• Gigs & Proposals
Questions?