SCALE YOUR DATA COLLECTION ON THE CLOUD LIKE A CHAMPMoty Michaely, VP R&D Xplenty

SCALING DATA COLLECTION = A PAINPlenty of companies are limited by their data collection methods when it comes to scalability.

Once they need more detailed data and in larger quantities, scaling the system can become a major pain.

THREE COMMON METHODS FOR COLLECTING BIG DATA... IS YOUR COMPANY USING THE RIGHT ONE?

▪ Storing directly in the DB

▪ Keeping it in a local file

▪ S3/CloudFront logging

STORING DIRECTLY IN THE DBThis is what companies usually start with. As the name suggests, data is inserted right into the DB.

There are two ways to do it:

▪ Row by row means the data is added as a row to the DB in real time.

▪ Bulk insert adds multiple rows to the DB in one transaction. (It’s faster than row by row, but insertion of the entire batch may fail, thus having to re-insert a big chunk of data.)

PROS FOR STORING DIRECTLY IN THE DB▪ Better performance than other methods for inserting data.

▪ Real-time data available when adding row by row.

CONS FOR STORING DIRECTLY IN THE DB▪ Schema changes are required to add new types of data.

▪ Scaling is required in two layers - application and database. Scaling the application is usually easier (using a network load balancer for example) but scaling the database requires hiring an expert DBA, partitioning the DB, and scaling up the server. (Relational DBs that scale out to multiple nodes are expensive and require a lot of maintenance.)

BOTTOM LINEStoring directly in the DB gives you fast performance, but it doesn’t scale.

KEEPING IT LOCALData is dumped in big local files. These files are periodically uploaded via a program to S3 or inserted in batches into a NoSQL DB, such as Amazon DynamoDB or a data warehouse like Amazon RedShift.

PROS FOR KEEPING IT IN A LOCAL FILE▪ New types of data can be added easily since no schema

changes are required.

▪ Compatible with all applications because any file format can be used.

▪ Quicker filtering via customized directory/file names, e.g. with date/time indication.

CONS FOR KEEPING IT IN A LOCAL FILE▪ One needs to develop a tracking program to deal with the

files - rotating logs while more data is incoming, handling failures, and transactionality. Even if you have the manpower, time, and money, it’s hard to develop such a program.

▪ Scaling means adding more servers, more maintenance, and more money.

▪ Data is not as query-able compared to storage in a DB.

▪ Staging and production environments require extra servers.

BOTTOM LINEMore flexible than direct DB storage, but requires more development, and scaling is still an issue.

S3/CLOUDFRONT LOGGINGThis old school solution goes back to the early days when visitor counters and burning “hot!” animations ruled the web. To track an event, an HTTP request is sent for a 1x1 pixel image from a relevant S3 directory. Accessing the image automatically generates a W3C log with all HTTP request parameters: IP address, browser, date/time, etc. Extra session level data like username or mouse position is passed via the query string. To differentiate between event types, images are placed in accordingly named directories, e.g. /click/.

PROS FOR S3/CLOUDFRONT LOGGING▪ No tracking server required - data reaches S3 automatically.

▪ No file management - Amazon handles all file monkey business.

▪ No servers - Amazon provides them.

▪ Cost effective - only log storage and bandwidth are paid for. The logs take little space since they are all GZipped and the bandwidth for 1x1 pixel images is marginal.

PROS FOR S3/CLOUDFRONT LOGGING CONTINUED

▪ Easily scalable with practically infinite space and firepower.

▪ Quick and easy to implement.

▪ Simple setup for staging/production environments via additional distributions and a prefix.

▪ Web application performance unharmed, especially using the CloudFront CDN.

CONS FOR S3/CLOUDFRONT LOGGING▪ Slower filtering performance compared to local setup. Amazon handles

log file/directory names automatically and no customization is available.

▪ Not suitable for real time or impatience. Data is aggregated into a new file in the bucket only once per hour, and that’s Amazon’s best effort so it could take longer.

▪ Data is not as query-able compared to storage in a DB.

▪ Vendor dependent. Having your servers outside of Amazon will decrease performance.

▪ No control over the file format. W3C Extended Log File Format is mandatory and some applications may not like that.

BOTTOM LINEQuick, cheap, and scalable though it doesn’t provide the best performance and customization.

WHAT’S RIGHT FOR YOU?So much emphasis has been put on the technologies used for processing, analyzing, and visualizing data. But so often getting lost in the shuffle is the importance of the collection of this data. The two go hand in hand. To get good output from your data, you must first have proper input. Only once you have achieved the synergy between the two will you fully be able to tap into your data’s potential.

XPLENTYWWW.XPLENTY.COM