big data forum 19 march 2014
TRANSCRIPT
What about Little Data? Big Data Forum Lightning Brief
Matthew Carroll, GM of 42six March 19, 2014
Who the heck are we?
2!
Technologies
Accumulo (active contributor) Ozone (active contributor) Storm Niagara Files Apache Kafka Titan (active contributor) CouchDB MongoDB (active contributor) Cassandra HBase Neo4J OpenStack Puppet
Programs
RTRG Intelink Kickawesome * Ozone & Apps Mall * Ozone Mobile (DISA) * Red Disk (Army-INSCOM) * ORION (DIA) FireTruck (DIA-DCTC) Coral Reef (Army/NMEC)
WE ARE CSC’S BIG DATA SERVICES ARM PROVIDING OPERATIONALLY FOCUSED SOLUTIONS AND CONSULTING TO THE U.S. INTELLIGENCE COMMUNITY, INTERAGENCY & PRIVATE SECTOR.
Big Data’s Little Brother
3!
Big Data is what organizations know about entities — be they people, places, things, etc. Data is aggregated from a large number of sources, assembled into a massive data store, and analyzed for patterns. The results are more accurate predictions, more targeted communications, and more personalized services. Good For: General questions, recommendations for all users, SNA, anomaly detection
Little Data is what we know about ourselves. What we search. Who we know. What we care about. How we spend our time. We’ve always had a sense for these things — after all, it’s our jobs. But thanks to the combination of social and cloud technologies, it’s easier than ever to gain insight into our own behavior. Good For: Individual recommendations, personal goals, efficiency analysis, individual pattern detection
Big Data gets all the attention but there is value, if not more, in Little Data…
So how do I collect Little Data?
4!
AS THE GOVERNMENT MIGRATES TO DISTIRBUTED SYSTEMS, SPECIFICALLY PaaS, ENGINEERING TEAMS NEED TO FOCUS ON INDIVIDUAL BASED LOGGING & ANALYTICS.
1. Simple APIs to insert logging of user activity including time in app, how many searches executed, common search terms, etc.
2. Insert “actions” into traditional web apps like was this report interesting or value percentage.
3. Hook into task management systems when you can…track what the user is working on, where they are and for how long.
4. Build personalized analytics pages with time and space visualizations to help users see context in their activity
Some Examples
5!
Graph Clustering: What are the groups that make up my activity?
Geospatial: What are the locations associated with my activity?
Temporal Activity: When are you active and what are you doing?
Textual: What do you search, what are your common topics?
Ok I have my Little Data
6!
ONCE HOOKED INTO THE ARCHITECTURE AND USERS HAVE ACCESS TO THEIR DATA, SETUP PERSONALIZED ANALYTIC ENGINES FOR REMINDERS AND AUTOMATED ALERTING
1. Design GUI for event-based processing first, i.e. node.js, etc. Think through dynamic updates based on user actions not necessarily on new data in the system
2. Think like IFTTT – set the stage for the user to define desired personalized goals, e.g. 20% search, 50% read, 30% write.
3. Design data models with users in mind. Index data in a way to key off of users. Treat users as first class citizens.
4. Think about useful information to guide general user experience, i.e. reminder that you typically search for “X” term when in this app on Fridays.
