big data and your career final
TRANSCRIPT
Big Data and Your Career Beyond the Classroom
Marina Kerbel, Principal Information Architect at Deluxe Corporation
November 12, 2013
Big Data Job Boom?
What is Big Data?
Data that is obtained, processed, and analyzed using emerging technologies that can get any data in any format from any place and interpret it to get value.
‘hear what someone is whispering backstage while you're attending a booming outdoor concert.’ *
* Source: TechRepublic 'Big Data basic concepts and benefits explained
Big Data – Hype vs. RealityHype Reality
1. What makes the data ‘Big Data’?
• Very large data sets
• New technologies that help to obtain and process diverse, complex and dispersed data.
2. Who is using Big Data?
• Internet giants like Google
• Companies of all sizes and from all industries
3. Why do companies use Big Data?
• Better decision making
• Business - Customer Experience, Operational Efficiency
• IT - Agility, Lower Cost
4. What data is being used?
• Social and mobile data • Core transactional data
5. How do architecture & technology change?
• Hadoop revolution
• Incremental adds of Hadoop and other new technologies
6. What skills are in-demand?
• Data Scientists with Ph.D. in Statistics
• Big Data Architects, Software Engineers, or Business Analysts with a mix of old and new skills
What makes the data ‘Big Data’?
Source: IBM ‘Big Data: Making the World go Round’
Big Data Platform
helps reliably store, access and analyze any data regardless
of how fast it is moving, what type
it is, or where it resides. It's today's
emerging technology that will help make
sense of the Big Data Explosion.
Who is using Big Data?• Gartner 2013 Big Data Survey Reveals That 64
Percent of Organizations Have Invested or Plan to Invest in Big Data in 2013
• …The industries with the most planned investments over the next two years are transportation, with 50% planning to invest in big data technology, followed by healthcare at 41% and insurance companies at 40%.
• However, every vertical industry again shows big data investment and planned investment…..
Source: Gartner ‘Press Release - Gartner Survey Reveals That 64 Percent of Organizations Have Invested or Plan to Invest in Big Data in 2013’
Who is using Big Data?
Source: Forrester ‘Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012’
Why do companies use Big Data?
Source: 2012 IBM and Said Business School study 'Analytics: the real world use of big data'
Why do companies use Big Data?
Source: Forrester Blogs - Brian Hopkins' Blog
I've found, in my direct interactions with firms, that leaders know the real value of big data is lower cost and greater agility. When I interviewed 11 firms with production-class big data implementations, all of them told me the same thing — they got into big data when they couldn't figure out how to accomplish what their business wanted in an affordable way with their existing technology.
What data is being used?
Source: Forrester ‘Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012’
How do architecture & technology change?
Sour
ces
Data
Sy
stem
sAp
plic
ation
s
Traditional RDBMS Repositories(Oracle, DB2, SQLServer)
Traditional Sources(Transactional data - CRM, ERP,
custom applications)
New Sources(Interaction data - Social Media, Web Logs , Emails, Video, Images, Sensors)
Analytics
OLTP EDW
Custom Applications
Packaged Applications
Hadoop: MapReduce &
HDFS
NoSQL(HBase, MongoDB)InMemory(SAP Hana)
Use Case 1 - Implement Bill Payment service where customers can enter variable biller data in any order Who was using Big Data?
Sales & Operations (Order Capture & Management)
Why Big Data was used?
Customer Experience, Operational Efficiency, Agility
What data was used?
Transactional data from corporate custom build application
How did architecture & technology change?
MongoDB added on front-end to allow capturing data incrementally and in variable format
What skills were in-demand
Programming, Databases, Data Modeling
Use Case 2 - Enable personalized campaigns that leverage prospect data from diverse sources
Who was using Big Data?
Marketing (Personalized Campaign)
Why Big Data was used?
Agility, Lower Integration Cost
What data was used?
Proprietary Prospect data + Third party data + Social Network data
How did architecture & technology change?
IBM BigInsighs and Amazon Web Services used to integrate and analyze data from diverse, frequently changing sources
What skills were in-demand
Integration, Cloud, Analytics, BI, Visualization
Use Case 3 - Leverage data from dropped website visits to entice customers to complete orders
Who was using Big Data?
Sales (User Experience)
Why Big Data was used?
Customer Experience, Performance
What data was used?
Transactional data from corporate packaged application + Weblog data from public facing site
How did architecture & technology change?
SAP Hana and Amazon Web Services used to integrate data real-time
What skills were in-demand
Integration, Cloud, Data Modeling
What skills are in-demand?Software Engineers
Data Scientists
Business Analysts
Technical Business
Capture, IntegrateDiscover, Explore
Analyze, Present
Programming Databases
Data Modeling Cloud
Integration
Math Statistics Curiosity
Data Story Analytics, BI
Visualization Business Acumen
Big Data Architects
What skills are in-demand?
Source: Talent Neuron 'Understanding Big Data Skills Taxonomy'
What skills are in-demand?
Source: Data Science Central ‘Top Big Data Skills in Demand’ posted by Fari Payandeh
Immediate Needs
What skills are in-demand? Long-Term Skills• As Big Data matures, too much focus on a specific
product might be counterproductive• The skills needed today and tomorrow:
– Foundational data competencies (Data Modeling, Databases, Programming)
– Knowing when to use traditional standards and when to break them
– Some knowledge of math, statistics, analytics – Business acumen and ability to see the big picture – Ability to quickly learn various new approaches and
technologies
How to Acquire Skills - Examples – Foundational data competencies – C.S. classes– Knowing when to use traditional standards and when to
break them – published vendor documentation – Some knowledge of math, statistics, analytics – elective
classes– Business acumen and ability to see the big picture – MBA
classes, webinars and publications with business use cases from major vendors, on-the-job experience
– Ability to quickly learn various new approaches and technologies – many free education sites and downloads such as http://bigdatauniversity.com/
What skills are in-demand?
Big Data – Hype vs. RealityHype Reality
1. What makes the data ‘Big Data’?
• Very large data sets
• New technologies that help to obtain and process diverse, complex and dispersed data.
2. Who is using Big Data?
• Internet giants like Google
• Companies of all sizes and from all industries
3. Why do companies use Big Data?
• Better decision making
• Business - Customer Experience, Operational Efficiency
• IT - Agility, Lower Cost
4. What data is being used?
• Social and mobile data • Core transactional data
5. How do architecture & technology change?
• Hadoop revolution
• Incremental adds of Hadoop and other new technologies
6. What skills are in-demand?
• Data Scientists with Ph.D. in Statistics
• Big Data Architects, Software Engineers, or Business Analysts with a mix of old and new skills
Questions
References and Resources• TechRepublic 'Big Data basic concepts and benefits explained' • Gartner ‘Information 2020: Evolving Beyond Big Data’ (Free Webinar) • Gartner ‘Press Release - Gartner Survey Reveals That 64 Percent of
Organizations Have Invested or Plan to Invest in Big Data in 2013’• IBM ‘Big Data: Making the World go Round’• IBM ‘Analytics: The real-world use of big data’• Forrester ‘Forrsights Strategy Spotlight: Business Intelligence And Big
Data, Q4 2012’• Forrester Blog: Brian Hopkins• Talent Neuron 'Understanding Big Data Skills Taxonomy'• Data Science Central ‘Top Big Data Skills in Demand’ • Big Data University • MongoDB Introduction• MongoDB Data Modeling