big data challenges - fcw.com
TRANSCRIPT
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
SIZING UPBIG DATA: 4 WAYS TO SUCCEED
BIG DATAâS SECURITY IMPERATIVE
BIG DATA HAS BIG POTENTIAL IN THE CLOUD
BIG DATA GOES MOBILE
VISUALIZATION DRIVES HOME BIG DATAâS VALUE
P2 P4 P5 P8 P9
ONLINE REPORTSPONSORED BY:
CDW-G and EMC
Special Report
FCW SPONSORED REPORT 2
Big data has been around long enough that there have been very notable successes â and resounding failures â related to its use.
Today 81 percent of IT executives surveyed say that big data is a top-ďż˝ ve IT priority for 2013 and only 6 percent of IT shops do not have big data on their top 10 priorities list. But 55 percent of respondents also say that they have already had big data failures â projects that were not completed or fell short of their objectives.
There are myriad reasons why big data projects fail, according to a recent industry report. The top two are a lack of expertise to connect the dots and a lack of business context around data. But nearly every organization faces similar challenges, including ďż˝ nding the right tools, dealing with time constraints, understanding the platforms and training staff.
The good news is that there are things that IT executives can do ahead of time to improve their success rate and end user satisfaction related to big data. Here the top four strategies that can help organizations get the most out of their big data efforts.
1) Get the scope rightAccording to the report, 58 percent of respondents say inaccurate scope is responsible for their failed big data projects. Alex Rossino, principle research analyst at Deltek, says that the bigger and more unlimited the mission of an organization is, the more complex its data requirements will be â and consequently the more work it will take to get the scope right.
âFor example, if you think about trafďż˝ c enforcement in Suffolk County, N.Y. a big data project might require making sure that data collected on red light cameras, ticket issues and any related data is what is needed, and the rest would theoretically be discarded,â he says. âHowever, agencies run into trouble when they start with
one outcome in mind and it spirals past what they have already decided is mandatory to collect and analyze.â
Rossino suggests talking to everyone who might be affected by the analysis, from the secretary or commissioner of an agency down to the heads of ofďż˝ ces and programs. âIt needs to be discussed and then kicked over to the CIO to determine the resources that are needed to make it happen and put in writing so that no one is expecting more out of a project than has previously been discussed.â
Thatâs not to say a projectâs scope might not change at a future date. But by sticking to the agreed upon scope â at least in the beginning â agencies and organizations are more likely to ďż˝ nd success.
2) Get the business users involvedIn the case of big data, success hinges on producing information that is of value. So it only makes sense to involve the people who will be using that information.
âIT must recognize that big data means something different to every business and IT user,â says Evan Quinn, a senior principle analyst with research ďż˝ rm Enterprise Strategy Group (ESG). âThe ďż˝ rst question every agency needs to ask itself is, âWhat am I trying to get out of this. What is the value?ââ
The value should be reďż˝ ected in the queries that people make, said Mukul Krishna, director for digital media at Frost and Sullivan, a market research ďż˝ rm. People might run ad hoc queries, but on the whole the payoff comes from asking the right questions.
Itâs also important to have someone who can sift through the results and understand what it means. Some agencies might ďż˝ nd themselves adding a chief analytics ofďż˝ cer that has both business and IT knowledge so they can understand how to turn raw data into speciďż˝ c deliverables.
âData is only worth something if IT can work with
SIZING UP BIG DATA: 4 WAYS TO SUCCEED
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 3
someone who understands the agencyâs mission and can explain why it even needs a big data project to begin with,â Krishna said.
3) Hire the right talentIn an August 2012 1105 Government Information Group survey, more than half of nearly 200 government agencies reported that they are having difďż˝ culty ďż˝ nding and keeping knowledge workers and data scientists for their big data efforts. This challenge will only get worse as private and public organizations expand their big data efforts.
The difďż˝ culty, says David Loshin, president of Knowledge Integrity Inc., a consultancy that focuses on business data management advice and guidance, is that big data is a departure for most IT professionals. âThere is a major learning curve to understand the opportunities that big data affords,â he said.
Thatâs not to say every agency will need to hire new data scientists or mathematicians. Loshin suggests identifying the trainable staff within your current organization by looking for existing skills such as a love for statistics and training in computer science. A security background is also important since some of the data that your teams will be using may be classiďż˝ ed or contain personal identiďż˝ cation.
4) Size the infrastructure rightThose agencies and organizations that are doing big data analysis on-site will need to make sure that they have the necessary storage and compute power, says Loshin.
âTiered storage may work well for some since you have the capacity to ďż˝ ow data between disk and share memory,â Loshin said. âThe most important thing is to make sure that, if youâre pulling data from multiple sources, performance does not lag.â Evaluating network
bandwidth should also be on your to-do list since low latency streaming will be key to end user satisfaction.
Many organizations may end up underestimating the amount of storage they will need, said Mike Gualtieri, an analyst with Forrester Research. âYou may know how much data you have, but you might not realize that you need to duplicate everything in an analytical sandbox for those who want to do advanced analysis,â he said.
Gualtieri suggests that clustered systems can help. Still, many organizations may ďż˝ nd that cloud-based services are the easiest way to provide the scaling and elasticity needed when dealing with big data.
âFor very large data sets it might be more economic, especially if youâre launching something completely new,â he says. âYou wonât have to invest millions of dollars if youâre keeping everything in the cloud.â
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 4
When big data began catching on, the tool of choice for many organizations was Hadoop, an open source programming framework that supports the
processing of large data sets in a distributed computing environment.
Some IT experts were concerned about the security risks that came with Hadoop. Still, from the beginning companies such as Facebook and eBay turned to the technology to do what nothing else could do: Collect, aggregate, analyze, and share structured and unstructured data from a variety of disparate sources.
In fact, Hadoopâs beneďż˝ t was its biggest Achillesâ heel: When bringing together data the security clearance for one set might be completely different than anotherâs. With big data being made available to multiple stakeholders and, in some cases, constituents, there was no way to protect its sources.
Today, while Hadoop has more security features â mostly software add-ons from other developers and service providers â security remains a serious concern for any IT executive considering a big data project. They must consider the security both of data within a cluster of servers and the cluster itself.
IT is charged with setting up encryption for the data and authentication or identity management for the clusters. In addition, especially for those organizations allowing mobile or web-based access, they must ensure the security of the applications and of the data that is produced. In particular, IT must make sure that crucial and classiďż˝ ed data doesnât end up being downloaded on to a mobile device, which can be lost or stolen at any time.
Itâs everywhereAnother problem, says Forrester analyst Mike Gualtieri, is that IT might not know which data is important for a speciďż˝ c query. âSo when the line of business or end users say, âGive me all youâve got,â
IT may do so without making sure the appropriate security standards are in place.â
Gualtieri recounts a client that had attempted to âanonymizeâ medical data that was being used in predictive analytics. However, although identifying data had been scrubbed from the sets, users were still able to infer and inadvertently exposed some information that never should have been allowed to come into the public view.
âThe challenge becomes you want to give access to as much data as possible, but you still need to protect privacy,â he says.
Mobile data is even more problematic because it is not only real-time, but also attached to GPS positioning. âYouâve got to do a lot more governance when youâre dealing with so much personally identiďż˝ able information, says Gualtieri.
Cloud installations are also giving IT pause, says Evan Quinn, a senior principal analyst with ESG. When data is on premise, you have more control over the data. Once it is put into the public cloud, IT must worry about whether or not the data is on a shared resource, if the servers are inside or outside of this country, and whether or not the cloud providerâs security and governance practices could withstand an audit.
âYou need to go to the cloud provider and ask point blank to see a checklist of its practices,â he said. He also suggests ensuring that employees and, in some cases, constituents are accessing data only via secure connections.
And ďż˝ nally, IT executives need to think carefully about the accessibility of big data query results. Sometimes, employees are so excited about what theyâre seeing that they extend access to others in their department or those who are working with them on projects. âYou donât want to get any surprises and ďż˝ nd out that youâve just distributed data to 1,000 people who were never cleared to see it in the ďż˝ rst place,â said Quinn.
BIG DATAâS SECURITY IMPERATIVE
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 5
This past January, IT heavyweights from the National Institute of Standards and Technology (NIST) as well as other government agencies, industry, and
academia, got together to discuss the critical intersection of big data and the cloud. Although government agencies have been slower to adopt new technologies in the past, the event underscored the fact that the public sector is leading â and in some cases creating â big data innovation and adoption.
Cloud is a multiplier when used with other technology, and is capable of big things when it comes to big data, according to U.S. CIO Steven VanRoekel. Combining big data with cloud delivery and compute power might help create new industries and provide beneďż˝ ts to every citizen.
âThe government is sitting on a treasure drove of data,â he said during a speech at the two-day NIST Joint Cloud and Big Data Workshop event. âWeâre opening data, and looking at what we can do. We can greatly impact the lives of every American by just unlocking simple prices of data.â
He also pointed to the formation of companies built and founded completely on government data.
Nowhere to go but upResearch and consulting ďż˝ rm Deltek Inc. says cloud environments are âoptimalâ for using analytics since cloud providers are investing in the best analytical and visualization tools available today. In addition, big data projects require processing speed and the most up-to-date technologies, two other cloud provider specialties.
But most important, said Richard Blake, senior technical advisor in the Enterprise GWAC Division of the Integrated Technology Service at the General Services Administrationâs Federal Acquisition Service, is that big data in the cloud enables something that is crucial for innovation, analysis and return-on-investment: The
ability to share resources between agencies. Evan Quinn, a senior principle analyst with research
ďż˝ rm ESG agrees. âBig data is an incremental learning process and when you can share expertise and resources you can score more small wins and progress more quickly than you may have on your own,â he says.
As highlighted in Deltekâs Federal Big Data Outlook 2012-2017, data is available from a wide variety of sources, including agency data such as data logs, space telescopes, reconnaissance, citizen information, and mission critical apps. However, data is also being sourced from the outside as well â in the form of email, text messages, pictures, embedded sensors, social media, GPS data, purchase data, trafďż˝ c cams and market research. With the cloud, it becomes easier to store, analyze, and access this information.
In addition, the same data can be analyzed and used in different applications and analytic projects, since one agency no longer âownsâ that data. With these tenets in mind going forward, every piece of data will be examined as a potential resource and used as the basis of future applications, explained VanRoekel during the conference.
One promising approach, he said, is to ensure that data is machine-readable â that is, that it can move from system to system without requiring human intervention or translation.
âGovernment agencies are ordered to look at the data they produce, catalog data, start to publish data, and think about machine-readable as the new default inside government,â VanRoekel explained. âAny time weâre building a new system, or amending a system, we focus on machine readability both on the collection, as well as getting agencies to develop [application programming interfaces] around their data.â
However, as promising as it is, agencies do not want
BIG DATA HAS BIG POTENTIAL IN THE CLOUD
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 6
to rush into putting big data in the cloud. Although it might mean that IT no longer has to worry about buying, maintaining, and managing those databases and associated storage, big data is a very specialized workload, says ESGâs Quinn. Agencies must ensure that their cloud providers can handle big data and that they understand the public sectorâs security and compliance requirements, he said.
The other issue is expertise. Most if not all agencies are currently doing some form of business intelligence (BI) on their data, but what works for most organizations with BI is not going to work for big data. Expectations on a whole must be realistic and in line with what is currently technologically possible, said Quinn.
âIf any agency thinks itâs going to go in and put $1 million and six months of work into a big data cloud project and change how the agency works from the inside out they are misinformed,â he said. âFor most organizations this is a completely new discipline so they need to think in terms of small wins.â This past January, IT heavyweights from the National Institute of Standards and Technology (NIST) as well as other government agencies, industry, and academia, got together to discuss the critical intersection of big data and the cloud. Although government agencies have been slower to adopt new technologies in the past, the event underscored the fact that the public sector is leading â and in some cases creating â big data innovation and adoption.
Cloud is a multiplier when used with other technology, and is capable of big things when it comes to big data, according to U.S. CIO Steven VanRoekel. Combining big data with cloud delivery and compute power might help create new industries and provide beneďż˝ ts to every citizen.
âThe government is sitting on a treasure drove of data,â he said during a speech at the two-day NIST Joint Cloud and Big Data Workshop event. âWeâre opening
data, and looking at what we can do. We can greatly impact the lives of every American by just unlocking simple prices of data.â
He also pointed to the formation of companies built and founded completely on government data.
Nowhere to go but upResearch and consulting ďż˝ rm Deltek Inc. says cloud environments are âoptimalâ for using analytics since cloud providers are investing in the best analytical and visualization tools available today. In addition, big data projects require processing speed and the most up-to-date technologies, two other cloud provider specialties.
But most important, said Richard Blake, senior technical advisor in the Enterprise GWAC Division of the Integrated Technology Service at the General Services Administrationâs Federal Acquisition Service, is that big data in the cloud enables something that is crucial for innovation, analysis and return-on-investment: The ability to share resources between agencies.
Evan Quinn, a senior principle analyst with research ďż˝ rm ESG agrees. âBig data is an incremental learning process and when you can share expertise and resources you can score more small wins and progress more quickly than you may have on your own,â he says.
As highlighted in Deltekâs Federal Big Data Outlook 2012-2017, data is available from a wide variety of sources, including agency data such as data logs, space telescopes, reconnaissance, citizen information, and mission critical apps. However, data is also being sourced from the outside as well â in the form of email, text messages, pictures, embedded sensors, social media, GPS data, purchase data, trafďż˝ c cams and market research. With the cloud, it becomes easier to store, analyze, and access this information.
In addition, the same data can be analyzed and used
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 7
in different applications and analytic projects, since one agency no longer âownsâ that data. With these tenets in mind going forward, every piece of data will be examined as a potential resource and used as the basis of future applications, explained VanRoekel during the conference.
One promising approach, he said, is to ensure that data is machine-readable â that is, that it can move from system to system without requiring human intervention or translation.
âGovernment agencies are ordered to look at the data they produce, catalog data, start to publish data, and think about machine-readable as the new default inside government,â VanRoekel explained. âAny time weâre building a new system, or amending a system, we focus on machine readability both on the collection, as well as getting agencies to develop [application programming interfaces] around their data.â
However, as promising as it is, agencies do not want to rush into putting big data in the cloud. Although it might mean that IT no longer has to worry about buying, maintaining, and managing those databases and associated storage, big data is a very specialized workload, says ESGâs Quinn. Agencies must ensure that their cloud providers can handle big data and that they understand the public sectorâs security and compliance requirements, he said.
The other issue is expertise. Most if not all agencies are currently doing some form of business intelligence (BI) on their data, but what works for most organizations with BI is not going to work for big data. Expectations on a whole must be realistic and in line with what is currently technologically possible, said Quinn.
âIf any agency thinks itâs going to go in and put $1 million and six months of work into a big data cloud project and change how the agency works from the
inside out they are misinformed,â he said. âFor most organizations this is a completely new discipline so they need to think in terms of small wins.â
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 8
For CIOs in the public and private sectors looking for ways both to innovate and save money, the combination of big data and mobile is increasingly
appealing solution. For example, Westminster, U.K., recently posted
online all of its geospatial data related to a state-run, London-based bicycle rental â where the cityâs bike racks were, how many bikes were available for rental in real time and other mapping and bike path data. Within days of the release, one of the agencyâs constituents had taken that information and built a mobile app to help people plan future bicycle journeys. Today, itâs easier for citizens to get around and the app is also helping reduce pollution, boost public health, and make more room on public transportation.
This is just one example of why CIOs are so interested in big data and mobility, according to venture capital ďż˝ rm Sierra Ventures, which recently released a study titled âSeizing Opportunity: The Transition from Legacy to Innovation in Enterprise IT.â
According to that report, which was conducted in conjunction with its CIO Advisory Board, when it comes to innovation many CIOs are making big data and mobility an even higher priority than cloud computing and social media.
Meanwhile, a TechAmerica Foundation study in January found that 75 percent of federal IT ofďż˝ cials surveyed think that real-time data â something that mobile devices can enable â is already âhelping government improve the quality of citizensâ lives.â
Another study from IDC Government Insights forecasts that this year, 35 percent of new Federal and state applications will be mobile.
Where the toys areThe reason that big data is intersecting with mobile is simple: Reach.
In order for big data to be successful there needs to be
a common way to disseminate information to a wider audience. Mobile in particular has also been a huge data sources as billions of data points have been collected by carriers, and those data points will grow exponentially as cell phone usage continues to grow.
Today, 78 percent of people between the ages of 12 and 17 own cell phones and 47 percent of those own smartphones, according to a Pew Research report released in March.
Likewise, the number of devices per U.S. household with online access is up to 5.7 â almost a half a percentage point growth in three months, according to the NPD Groupâs âConnected Home Report.â During the same three-month period, the installed base of tablets grew by almost 18 million, with almost 60 percent of U.S. households now owning a tablet device. Combined, connected devices now number about a half a billion and growing.
The challenge for government IT executives is how to extend access to big data initiatives without compromising the security or quality of the data or other agency resources.
In fact, the TechAmerica Foundationâs report found that âprivacy and policy concernsâ trumped âdemonstrating the level of return on investmentâ by a wide margin at the state level â 40 percent versus 22 percent, respectively. At the federal level, privacy and policy was also the top concern, cited by 47 percent of federal IT ofďż˝ cials as opposed to 42 percent of ofďż˝ cials who cited ROI.
The concept of combining big data and mobility is still evolving. But despite its limitations, the combination, when done successfully, can provide a big payoff, as Westminster found out. That might be why, at the end of last year, David Willetts, minister for universities and science in the U.K., announced big funding for big data, with more than ÂŁ189 million going to fund research related to the technologies. After all, when you can harness data and disseminate it in a way that democratizes it, everyone wins.
BIG DATA GOES MOBILE
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
FCW SPONSORED REPORT 9
Analytics and business intelligence is the fastest growing software market today, according to research ďż˝ rm Gartner. The global market for BI is
expected to reach $13.8 billion this year and shoot to more than $17 billion in 2016. Big data and âdata-as-a-serviceâ are the growth factors for the segmentâs upward trajectory.
It might stand to reason then that all it takes to start a big data project is some basic analytics software and a few good on-staff experts. However, experts say basic analytics is only part of the equation.
Another essential part of that equation is visualization software, which translates raw data into a graphical presentation, which humans can understand more intuitively.
âBig data needs an integrated approach â one that combines analytics and visualization software,â explains Alex Rossino, a principle research analyst at Deltek. âYou canât just have analytics and expect it to solve all your problems.â
Picture thisVisualization is important because data, in its raw form, can be difďż˝ cult to analyze, even for the most savvy IT and knowledge workers. In addition, traditional BI tools are designed for traditional data thatâs stored in relational databases. Visualization provides a framework for working with the wider range of data types that might be combined in big data applications.
For example, an application could take a county map and lay speciďż˝ c data points over it: incidents of driving-while-intoxicated fatalities, socioeconomic data, the locations and coverage of local law enforcement. Presented visually, the results might help government ofďż˝ cials determine to beef up policing or where to install extra safety measures, such as guard rails or impact-softening barriers.
Agencies also might extend visualization applications to constituents, says David Loshin, president of Knowledge Integrity Inc, a consultancy that focuses on business data management advice and guidance.
For example, an application might be created to show how the U.S. House of Representatives is voting. âPeople complain that lobbyists have that information, and they do because theyâre on the ďż˝ oor, but you could take that voting data and make a graphical representation so that constituents were as much in the know as lobbyists.â
Visualization also helps IT and its users uncover what might not be obvious to the untrained eye. The Department of Homeland Securityâs Cyber Security Research and Development Center uses visualization to identify patterns in network ďż˝ ow data and metadata that could signal denial of service attacks and other cyber attacks and criminal behavior.
Once people can see information in graphical format, the message of data can really hit home, says Ed Parsons, a fellow of the Royal Geographical Society.
Parsons worked on a project about with the U.K.âs Hadley Centre, a government agency charged with tracking climate change. At the time, there was no way to take the data that had been collected and visualize it in an impactful way, he explained at the Aspen Ideas Festival session, âVisualizing our Future through the Lens of Big Data.â Few people watched the YouTube clips of the climate change announcement or went to the website detailing the news. However, only a few years later he was able to use big data and visualization to demonstrate at the street level the impact and water level after the 2011 Japanese earthquake and tsunami.
âOften our heart is more powerful than our brain when we really want to drive home a message,â Parsons said at the conference. People, whether c-level executives, constituents or voters, are more moved by what they can see than what they can hear. â˘
VISUALIZATION DRIVES HOME BIG DATAâS VALUE
Special Report
www.fcw.com/2013BigDataChallenges>>>
BIG DATA CHALLENGES
BEST PRACTICESFOR
MEETING
+
Conquer the chaos with storage management solutions from CDWG. Utilizing our partnership with EMC, our solution architects can help design and implement a custom storage solution that meets your needs. And once all of the excess has gone, your data center can achieve perfect harmony.
Get organized at CDWG.com/emc
SOLVED.
EVErybODy WantS tO StOrE EVErythinG EVEryWhErE fOrEVEr.
Š2013 CDW Government LLC. CDWÂŽ, CDWâ˘GÂŽ and PEOPLE WHO GET IT⢠are trademarks of CDW LLC.
124542_CDWFed_SnapBigData_043013.indd 1 3/20/13 2:30 PM