광주 주요 분야의 빅데이터 구축 및 가공 활용방안 연구 file광주 주요...
TRANSCRIPT
-
A Study on the Construction and Application
of Big data Analysis System in Gwangju
( )
-
1 1
1. 1
2. 5
2 7
1. 7
2. 8
3. 12
4. 32
3 36
1. 36
2. 40
3. 51
4. 53
4 56
1. 56
2. 63
3. 74
4. 78
-
5 80
1. 6 80
2. 81
3. 91
4. 96
5. 99
6 102
104
107
-
10
11
29
HSC 38
42
(10 ) 44
48
49
50
51
60
60
10 61
IT 63
65
69
70
79
87
99
-
3
7
8
9
11
13
14
15
16
ETL 18
18
OLAP 23
24
24
25
26
26
35
39
45
46
47
47
53
54
/ 64
64
-
68
DB 70
71
72
72
73
73
74
93
94
94
95
() 95
4 10 96
() 98
() 98
100
-
- i -
, 3.0 , , ,
(ICT)
, 6
6
,
-
- ii -
,
,
, ,
, , ,
,
, , 5
,
-
- iii -
1980 , KDD
(Knowledge Discovery in Databases)
SNS
,
(Near Real Time)
(Big) , ,
, (Big data) (Big insight)
, , , ,
, , , , ,
(Fortune) (data scientist) 2015
200, 400
-
- iv -
IT IDC(International Data Corporation)
27%
2017 324
IDC ( ) 46.8% 2016
17 6
2015 2.6 , 2020 8.9
24% 2015 1.6%
(, 2013)
DDI(Data-Driven Innovation) , DDE(Data-Driven
Economy) , ICT ,
,
,
, ,
2014 ()
,
-
- v -
5
, , ,
, ,
,
, 3
5
, ,
,
, ,
,
5
, ,
-
- vi -
, , , 4
10
- : , ,
- : , ,
- : ,
- : ,
5
, ,
( ,
, )
IT ,
()
, , ,
-
1
1
1.
1.1.
,
, 3.0 , , ,
(ICT)
, 6
6 4
, (Big-Fi)
-
2
2014-04
5
,
6
21
,
, , , , , , ,
-
3
,
,
, ,
, , ,
-
4
2014-04
1.2.
, ,
, , ,
5
, , ,
,
-
5
,
2.
2.1.
,
, ,
2.2.
, 6
-
6
2014-04
1 ,
2 ,
3
4
5 , ,
,
6
-
7
2
1.
1980 , KDD
(Knowledge Discovery in Databases)
- KDD (Data Mining)
:
: , , , , ,
,
-
KDD(Knowledge Discovery in Databases) Data Mining Business
Intelligence Big Data(Big Insight)
-
8
2014-04
2000 ,
BI(Business Intelligence)
SNS
,
(Near Real Time)
()
-
3.0(, , , )
2.
,
-
9
: (2013)
- 21 :
-
- Brainware
- ,
-
(volume)
(velocity)
(variety)
-
-
10
2014-04
Gartner
(insight)
, , ,
McKinsey SW
IDC
(Big) , ,
, (Big data) (Big insight)
- vs.
- vs.
- vs.
, , ,
: (2013)
-
11
,
,
: (2012),
,
-
, ,
, ,
,
: (2013)
-
12
2014-04
- (, , )
- :
- :
- :
3.
3.1.
, , , ,
,
:
:
(pre-processing) :
-
13
- (On-Line Analytical Processing)
-
- (Social Network Analysis)
- (Natural Language Processing)
- (Machine Learning)
- , , , ,
- : , ,
, ,
-
14
2014-04
- : , ,
-
15
- IBM 100 THINK () 40
, , ,
- : 1 800 , 52PB
Tableau :
-
16
2014-04
3.2.
ETL (Extraction), (Transformation),
(Load) ,
- , (ODS),
- , ,
- ETL
ETL : ETL :
-
17
-
-
- , , ,
- :
- :
: ,
: , ,
-
- ,
- :
- :
(MDM)
-
18
2014-04
: (2014),
ETL
: (2014),
-
19
3.3.
:
-
,
- Interface :
- Staging :
- Profiling : , ,
- Cleansing :
- Integration : ODS
- Export : DBMS ,
,
3.4.
, (EIS),
: ,
,
-
20
2014-04
DB
-
-
-
-
-
,
-
21
- :
,
Decision support database
- :
Departmental data warehouse
-
-
-
-
-
-
22
2014-04
-
: 3
: 2()
- , ,
-
- 3
-
-
-
- :
- :
- , /
, SQL , BI
-
23
3.5.
(OLAP) :
( )
- (DW)
, ,
On-Line Transaction Processing
DW OLAP
-
- Drill-down Roll-up
OLAP ( ),
( ),
OLAP
: Slice and dice
OLAP
-
24
2014-04
-
25
3.6.
DB
- (neural networks)
- (classification)
- (association rules)
- (sequential pattern)
- (genetic algorithms)
- (clustering)
- (visualization)
-
26
2014-04
-
-
-
27
-
,
,
-
-
-
-
-
, , ,
-
Topic tracking :
Question answering :
Duo mining :
Opinion mining : ,
-
28
2014-04
- ,
,
-
Web structure mining :
Web usage mining :
Web contents mining :
3.7.
-
-
, , , ,
, , , , ,
-
29
EMC
, ,
,
(, ),
()
HP
BI ,
,
IBM
(), (
), ()
: InfoSphere BigInsight(Hadoop),
InfoSphere Streams
(Oracle)
DB
SAS
(Advanced Analytics) HPA(High
Performance Analytics) SEMMA
IT++ (SAS
Solution MAP)
IT
- IT
-
30
2014-04
(Teradata)
(BI)
(
)
(Gruter)
, ,
,
,
(NexR)
(NDAP : NexR Data
Analytics Platform), (RHive)
(Daumsoft)
SNS , ,
,
,
(CYRAM)
(NetMiner) :
(Saltlux)
(truestory),
(IN2),
(STORM), (O2)
(Sensology)
( ), ()
-
31
(ArchoneSoft)
(Easy-Up), SI ,
(Altair)
, , BA
BA(HiQube) (PBS
works)
(YainSoft)
/
BI(OctagonTM EnterpriseBI Server)
(SM2 Networks)
BI SW
(Radian6)
(SK Telecom)
/ (Smart
Insight)
NLP ,
(NFLabs)
,
BI/BA (PelotonTM)
(WISE-I-TECH)
BI// /CRM
(CampaignTM), /
(Cloud BITM), / (Smart BITM),
(Social AnalyticsTM)
(ECMiner)
,
(ECMinerTM), (IMSTM), /
(RuleTM), (SISTM)
(e2on)
/(SNSpiderTM),
(UNINANTM)
-
32
2014-04
(Cardinal Info.
Tech.)
, , ,
(Monad Storage), (Monad Integration),
(Monad Management)
(Konan Technology)
, , /
(pulse-K)
(Clunix)
/ /
(Gridcenter Hadoop),
(Teragon-Hadoop), (RNTier)
(2E Consulting)
,
( )
: (2012)
4.
2011
1.9 5 9
(IDC, 2011)
1 15MB 2020 1GB
-
33
SNS(Social Networking Service)
, /
IT
(Fortune) (data scientist) 2015
200, 400
, , ,
IT IDC(International Data Corporation)
27%
2017 324
IDC ( ) 46.8% 2016
17 6
2015 2.6 , 2020 8.9
24% 2015 1.6%
(, 2013)
-
34
2014-04
80% ,
2015 1
- , , IBM 2~6
(, 2013)
2017 1.4
- 2012 3 , 100
- , , KAIST
- 2013 2017 5 52
, ,
- : , ,
- : , ,
-
, ,
-
35
- , ,
: (2013.06.03 14)
-
36
2014-04
3
1.
DDI(Data-Driven Innovation) , DDE (Data-Driven
Economy) , ICT ,
- 2014 7
1.1.
- 2012 3 2
, ,
-
2 5
, ,
(XDATA)
-
37
2,500 , ,
1.2.
, , /
,
,
,
-
- (data.gov.uk)
- HSC(Horizon Scanning Center)
HSC
HSC
-
38
2014-04
10~15
50
HSC
: (2013)
-
5,000 2012 9
, , ,
1.3.
-
- RAHS(Risk Assessment Horizon Scanning)
-
39
Horizon Scanning : , , ,
, ,
- (RAHS Programme Office)
, , 3
1.4.
- 2012 5
- ICT 5
-
40
2014-04
2.
( 45, '10.07)
- (open) (provide)
- 3
-
(, '11.11)
-
- , ,
(, '12.06)
- ,
- ,
,
- 7
-
41
(, '12.11)
- 5 (, , , ,
)
- 2017 ,
, ,
-
- , 6
16 3
:, , , , ,
6 16
: ,
,
-
42
2014-04
()
, ,
,
(), ,
,
, ,
,
, ,
- , ,
,
, ,
,
,
(), ,
,
, ,
,
,
,
, ,
, ,
,
, ,
: (2012)
-
43
3.0 ( , '13.06)
- 15 24
-
- , ,
10
: ,
,
: ,
,
: ,
, ,
-
44
2014-04
, , ( )
, ,
GPSGIS, 3D,
GPS, CCTV , ,
, ,
, ,
GPS
, ,
, ,
, ,
,
,
, ,
,
(
, ), ,
,
, ,
,
,
, ,
,
,
,
, , ,
, ,
, ,
,
, , ,
,
, ,
(10 )
: 3.0 (2013)
-
45
(, '13.06)
- , ,
: (2013),
( ,
'13.06)
- ,
,
-
46
2014-04
-
,
( , '13.12)
- 3.0
- '17 2
- 7
- 5
, ,
- 10
- , ,
-
47
-
48
2014-04
Flagship Project
-
( )
6
- 6
* , , , , ,
-
Shared Service
- *
*
-
( )
-
, -
(
)
-
49
7
- 7
* , , , ,
, ,
- , (ITU )
IPR
5
- , , 5
Job ,
*
- DB
-
-
50
2014-04
DB
- DB , DB
DB , LOD*
* LOD : Linked Open Data
-
-
, ,
-
,
2014 12, 20
- 2025 5 10
5 : , ,
, ,
10 : ,
, ,
, ,
, , ,
,
-
51
, , ,
, , ,
,
, ,
3.
(, 2014)
- 302 IT
73% 2 (2013 64%
)
31%(2013) 24%(2014)
(, 2014)
- 1,000
92% , 89%
(94%), (90%),
(89%)
-
52
2014-04
,
-
-
e- IT (, 2013)
- 3,000
2.1%, 7.1%, 1.2%, 7.7%,
1.7%, 50 1.0%, 1,000
8.6%
/ 4.2%, 3.7%, / 2.3%
(, 2014)
- 500
81.6%
18.4%( )
: (47.3%), (41.9%), (36.6%),
(24.7%), (20.4%)
(2012) ,
56.4%,
19.4%
-
53
4.
- 2013 10,
- , ,
,
- 300TB , ,
,
: (2013), 2013
-
54
2014-04
- 2014 1 10 10 5 , 2,138 ,
950
: (http://www.kbig.kr)
- 2012 8, , ,
, /
-
-
-
55
- 2013 9,
- ,
, ,
- 2014 4,
- , ,
- 2013 12,
-
-
56
2014-04
4
1.
,
,
,
, , , , , , ,
, 2012 500 90%
1.1.
- , ,
-
-
57
- FBI , , 10
1,000
T-Mobile
- 3 170
-
3,450
-
1,000
- (AWS) 1,700
- 1,000
2008 1 , ,
,
,
-
58
2014-04
- 1
,
- , ,
1
11%, 8% 56%
: , ,
1.2.
-
1.3.
- (GPS: Global Positioning System)
-
59
1.4.
(RAHS: Risk Assessment & Horizon Scanning)
- ,
1.5.
- ()
- TSG , ,
, , 60
-
60
2014-04
911
, ()
1 DNA DNA
CODIS (FBI)
Pillbox
Health 2.0 ()
(data.gov.uk)
, ,
RAHS
2.0
data.gov.au ()
(BI
)
SAS
BI ,
,
EMC
-
- 6
-
-
6
-
3
-
( 2
)
-
- 1
- 100
(ROI)
: (2011)
: , (2012.07.15 etnews.com)
-
61
/
- ,
,
-
-
-
-
-
-
-
- ,
Pillbox
-
-
10
-
62
2014-04
/
(WellPoint)
-
-
-
,
-
-
-
-
FBI
-
-
-
: (2012), Big Data 10
-
63
, OS , +
,
iOS , ,
iCloud
,
,
, /
1.6.
, , ,
IT
: (2012)
2.
2.1.
- 1 1,252 5
30 KT
-
64
2014-04
: (2013), 2013
/
: (http://data.seoul.go.kr)
-
65
, , , , , ,
, , , , TV
, , , , , ,
, , , , ,
, , , , ,
, , ,
, , , , ,
, , , , ,
, , , , ,
, , , , , ,
, , , ,
, , , ,
, , , , ,
, , , , ,
, , , ,
, , , , ,
, , , , ,
, , , , ,
, , , , ,
, ,
: (http://data.seoul.go.kr)
-
66
2014-04
2.2.
278 GIS ,
- , ,
2.3.
, ,
2011
- ,
2.4.
DB, DB,
,
2.5.
2011 ,
- ,
,
-
67
-
2.6.
SK
- , (GPS)
- , ,
-
BC
-
- 2,200
18
-
68
2014-04
LG
-
- 5 169
2012 6 90%
2.7. , , ,
1
2 , 3
: (http://www.mospablog.net/11810946)
-
69
29 23
- ,
: (2013)
-
70
2014-04
1,634 4.2% , , ,
5,870 15.1%, , , ,
(, , )
30,978 79.7% , , , , ,
366 0.9% ,
38,848 100.0% -
, ,
DB
- DB (DW)
DB
26 38,000
: (2012)
-
71
2.8. 2013
2013 12 , 44
(, 2013)
- 36, 3, 5
- 50%
- 225 1.87%
( 2013)
- : + KT
KT
-
72
2014-04
- : () + ()BC
+
,
,
- : +
DB SNS , ,
-
73
- : () + +
DB, SNS
,
- : + ETRI, , ,
,
,
,
-
74
2014-04
- : () +
, ()
,
3.
3.1.
: 2014.01.03 ~ 2014.02.22
-
-
-
75
- : 2005.01 ~ 2013.12
67,930
- 120 : 2009.04 ~ 2013.12 120
632,696
- : 2003.01 ~ 2013.12 , ,
, SNS 15,248,722
- , , ,
-
- 120
-
-
-
-
-
-
76
2014-04
3.2.
- 1
('13.7.19 ~ '14.7.18)
-
-
-
-
-
- ,
, ,
, IC , ,
, ,
, 7
- , ,
, ,
2
-
77
- / , ,
19
, 17, 29, 22 .
/ 33
14
28, 27
- / ,
, 19, 72,
36 . 36
-
-
14
LED
-
'15 33() 15
,
-
78
2014-04
15
14
-
'14 10 2015 80
11
5
,
4.
2012 , , , ,
, 2014 2 , , 9
2014 6, 10
- : , , ,
, , DB,
, , BI,
-
79
( ) 2012 4
() 2012 4
(SK ) 2012 8
( ) 2013 1
( ) 2013 3
( ) 2013 5
( ) 2013 8
( ) 2013 10
( ) 2014 3
-
80
2014-04
5
1. 6
1.1.
6 ,
, , ,
, , 6
1.2.
15
-
-
81
-
SNS
GIS
( )
2.
2.1.
SNS : /
- , ,
-
-
: , ,
-
82
2014-04
:
, , ,
- ,
: ,
, ,
: , , ,
- (, ), (, , , ),
-
-
- ,
, , ,
:
-
83
: ,
: , , ,
-
- , ,
,
2.2.
- , , ,
:
: , ,
: ,
-
84
2014-04
: GPS
, , (), ,
-
: GPS
2.3.
/ : SNS
:
: /
-
85
:
SNS :
2.4.
: , ,
- , ,
- , ,
SNS : , , ,
() :
:
2.5.
:
- , CCTV, ,
-
86
2014-04
: , ,
: , ,
: CCTV, , SNS,
- , , , , , CCTV
,
:
- ,
: , , , /
-
87
, , ,
, , ,
SNS /
, , ,
,
, ,
,
, ,
/
SNS
/
,
,
SNS , , ,
()
-
88
2014-04
CCTV, , SNS,
, ,
, , , /
2.6.
- (, )
- : GIS
- :
, ,
Deep data
-
- :
()
-
89
, , ,
GIS
-
- RFI(Request For Information)
-
-
-
-
-
-
-
- Tool : (1), (1~2), &
(2), (1)
- : , (PM) 4 3
-
90
2014-04
- : , , , , ,
- :
- :
- (IoT: Internet of Things)
2.7.
- ,
(M2M : Machine-to-Machine),
, , ,
- (BIC)
BIC : Big data, Internet of Things, Cloud
-
91
3.
3.1.
, ,
,
-
-
-
-
-
-
-
-
-
-
-
-
-
-
92
2014-04
3.2.
, ,
, 5
,
-
-
-
-
- , ,
-
- ,
-
-
-
-
-
,
, 3
-
93
, , ,
,
-
-
- pool
-
-
-
94
2014-04
-
-
- ICT
-
95
2014 5 1 , ,
, 1~2
()
-
96
2014-04
, , , 4
10
4 10
4.
-
97
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
98
2014-04
()
, ,
, , ,
-
()
-
99
- : ()
-
2~4
- : ()
-
1~2
- :
-
6
5.
5
-
, ,
,
,
: (2014)
-
100
2014-04
4 , 4 ()
34
- , , 3 34
: (2014)
- Hard Skill
:
:
- Soft Skill
: , ,
: ,
:
-
101
- (Advanced Data Analytics Professional)
,
,
: , ,
- (Advanced Data Analytics semi-Professional)
, ,
- , ,
IT ,
()
-
102
2014-04
6
,
, ,
,
,
,
5
, , ,
,
-
103
, ,
, , ,
,
Deep
data
, , ,
-
, :
, , 20 2, pp.71-79, 2012.
, 3.0 , 2013a.
, 3.0 , 2013b.
, (), 2011.
, , 2013
, 2013.
, , , 2012.
, , , 16
3, pp.13-41, 2013.
, , 2013 Local Info. Issue Vol.8,
, 2013.
, , , 2011.
, ICT , , 2009.
, , Journal of The Korea
Society of Computer and Information, Vol.19 No.3, pp.73-85, 2014.
, , Journal of Digital Convergence,
12(1), pp.89-97, 2014.
, , 2014.
, , , 2013.
, , 2013a.
, 5 (2013~2017), 2013b.
, 2013 e- IT ,
2014.
, STRONG KOREA 2013 , 2013.
, 2013 IT , , Vol.30, No.1, pp.5-9, 2012.
, : , SERI , 177, , 2013.
, , , 25 10 555,
pp.37-74, , 2013.
-
, : , IT&Future Strategy, 6,
, 2012.
, : 70 , ,
2012.
, ,
, 2008.
, , , 2012.
, , , 57(5), pp.398-404, 2014.
, , , 2013.
, , , 30 6, pp.10-17, 2012.
, , ,
Vol.13 No.4, pp.665-674, 2013.
, Big Data 10 , , 2012.
, (Knowledge Base) ,
, 30 6, pp.40-46, 2012.
, ,
2013-21, , 2013.
, , 2013
, 2013.
, , 2014-04-537,
, 2014.
, v1.0, ,
2012.
, 2013 , EA, 2013.
, , : , 29 11,
pp.30-35, 2012.
, 2013 10 , , 2013.
, , , 2012.
, , 2013
, 2013.
, : , CEO Information, 851, , 2012.
-
, ICT , KISDI Premium Report, ,
2012.
, , 2014.
, , 2011.
, : , IT & Future Strategy, 6
, 2012.
, : , 2013a.
, Ver 1.0, 2014.
, , 2013b.
, 15 , 2010.
, , , 2013.
http://www.data.go.kr
http://www.dbguide.net
http://www.msip.go.kr
http://www.bicdata.com
http://www.bigdataforum.or.kr
http://www.kbig.kr
http://www.bpa.re.kr
http://www.kbd.or.kr
http://bigdata.snu.ac.kr
http://data.seoul.go.kr
http://www.mospa.go.kr
http://www.mospablog.net
http://www.nipa.kr
http://www.kisdi.re.kr
http://www.kodb.or.kr
http://www.kisa.or.kr
http://www.nia.or.kr
-
107
(2014)
-
-
-
- :
- :
Flume-NG :
-
:
, , ,
- ,
-
108
2014-04
(Impala)
- SQL
(Hadoop) (Hive) SQL
- HBase HDFS(
)
(HiveQL)
-
C++ ,
- SQL , SQL
- (1.2) UDF(User Defined Function)
UDAF(User Defined Aggregate Function)
-
: , ,
:
: /
:
:
-
- (HDFS),
-
109
-
-
:
:
-
-
-
-
2
4
-
, :
JobTracker :
TaskTracker :
SQL on Hadoop
- 2012
-
110
2014-04
-
(Shuffling)
:
DW
Hive
- Hive : ANSI SQL
- Hive : SQL
SQL
- (Pig) (Hive)
2008
:
: SQL
-
111
(
)
- GB TB
- :
HDFS
- :
2005
-
:
5
( )
100
2
-
3
-
112
2014-04
-
: 2
,
-
: Flume-NG
: Sqoop
: NoSQL(Hbase), SQL (Hive, Pig),
SQL (Impala, Tajo), (Oozie, Azkaban)
-
Sqoop :
- Sqoop
-
113
, MySQL, postgreSQL, JDBC
RDB
Hbase NoSQL
- Hive : SQL
Batch
- SQL on Hadoop
SQL :
Drill : (Dremel) 2011
Stinger :
:
:
: 2
: ,
- MapReduce
- Sawzall
MapReduce
MapReduce 30%
-
114
2014-04
- Pig
- Hive
DDL, DML, Query
-
A Study on the Construction and Application
of Big data Analysis System in Gwangju
2015 1 2015 1 ( ) [http://www.gji.re.kr] 152 53-27
TEL. 062) 940-0500 / FAX. 062) 940-0523
(062-225-9633) ISBN 979-11-85580-15-9
. .
2014-04