time series data with influxdb
TRANSCRIPT
![Page 1: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/1.jpg)
Working with time series data with InfluxDB
Paul Dix @pauldix
![Page 2: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/2.jpg)
What is time series data?
![Page 3: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/3.jpg)
Stock trades and quotes
![Page 4: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/4.jpg)
Metrics
![Page 5: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/5.jpg)
Analytics
![Page 6: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/6.jpg)
Events
![Page 7: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/7.jpg)
Sensor data
![Page 8: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/8.jpg)
Two kinds of time series data…
![Page 9: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/9.jpg)
Regular time series
t0 t1 t2 t3 t4 t6 t7
Samples at regular intervals
![Page 10: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/10.jpg)
Irregular time series
t0 t1 t2 t3 t4 t6 t7
Events whenever they come in
![Page 11: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/11.jpg)
Inducing a regular time series from an irregular one
query: select count(customer_id) from events where time > now() - 1h group by time(1m), customer_id
![Page 12: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/12.jpg)
Data that you ask questions about over time
![Page 13: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/13.jpg)
InfluxDB is an open source distributed time
series database* still working on the distributed part
![Page 14: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/14.jpg)
Why would you want a database for time series
data?
![Page 15: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/15.jpg)
Scale
![Page 16: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/16.jpg)
Example from DevOps• 2,000 servers, VMs, containers, or sensor units
• 200 measurements per server/unit
• every 10 seconds
• = 3,456,000,000 distinct points per day
![Page 17: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/17.jpg)
Sharding Datausually requires application level code
![Page 18: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/18.jpg)
Data retentionapplication level code and sharding
![Page 19: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/19.jpg)
Rollups and aggregation
![Page 20: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/20.jpg)
InfluxDB features
![Page 21: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/21.jpg)
SQL style query language
![Page 22: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/22.jpg)
Retention policiesautomatically managed data retention
![Page 23: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/23.jpg)
Continuous queriesfor rollups and aggregation
![Page 24: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/24.jpg)
HTTP API - 2 endpoints
![Page 25: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/25.jpg)
HTTP API - 2 endpoints
/write?db=mydb&rp=fooWrite: HTTP POST
![Page 26: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/26.jpg)
HTTP API - 2 endpoints
/write?db=mydb&rp=foo
/query?db=mydb&rp=foo&q=
Write: HTTP POST
Read: HTTP GET
![Page 27: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/27.jpg)
InfluxDB Schema• Measurements (e.g. cpu, temperature, event,
memory)
![Page 28: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/28.jpg)
InfluxDB Schema• Measurements (e.g. cpu, temperature, event,
memory)
• Tags (e.g. region=uswest, host=serverA, sensor=23)
![Page 29: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/29.jpg)
InfluxDB Schema• Measurements (e.g. cpu, temperature, event,
memory)
• Tags (e.g. region=uswest, host=serverA, sensor=23)
• Fields (e.g. value=23.2, info=‘this is some extra stuff`, present=true)
![Page 30: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/30.jpg)
InfluxDB Schema• Measurements (e.g. cpu, temperature, event,
memory)
• Tags (e.g. region=uswest, host=serverA, sensor=23)
• Fields (e.g. value=23.2, info=‘this is some extra stuff`, present=true)
• Timestamp (nano-second epoch)
![Page 31: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/31.jpg)
All data is indexed by measurement, tagset,
and time
![Page 32: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/32.jpg)
Influx CLI
$ ./influx Connected to http://localhost:8086 version 0.9 InfluxDB shell 0.9 >
![Page 33: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/33.jpg)
Create a database
CREATE DATABASE foo
![Page 34: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/34.jpg)
Create a retention policy
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
![Page 35: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/35.jpg)
Create a retention policy
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT
![Page 36: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/36.jpg)
Create a retention policy
CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT
Writes will go into this RP unless otherwise specified
![Page 37: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/37.jpg)
Discovery
![Page 38: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/38.jpg)
Inverted indexof measurements and tags
![Page 39: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/39.jpg)
DiscoverySHOW MEASUREMENTs
![Page 40: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/40.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
![Page 41: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/41.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
SHOW TAG KEYS
![Page 42: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/42.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
SHOW TAG KEYS
SHOW TAG KEYS from CPU
![Page 43: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/43.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
SHOW TAG KEYS
SHOW TAG KEYS from CPU
SHOW TAG VALUES from CPU WITH KEY = 'region'
![Page 44: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/44.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
SHOW TAG KEYS
SHOW TAG KEYS from CPU
SHOW TAG VALUES from CPU WITH KEY = 'region'
SHOW SERIES
![Page 45: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/45.jpg)
DiscoverySHOW MEASUREMENTs
SHOW MEASUREMENTS where host = 'serverA'
SHOW TAG KEYS
SHOW TAG KEYS from CPU
SHOW TAG VALUES from CPU WITH KEY = 'region'
SHOW SERIES
SHOW SERIES where service = 'redis'
![Page 46: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/46.jpg)
Queries
![Page 47: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/47.jpg)
SQL-ish
select * from some_series where time > now() - 1h
![Page 48: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/48.jpg)
Aggregates
select percentile(90, value) from cpu where time > now() - 1d group by time(10m)
![Page 49: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/49.jpg)
Aggregates
select percentile(90, value) from cpu where time > now() - 1d group by time(10m), region
Group by a tag
![Page 50: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/50.jpg)
Where against Regex (field)
select value from some_log_series where value =~ /.*ERROR.*/ and time > "2014-03-01" and time < "2014-03-03"
![Page 51: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/51.jpg)
Where against Regex (tag)
select value from some_log_series where host =~ /.*asdf.*/ and time > "2014-03-01" and time < “2014-03-03" group by host
![Page 52: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/52.jpg)
Functionsmin max percentile first last stddev mean count sum median distinct count(distinct)
more soon: difference, histogram, moving_average
![Page 53: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/53.jpg)
Continuous queriesCREATE CONTINUOUS QUERY "10m_event_count"ON mydbBEGIN SELECT count(value) INTO "6_months".events FROM events GROUP BY time(10m)END;
![Page 54: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/54.jpg)
Other tools
![Page 55: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/55.jpg)
Telegrafdata collection
![Page 56: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/56.jpg)
Chronograf
![Page 57: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/57.jpg)
Grafana
![Page 58: Time Series Data with InfluxDB](https://reader036.vdocuments.mx/reader036/viewer/2022082218/55cad513bb61eba8438b4660/html5/thumbnails/58.jpg)
More coming• Compression
• Clustering
• Custom functions