devcommerce conference 2016: big data com elastic search
TRANSCRIPT
![Page 1: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/1.jpg)
Big data com Elasticsearch
Roger MattosCo-fundador e CTO
![Page 2: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/2.jpg)
Quem são os seus consumidores e quais são seus interesses?
![Page 3: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/3.jpg)
E-Commerce database
1
1
1
Automação de campanhas personalizadas
![Page 4: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/4.jpg)
Volume atual
~1miLogs de comportamento por dia
~8k
Logs de comportamento por minuto
![Page 5: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/5.jpg)
Como coletamos e armazenamos nossos dados
![Page 6: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/6.jpg)
Desafios
● Filtros de audiencia em tempo real
● Consolidação e processamento de dados diários
● Inferir interesse de usuários baseado em navegação
![Page 7: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/7.jpg)
Segmentação de Audiência
![Page 8: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/8.jpg)
Campanha de Trending
![Page 9: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/9.jpg)
Bottlenecks
● Lentidão nas consultas de audiencias
● Demora no processamento de campanhas
● Queda de CTR (conversão) devido ao delay de entrega
![Page 10: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/10.jpg)
Big data com Elasticsearch
Leandro LustosaSoftware Engineer
![Page 11: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/11.jpg)
You Know, for search.
![Page 12: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/12.jpg)
![Page 13: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/13.jpg)
Para quê?
Busca
- FullText Search
-GeoÍndices
-Agregações
Analytics
-Near Realtime
-Ferramentas de Ingestão e Vizualização de
Dados
Fazer os dados terem sentido
![Page 14: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/14.jpg)
● Arquitetura Distribuida● API RESTFul ● Facilidade de Uso
Pontos Fortes
● Document Based● Open Source● Ecosistema
![Page 15: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/15.jpg)
Index 1 Index 2 Index 3 Index 4
Arquitetura Distribuída
![Page 16: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/16.jpg)
![Page 17: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/17.jpg)
![Page 18: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/18.jpg)
![Page 19: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/19.jpg)
![Page 20: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/20.jpg)
![Page 21: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/21.jpg)
Exemplo de Consulta
![Page 22: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/22.jpg)
![Page 23: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/23.jpg)
![Page 24: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/24.jpg)
![Page 25: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/25.jpg)
![Page 26: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/26.jpg)
![Page 27: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/27.jpg)
![Page 28: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/28.jpg)
![Page 29: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/29.jpg)
![Page 30: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/30.jpg)
![Page 31: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/31.jpg)
Na Social Miner
● Processamento de dados de navegação
● Monitoramento de Serviços e Infraestrutura
● Visão para a equipe de Negócios
![Page 32: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/32.jpg)
● ~6 milhões de usuários
● ~60 milhões de logs (2 meses)
● ~7 milhões de páginas processadas e indexadas
O que e como nós processamos
● 3 Máquinas
● 32 GB de RAM
● 4 Cores
![Page 33: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/33.jpg)
● Configuração Padrão vs Produção
● Alterações de Schema
● Quanto Hardware utilizar
● Como dividir os Shards
Dificuldades
![Page 34: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/34.jpg)
● Dados Quentes x Frios
● Múltiplos Shards x Múltiplos Índices
Modelagem de Dados
![Page 35: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/35.jpg)
● Na dúvida, escolha memória.
● Metade para a aplicação, metade para o sistema operacional.
● Sempre tenha uma réplica de cada shard, de preferência em
zonas de disponibilidade distintas.
● Proteja seu Endpoint.
Deploy
![Page 36: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/36.jpg)
Monitoramento de Serviços e Infraestrutura
![Page 37: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/37.jpg)
Visão de Negócio
![Page 38: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/38.jpg)
Quem usa?
![Page 39: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/39.jpg)
Conclusão
Memória Importa
Pense nos Shards
Sempre tenha uma réplica
![Page 40: DevCommerce Conference 2016: Big data com Elastic Search](https://reader034.vdocuments.mx/reader034/viewer/2022051300/587852ac1a28ab68198b699f/html5/thumbnails/40.jpg)
Roger MattosCo-fundador e CTO
Obrigado!
@roger_mattos
linkedin.com/in/rmattos
Leandro LustosaSoftware Enginner
@llustosa_s
linkedin.com/in/leandro-lustosa-24672121