login with aws elasticsearch servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf ·...

33
Login with AWS Elasticsearch Services .NET CORE

Upload: others

Post on 22-Aug-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Login with AWS Elasticsearch Services.NE T CO R E

Page 2: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Overview

• Released in 2006

• Previous version not supported for structured login

• Introduced structured login in 2017 which is support for .NET standards

Page 3: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Overview

• The newest login framework

• Released in 2013

• The big difference between Serilog and the other frameworks is that it is designed to do structured login out of the box

• Serilog, was.. NLog also supports structured login, but it is was only added recently and the benchmarks suggest that using it has some serious performance implications.

Serilog

Page 4: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Structured login support in Serilog

• Ease of configuration

• API is more modern

• Ability to add enrichers give you the ability to intercept and modify the messages (append properties to the log messages)

• Can store logs in different destinations (Sinks)

• Ex: Amozon Cloudwatch, Azure Blob Storage, Azure Analytics, Rolling File, Azure Event Hub, Elasticsearch

Serilog

Page 5: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

NLog Benchmark

Page 6: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Serilog Benchmark

Page 7: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Conclusion

• Serilog log provider and the NLog log provider are similarly configured

• The results for Serilog are much better than the ones for NLog in both throughput and latency.

• In most cases latency for Serilog is half as much as the latency for NLog, and throughput twice as much.

Page 8: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Serilog

• Advantages

• Widely adopted

• A lot of documentation

• Great community

• Disadvantages

• A bit harder to learn when coming from log4net

NLog

• Advantages

• A lot of documentation

• Having been around a long time, there are lots of blog posts

• Easy to get started when coming from other login frameworks

• Disadvantages

• Structured login is still a bit behind Serilog

• C#-based API is harder to use than Serilog's fluent API

Page 9: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Centralized login in .Net Core using AWS Elasticsearch Service

Serilog & Kibana

Page 10: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of
Page 11: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

• It is No Sql, distributed, full text database

• built on Apache Lucene and was first released in 2010

• Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of open source tools for data ingestion, enrichment, storage, analysis, and visualization

• Referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana)

Page 12: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Elasticsearch Used For?• Application search

• Website search

• Enterprise search

• login and log analytics

• Infrastructure metrics and container monitoring

• Application performance monitoring

• Geospatial data analysis and visualization

• Security analytics

• Business analytics

Page 13: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

How does Elasticsearch work?

• Raw data flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications

• Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch.

Page 14: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Elasticsearch Index

• Collection of documents that are related to each other

• Stores data as JSON documents

• Document correlates a set of keys (names of fields or properties) with their corresponding values (strings, numbers, Booleans, dates, arrays of values, geolocations, or other types of data).

Page 15: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Logstash

• Core products of the Elastic Stack

• Used to aggregate and process data and send it to Elasticsearch

• An open source, server-side data processing pipeline

• Enables you to ingest data from multiple sources simultaneously and enrich and transform it before it is indexed into Elasticsearch

Page 16: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of
Page 17: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Kibana

• Data visualization and management tool for Elasticsearch

• Provides real-time histograms, line graphs, pie charts, and maps

• Includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.

Page 18: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of
Page 19: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Why use Elasticsearch?

• Elasticsearch is fast

• Elasticsearch is distributed by nature

• Elasticsearch comes with a wide set of features

• The Elastic Stack simplifies data ingest, visualization, and reporting

Page 20: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Elasticsearch Logging Process

Page 21: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

How to Setup AWS Elasticsearch Service

Page 22: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Setup Elastic Search Domain

Page 23: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Configure Cluster

Page 24: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Setup Access

Page 25: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Setup Access Policy

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": "*"

},

"Action": "es:*",

"Resource": "arn:aws:es:us-east-1:762493928111:domain/youcab/*"

}

]

}

Page 26: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Elasticsearch Service Summary

Page 27: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

.Net Core Application Setup

• Required Packages

• Elasticsearch.Net

• Elasticsearch.Net.AWS

• Serilog.Sinks.Elasticsearch

Page 28: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Setup AWS Elasticsearch Credentials (appsetting.config)

"AWS": {

"AccessKey": "AKIAJOUREKVWNFMYLKOA",

"SecretKey": "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",

"Region": "us-east-1",

"ElasticUrl": "https://search-youcab-paf2n6jkgms3y5kxtzajjywspm.us-east-1.es.amazonaws.com"

}

Page 29: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Logger Service Project Setup (.Net Core)

• Setup AWS credentials in Startup class

Page 30: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

• Configure AWS Elasticsearch Connection String

Page 31: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Demonstration

Page 32: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Elasticsearch Available Methods

• AWS Elasticsearch

• Azure Elasticsearch

• Elasticsearch Cluster

Page 33: Login with AWS Elasticsearch Servicesfintechasiapacific.com/pdf/techtalk_anushka.pdf · •Elasticsearch is a distributed, open source search and analytics engine for all types of

Any Questions???