SQS ingress for AWS Lambda

Download SQS ingress for AWS Lambda

Post on 23-Jan-2018

425 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>for</p><p>AWS LAMBDAAWS LAMBDA</p><p>SQS ingressSQS ingress</p></li><li><p>hi, Im Yan Cui</p></li><li><p>AWS user since 2009 </p></li><li><p>can I haz SQS as ingress source?</p></li><li><p>DO ITwe canDO IT</p><p>ourselves!</p></li><li><p>retried 3 times</p><p>dead letter queue</p><p>configurable</p><p>dead letter queue</p></li><li><p>retried 3 times</p><p>dead letter queue</p><p>one invocation per message</p><p>configurable</p><p>dead letter queue</p><p>no. of pollers</p></li><li><p>retried 3 times</p><p>dead letter queue</p><p>one invocation per message</p><p>highest parallelism!</p><p>configurable</p><p>dead letter queue</p><p>no. of pollers</p></li><li><p>retried 3 times</p><p>dead letter queue</p><p>one invocation per message</p><p>consider impact on downstream systems consider Lambda throttlingwhich is per region, so other</p><p>functions can get throttled too</p><p>configurable</p><p>dead letter queue</p><p>no. of pollers</p></li><li><p>retried until success/expiry</p><p>NO dead letter queue</p><p>configurable</p><p>dead letter queue</p></li><li><p>retried until success/expiry</p><p>NO dead letter queue</p><p>configurable</p><p>dead letter queue</p></li><li><p>retried until success/expiry</p><p>NO dead letter queue</p><p>one invocation per shard</p><p>configurable</p><p>dead letter queue</p><p>no. of pollers</p></li><li><p>retried until success/expiry</p><p>NO dead letter queue</p><p>one invocation per shard</p><p>charged by shard hour + reqs</p><p>configurable</p><p>dead letter queue</p><p>no. of pollers</p><p>charged by reqs</p></li><li><p>retried until success/expiry configurable</p><p>NO dead letter queue dead letter queue</p><p>one invocation per shard no. of pollers</p><p>charged by shard hour + reqs charged by reqs</p><p>high baseline cost (1 shard) butorder of magnitude cheaper at scale</p></li><li><p>1 msg/s for a month (1K msgs)</p><p>1 x 60s x 60m x 24hr x 30days @ $0.4 per mil</p><p>= $1.0368</p><p>1 x 60s x 60m x 24hr x 30days @ $0.014 per mil+ 24hrs x 30days x $0.015 per hr</p><p>= $10.836</p></li><li><p>1000 msg/s for a month (1K msgs)</p><p>1000 x 60s x 60m x 24hr x 30days @ $0.4 per mil</p><p>= $1036.80</p><p>1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil+ 24hrs x 30days x $0.015 per hr</p><p>= $47.088</p></li><li><p>none of them is the right choice </p><p>for every situation</p></li><li><p>SQS is right choice sometimes, no?</p><p>none of them is the right choice </p><p>for every situation</p></li><li><p>cron jobslevel 1</p></li><li><p>every X mins</p></li><li><p>simple cheap</p></li><li><p>simple cheap </p><p>good for predictable traffic</p></li><li><p>unpredictable traffic leads to unpredictable latency</p></li><li><p>unpredictable traffic leads to unpredictable latency</p><p>manual scaling</p></li><li><p>recursive functions</p><p>level 2</p></li><li><p>poll (long polling) </p><p>process </p><p>recurse</p></li><li><p>POLL</p><p>long polling</p></li><li><p>PROCESS</p></li><li><p>RECURSE</p></li><li><p>predictable latency even when traffic is unpredictable</p></li><li><p>unhandled exception kills loop</p></li><li><p>unhandled exception kills loop</p><p>cost</p></li><li><p>autoscalinglevel 3</p></li><li><p>poll (long polling) </p><p>process </p><p>continue? </p><p>recurseyes</p><p>stopno</p></li><li><p>token last_heartbeat</p><p>my_loop_1 2017-05-17 20:47:32.142</p><p>my_loop_2 2017-05-17 20:47:29.923</p><p>passed to a recursive function at the start</p></li><li><p>every X mins</p><p>look for dead loops</p></li><li><p>every X mins</p><p>look for dead loops</p><p>recurse</p><p>start</p></li><li><p>too many messages</p></li><li><p>too many messages</p><p>add token</p></li><li><p>too many messagesrecurse</p><p>start</p><p>add token</p></li><li><p>http://bit.ly/2npomX6</p><p>http://bit.ly/2npomX6</p></li><li><p>DE moDE moshow me </p><p>the</p></li><li><p>@theburningmonktheburningmonk.comgithub.com/theburningmonk</p></li><li><p>Process SQS with Lambda</p><p>part 1 : http://bit.ly/2rvXYdW </p><p>part 2 : http://bit.ly/2pY0Grj</p><p>http://bit.ly/2rvXYdWhttp://bit.ly/2pY0Grj</p></li><li><p>QUESTIONS?</p></li></ul>