architecting for huper growth and great engineering culture
TRANSCRIPT
Architecting for Hyper Growth and Great Engineering CultureSoftware Architect Indonesia Community #2 Meetup
Ifnu, 26 November 2016
Who am I?
• Ifnu Bima• Ilmu Komputer IPB• Join blibli in 2012• Was working for Deustche Bank Singapore• Employee no 69• Engineers no ~13• Development Manager
– blibli.com Web UI– Mobile Apps : Android, iOS, Windows Phone– Content Management System – Search– Product Discovery : Wishlist, Product Review
Architecture Goals
• Hyper Growth
– X times every year: traffic, transaction, feature, codebase and head count
• Great Engineering Culture
– Foster innovation
– Iteration speed
Software Architecture
Software architecture refers to the fundamental structures of a software system, the discipline of creating such structures, and the documentation of these structures
Software architecture is about making fundamental structural choices which are costly to change once implemented
Source : https://en.wikipedia.org/wiki/Software_architecture
Software Architect / Principal
• Design architecture– Monolithycs vs Microservices
– SQL vs NoSQL
– Datacenter vs Cloud
• Pick Platform, library and provide template– Java vs PHP vs Ruby vs Scala vs Go Lang
• Hiring standard– Technical interview
– Hiring decision
– Training and Certification
Functional vs Non Functional Requirement
• Functional requirement in BussinessRequirement Documentation (BRD)
• Non Functional Requirement– External Quality
• Speed, Performance
• Scalability, Elasticity, Reliability
• Security
– Internal Quality• Unit testing coverage
• Code readability, design pattern
• Learning curve for new hire
Speed – Performance : Server Side
• HardDrive & Database is the slowest link
• Lots lots lots of caching :
– Redis : small but lots of item
– Guava : big and slow item, refresh cache with background process
• SSD for IO intensive operation : Database, static & image server
• Non blocking IO using reactive system
• Slow Query monitoring and indexing
Speed – Performance : Client Side
• Bandwith is the slowest link
• Small bandwith : gzip, image compression, minimize css & js, image & html lazy load
• Render first, script later: defer 3rd party script
• Layout first, content later: layout only using CSS, inline CSS for viewport
• Reduce page load & render from 12s to 4s
Scalability, Elasticity, Reliability
• Stateless application• Externalize session to repository. Currently we use Redis• Externalize scheduler• Microservices
– http://martinfowler.com/articles/microservices.html
• Reactive System– Non blocking IO– Parallel execution– Future / callback
• NoSQL – Solr, elasticsearch– Redis– MongoDB
• Go lang ?
Performance Monitoring
• Newrelic.com
• AppDynamic
• Realtime and Archived performance monitoring
• Performance metrics
– Response time, both server and client side
– Drill down of time spent in application
– Slow query monitoring
– Cache monitoring. Cache hit ratio.
– Server monitoring : CPU, Memory, I/O and Network
Performance Testing and Profiling
• JMeter and WAPT, other alternative is Gatling
• Dedicated team to run it
• Java Profiling
– VisualVM
– XRebel
Security
• Security Appliance
• Application access log analysis
• Risk & fraud team
• PCI DSS compliance process
What we learn about customer
• Fast UI – Speed & Performance
• Website stay up during promotion and flash sale – Scalability & Elasticity
• Consistently good user experience– Great UI/UX
– Timely order delivery
– Original and high quality product
• New features– Rapid Deployment
– Continuous Delivery
Architecting for Great Engineering Culture
• Optimize for iteration speed
• Push relentlessly toward automation
• Build the right software abstractions
• Develop a focus on high code quality with code reviews
• Maintain a respectful work environment
• Build shared ownership of code
• Invest in automated testing
• Allot 20% time
• Build a culture of learning and continuous improvement
• Hire the best
Optimize for iteration speed
• Microservices Architecture
• Blibli.com SOA architecture circa 2013
Store Front
Back officeMerchant
ESB
Monolithic vs SOA vs Microservices
Source : http://www.pwc.com/us/en/technology-forecast/2014/cloud-computing/features/microservices.html
Monilithic team
Source : http://www.martinfowler.com/articles/microservices.html
Microservices Team
Source : http://www.martinfowler.com/articles/microservices.html
• Decentralized Governance
Microservices Team
• Decentralized Data Management
Source : http://www.martinfowler.com/articles/microservices.html
Microservices Team
Microservices Team
Push relentlessly toward automation
• Compile, build, unit test every commit using maven / sbt run automaticly using bamboo CI
• Functional test by QA, automated using JBehave and Appium• API Test using JMeter• Release versioned artifact (war / zip) to artifactory• Deploy to UAT, Performance environment and Production using Chef and Jenkins
– Install latest java– Setup properties– Install latest tomcat– Do additional steps : setup JCE, deploy static file to static server, run database change using
Flyway / MongoBee, etc– Deploy application
Source : http://www.martinfowler.com/articles/microservices.html
Build the right software abstractions
• Microservices template – Java with Spring
– PlayFramework Scala
– Java with Spring and RXJava
– Go Lang ??
• SSO using CAS
• OAuth API abstraction using CAS
• Example of software abstractions– Google BigTable and MapReduce
– Facebook ReactJS and ReactNative
– Amazon AWS
Develop a focus on high code quality with code reviews
• Use GitFlow for code management– http://nvie.com/posts/a-successful-git-branching-model/
– Every feature and bug will have their own branch
– Developer commit to their own branch and pull request to feature / bug / develop branch
– Peer code review and successful build from bamboo is mandatory to merge branch
• Pair Programming if possible
Develop a focus on high code quality with code reviews
• TDD and Unit testing using JUnit and KarmaJS
• Static code analysis using Sonar Cube– Calculate unit test coverage
– Calculate code complexity, less than 3
– Code smell
– Security analysis: SQL Injection vulnerability, XSS, etc
– Potential bug in runtime: NullPointerException, Forever loop, etc
• 90% test coverage expected. Build failed if code coverage drop compared to previous build. If build failed, cannot merge to mainstream branch.
Maintain a respectful work environment
• Open culture
• Using Slack for open communication
• Share everything
• RESPECT as core value– Risk taking
– Excellent
– Serving
– Passionate and Proud
– Employee great place to work
– Customer focus
– Teamwork
Build shared ownership of code
• Access to source code from day 1
• Some degree of rotation to make sure no part of code that only known to 1 person
Invest in automated testing
• We do not have QA as title, it is Software Development Engineer in Test (SDET)
• 2017 we really focus on automation– Bamboo CI – Chef and Jenkins– Nightly Build– Data Preparation– JBehave & Appium– JMeter, WAPT & gatling– Monitoring and alert– Elasticity on Datacenter– Etc
Allot 20% time
• Internal coding competition
• Internal Hackaton
• Skunk Work
Build a culture of learning and continuous improvement
• Rotation for people to learn different things• Survey and Management action for people who want to change their role or team• Weekly sharing session : BrownBag• Training
– Java Training and Certification, Sun Certified Enterprise Architect– iOS development & testing– SOLR training– 7 Habit of effective people– Presentation and public speaking– Project management– Leadership essential – Management assessment– Evident base interview– User Experience from Somia
• Conference– JavaOne in San Francisco– Solr Revolution in Austin, Texas– TechInAsia Jakarta
• Scholarship for Master Degree
Hire the Best
• Blibli.com future program
– https://www.blibli.com/page/future-program/
• Employee get Employee
– Best people will attract best people, law of attraction
• Excruciating round of interview
• Very high hiring bar
We Are Hiring!!
THANK YOU