WHATs & WHYs

What you need and why you need it!

Harun YardımcıSoftware Architect @ eBay

www.linkedin.com/in/harunyardimci/

Notes on Notes

• Slides are unordered• There are always alternative tools. Concentrate

on the idea!• Not talking about technology

HELP!!Someone please make this presentation beautiful

Release Management

• Release NotesYou have to know exactly what is deployed and when

it is deployed. There is no exception.

• Release ManagerYou need this title to give the responsibility. So you can

blame someone.Can’t hire a new guy, just hand it to smartest one.

• Project ManagerUsually they will tell you what needs to be deployed.

Code Review

• How do you know you write a good quality code?Share your code with your colleagues, they will enjoy

to embarrass you

• Tell others that their code smellsSo you have a chance to get your revenge

• Standards and best practicesThere is always a better way and probably someone

did it

Network Bandwidth

• Measure the network bandwidth on each machine.

You will need to know it just before pulling hairs out of your head when your application is not responding.

o volker groppo Iptraf

There are many others. Choose one and go.

MHA for MySQL

• Master High Availability Manager and tools for MySQL

Tools are faster than you! They don’t get tired and don’t complains.

There is nothing to loose since it works when your master DB is crashed

Load Balancer

• Hardware Balancero Citrix NetScalero F5

More features and better scalability. Costs you more.

• Software Balancero HAProxy

Software load balancers seems cheaper and they perform quite well but you need a good hardware to run it!

Online Schema Change

Who wants to shutdown the site and alter a table for an hour, then wait for the slaves to catch up?

No one! Not any more.

o pt-online-schema-changeo oak-online-alter-table

Openark and Percona has many more useful tools. Use them! Tools are they to make your life easier.

Archiving

• Move old and expired data to archive

o Small dataset and index = $$ CASH $$o Faster response, low latency = $$ CASH $$o Maintainable database = $$ CASH $$o Faster dumps and restores = $$ CASH $$

Don’t forget to write scripts to get it back when needed!

No Database Dependency

• NO TRIGGERS• NO STORED PROCEDURES• NO VIEWS

o Nobody will know the business logico Impossible to learn how a piece of code is functioningo Not easy to maintain

Forget immediately what your old-school database teacher told you in the college.

Don’t use them.

CACHING

• Cache everything• More importantly evict everything

if you can’t evict do not cache

• Web Developers Toolbox:o Varnish, Squid, etc.o Memcached, Redis, Couchbase, etc.o Hazelcast, Terracotta, Coherence, EhCache, etc.

CPU cache, disk buffer, page cache, browser cache, DNS cache, Proxy cache, HTTP Etag, Expire Headers, ….

Server Configuration Management

• Declare once execute many timesWrite a manifesto file as an configuration and define

the standards. Each servers configuration will be identical.

o Puppeto Chef

Easy and fast to configure a new serverVery fast to change any settingsIdentical configurationsCentral ControlMore holidays for system admins

LOGGING• Don’t generate logs if you don’t read them.

At least you can save I/O on disks.

• Centralized logs enables you to search every action of a component (user, product, sale, etc)

Writing a proper log in each application, makes you enable to search for actions of userId:101.

o Logstasho Graylog2

Your boss will love you as long as you can read the past!

Capped collection, Elastic Search, Gelfino, Grok, Kibana, etc.

Auto Deployment

This is 21st Century! Manual deployment is unacceptable.

Don’t wait anyone to ask you to do it.

Use Jenkins, Maven, Ant, Bash Script, Fabric, Capistrano, Python, Perl or whatever you can.

Still reading this? Don’t!

Go and write a script.Now!

Test Environments

• Separated Test Environments• Create Test Data• Test more than once:

o Staging

Have more than one test environment

Testing affects to number of bugs dramatically.

Test Data

• Every application has to create its own test data. o Your application, your test data. You know better and It is your

responsibility.

• Easy to test it on every environment• Reduces dependencies• No need to database syncs to test DBs

Data Sanitizing• Some users have same passwords for Facebook, Gmail and so

• Users emails, bank accounts, what they bought, what they like are valuable data

Can be sold to third parties or can be used to threaten someone

• Users data is personal and it should be protected

Some employees are very curious about others, specially about their friends.

Block DB Access, Authentication, Auditing, Data Encryption

There is no “Freedom of Information Law” on your users data

Development Environments

• Own Machines (Leads to “but it works on my machine” problem)

• Virtual Machines• Dedicated Servers

Give them options and let them choose one they feel comfortable.

Should be Simple and Fast!

Agile Methodology

Developers loves it!

• Basic Idea: o Small, well defined tasks o Solve problems and unclear points immediatelyo Be a communication bridgeo Remove disturbanceo Plan aheado Measure performanceo Learn the limits

Scrum, Kanban, Scrumban, XP, Kaizen, Six Sigma, etc..

Monitoring

• Monitor everything load, disk, network, memory, db response, cache response, data size, counts, 404s, 500s, every little tiny bit can be important.

• Define milestones• Watch after deployments• Write Health Check services, checking the port

not enough• Analyze Trends

Forecast the load on next Christmas, campaign, holiday

Source Control

• Use GIT

If you think that you don’t need it, then most probably you are doing something wrong.

GitHub, Gitolite, Gitosis, Gitstash, Gitlist, Gitweb, etc..

Define Branching Model Authenticate and Authorize the Access

Project Managers

• You need someone who can move some excel boxes from one to another cell

• Sometimes some cells needs to be merged together

Those excel cells are project plan, execution plan, priority of tasks, risk analysis, etc.

They typically have extensive background in development

I wish so!

Asynchronize

• Don’t keep users waiting• Add some actions to a queue• Reduce Race-Conditions

Everything will be okay!

In time

Enable / Disable

• Flexibility• Partially Maintenance• Less Downtime• Decoupled features

Enable / Disable messages for a time while payment is working

or any feature..

Budget For Teams• Go for lunch, take them to cinema, do some team building

activitiesThey will never forgetLoyalty will increase

Know each other betterIncrease communication between members

• Have a half day off once a month to:Do something fun togetherGet relaxedListen their problems

They need to talk each other and to do something together!

Legacy Codes and Systems

• Laugh at them• Make fun of them• Delete them• Remove them• Burn them

Be courage about designing and creating them from scratch.

They are not so big things as you think!

Meetings

• Don’t have meetings

• If you really need ito Stand-up meetings (Less time)o Invite very few peopleo Have someone to take noteso Define time box (30 min. max)

• Need more? Break it into small pieceso Share outcome and result of the meeting

Other Headaches• Cross Functional Teams Good!

• Outsourcing Generally Bad!

• Centralization / DecentralizationVaries case by case. Just think about it before

you start.• Details and early optimizations blocks you to start

something• People need uninterrupted time to finish their tasks• Context Over Consistency

What makes sense here may not makes sense there*

• Spend some time with other departments* Rework

Thanks!

• Have a question? Harun.yardimci@gmail.comhyardimci@ebay.com

• Liked it?o Use it, share it, show it to your boss

• Didn’t like it? o Pretend that you liked it and help me to make it better