33degree krzysztof debski - let's build a solid base for a scale

@DebskiChris

#33rdDegree

Krzysztof Dębski Allegro Group

Let’s build a solid base for a scale

#33rdDegree @DebskiChris Allegro The New World Improvement WIIFY

Agenda

The Allegro Situation How to build a new World? The way to improve What’s in it for you?

Allegro The New World Improvement WIIFY #33rdDegree @DebskiChris

The Allegro Situation


Allegro


Allegro

6 million LOC


400 people in IT

Allegro


A New Hope


A New Hope?

Service

Oriented

Ambiguity


Domain Driven Design


(Micro)Services


(Micro)Services

Business needs

User

Transaction

Offer


(Micro)Services

Independent

User

Transaction

Offer


(Micro)Services

API

User

Transaction

Offer


(Micro)Services

Polyglot

Cassandra

MongoDB

Oracle

User

Transaction

Offer


(Micro)Services

Smart Endpoints

Cassandra

MongoDB

Oracle

Offer

Transaction

User


The first approach

Service

Auto deployable Auto monitored Auto scalable Auto healable Auto discoverable … and many more features


How to build a new World?


The first project

Service

Jetty Server

War file

Spring

Gradle


Let’s REST

JAX-RS / JSR Compliant @Path("/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Let’s expose our methods

Swagger @Path("/users") @Api(value = "/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @ApiOperation(value=“Get all users”, response=UserCollectionResponse.class) @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Let’s expose our methods

Swagger


And make them discoverable

Offer

User

Discovery ZooKeeper



Offer

User

Discovery ZooKeeper

Register



Offer

User

Discovery ZooKeeper



Offer

User

Discovery ZooKeeper

Get User



Offer

User

Discovery ZooKeeper

Get User Get offer for user


Multiple API versions

Content negotiation @Path("/users") @Consumes(”application/json”) @Produces(”application/json”) public class UsersEndpoint {


}



Content negotiation @Path("/users") @Consumes(”application/vnd.allegro.user.v1+json”) @Produces(”application/vnd.allegro.user.v1+json”) public class UsersEndpoint {


}

curl --dump-header - -H ”Accept: application/vnd.allegro.user.v1+json" -X GET http://localhost:8080/users



Content negotiation @Path("/users") @Consumes(”application/vnd.allegro.user.v1+json”) @Produces(”application/vnd.allegro.user.v1+json”) public class UsersEndpoint {

// [...] @GET @Produces(“application/vnd.allegro.user.v2+json”) public UserCollectionResponse findAllUsers() { //[...] }

}

curl --dump-header - -H ”Accept: application/vnd.allegro.user.v2+json" -X GET http://localhost:8080/users


And it became slow…


Lower startup time

External Jetty

Service

Jetty Server

War file


Lower startup time

Embedded Jetty

Service

Jetty Server

Jar file


Lower startup time

Embedded UnderTow

Service

Jetty Server Undertow

Jar file


Lower startup time

Immutable Images

Service

Undertow

Zip file

Server

Immutable image


How to configure it?

Service

java –jar service.jar



Service ZooKeeper

java –jar service.jar –Dzookeeper.address=zookeper.service



Service ZooKeeper

java –jar service.jar –Dzookeeper.address=zookeper.service –Denvironment=prod

prod.properties



Service ZooKeeper

java –jar service.jar –Dzookeeper.address=zookeper.service –Denvironment=prod -Dservice.name=testService

prod.properties


Tests


Do you test your tests?

Pitest


Integration tests

Test without mocks Run from IDE

public class UsersIntegrationTest {

private static Map<String, String> overrideConfiguration = Maps.newHashMap();

static { overrideConfiguration.put("property.name", "This was overwritten value"); }

@ClassRule public static final RestServiceStarted DEPLOYED_SERVICE = new RestServiceStarted(overrideConfiguration);

private WebTarget getUsersResourceWebTarget() { return DEPLOYED_SERVICE.getWebTarget().path("users"); }

}


Monitoring


Let’s see what’s happening

NxLog LogStash Kibana


Let’s see what’s happening

Graphite # Metrics metrics.reporters.graphite.enabled=true metrics.reporters.graphite.host=graphite.service metrics.reporters.graphite.port=2003 metrics.reporters.graphite.prefix=stats.Prod.service metrics.reporters.interval=30


Monolith Alert


Monolith Alert

Multi module project support


The way to improve


Authentication

OAuth2


Patch Support

Standardized by IETF https://tools.ietf.org/html/rfc6902

PATCH /user/1 HTTP/1.1 Host: user.service.local Content-Length: 312

Content-Type: application/json-patch+json [

{”op”: ”test”, ”path”: ”/firstname”, ”value”: ”Jane”}, {”op”: ”add”, ”path”: ”/maidenname”, ”value”: ”Smith”}, {”op”: ”replace”, ”path”: ”/lastname”, ”value”: ”Doe”},

{”op”: ”remove", ”path”: ”/meetings”}, {”op”: ”move”, ”from”: ”/balance”, ”path”: ”sharedbalance”}, {”op”: ”copy”, ”from”: ”/a”, ”path”: ”/b”}

]


Simplify annotations

Swagger? @Path("/users") @Api(value = "/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @ApiOperation(value=“Get all users”, response=UserCollectionResponse.class) @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Deployment

Docker support


What’s in it for you?


Use the latest tools

GradleW in CI ENV and parent POM issues. Don’t do DDOS yourself and your partners.


Change is the only constant

Deployment tools Service API provider Dependency injection HTTP Server

Cargo Docker

Ecosystem

Swagger Jersey addon

Guice Spring

Jetty Undertow


Focus on right metrics

How to have 100% code coverage? public class UserMediaType {

public static final String V1_JSON = "application/vnd.allegro.user.v1+json” public static final String V2_JSON = "application/vnd.allegro.user.v2+json”

}


Focus on right metrics

import org.junit.Test; import static net.trajano… public class MediaTypesTest { @Test public void mediaTypesShouldBeValidUtilityClasses()

throws Throwable { assertUtilityClassWellDefined(UserMediaType.class); } }

You don’t need 100% code coverage!


Tools also lie

Metrics tend to lie

Configuration: default PHP metrics

Tested file: 4535 CLOC Imperative programming


Tools also lie

How many violations are there?


Tools also lie

How many violations are there?

4 At least according to Sonar.


Learn to REST

REST is not so obvious

/api/bi/XYZ123/1

/recommendations/1/items

/offer/100?output=ESI/JSON


Community

Involve all developers in building the Bootstrap. Or they will build their own tools.

Łukasz Drumiński Mateusz Gajewski @wendigo


Q & A

33degree krzysztof debski - let's build a solid base for a scale

Technology

debskichris allegro

allegro situation

new hope

debskichris microservices

json x

deployable auto

service oriented ambiguity

jaxrs jsr compliant