refactoring toward deeper insight java forum
DESCRIPTION
Refactoring Toward Deeper Insight DDD Findings in Batch Processing, a Case Study When I was introduced to the Domain-Driven Design (DDD) approach close to ten years ago, it provided me with some of the missing pieces I needed to implement Object-Orientation in an effective way. And over the years I've been coming back to Eric Evans' very rich and deep book many times to discover something new to help me design better software - thinking tools and practical design advice, in the small and in the large. Over the years Object-Orientation has become less important to me, but DDD is still my default starting point when I am helping teams to refactor their architectures and take control over their code bases. Many teams have already made attempts to implement DDD, but very often they don't get the effects they were hoping for. It turns out that DDD is hard to get right. In a current project I have been involved in yet another effort to implement DDD on a legacy code base. And I have made some interesting findings. Batch processing scenarios opened up my eyes to some intrinsic problems with the DDD approach. Issues that have been have been nagging me over the years became very clear. And yet again I managed to gain deeper insight in the DDD approach and come up with some quite interesting ways to implement it. Andreas Brink, factor10TRANSCRIPT
Refactoring Toward Deeper Insight
DDD Findings In Batch Processing A Case Study
Andreas Brink, factor10
Disclaimer!!! .NET not Java
Work In Progress…
Danica Pension
• Core financial business system.
• ASP.NET / RDBMS / Integrations
• Generations of technologies / architectures.
• Lots of financial business rules.
• Lots of Batch Processing.
• Mission: Taking Control of the Code Base – DRY, Understandability, Testability, Automation
• DDD/DM + ORM fits well!
My View of the DDD Toolbox
Philosophy & Principles • Ubiquitous Language • Model Driven • Declarativity • Distilling The Model • Breaktrhough • …
Model Design & Impl. • Entity • Aggregate • Value Object • Respository • Service • Specifiction • Side-Effect Free Functions • …
Strategic Design • Bounded Contexts • Responsibilty Layers • …
Domain Model Pattern (DM)
Supple Design • Assertions • Side-effect free
Functions • Standalone Classes • Closure of Operations
Design Sweet-Spot • Understanding & Communication • Testability, Executable Specifications But, not like the Office Object Model… • Must scale • Not one single file on disk
Basic Building Blocks • Entity, • Aggregate • Value Object • Specification • Factory
Implementability
Service
Repository
• Object Navigation does not scale Repositories
• DM does not scale well with composition & coupling Services
• Problem Solved !?!?
Some Layer…
ORM
Implementation Mess
Service
Repository
• Less focus on Domain Model • Services – The Ultimate Loophole
– Touches big parts of the system – horizontally, vertically – Side Effects Understandability & Testing Problems
• Decentralized Flow Control & Data Access… Global Optimization & Generic Processing hard or impossible Performance Problems
ORM Some Scenario…
Why DDD is Hard
Service
Repository
• Model Design is hard to begin with – OO Expertise is still quite rare
• Have to be a Design Patterns / ORM / Architecture Expert
• Fail to get an Easily Consumable & Testable Model
• Same old Procedural Service Soup (+ some entities…)
ORM Some Scenario…
My DDD Challenge
• Reclaiming the Domain Model – Easy Reasoning, Consumption & Testing
• REAL Separation of Concerns – Not just a complex web of objects and method
calls behind superficially simple interfaces
• And with Batch Processing in the mix…
IS THIS POSSIBLE??
Batch in the mix…
• ”ORM is not for Batch, use the right tool…”
• DDD/ORM vs Stored Procedures
• Service Chattiness Performance Problem
• Batch becomes a domain service in itself
– Business Workflow as a mini program
– Hard to decompose/compose without Service
– I want the business rules in the Domain Model…
Billing Batch – Pseudo Code
foreach (PlanAgreement planAgreement in GetPlanAgreements())
{
Agreement agreement = GetAgreement(planAgreement.Id);
foreach (PlanCategory planCategory in GetPlanCategories(planAgreement.Id))
{
PremiumTable premiumTable = GetPremiumTable(planCategory.Id);
foreach (PlanInsurance planInsurance in GetPlanInsurances(planCategory.Id))
{
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);
double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);
List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);
...
...
...
}
}
}
Billing Batch – Levels
Agreement
Category
Insurance
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable
• PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
• …
Misc…
• …
• …
Batch Observations
• High input/output entity ratio – 11 Entity Types as Input – Often Complex
– 2 as Output (1 Create, 1 Update) – Simple
– Simple State Semantics
– Opportunities for Caching
– (Responsibility Layers Analysis…)
• Data is centered around a few central business keys.
Potential for generalizing / streamlining the batch processing pipeline??
Billing Batch – Loops Flattened
PlanAgreement planAgreement = null;
Agreement agreement = null;
PlanCategory planCategory = null;
PremiumTable premiumTable = null;
foreach (PlanInsurance planInsurance in GetPlanInsurances()) {
if (planInsurance.PlanAgreement != planAgreement) {
planAgreement = planInsurance.PlanAgreement;
agreement = GetAgreement(planAgreement.Id);
}
if (planInsurance.PlanCategory != planCategory) {
planCategory = planInsurance.PlanCategory;
premiumTable = GetPremiumTable(planCategory.Id);
}
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);
double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);
List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);
...
}
Billing Batch – Levels Flattened
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable • PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
Agreement Category Insurance Misc…
• …
• …
Agreement Category Insurance Misc…
Agreement Category Insurance Misc…
Billing Batch – Entity Level Keys
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable • PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
Agreement Category Insurance Misc…
• …
• …
Agreement Category Insurance Misc…
Agreement Category Insurance Misc…
Agreement Level Key
Category Level Key
Insurance Level Key
Entities
Billing Batch – Generic Cursor Style
Plan Agreement
Plan Category
Insurance History
Master Keys
Agreement Level
Category Level
Insurance Level
Agreement
Plan Insurance History
Advice History Premium Table
… …
• Cursor Semantics • A Set of Master Keys Drives the Cursor • Entities Associated with Keys in Master • Each Row Contains Entities for a Unit-Of-Work
Entity Level Keys
Level Keys
Agreement Level = 4501
Category Level = 78
Insurance Level = ”56076”
Plan Insurance
Agreement ID = 4501
Category ID = 78
ID = ”56076”
• Map of EntityLevel & Values
– Dictionary<EntityLevel, object>
• Or derived from Entity Properties
The Entity Level Abstraction
public class PlanAgreement
{
[Level(typeof(AgreementLevel), IdentityType.Full)]
public int Id;
}
class AgreementLevel : EntityLevel {}
class CategoryLevel : EntityLevel {}
class InsuranceLevel : EntityLevel {}
Entity Cursor: Master + Entities
void Initialize()
{
var cursor = EntityCursor.For(SessionFactory, MetaData);
// MASTER: IEnumerable<object[]> OR IEnumerable<TEntity>
cursor.Master = GetMyMaster();
cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel());
cursor.Add(Query.For<PlanAgreement>());
// ADD MORE ENTITIES TO THE CURSOR...
while (cursor.MoveNext()) {
var currentPlanAgreement = cursor.Get<PlanAgreement>();
// PROCESS EACH ROW IN THE CURSOR...
}
}
IoC Style + Syntactic Sugar class MyBatch : BaseBatch
{
PlanAgreement planAgreement;
EntityLevel[] Levels() { return ... }
object[] Master() { return ... }
void Initialize() {
// Query Defintions that are not simple
// Query.For<MyEntity>()
Add<PlanAgreement>()
.Where(pa => pa.Foo != null);
}
void ProcessRow() {
var foo = this.planAgreement.Foo ...
// PROCESS THE ROW...
}
}
Row Processing
Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No
(1, “InNo-1")
(1, “InNo-2")
(1, “InNo-3")
(2, “InNo-4")
(2, “InNo-5")
(3, “InNo-6")
...
(n, “InNo-n")
Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") -||- Insurance(1, “InNo-2")
(1, “InNo-3") -||-
(2, “InNo-4") Agreement(2)
(2, “InNo-5") -||-
(3, “InNo-6")
...
(n, “InNo-n")
Row Processing Chunked Data Fetch
Query<Agreement>()
.Where(a => a.Id)
.IsIn(1, 2)
Query<Insurance>()
.Where ...
• Entities are fetched in Chunks
• Multiple chunk queries executed in one DB round-trip.
• NHibernate MultiCriteria (or Futures).
Master Keys Agreement Insurance
Key 1: Agreement ChunkSize: 2 ChunkSize: 2
Key 2: Insurance
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") Agreement(2) Insurance(1, “InNo-2")
(1, “InNo-3") Insurance(1, “InNo-3")
(2, “InNo-4") Insurance(2, “InNo-4")
(2, “InNo-5") Insurance(2, “InNo-5")
(3, “InNo-6") Agreement(3) Insurance(3, “InNo-6")
... ...
(n, “InNo-n")
Row Processing Indexing
• Each entity is indexed with the identifying level key(s). • Entities in chunks synced with key for current row as the
cursor proceeds forward.
Entity Grouping
class InsuranceHistory : GroupingEntity<Insurance>
{
static readonly Grouping Grouping = Grouping.For<Insurance>()
.By(i => i.AgreementId)
.By(i => i.InsuranceNumber);
public InsuranceHistory(IList<Insurance> values) { ... }
}
• Groups as a 1st class modeling concept
• Enriching the Domain Model
• “Virtual Aggregate Root” – Model Integrity
• Declarative expression (By, Where, Load)
Cursor.Add<PlanInsuranceHistory>();
Cursor.Add<PlanInsuranceHistory, PlanInsurance>()
.Where(...); // OK to override filter??
Complex Grouping – PremiumTable
• Rich Model Abstraction
• Complex data structure with lookup semantics
• No natural aggregate root
• Not cacheable in NHibernate session
• Fits well as a GroupingEntity
Value
10-22 23-30 31-45
0-20 100 120 135
20-40 110 130 150
40-65 130 160 190
Row Interval
ColumnInterval
Querying
• Filter per Entity – Cursor “Joins” using Shared Level Keys • ORM-semantics: Where, Load • Grouping Entity has query like qualities • Level Queries are statically defined query using Entity Levels
Keys to construct underlying ORM query (yes, coupling)
Conceptual API:
Cursor.Add(Query entityProvider)
Query.For<PlanInsurance>()
.Where(insurance => insurance.IsActive)
.Load(insurance => insurance.Category)
Query.For<AdviceHistory>()
Query.For(PremiumTable.ByAgreement)
.IndexBy(table => table.TableId)
Versioning
public class PlanInsurance
{
[Level(typeof(AgreementLevel), IdentityType.Partial)]
public int AgreementId;
[Level(typeof(InsuranceLevel), IdentityType.Partial)]
public string InsuranceNumber;
[VersionLevel(typeof(PlanInsurance), IdentityType.Partial)]
public int Version;
}
• Core to many business domains
• Has its own set of semantics
• Common in Groups – Latest<Insurance> vs InsuranceHistory
• Implemented in different ways in the DB
• Expressed declaratively
• Uniform Query Semantics
What About The Services? void ProcessRow()
{
...
var premiumService = new PremiumService
{
PlanAgreement = Cursor.Get<PlanAgreement>(),
PlanInsurance = Cursor.Get<PlanInsurance>(),
Insurance = Cursor.Get<Insurance>(),
Insured = Cursor.Get<Person>(),
PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(),
PremiumTable = Cursor.Get<PremiumTable>(),
RiskTable = Cursor.Get<RiskTable>()
};
var premium = premiumService.CalculatePremium(advicePeriod);
...
} • Service has pure calculation responsibility
• Dependencies are injected by client
• Coupling…? Boilerplate Smell…?
Conclusions
• Data Access Abstraction with Power & Ease of Use • Declarative & Composable Entity Pipeline • Minimizes DB Round-trips; Favors Eager Loading • Repositories Become Redundant • No More Unconstrained Services – “Calculators” / …??? • Richer Domain Model – Less Supporting Objects, More Domain
Related Objects • DDD/ORM + Legacy DB == True • Composite DB Key Thinking Essential to the Solution • Patching the DB Model with Entity Level Abstraction… • What’s Next? – Lots of Low Hanging Fruit… TOWARDS AN EXECUTABLE ARCHITECTURE…???
What’s Next? – Entity Injection
Cursor.Add<PremiumCalculator>();
void ProcessRow()
{
...
var calculator = Get<PremiumCalculator>();
var premium = calculator.Calculate(advicePeriod);
...
}
• Cursor can inject entity dependencies automatically
• Calculators dependencies can be inferred and added to cursor automatically
• ”Calculator” define Cursor Entities Implicitly
What’s Next? – Stateful Calculators?
class PremiumCalculator
{
...
double CalculatePremium(...) {}
...
}
• What if we treated a calculation as a stateful object? • Calculations become data flows through the system • Stateful Objects as the Uniform Expression – Simplifies
declarative programming • Captures Multiple / Intermediate Calculation Results • Can be bound to a UI • Additional state in the cursor – UI could add presentation
model/wrapper to the cursor
class PremiumCalculation
{
...
double Premium;
...
}
What’s Next? – Entity Pipeline
class BillingCalculation : EntityPipeline
{
void Initialize() {
Add<PlanAgreement>();
...
}
}
var monthlyBatch = new BillingCalculation();
monthlyBatch.Master = GetMasterForMonthlyBatch();
monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);
monthlyBatch.Execute();
var singleInstance = new BillingCalculation();
singleInstance.Master = new object[]{ 24, "InNo-1"};
singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);
singleInstance.Execute();
var nextUIPage = new BillingCalculation();
nextUIPage.Add<MyUIModel>();
nextUIPage.Master = GetMasterForNextPage();
myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())
What’s Next? – New Data Providers
• File Processing for Data Imports – Prototyped batch framework
• Document Based Persistence – Premium Table for example
• Hybrid Persistence – Serialized object graphs in SQLServer
• SOA Integrations – Loosely Coupled Bounded Contexts
• Parallel data fetch – Multiple DBs / Data Services
What’s Next? – Business Events
• Entity Processing Pipeline seems to be a good environment for triggering and/or handling business events based on persistence events.
• Poor man’s Business Events!?!?
What’s Next? – Greenfield
• Search the Core Domain/Application Semantics – Built-in Versioning from the start e.g. – Semantic Storage…
• Streamline – Uniform Expression – Semantics – Patterns
• Be Opinionted – Constraints are Liberating
• Executable Architecture
Thanks For Listening!!!
Questions?