Download - Refactoring toward deeper insight java forum
Refactoring Toward Deeper Insight
DDD Findings In Batch Processing A Case Study
Andreas Brink, factor10
Disclaimer!!! .NET not Java
Work In Progress…
Danica Pension
• Core financial business system.
• ASP.NET / RDBMS / Integrations
• Generations of technologies / architectures.
• Lots of financial business rules.
• Lots of Batch Processing.
• Mission: Taking Control of the Code Base – DRY, Understandability, Testability, Automation
• DDD/DM + ORM fits well!
My View of the DDD Toolbox
Philosophy & Principles • Ubiquitous Language • Model Driven • Declarativity • Distilling The Model • Breaktrhough • …
Model Design & Impl. • Entity • Aggregate • Value Object • Respository • Service • Specifiction • Side-Effect Free Functions • …
Strategic Design • Bounded Contexts • Responsibilty Layers • …
Domain Model Pattern (DM)
Supple Design • Assertions • Side-effect free
Functions • Standalone Classes • Closure of Operations
Design Sweet-Spot • Understanding & Communication • Testability, Executable Specifications But, not like the Office Object Model… • Must scale • Not one single file on disk
Basic Building Blocks • Entity, • Aggregate • Value Object • Specification • Factory
Implementability
Service
Repository
• Object Navigation does not scale Repositories
• DM does not scale well with composition & coupling Services
• Problem Solved !?!?
Some Layer…
ORM
Implementation Mess
Service
Repository
• Less focus on Domain Model • Services – The Ultimate Loophole
– Touches big parts of the system – horizontally, vertically – Side Effects Understandability & Testing Problems
• Decentralized Flow Control & Data Access… Global Optimization & Generic Processing hard or impossible Performance Problems
ORM Some Scenario…
Why DDD is Hard
Service
Repository
• Model Design is hard to begin with – OO Expertise is still quite rare
• Have to be a Design Patterns / ORM / Architecture Expert
• Fail to get an Easily Consumable & Testable Model
• Same old Procedural Service Soup (+ some entities…)
ORM Some Scenario…
My DDD Challenge
• Reclaiming the Domain Model – Easy Reasoning, Consumption & Testing
• REAL Separation of Concerns – Not just a complex web of objects and method
calls behind superficially simple interfaces
• And with Batch Processing in the mix…
IS THIS POSSIBLE??
Batch in the mix…
• ”ORM is not for Batch, use the right tool…”
• DDD/ORM vs Stored Procedures
• Service Chattiness Performance Problem
• Batch becomes a domain service in itself
– Business Workflow as a mini program
– Hard to decompose/compose without Service
– I want the business rules in the Domain Model…
Billing Batch – Pseudo Code
foreach (PlanAgreement planAgreement in GetPlanAgreements())
{
Agreement agreement = GetAgreement(planAgreement.Id);
foreach (PlanCategory planCategory in GetPlanCategories(planAgreement.Id))
{
PremiumTable premiumTable = GetPremiumTable(planCategory.Id);
foreach (PlanInsurance planInsurance in GetPlanInsurances(planCategory.Id))
{
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);
double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);
List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);
...
...
...
}
}
}
Billing Batch – Levels
Agreement
Category
Insurance
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable
• PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
• …
Misc…
• …
• …
Batch Observations
• High input/output entity ratio – 11 Entity Types as Input – Often Complex
– 2 as Output (1 Create, 1 Update) – Simple
– Simple State Semantics
– Opportunities for Caching
– (Responsibility Layers Analysis…)
• Data is centered around a few central business keys.
Potential for generalizing / streamlining the batch processing pipeline??
Billing Batch – Loops Flattened
PlanAgreement planAgreement = null;
Agreement agreement = null;
PlanCategory planCategory = null;
PremiumTable premiumTable = null;
foreach (PlanInsurance planInsurance in GetPlanInsurances()) {
if (planInsurance.PlanAgreement != planAgreement) {
planAgreement = planInsurance.PlanAgreement;
agreement = GetAgreement(planAgreement.Id);
}
if (planInsurance.PlanCategory != planCategory) {
planCategory = planInsurance.PlanCategory;
premiumTable = GetPremiumTable(planCategory.Id);
}
Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);
InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);
AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);
double premium = CalculatePremium(planAgreement, agreement, planCategory,
premiumTable, planInsurance, insurance);
List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);
...
}
Billing Batch – Levels Flattened
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable • PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
Agreement Category Insurance Misc…
• …
• …
Agreement Category Insurance Misc…
Agreement Category Insurance Misc…
Billing Batch – Entity Level Keys
• Agreement
• PlanAgreement
• PlanCategory
• PremiumTable • PlanInsuranceHistory
• LatestPlanInsurance
• InsuranceHistory
• LatestInsurance
• InsuranceAccount
• AdviceHistory
Agreement Category Insurance Misc…
• …
• …
Agreement Category Insurance Misc…
Agreement Category Insurance Misc…
Agreement Level Key
Category Level Key
Insurance Level Key
Entities
Billing Batch – Generic Cursor Style
Plan Agreement
Plan Category
Insurance History
Master Keys
Agreement Level
Category Level
Insurance Level
Agreement
Plan Insurance History
Advice History Premium Table
… …
• Cursor Semantics • A Set of Master Keys Drives the Cursor • Entities Associated with Keys in Master • Each Row Contains Entities for a Unit-Of-Work
Entity Level Keys
Level Keys
Agreement Level = 4501
Category Level = 78
Insurance Level = ”56076”
Plan Insurance
Agreement ID = 4501
Category ID = 78
ID = ”56076”
• Map of EntityLevel & Values
– Dictionary<EntityLevel, object>
• Or derived from Entity Properties
The Entity Level Abstraction
public class PlanAgreement
{
[Level(typeof(AgreementLevel), IdentityType.Full)]
public int Id;
}
class AgreementLevel : EntityLevel {}
class CategoryLevel : EntityLevel {}
class InsuranceLevel : EntityLevel {}
Entity Cursor: Master + Entities
void Initialize()
{
var cursor = EntityCursor.For(SessionFactory, MetaData);
// MASTER: IEnumerable<object[]> OR IEnumerable<TEntity>
cursor.Master = GetMyMaster();
cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel());
cursor.Add(Query.For<PlanAgreement>());
// ADD MORE ENTITIES TO THE CURSOR...
while (cursor.MoveNext()) {
var currentPlanAgreement = cursor.Get<PlanAgreement>();
// PROCESS EACH ROW IN THE CURSOR...
}
}
IoC Style + Syntactic Sugar class MyBatch : BaseBatch
{
PlanAgreement planAgreement;
EntityLevel[] Levels() { return ... }
object[] Master() { return ... }
void Initialize() {
// Query Defintions that are not simple
// Query.For<MyEntity>()
Add<PlanAgreement>()
.Where(pa => pa.Foo != null);
}
void ProcessRow() {
var foo = this.planAgreement.Foo ...
// PROCESS THE ROW...
}
}
Row Processing
Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No
(1, “InNo-1")
(1, “InNo-2")
(1, “InNo-3")
(2, “InNo-4")
(2, “InNo-5")
(3, “InNo-6")
...
(n, “InNo-n")
Master Keys Agreement Insurance
Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2
Key 2: Insurance No
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") -||- Insurance(1, “InNo-2")
(1, “InNo-3") -||-
(2, “InNo-4") Agreement(2)
(2, “InNo-5") -||-
(3, “InNo-6")
...
(n, “InNo-n")
Row Processing Chunked Data Fetch
Query<Agreement>()
.Where(a => a.Id)
.IsIn(1, 2)
Query<Insurance>()
.Where ...
• Entities are fetched in Chunks
• Multiple chunk queries executed in one DB round-trip.
• NHibernate MultiCriteria (or Futures).
Master Keys Agreement Insurance
Key 1: Agreement ChunkSize: 2 ChunkSize: 2
Key 2: Insurance
(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")
(1, “InNo-2") Agreement(2) Insurance(1, “InNo-2")
(1, “InNo-3") Insurance(1, “InNo-3")
(2, “InNo-4") Insurance(2, “InNo-4")
(2, “InNo-5") Insurance(2, “InNo-5")
(3, “InNo-6") Agreement(3) Insurance(3, “InNo-6")
... ...
(n, “InNo-n")
Row Processing Indexing
• Each entity is indexed with the identifying level key(s). • Entities in chunks synced with key for current row as the
cursor proceeds forward.
Entity Grouping
class InsuranceHistory : GroupingEntity<Insurance>
{
static readonly Grouping Grouping = Grouping.For<Insurance>()
.By(i => i.AgreementId)
.By(i => i.InsuranceNumber);
public InsuranceHistory(IList<Insurance> values) { ... }
}
• Groups as a 1st class modeling concept
• Enriching the Domain Model
• “Virtual Aggregate Root” – Model Integrity
• Declarative expression (By, Where, Load)
Cursor.Add<PlanInsuranceHistory>();
Cursor.Add<PlanInsuranceHistory, PlanInsurance>()
.Where(...); // OK to override filter??
Complex Grouping – PremiumTable
• Rich Model Abstraction
• Complex data structure with lookup semantics
• No natural aggregate root
• Not cacheable in NHibernate session
• Fits well as a GroupingEntity
Value
10-22 23-30 31-45
0-20 100 120 135
20-40 110 130 150
40-65 130 160 190
Row Interval
ColumnInterval
Querying
• Filter per Entity – Cursor “Joins” using Shared Level Keys • ORM-semantics: Where, Load • Grouping Entity has query like qualities • Level Queries are statically defined query using Entity Levels
Keys to construct underlying ORM query (yes, coupling)
Conceptual API:
Cursor.Add(Query entityProvider)
Query.For<PlanInsurance>()
.Where(insurance => insurance.IsActive)
.Load(insurance => insurance.Category)
Query.For<AdviceHistory>()
Query.For(PremiumTable.ByAgreement)
.IndexBy(table => table.TableId)
Versioning
public class PlanInsurance
{
[Level(typeof(AgreementLevel), IdentityType.Partial)]
public int AgreementId;
[Level(typeof(InsuranceLevel), IdentityType.Partial)]
public string InsuranceNumber;
[VersionLevel(typeof(PlanInsurance), IdentityType.Partial)]
public int Version;
}
• Core to many business domains
• Has its own set of semantics
• Common in Groups – Latest<Insurance> vs InsuranceHistory
• Implemented in different ways in the DB
• Expressed declaratively
• Uniform Query Semantics
What About The Services? void ProcessRow()
{
...
var premiumService = new PremiumService
{
PlanAgreement = Cursor.Get<PlanAgreement>(),
PlanInsurance = Cursor.Get<PlanInsurance>(),
Insurance = Cursor.Get<Insurance>(),
Insured = Cursor.Get<Person>(),
PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(),
PremiumTable = Cursor.Get<PremiumTable>(),
RiskTable = Cursor.Get<RiskTable>()
};
var premium = premiumService.CalculatePremium(advicePeriod);
...
} • Service has pure calculation responsibility
• Dependencies are injected by client
• Coupling…? Boilerplate Smell…?
Conclusions
• Data Access Abstraction with Power & Ease of Use • Declarative & Composable Entity Pipeline • Minimizes DB Round-trips; Favors Eager Loading • Repositories Become Redundant • No More Unconstrained Services – “Calculators” / …??? • Richer Domain Model – Less Supporting Objects, More Domain
Related Objects • DDD/ORM + Legacy DB == True • Composite DB Key Thinking Essential to the Solution • Patching the DB Model with Entity Level Abstraction… • What’s Next? – Lots of Low Hanging Fruit… TOWARDS AN EXECUTABLE ARCHITECTURE…???
What’s Next? – Entity Injection
Cursor.Add<PremiumCalculator>();
void ProcessRow()
{
...
var calculator = Get<PremiumCalculator>();
var premium = calculator.Calculate(advicePeriod);
...
}
• Cursor can inject entity dependencies automatically
• Calculators dependencies can be inferred and added to cursor automatically
• ”Calculator” define Cursor Entities Implicitly
What’s Next? – Stateful Calculators?
class PremiumCalculator
{
...
double CalculatePremium(...) {}
...
}
• What if we treated a calculation as a stateful object? • Calculations become data flows through the system • Stateful Objects as the Uniform Expression – Simplifies
declarative programming • Captures Multiple / Intermediate Calculation Results • Can be bound to a UI • Additional state in the cursor – UI could add presentation
model/wrapper to the cursor
class PremiumCalculation
{
...
double Premium;
...
}
What’s Next? – Entity Pipeline
class BillingCalculation : EntityPipeline
{
void Initialize() {
Add<PlanAgreement>();
...
}
}
var monthlyBatch = new BillingCalculation();
monthlyBatch.Master = GetMasterForMonthlyBatch();
monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);
monthlyBatch.Execute();
var singleInstance = new BillingCalculation();
singleInstance.Master = new object[]{ 24, "InNo-1"};
singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);
singleInstance.Execute();
var nextUIPage = new BillingCalculation();
nextUIPage.Add<MyUIModel>();
nextUIPage.Master = GetMasterForNextPage();
myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())
What’s Next? – New Data Providers
• File Processing for Data Imports – Prototyped batch framework
• Document Based Persistence – Premium Table for example
• Hybrid Persistence – Serialized object graphs in SQLServer
• SOA Integrations – Loosely Coupled Bounded Contexts
• Parallel data fetch – Multiple DBs / Data Services
What’s Next? – Business Events
• Entity Processing Pipeline seems to be a good environment for triggering and/or handling business events based on persistence events.
• Poor man’s Business Events!?!?
What’s Next? – Greenfield
• Search the Core Domain/Application Semantics – Built-in Versioning from the start e.g. – Semantic Storage…
• Streamline – Uniform Expression – Semantics – Patterns
• Be Opinionted – Constraints are Liberating
• Executable Architecture
Thanks For Listening!!!
Questions?