schema less table & dynamic schema

Schema-less table & Dynamic SchemaDavide Mauridmauri@solidq.com@mauridb

Davide Mauri

• Microsoft SQL Server MVP

• Works with SQL Server from 6.5, on BI from 2003

• Specialized in Data Solution Architecture, Database Design, Performance Tuning, High-Performance Data Warehousing, BI, Big Data

• President of UGISS (Italian SQL Server UG)

• Regular Speaker @ SQL Server events

• Consulting & Training, Mentor @ SolidQ

• E-mail: dmauri@solidq.com

• Twitter: @mauridb

• Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

Agenda

• Schema, Schemaless & Implicit Schemas

• Possible solutions

• Conclusion

Schema

• “A priori” definition of data structures

• Allows data to be inserted if and only if it is compatible with the schema

• Es: RDBMS Table, XML Schema, Class, Struct

Schemaless (?)

• No definition at all on the data you expect to have. • Unstructured data.

• For example: text files, binary files• with no metadata and no position-based format

• In one word: chaos

Implicit Schema

• In reality a schema always exists, albeit implicit• Otherwise it would be impossible to handle data

Implicit Schema

Any data that doesn't fit this implicit schema will not be manipulated properly, leading to errors.

(Schemaless data structures, Martin Fowler)

Pros

• Flexibility• Easy to manage

• actually, almost no management at all

• Easy to be extended• Just add a new element and you’re done

• Easy to be used• No mismatch between OOP and other models

Cons

• Schema information are hidden somewhere• Scattered all across the codebase

• It’s really difficult to keep under control the chaos that can emerge• For example two different element that contains the same information

• CustomerName and Customer_Name

• You still need to have a sort of «First Normal Form» in order to avoid inconsistency and code inefficiencies

Cons

• It’s really difficult to define and maintain integrity constraints • Data Integrity is a value that must be preserved!

• Otherwise we’ll have data, not information

• XML Schema were born for that specific reason

• Without Data Integrity, the process of extracting information from data becomes• Difficult

• Expensive

• Untrustable

Words of Wisdom

«Schemaless => implicit schema = bad.

Prefer an explicit schema»(Schemaless data structures, Martin Fowler)

But if we need it anyway?

• What if my use case is one that perfectly fits the need for a implicit schema?

• The only possible solution are the so-called «No-SQL» databases• Document Database or Key-Value store?

• How can I integrate it into already existing database?

• Integration does not come for free!

Schemaless & RDBMS

• (Usually) Are the exact opposite extremes

• Still is a very common request• CRM, eCommerce, ERPs….

• Schemaless is used not only for pure data persistence

Solution within an RDBMS

• «Custom» columns• Custom1, Custom2

• In-Table Data Structures• BLOB, XML, JSON, «Complex» columns

• Entity-Attribute-Value Models

«Custom» Columns

• A problem until SQL Server 2008• Space is still used for fixed length column even if they contain a NULL

value

• With SQL Server 2008 the «Sparse Column» feature comes to help• Helps to make the schema easily modifiable, even in presence of

existing data

• Changes to the schema must still be done with «ALTER TABLE»

«Custom» Columns

• Sparse Columns• Are Columns at 100%

• Optionally you can have *all* the Sparse Columns returned as a single XML column• «Column Set»

• Make development easier

• Do not take space if not used • But use more space when used

DemoDynamic Schema & Sparse Columns

In-Table Data Structures

• Complete support for XML• XPath/XQuery

• XML Index

• Performance «Good Enough»• But not optimal (compared with the equivalent relational approach)

• Use a lot of space

In-Table Data Structures

• XML Sometimes needs some help to boost performance

• Would be nice to be able to «promote» elements to turn them into real columns• Must be done manually using a choice of

• Triggers

• Stored Procedure

• Data Access Layer

• Service Broker

In-Table Data Structures

• JSON support is still missing in SQL Server• But others database like PostgreSQL already have it…• …so we can see it coming to MS Platform too

• Right now one solution is to use SQLCLR• Solutions available surfing the web:• http://www.sqlservercentral.com/articles/SQLCLR/74160/• http://www.json4sql.com/examples.html

• There is also a pure T-SQL solution• https://www.simple-talk.com/sql/t-sql-programming/consuming-json-strings-

in-sql-server/

In-Table Data Structures

• Blob is an option if you just need to do persistence

• Blob can be stored in different way• «Classic» blob inside SQL Server pages & extents

• Blob in a filestream

• Blob in a filetable

DemoDynamic Schema & In-Table Data Structures

Entity-Attribute-Values

• Old and very common technique to store attribute-value pairs• Some well-known samples: Wordpress

• Works on any RDBMS• No «special» features required

• There’s a huge debate around it • http://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_mo

del

• But until SQL 2005 no true alternative

Entity-Attribute-Values

Entity-Attribute-Values

• Offers maximum flexibility• No real control over data types.

• Options to deal with data types• All strings• SQL Variant• One-Column-Per-Type

• Complex query pattern for «AND» predicates between attributes• «Return all the entities that have «CPU=i7» and «Display=15.4’»

Entity-Attribute-Values

• Queries requires the implementation of a relational operator not implemented in common RDMBS• «Relational Division»

• Document and well explained in theory• It is quite easy to implement it. Follow theory + add some pepper to boost

performances

Relational Division

• Let’s get back to theory a little bit, in order to see the problem from a more open perspective:

Dividend

Divisor Result

Remainder

𝛼

𝛽

Relational Division

• How do we implement the division?

• Thanks to Codd and the relational theory we already have the solution

28

Relational Division

• Thanks to relational algebra we know that the division is expressed as

• Generate all possibile pairings

• Remove existing pairing• (Now we’ve found all pairings that are NOT answers)

• Remove the non-answers from the dividend

29

DemoDynamic Schema & EAV

Conclusions

• It works! • Performance more than good

• Choose the solution that better fits your use-case• Search for attributes only?

• Persistence only?

• Search for attributes & values?

• Performance read, write, read/write?

Conclusions

• Use it if and only if when really needed

• Always remeber the «Words of Wisdom» • If you can define and use a schema.

• It may seem «not cool» and convoluted but in the long term is the best solution.• *data* *must* *be* *turned* *into* *information*

• Sooner or later

• Without metadata (a schema) it’s really really really hard!

Questions?

Thanks!

• If you want to rate this session on my SpeakerScore page:

• www.speakerscore.com

• Feedback Key: TZQL

Demo Material

• Can be found here• http://1drv.ms/1Av5mb5

• Everything is release under the Creative Common Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) • http://creativecommons.org/licenses/by-nc-sa/4.0/