Test-Driven Architecture Article Accepted by InformIT

As you may recall some friends of mine and I are working on articles and blog entries intended to reform architecture for the modern age.  This is a quick note that the article Mike Brown and I wrote regarding the impact of TDD on architecture has been submitted to the publisher and is in the editorial phase.

I'll post something here when it is published.

The Version Manager

One of the more recent phases in my journey to a class of databases was understanding that the best way to codify a database's design was as the series of revisions that could get you there rather than in terms of specific design elements.  It's harder to find fault in this way of thinking because I still use and recommend it today but it was incomplete; sufficient to enable some level of sustainable database development but not true test-driven database development.

Let me refresh your memory on the building technique to which I am referring.  You store an ordered list of scripts that are used to build a database.  You build some infrastructure that ensures only the right scripts are executed against any given database and in the right order.  So, if you have a version 2 database instance and you want to upgrade it to version 5, the infrastructure will execute the steps to upgrade to version 3, then 4, then 5.

Digging the nugget of truth out of this way of doing things is weird because the thing is its own nugget of truth and, at the same time, is not sufficient to support a modern test-driven environment.  The reason for this weirdness is that version-based builds are the best way I know to do things but they aren't enough.  More was required.

Anyway, the grain of truth in this way of thinking was the recognition that the actual revisions applied to real production databases should govern how we organize database build scripts.  That is, while I had not yet discovered the true class of database concept - at least not as I understand it today - I had discovered one of the principles that drives test-driven database development: the historical reality of important database instances must be respected and always trumps our wishes, hopes, and ideals.

Knowledge, Behavior, and Information

I mention this in my book but I thought I might elaborate.  I think a useful way to conceptually divide the parts of a database is into three groups of design elements: information, knowledge, and behavior.

The Three Concepts

Information and knowledge are often confused in everyday language.  So, first, I'll disambiguate those two words.  Not all data are information or knowledge and rarely are the two interchangeable.

Information is a special subclass of data.  I'm sure someone who is an expert in communication theory or some other kind of academic would be glad to correct me and I won't fight them on what the technical definition of the word is.  I'm only interested in what the useful definition for my own purposes is, not the officially right definition.  For the purposes of this blog entry, and of everything I say and write, it is the part of a signal that the recipient did not know in advance.  Simply put, information is data which informs its recipient.

If information could be thought of as facts in transit, then knowledge would be facts at rest.  In essence, knowledge is potential information but it is also a potential driver for action.  That is, the two uses of any given object's knowledge are to inform other objects, thereby adding to their knowledge, and to inform decisions, thereby improving the value of an action taken.

That latter purpose is the perfect opening to briefly introduce the third player in the database design world: behavior.  If knowledge and information are facts at rest and in motion, behavior can be thought of as how something responds to knowledge or information.  For instance, you drive on the correct side of the road because you know you will slam into something if you do.  Likewise, you yank your hand away from a too-recently poured cup of coffee have become informed that the cup is too hot to touch without damaging tissue.

Information as Pertains to Databases

In the database world, information is the set of signals sent or received by a database.  A query and its parameters, the invocation of a stored procedure, the results set, and any errors that occurred are all examples of information as a database sees it.

In essence, information is the "surface" of a database's design.  It is impossible for external parties to access the value of a database except by sending and receiving signals.

Moreover, it is the means by which value is conveyed between a database and its clients.  It is pointless to update a database with information it already knows.  It is useless to query a database for what you already know.  Value is created by such actions that results in one of the other entities "learning" something.

Knowledge as Pertains to Databases

Knowledge is the reason why databases cannot be maintained using the simple "blast and rebuild" upgrade path we apply to most software deployment problems.  All the facts stored in a database are knowledge; not all the data, because you can introduce noise into a database's design, but all the facts.

Knowledge is the purpose of a database.  Most software products and components exist to convey facts between parties or to process data and discover new facts.  Some software exists to entertain.  Databases exist to preserve knowledge.  Each production database is a modern day Library of Alexandria, complete with the ability for some asshole to burn it down and, in so doing, to cause irreversible damage.

We have known this for a long time - as close to "forever" as matters.  Databases have always been designed around the knowledge they capture and preserve.  Those design decisions stand as a reflection of our implicit understanding that databases aren't merely data bases, but knowledge bases.

Behavior as Pertains to Databases

So what is the role of behavior in database design?  It's another one of those things that can be put simply or drawn crisply, but can take a lot of work to implement correctly.  The role of behavior in a database is to mediate between knowledge and information.

All the information that a database receives needs to be translated into knowledge and stored for safekeeping.  Why?  So that, later, that knowledge can be translated back into information to help other actors make decisions or discoveries.

You can codify the behavior of a database in many different ways.  At the time of this entry, the most common way is to couple the behavior offered by a database directly to the kinds of knowledge that database can store.  This is accomplished by creating, publicly exposing, and coupling clients to table structures and relationships.

The Relationship Between the Three

I find it useful to divide database design into three parts.  The information layer of design is where the interactions between databases and other objects are defined.  The knowledge layer of design is where the facts you want to store in a database are housed.  The behavior layer is where one codifies the manner in which facts are absorbed or emitted.

behavior translates between knowledge and information

I'll post more on each of the specific layers of design with some implementation recommendations later.

Division of Ideas

We choose how we organize ideas in our heads.  Take the concept of a wrench.  For me, there are two kinds of wrenches: regular wrenches and monkey wrenches.  That distinction exists because one kind of wrench is useful for binding while the other is good for bashing.  For a mechanic, however, I'm sure there are many kinds of wrenches.

I'm not sure either of us is wrong.  Now, I want to be clear: I am not saying that everyone who believes anything is correct because it's what they believe.  I am saying that someone who looks at a room full of people and sees two groups is every bit as right as someone standing right next to him who sees three groups in that there is no "right" way to divide things into groups, only "useful" ways to do so.

This may be obvious to others but it wasn't obvious to me.  I'm documenting it here as much to ensure I'll have the discovery available for myself when I forget it later as to share it with you.

The Database Installer

Another fallacious idea I, and many others, had was to treat database instances like programs that need to be installed.  Again, there are many things wrong with this line of reasoning, but something positive came from it.

You know me... "Mr. Positive."

This particular step in my journey to a class of databases bore what was, at least for me, a pretty subtle value.  Part of the subtlety came from the fact that the installer paradigm looks like it works for longer than a lot of its predecessors, which tended to break down extremely early.  Part of it was my own stubbornness - I was spending so much energy arguing the small improvement that I couldn't see the bigger improvements waiting just around the corner.

I'm pretty sure that's irony: that this way of thinking was so successful kept me from seeing other, more successful, ways of understanding a problem.  Wait.  Maybe that's not irony.  Maybe that's the human condition.  ...or maybe those things do not really oppose one another.

Anyway, grain of truth in this way of imagining database build technologies is that it recognizes the importance of discrete, tracked, testable deltas in design and highly controlled, repeatable ways of introducing those changes.  That ends up being a pretty fundamental concept.  It serves as the basis for building a testable class of databases, enabling test-driven database development, and ultimately unlocking database agility.

So there you have it.  Another step in the journey.  Another failure to hit the mark.  Another lesson that built to what we know today.