Another Generally Positive Review for Test-Driven Database Development

I would be a fool to think that my first book would receive only positive feedback.  This review, I think, is excellent.  I like it for three reasons:

1. My own self-interest
2. The reviewer clearly understood what he was reviewing
3. It contains actionable criticisms that will benefit a 2nd edition, if there is one

Vanity of the Database Authors

Mr. Carlson's review is, I think, generally positive.  This means that it will cause more people to buy my book, which is nice.  It also means that it fans the flames of my own vanity, which is really nice.  Most importantly, though, it means that more people are likely to actually read my book which, in turn, means more people are likely to take a sane approach to database development.

Deep Understanding

The author of the review in question clearly took the time to read, understand, and evaluate the concepts in my book.  I think a lot of people who buy books do this but, for some reason, only about half of people who write reviews appear to do it.

Mr. Carlson didn't necessarily understand all of my motivations or what I know about outside the scope of Test-Driven Database Development: Unlocking Agility but... hey... who's fault is that?  Right?

Actionable Criticisms

Not everything Mr. Carlson has to say is positive but those bits which are negative are highly actionable.  He says I give the appearance of not understanding normalization.  I do understand how to do it and why people do it in addition to why it actually should be done.  However, looking back, I can see how someone might get the impression I don't if they were basing their opinion solely on my book.  In the 2nd edition, I will remedy that by adding material that addresses those kinds of concepts.

Likewise, he makes the point that I don't address object-relational-mapping at all.  He rightly guessed that it is because ORM runs contrary the message I send - databases as instances of classes with tightly-controlled and rigorously-tested sets of exposed behaviors.  Nevertheless, his argument is that ORM is a very popular concept in designing interactions with databases is true.  If I ever get to do a second edition, which I happily think I might, and ORM is still popular, which I sadly believe it may be, then I will address it at that time.

I may address those concepts sooner, too, in the form of blog entries or articles.

Test-Driven Architecture Article Published

The article my friend, Mike Brown, and I wrote has been published.  The topic is how to apply test-driven development to architecture.  You can find it here.

Why Revision Control is Not Enough

Several people have suggested that revision control might be an alternative to creating a class of databases.  I really have no way to counter such an argument because I cannot fathom what they were thinking in the first place.

The two concepts target completely and utterly different things.  I guess that all I can really do is talk about how different they are and hope some people change their stance.

Revision control (e.g.: CVS, Subversion, or Perforce) allows you to track and manage the changes you are making to source code.  The point is to (a) provide a place of safekeeping for your documents of design and (b) to allow you to access previous versions of a design.  When you update the design of a class, you create a new version of it in source control and do not generally see older versions unless you go looking for them.

The point is to allow you to work on the most recent design in a particular branch while keeping a history of what was done and not betting on a single computer to keep your source code safe.  In short: Source control is about managing the relationship between developers and designs.

A class of databases has a totally different functions.  Like any other class, it's most interesting relationship is between it and its objects.  Yes, a database class must store the steps required to build older versions in it but for a totally different reason that why a version-control system stores older versions of source code.

Whereas a revision control system tracks versions for the benefit of the development process, a class of databases must track revisions for the benefit of the database-construction process.

Storing the incremental deltas required to get from each version to the next and forcing all databases in a class to follow the exact same path of growth enables test-driven, agile database development in two ways.  For one thing, it drastically simplifies the database deployment model; every single database of a given type grows exactly the same way.  For another, it allows you to properly test the transitions from one version to the next.

The versions of a source file are created because a developer has learned something, changed his mind, or otherwise revised design and they are created to ensure that developers can revisit previous designs.  The versions of a database class are created because a revision to design was actually released to production and they are created so that every database in a class is built in the exact same way as every other database in that class.

Encapsulation of Database Design

It seems to me like every software developer goes through a phase where they think stored procedures are bad and a phase where they think they are good.  Like me, some alternate through these positions several times before settling on a position that works.

In the end, I discovered that there was never a dichotomy in the first place.  The real question was not whether to use stored procedures but how to use them.  The struggle over whether or not to use stored procedures was never really about stored procedures.  It was always about properly separating concerns and encapsulating designs.

In the Beginning...

I started off loving the idea of stored procedures.  I loved them based on a single argument that, at the time, was possibly relevant: performance.  That pernicious specter that looms over our industry, the myth of proactive optimization, cast its spell over me and I was totally sold.

In the nineties, it might even have been true that there was a compelling performance-based reason to write a stored procedure.  In general, for modern database systems, that is no longer the case.

Flip

As with many people in the pro-stored-procedure camp, I ran into one of the many potential failings of heavy stored-procedure use.  Putting behavior in a stored procedure because it is "faster" or "more appropriate" often drives bad design decisions.

The more logic I added to a database, the harder it was to test.  This was before I stumbled into the ideas in my book but it would still be true, at least to some extent, today.  Also, the fatter I made my database layer, the thinner I made the other layers.  The coup de grace was the fact that, on a regular basis, large stored procedures were actually slower than performing dumb selects and processing the data into meaningful results in a middle-tier service.

Essentially, every time I transferred business logic from somewhere more appropriate to a database, I made my system harder to test, less flexible, and slower.  So I did what humans often do and switch from one extreme to its opposite: no stored procedures of any kind.

Flop

The problem with toggling between extremes is that it reduces the world to binary decisions.  Of course it is true that, in any decision, one is considering a finite set of options which can be reduced to a series of binary decisions.  There's nothing wrong with choosing between two things but there is something wrong with limiting your options early in the decision-making process.

The "no stored procedures" stance didn't work out very well either, although it does work better than "everything in the database" way of working.  Instead of a tightly-wound, change resistant machine that served as the center of an application's universe and ground change to a halt, I was building delicate arrangements of glass work that shattered at the first hint of change which served as the center of an application's universe and ground change to a halt.

The problem with not having any stored procedures is that something has to have an intimate relationship with the structure of a database.  When the stored procedures go away, that tends to become whatever is connected to the database (usually a middle-tier service).

That kind of intricate understanding of implementation details between one module and another is inherently more difficult to enforce that such an understanding residing exclusively within a module.  It drives the cost of change through the roof.

The Other Way

As is often the case, the artificial binary decision was hiding the true nature of the decision in the first place.  The question was never "should stored procedures be used or not?"  The right question to ask is "for what should we use stored procedures?"  Note the "yes" and "no" answers to the former question map to answers in the latter but there are many other potential answers as well.

I don't want to iterate through all the possible ways stored procedures could be used in a database design; mostly because that list is probably infinite in length.  So I'll just cut straight to the one that I think is right: encapsulation.

A while back, someone I was working with, Christopher Taylor, suggested to me that stored procedures were the equivalent of a class's interface for a database.  They are a contract between a database and its clients.  So long as the contract is fulfilled, you are free to revise the implementation details to your heart's desire.  He didn't use these words and I don't recall the exact words he used but it the point is that it was he who turned me on to this idea.

Database Designs Require Encapsulation

Probably the most sapient portion of the observation had little to do with stored procedures, specifically.  Stored procedures are merely a tool we use in recognizing that databases need encapsulation of their designs just the same way that any other object would.

Depending on the features available in your chosen database platform, you could conceivably create encapsulation other ways.  For instance, some platforms allow updateable views.  So long as you could keep satisfying the contract specified by an earlier set of exposed tables, you can change the design of your database behind that wall of encapsulation.

As Scott Bain says, encapsulation is the first principle of design.  It should come as no surprise that this is true for database designs.

The Version Manager

One of the more recent phases in my journey to a class of databases was understanding that the best way to codify a database's design was as the series of revisions that could get you there rather than in terms of specific design elements.  It's harder to find fault in this way of thinking because I still use and recommend it today but it was incomplete; sufficient to enable some level of sustainable database development but not true test-driven database development.

Let me refresh your memory on the building technique to which I am referring.  You store an ordered list of scripts that are used to build a database.  You build some infrastructure that ensures only the right scripts are executed against any given database and in the right order.  So, if you have a version 2 database instance and you want to upgrade it to version 5, the infrastructure will execute the steps to upgrade to version 3, then 4, then 5.

Digging the nugget of truth out of this way of doing things is weird because the thing is its own nugget of truth and, at the same time, is not sufficient to support a modern test-driven environment.  The reason for this weirdness is that version-based builds are the best way I know to do things but they aren't enough.  More was required.

Anyway, the grain of truth in this way of thinking was the recognition that the actual revisions applied to real production databases should govern how we organize database build scripts.  That is, while I had not yet discovered the true class of database concept - at least not as I understand it today - I had discovered one of the principles that drives test-driven database development: the historical reality of important database instances must be respected and always trumps our wishes, hopes, and ideals.

Knowledge, Behavior, and Information

I mention this in my book but I thought I might elaborate.  I think a useful way to conceptually divide the parts of a database is into three groups of design elements: information, knowledge, and behavior.

The Three Concepts

Information and knowledge are often confused in everyday language.  So, first, I'll disambiguate those two words.  Not all data are information or knowledge and rarely are the two interchangeable.

Information is a special subclass of data.  I'm sure someone who is an expert in communication theory or some other kind of academic would be glad to correct me and I won't fight them on what the technical definition of the word is.  I'm only interested in what the useful definition for my own purposes is, not the officially right definition.  For the purposes of this blog entry, and of everything I say and write, it is the part of a signal that the recipient did not know in advance.  Simply put, information is data which informs its recipient.

If information could be thought of as facts in transit, then knowledge would be facts at rest.  In essence, knowledge is potential information but it is also a potential driver for action.  That is, the two uses of any given object's knowledge are to inform other objects, thereby adding to their knowledge, and to inform decisions, thereby improving the value of an action taken.

That latter purpose is the perfect opening to briefly introduce the third player in the database design world: behavior.  If knowledge and information are facts at rest and in motion, behavior can be thought of as how something responds to knowledge or information.  For instance, you drive on the correct side of the road because you know you will slam into something if you do.  Likewise, you yank your hand away from a too-recently poured cup of coffee have become informed that the cup is too hot to touch without damaging tissue.

Information as Pertains to Databases

In the database world, information is the set of signals sent or received by a database.  A query and its parameters, the invocation of a stored procedure, the results set, and any errors that occurred are all examples of information as a database sees it.

In essence, information is the "surface" of a database's design.  It is impossible for external parties to access the value of a database except by sending and receiving signals.

Moreover, it is the means by which value is conveyed between a database and its clients.  It is pointless to update a database with information it already knows.  It is useless to query a database for what you already know.  Value is created by such actions that results in one of the other entities "learning" something.

Knowledge as Pertains to Databases

Knowledge is the reason why databases cannot be maintained using the simple "blast and rebuild" upgrade path we apply to most software deployment problems.  All the facts stored in a database are knowledge; not all the data, because you can introduce noise into a database's design, but all the facts.

Knowledge is the purpose of a database.  Most software products and components exist to convey facts between parties or to process data and discover new facts.  Some software exists to entertain.  Databases exist to preserve knowledge.  Each production database is a modern day Library of Alexandria, complete with the ability for some asshole to burn it down and, in so doing, to cause irreversible damage.

We have known this for a long time - as close to "forever" as matters.  Databases have always been designed around the knowledge they capture and preserve.  Those design decisions stand as a reflection of our implicit understanding that databases aren't merely data bases, but knowledge bases.

Behavior as Pertains to Databases

So what is the role of behavior in database design?  It's another one of those things that can be put simply or drawn crisply, but can take a lot of work to implement correctly.  The role of behavior in a database is to mediate between knowledge and information.

All the information that a database receives needs to be translated into knowledge and stored for safekeeping.  Why?  So that, later, that knowledge can be translated back into information to help other actors make decisions or discoveries.

You can codify the behavior of a database in many different ways.  At the time of this entry, the most common way is to couple the behavior offered by a database directly to the kinds of knowledge that database can store.  This is accomplished by creating, publicly exposing, and coupling clients to table structures and relationships.

The Relationship Between the Three

I find it useful to divide database design into three parts.  The information layer of design is where the interactions between databases and other objects are defined.  The knowledge layer of design is where the facts you want to store in a database are housed.  The behavior layer is where one codifies the manner in which facts are absorbed or emitted.

behavior translates between knowledge and information

I'll post more on each of the specific layers of design with some implementation recommendations later.

The Database Installer

Another fallacious idea I, and many others, had was to treat database instances like programs that need to be installed.  Again, there are many things wrong with this line of reasoning, but something positive came from it.

You know me... "Mr. Positive."

This particular step in my journey to a class of databases bore what was, at least for me, a pretty subtle value.  Part of the subtlety came from the fact that the installer paradigm looks like it works for longer than a lot of its predecessors, which tended to break down extremely early.  Part of it was my own stubbornness - I was spending so much energy arguing the small improvement that I couldn't see the bigger improvements waiting just around the corner.

I'm pretty sure that's irony: that this way of thinking was so successful kept me from seeing other, more successful, ways of understanding a problem.  Wait.  Maybe that's not irony.  Maybe that's the human condition.  ...or maybe those things do not really oppose one another.

Anyway, grain of truth in this way of imagining database build technologies is that it recognizes the importance of discrete, tracked, testable deltas in design and highly controlled, repeatable ways of introducing those changes.  That ends up being a pretty fundamental concept.  It serves as the basis for building a testable class of databases, enabling test-driven database development, and ultimately unlocking database agility.

So there you have it.  Another step in the journey.  Another failure to hit the mark.  Another lesson that built to what we know today.

The Database Design Applicator

Another step in my journey to a class of databases was believing that a design tool could properly maintain all my database instances for me.  Again, I want to try and dig the nugget of truth out of this belief rather than beat it to death for what is wrong about it.

I think that, in this case, the nugget of truth is actually pretty self-evident.  The idea is to have a document that specifies the current design of a database and a tool that can update any database to have that design.  If you scrape away the part of that sentence that is obviously magical thinking, you are left with this:

"The idea is to have a document that specifies the current design of a database."

The part that is not quite as readily-apparent is that the motivation runs just a little bit deeper than that.  The real drive behind a tool that can update any database to a well-understood definition of the most recent design is that you have a document that codifies the current design and a tool that enforces the current design.

That's a noble goal, actually and an attainable one at that.  Think about it...

Disregarding all the ancient ideas, don't we still have a way of doing that when defining a "regular" object model?  We just do it a different way.  Instead of technical specifications and UML diagrams, we have unit tests.  Instead of a tool that generates code from high-level specifications, we have test runners.

Given a class of databases, using unit tests as the primary specification of design is a discipline that transfers straight over to database development; practically unmodified.

So that's the positive takeaway from the era of magical design-application tool mysticism: that we do in fact need a way to specify and enforce the current design of a class of databases and that, like Java, .Net, or C++ classes, that should be automated unit tests specifying how a database behaves and enforcing that specification on a regular basis.

Wrangling Wild Databases

I've received some interesting questions lately.  One question at my most recent talk for the DAMA group in Portland rang very familiar.  It got me thinking.

There are a lot of database instances out there that were developed before my book was published.  Even if everyone adopted the techniques therein, that would still leave trillions of rows of data in databases that were not in accordance with the book.  Among other things, these databases are generally created with insufficient automated test coverage so, in the spirit of Working Effectively with Legacy Code, I call such databases "legacy databases."

I cover legacy databases a little bit in the book but there is a limited number of scenarios.  This blog seems like a natural place to start addressing other issues.

The Scenario

Imagine the following:

You work for a company that sells an enterprise product; we'll call that product "Calm Cheddar."  As with most enterprise software products, Calm Cheddar has a database back end.  Calm Cheddar has been successful in several applicable markets and has been sold to numerous customers over the course of several years.  In that same time, it has grown as a product.  Along with the overall design of Calm Cheddar, the design of its database has grown and morphed over time.

manual deployment and broad customer base leads to
deviation in database builds

The final result is that there is an array of customers with varying versions of the software and each with a different path to their current deployment.  The saving grace is that Calm Cheddar's customers tend to upgrade to the latest version.  They may or may not upgrade often, but then never upgrade to something that is already outdated.

Now let's say you want to start emerging a class of databases in this scenario.  What I've shown in the past and what I teach in the first two thirds of my book do not cover scenarios like this.  This specific scenario is not covered anywhere and I imagine that there are several people in a similar situation.

The Ideal and the Real

The ideal class of databases is expressed as a linear sequence of versions created using a linear series of revisions.  The class of databases is simple, knows how to perform any reasonable upgrade, and is very robust.

Circumstances, however, are rarely ideal.

just because you want it, doesn't make it so

There are at least two lessons from Test-Driven Database Development: Unlocking Agility that apply to this scenario.  First: drive variation out of your database build process as much as you possibly can.  Second: above all else, make your database build process reflect the actual transitions that have really been applied to real databases.

These two forces might appear to be contradictory but, in fact, they align perfectly.  You should drive all variation out of your database development process, yes, but that does not mean you will succeed or will have started autonomating your database build mechanism at the very beginning of your very first database's life.  That you should drive all variation from a system does not mean that none will ever be there.

The Reckoning

The key to reconciling these two forces - the impulse to minimize variation in database build paths and the need to recognize the true database upgrade steps that have actually occurred - with the reality that there is a large, diverse population of databases within a given class is in understanding that you must merge these disparate paths together.  What was once a vast array of trickling creeks should, over time, be coalesced into a single coursing river of features.

there can be only one

Two techniques must be applied to resolve any differences between the various deployment paths.  One is taming a legacy database.  The other is remediation deviations.

Without getting into the details, the former amounts to creating a new class of databases that has conditional build logic in its very first version to address the potential of a database that has been built prior to the class's creation.  The latter consists of documenting variations in a database's historical construction patterns, then using transition tests to drive conditional logic that reconciles the various "flavors" of a database design.

Sometimes it will be a great deal of time.  Imagine that eighty percent of your database instances are almost exactly alike, fifteen percent fall into a few other distinct categories, and the remainder are "lone wolves" with highly deviant paths.

In such a case, you might want to phase your database wrangling activities, starting with the large body of highly similar databases first, moving on to the smaller groups second, and start picking off the lone wolf types on an "as-needed" basis.

This technique works great in a relatively controlled environment where, among other things, all of the databases are roughly the same version.  They don't need to all have exactly the same design to start but it does make things a lot easier if they have approximately the same design.

In the Calm Chowder scenario, however, we don't have the luxury.  Remember: we have variation in both the version and the manner of construction.

The Conditioner

A friend of mind, Seth McCarthy, has also come up with an interesting twist on this way of doing things; one that, I think, addresses the extra kind of variation.

He suggested adding a proxy over the simple, linear-step-oriented database builder at the heart of a class of databases.  This proxy's job is twofold.  First and foremost, it detects the version of a database's design then conditions it to look as though the infrastructure for a class of databases has been used to build it.  Typically, that means creating and populating some kind of version registry table.

Naturally, if acting on an empty database, the proxy does nothing but delegate to the core database builder.

rectify...

Also, the conditioner code is in a position to perform conditional transformations before delegating to the core class of databases.  It can even inject custom transitions in between steps if necessary.

The conditioner represents almost the exact opposite of the linear sequence of database upgrade steps I ordinarily recommend but special circumstances demand special responses.  Ordinarily, one would be adding a new version to a class of databases on a very regular basis and one would have to manage an ever-growing number of possible historical versions as a starting point.  There should be very little variation between one instance and another of any given version.  In that case, having special transformations that go from each version to each other version adds complexity and work.

In this scenario, however, the conditioner proxy exists in the exact opposite context.  There's only one target version with which the proxy is principally concerned.  There are many source versions and there is the potential for some amount of variation between two instances of any given historical version.  So having a special place to handle one or more of the special cases makes perfect sense.

So There You Have It

At a very high level, this should serve as a strategy one could apply to the problem of handling a large, diverse population of databases in various historical states and previously managed with an at least somewhat unreliable process such has being built by hand.

The strategy can be stated simply, though it is not always easy to do.  Capture the historical versions of your class of databases.  Codify those versions, either as conditional logic in the initial version of a simple linear sequence of modification scripts or as a proxy to a similarly sustainable database class format.  Drive each conditional behavior from transition tests that model the variant starting points.  Force all of the variation out of existence in a controlled way.  If you have too much variation to handle all at once, ingest smaller segments of the source database space into your class of databases.

Why the Linear Chain of Deltas Is the Best Way to Define a Database

There are several ways to organize the revisions in a class of databases.  In fact, there are at least two distinct ways that one's strategy can vary: how one organizes the outcomes of applying a class of databases and how one organizes the implementation of a class of databases.  It is my stance that, in almost every case, the most effective solution is to have a class of databases codify a series of released versions and to use a sequence of delta scripts to get there.

What

The first order of business is to decide what a class of databases describes.  You could have it describe the components of a database.  You could have it describe the current state of a database only.  You could have it describe each released version of a database plus the one you are working on now.

As stated above, I think the latter is best.  Rather than disprove its competitors and every other possible competitor, I will demonstrate its superiority.

There is at least one database of consequence for most release products.  Usually, that is a production database acting as the source of record for one or more software applications.  Usually, if you were to take a time-lapsed video of that database's design diagram over, say, a decade condensed down to a few minutes, the image in the video would remain almost completely static for many seconds, then it change almost instantaneously to a new design before going back to being stable again.  This process would probably repeat for the course of the video.

The real database - the most important one in your product, organization, or design - is almost always expressed as a series of discrete versions.  Its content grows gradually over time but it's design alternates between long periods of stasis and short periods of violent change.

Databases Naturally Transition from Version to Version

These long, stable versions of your production database are the natural targets around which to organize your class of databases.  Why build anything that won't produce one of those versions?  Of course, the version you are working on now is a bit of a moving target but, when it gets released, it ceases to be the version you are working on now and becomes another in the series of versions in a deployed database.

How

So I've shown that the allowable targets of a class of databases should be past and future versions of a released database only because the reality of the most important databases is that they transition from one version to another.  Once you are doing that there are a few options for how you do it.

Define Every Possible Transition

One option is to manage every possible transition in design.  That is, if you've developed three versions of your database design already and you want to add a fourth, you would produce and test a script for how to get from each of those previous versions to the latest.

Aside from the fact that this strategy cannot be depicted without violating the cardinal rule of diagramming design (no crossing lines), this is a lot of work.  You've got to test four paths for your fourth version, five paths for your fifth version, six paths for your sixth... you get the idea.

There's another option: only define the transition from the most recently-released version of your database to the version you intend to release next.  That way, when you want to add a ninety-fifth version, you only need to add to your class and test one set of transition scripts, not ninety-five.

Only Define the Next Transition

How do you get from an older version to the latest?  By applying each of the intervening transitions in the correct order.  Of course, to ensure this is done consistently, you have to build a little bit of infrastructure but that infrastructure is a one time investment and costs almost nothing to write in the first place.  It usually pays for itself in months, not years.

On top of all that, I could make the same argument for how a class of database grows as I did for what its outputs can be.

The most important, valuable, and rigid database in your life probably transitions from to design to design by a sequence of transformations applied to it.  The linear nature of its development is nothing more than a reflection of the linear nature of time (as we are able to interact with it).  You made a series of changes in a particular order so there is a series of changes to applied in a particular order.

Do It

Unless you are in the two percent of people for whom this way of doing things actually doesn't make sense, you should start managing your database designs this way: a linear chain of deltas allowing you to move between discrete versions of design and managed by a lightweight infrastructure.

I'll Be Speaking At DAMA Iowa In May, 2014

This is just a quick note that I will be speaking at the Iowa chapter of DAMA in May of 2014.  I'll announce the official date and place when they have been set.

Databases as Monuments

In an earlier post, I mentioned wanting to transfer the focus of database developers from individual databases over to classes of databases.  Focusing on the care and feeding of individual database instances was the starting point for my journey to a class of databases.

I want to drill in to each one of the steps I took in the aforementioned post, including the place where I started.  However, rather than talk about what is wrong with each concept, I'm going to focus on what was right about it.  This is as much to challenge myself as it is to explore the problem in an interesting way.

Thousands of Years Ago, Before the Earth Had Cooled

you change your course for a
monument, not the other way around

Once, long ago, we thought of databases primarily in terms of the ultimately deployed instance.  This corresponds with the Lean concept of a monument - a piece of equipment that is so much a fixture that it controls your process rather than conforming to it.

Often, individuals were given caretaker positions and these caretakers quickly became responsible for the maintenance of both health and design.  The caretakers became gatekeepers in very short order.

Even early on, before agility was formalized, patterns-oriented development was codified, or test-driven development was understood, eye witnesses have told me this was a major impediment to the flow of new features in many software systems.

This is no surprise.  Monuments almost always impede flow.

When I was starting out as a developer, this was still a problem.  When I started capturing knowledge about test-driven database development, nearly a decade ago, this was still a problem.  I observe organizations today where this is still a problem.

The real question should never have been "How do we remove the impediment here?"  It should have been "Why does this problem exist everywhere I look?"

Lifeblood

The answer, as with many of the old ways that we so readily scorn today, is that there was something right about the gate-keeping behavior.  Not that it was completely right - I'm not saying that.  There was, however, one thing right about it: the motivation.

Databases are unlike most other kinds of software.  At least that is true in the case of production databases acting as the source of record for business data, which is what most of us think of when we hear the word "database."

Most kinds of software we write have two properties that are interesting in this post.  First, the designs are very complex.  Despite all our efforts to "keep it simple," we are addressing complex problems that demand sophisticated solutions.  Second, they contain little or no data other than their design.  In the old days there was some weird stuff about storing certain data as resources within binaries but that is mostly gone now.

By contrast, databases tend to have small, simple designs housing large, complex bodies of content.  That content is usually extremely important; like, "lose your job if it goes away" important.  As software becomes a more influential part of business, the contents of databases become more vital.

Cover Your Heart!  Cover Your Heart!

When something is vital, you protect it.  Sometimes, you protect it in a way that is otherwise detrimental to your health.

Imagine you are trapped under water but just barely.  Your wrist is caught in something and it is forcing you to face downward.  If you could just turn over, you can breath but you have to break your arm or dislocate your shoulder to do it.

You might think you're too weak to do anything about it - that you would drown.  That's certainly what we are all taught would happen.  Don't kid yourself: You're an animal.  Somewhere in your brain is a thing that wants to survive even if it hurts.  That thing doesn't care that you heard some words in school telling you that humans are different, special, and weak.

At some point, you'd start thrashing about until something gave and you'd stand a very good chance of flipping over and sucking in some air as a result.  It would hurt.  It would cost you something - maybe even a hand or a thumb - but you would live.

That is how I think of monuments, now.  Databases as monuments create gate keeping caretakers.  Those people are not malicious in their conservation of power.  They are not incompetents maintaining job security.

They are an organization's "lizard brain" (in a good way): the part of an organization that forces it to do anything - bend over backwards or even sacrifice a body part - in order to avoid the devastating effects of losing critical data.

Keeping the Faith

When I started getting good at "regular" TDD, I went through a phase where I believed that traditional testers with no programming skills were no longer needed.  While we may not need as many of them now as we did in the '80s (I hear), I freely admit that I was wrong in believing we didn't need them at all.

Testers were once essential to the software development process, providing vital feedback that ensured a product was at least worth showing a customer.  While we can get a lot of that feedback from automation, now, it is still difficult for a developer to have the same perspective as a real tester.

It's a skill.  As one of my coworkers at the time of this post would say, you have to be able to imagine what an end user might do.  So traditional testers have found a new place in the modern world.  One in which they are extremely effective; maybe even more so than before people did a lot of automation, because now they aren't stuck doing as many menial tasks.

Similarly, as we work to build a new practice of test-driven development in the database world, we should remember the vital function that the traditional DBA has filled.  That role and the perspective that comes with it should not be lost.

Part of learning how to create and test-drive classes of databases will be providing views into those structures that allow traditional DBAs to contribute, not to satisfy some political requirement, but to harvest the valuable knowledge and experience in their heads.

I'll Be Speaking At DAMA SoCal in February

This is just a quick announcement that I will be giving the same talk I mentioned earlier again at DAMA SoCal on February 24th, 2014.

Click here for more details.

My Journey to a Class of Databases

If you know anything about me or you read my article on InformIT, Ten Tips for Constructing an Agile Database Development Environment that Works, you know that I think the foundation of database TDD lies in creating a class of database.  That means transferring focus from the design of individual databases onto something that makes databases.  You want the ability to create or upgrade as many instances as you like and know that they all have exactly the same design.

Exactly the same design.

There are many possible ways to implement a concept like that, though.  When I began developing these concepts nearly a decade ago, before I even knew that I wanted a class of databases, I was just focused on controlling database creation.  I experimented with a lot of different kinds of infrastructure and a bunch of different patterns of database growth.

Since I started being a professional software developer, I've tried a bunch of different ways to control how databases grow and transform.  I think a lot of people have had similar thoughts and various times in their lives.  These are the ways and times I thought about these problems.

The Fool's Errand (pre-2005)

The most naive solution is the idea that you can specify what you want the design to be right now and have some tool that will update an existing database to have the new design.  Sometimes, it is a tool that compares two databases.  Sometimes, it is a diagramming tool that will inspect a database and figure out how to make it comply with a drawing.

magic will transmit design changes!

The problem is that this doesn't work.  It doesn't work for the same reason that you can't unscramble an egg.  There is no way for a software system to look at the current design, look at a new design, and figure out how to get from point A to point B.

At least, it's not possible to do that every time and with current technology.  Maybe, one day when we have computer systems that can infer intent, it will be possible.  Right now, however, that's too complex a task for a computer.

The Installer Fallacy (2005-2006)

Another way of thinking about the problem is the way we imagine installers.  Databases have components.  Components have dependencies.  You ask the installer to make sure the features you want are there and it ensures the dependencies are satisfied.

The problem is that there is always a meltdown.  In this case, I'm using that term a little less figuratively than usual.  In a healthy database design, things are changing.  Tables are splitting and recombining into newer, better shapes all the time.

The features all melt together more quickly than you could imagine.  Pretty soon, it's difficult to tell why you are creating separate features and components at all.  Eventually, all the components blend together and you wonder why you ever divided components in the first place.

Rise of the Versions (2006-2008)

After about my third database "feature" that depended on exactly one feature, which in turn depended on only one feature, I started to get the message.  I realized that the forces in the database world are telling us to organize around time, rather than around features.

It turns out that there is usually at least one database instance that as an extremely linear path of transformation, and it happens to be the absolute most important kind of database there is: a source of record database in production.

Production databases tend to metamorphose over time in a series of discrete transitions from one design to another.  At the same time, production databases are the most indispensable and long-lived databases of all.

Everything else (e.g.: test databases or development databases) tends to have a little more flexibility.  At the very least, nothing else has less flexibility.  So why shouldn't the most important and least flexible kind of database define how all databases of a particular kind are built.

Have a Little Class (2008-present)

When I started formulating these thoughts into something that I could start evangelizing, I realized there was more to this than just regulating the flow of design changes from a development environment out into a production environment.  That's an important feature but it's just an implementation detail of a much more critical shift in mindset.

a path of confidence

What really matters is having uniformity of design between all the different database instances filling the same role.  If you have that, tests executed against one instance allow you to make predictions about how another instance will behave.

That mechanism - that way of thinking - serves as a critical underpinning for test-driven development in the database world.

A Class of Databases for Non-Middle-Tier-Developers?

As I previously mentioned, you can use the sample code for my book as a starting point to establish a class of databases and you can also inexpensively write one for yourself.

However, if you don't want to do any non-SQL coding and you don't want to use the infrastructure I built and made freely available, it might get a little stickier.  At least it would for me because the test frameworks and expressiveness available for SQL-only solutions is kind of weak.

One option is to write something yourself.  If you do that, I'd love to hear about it and I bet a lot of other people would too.

Another option is to wait for the release of DataClass 3.  Maybe you could even participate in its design to make sure it meets your needs.  Right now, it is being made by a software developer for software developers.

If you need something now, though, you'll need to invest in developing something yourself or try downloading and compiling the sample code to see if it will work for you out of the box, which it easily could do.

Establishing a Class of Databases

A lot of people ask me about the infrastructure required to establish a class of databases.  In my opinion, that's not a lot of the work.  It takes a day or so here or there to write just what you need, just when you need it.

Nevertheless, I would like to remind everyone that the sample code for my book also contains sample infrastructure.  Feel free to take that as a starting point and go forward.  If you're not a .NET person, it could pretty easily be translated into Java.

Test-Driven Database Development Talk at the Portland Chapter of DAMA

Recently, I gave a talk on the subject of test-driven database development at the Portland chapter of the organization known as DAMA (DAta MAnagement).

I was surprised by a couple things during this talk.  For one thing, the group was a lot more engaged than I've ever seen at this kind of talk.  For another, I expected the audience to resist the ideas a lot more than they did.  Finally, I expected to encounter new classes of objections that I had not previously addressed but none were raised.

A Great Group

It's not often that I get to (read: "choose to") say positive things about an entire group of people but this is one of those times.  The group was attentive.  They were very focused on what I was saying.  That's not a back door complement for myself either.  I've delivered virtually the same talk to other crowds and seen a lot more droopy eyes.  This is just a voracious group that wants to hear someone's ideas.

That level of attention to detail, naturally, gave rise to an unprecedented level of engagement and involvement.  The questions were piercing and pointed.  They demanded good answers.  They gave many opportunities to elaborate.  Yet, there wasn't "that guy."

Everybody who's been involved in public speaking knows that guy.  There's almost always on in every crowd.  The person who objects for the wrong reasons.  Sometimes it's because they like the sound of their own voice.  Sometimes, because they want others to hear how smart they think they are.  On occasion, they object for objection's sake, subscribing to the same basic motives as a vandal.

This group, at least when I talked to them, didn't have that guy and it made the speaking process so much more enjoyable.  Not only were there a lot of good questions, but there were only good questions.  Not only was it a lot of engagement, but it was all thoughtful engagement.

Little Resistance

I was a little nervous because I have traditionally engaged programmers who are affected by database development processes and this was an entirely different audience: mostly people who's primary skill was database development, administration, or support.  In many cases, especially in large organizations with so-called "centers of excellence" (read: "centers of delay"), application developers and data managers are at odds with one another.  At least, the application developers with whom I converse present things that way.

This talk served as evidence that things are that simple.  The group was very open to new ideas and to enhanced collaboration.  In fact, some of their questions seemed to imply that they were concerned with how to get application developers to adopt these concepts.

I now suspect that, in many organizations, data manager types and application developer types each see the other group as stubborn and distant almost entirely on account of the inefficiency introduced by the management structure in which they work.  That is, the people are reasonable and are willing to work together but their organizations impede collaboration by concentrating into skills-based groups rather than product-based groups.  This invalidates some of what I wrote in a previous entry entitled "Breaking Down Barriers to Agile Database Development" but maybe not all of it.

The resistance I did meet was mostly good, coming in the form of tough questions posed by people with real obstacles to overcome.

Bases Covered

Apparently, I'm getting good at this or something.  There wasn't a single question asked that couldn't be answered either by something from my book or by basic TDD/Agile wisdom.  This was surprising.  I really thought that, given the new type of audience, I would be facing some new tough questions.

In reality, something like a third of the questions were directly addressed by the content I brought with me.  That was really gratifying because a lot of the time I got to say "That's covered in a future slide...<click>...this one!"

Another third was covered by chapters in Test-Driven Database Development: Unlocking Agility.  There's not a lot you can do about that.  You're allotted a certain amount of time.  If you have more material than that block will support, you either cut scope or depth.  This talk was already a surface-view of the problem and solution, so cutting depth wasn't an option.  Besides, there's nothing wrong with having concepts that entice people to buy my book, right?

The last third was just basic agility or TDD questions.  I could have covered more agility and TDD concepts at the beginning of the talk but that would have forced me to cut even more scope at the end.

Regardless, the questions that were asked were easily addressed.  That's not to say I convinced everyone of everything - especially people who were at the beginning of their journey to agility.  However, the fact that there was an answer to every question made me happy.

Conclusion

It was an enjoyable and rewarding talk.

If you've got a talk that you thin might apply to this group, I highly recommend them.  I know I'm not the only one, either, because their speaking calendar is filled something like a half year in advance.

On my way to the @agilepdx meeting tonight

The weather is a little dicey but we have enough of a head start to make it on time.

I will post a link to the slides and sample code tonight after the talk.

Also: While today's talk is not a sales pitch, I will have some books on hand should anyone want to buy one.

I Got on Some Top 100 Lists!

I just wanted to document that, at some point in my life, something I wrote broke #100,000 in paid books and was on the top 100 list in three paid categories.

At one point, this was one the above page

Test-Driven Database Development: Unlocking Agility Complete; on to DataClass 3

After a year of work on my book, I'm ready for a break.

Just kidding.  Now that the book is done, it's time to write DataClass 3.  How will it be different from DataClass or DataConstructor?  A lot of ways, actually.  I'm not going to plan everything out right here and now.  That wouldn't be very in character, would it?  However, I do have some high-level goals I want to meet.  Here they are in no particular order.

• Add a transparent data model
• Expose parsing capabilities for automation
• Restore a feature from DataConstructor: support for other kinds of data (XML, objects, etc.)
• Extension point for client platforms
• Extension point for database platforms
• First class support for the obvious structures in a database (tables, views, stored procedures, etc.)
• Inferred design

There are a bunch of other "nice to have" features I intend to implement later but for the initial rewrite, I'll stick with that scope.  Sadly, yes, you read correctly.  Once every two years, I run a little experiment to verify that people who think mocking is bad are crazy.  DataClass 2 was that experiment.  It's easier to rewrite it the right way with these new goals than to try and change directions with that code base.

There are other things I'm changing, too.  Most notably, I'm changing the profit model for DataClass.  The compiler and API will be freeware.  Anyone can download it and use it for whatever purpose they may have.  Then I'm going to write extensions for Visual Studio, Eclipse, and maybe IntelliJ IDEA.

I'll delve more into each topic in a separate entry later as I have time.