Why Revision Control is Not Enough

Several people have suggested that revision control might be an alternative to creating a class of databases.  I really have no way to counter such an argument because I cannot fathom what they were thinking in the first place.

The two concepts target completely and utterly different things.  I guess that all I can really do is talk about how different they are and hope some people change their stance.

Revision control (e.g.: CVS, Subversion, or Perforce) allows you to track and manage the changes you are making to source code.  The point is to (a) provide a place of safekeeping for your documents of design and (b) to allow you to access previous versions of a design.  When you update the design of a class, you create a new version of it in source control and do not generally see older versions unless you go looking for them.

The point is to allow you to work on the most recent design in a particular branch while keeping a history of what was done and not betting on a single computer to keep your source code safe.  In short: Source control is about managing the relationship between developers and designs.

A class of databases has a totally different functions.  Like any other class, it's most interesting relationship is between it and its objects.  Yes, a database class must store the steps required to build older versions in it but for a totally different reason that why a version-control system stores older versions of source code.

Whereas a revision control system tracks versions for the benefit of the development process, a class of databases must track revisions for the benefit of the database-construction process.

Storing the incremental deltas required to get from each version to the next and forcing all databases in a class to follow the exact same path of growth enables test-driven, agile database development in two ways.  For one thing, it drastically simplifies the database deployment model; every single database of a given type grows exactly the same way.  For another, it allows you to properly test the transitions from one version to the next.

The versions of a source file are created because a developer has learned something, changed his mind, or otherwise revised design and they are created to ensure that developers can revisit previous designs.  The versions of a database class are created because a revision to design was actually released to production and they are created so that every database in a class is built in the exact same way as every other database in that class.

The Database Installer

Another fallacious idea I, and many others, had was to treat database instances like programs that need to be installed.  Again, there are many things wrong with this line of reasoning, but something positive came from it.

You know me... "Mr. Positive."

This particular step in my journey to a class of databases bore what was, at least for me, a pretty subtle value.  Part of the subtlety came from the fact that the installer paradigm looks like it works for longer than a lot of its predecessors, which tended to break down extremely early.  Part of it was my own stubbornness - I was spending so much energy arguing the small improvement that I couldn't see the bigger improvements waiting just around the corner.

I'm pretty sure that's irony: that this way of thinking was so successful kept me from seeing other, more successful, ways of understanding a problem.  Wait.  Maybe that's not irony.  Maybe that's the human condition.  ...or maybe those things do not really oppose one another.

Anyway, grain of truth in this way of imagining database build technologies is that it recognizes the importance of discrete, tracked, testable deltas in design and highly controlled, repeatable ways of introducing those changes.  That ends up being a pretty fundamental concept.  It serves as the basis for building a testable class of databases, enabling test-driven database development, and ultimately unlocking database agility.

So there you have it.  Another step in the journey.  Another failure to hit the mark.  Another lesson that built to what we know today.

Why the Linear Chain of Deltas Is the Best Way to Define a Database

There are several ways to organize the revisions in a class of databases.  In fact, there are at least two distinct ways that one's strategy can vary: how one organizes the outcomes of applying a class of databases and how one organizes the implementation of a class of databases.  It is my stance that, in almost every case, the most effective solution is to have a class of databases codify a series of released versions and to use a sequence of delta scripts to get there.

What

The first order of business is to decide what a class of databases describes.  You could have it describe the components of a database.  You could have it describe the current state of a database only.  You could have it describe each released version of a database plus the one you are working on now.

As stated above, I think the latter is best.  Rather than disprove its competitors and every other possible competitor, I will demonstrate its superiority.

There is at least one database of consequence for most release products.  Usually, that is a production database acting as the source of record for one or more software applications.  Usually, if you were to take a time-lapsed video of that database's design diagram over, say, a decade condensed down to a few minutes, the image in the video would remain almost completely static for many seconds, then it change almost instantaneously to a new design before going back to being stable again.  This process would probably repeat for the course of the video.

The real database - the most important one in your product, organization, or design - is almost always expressed as a series of discrete versions.  Its content grows gradually over time but it's design alternates between long periods of stasis and short periods of violent change.

Databases Naturally Transition from Version to Version

These long, stable versions of your production database are the natural targets around which to organize your class of databases.  Why build anything that won't produce one of those versions?  Of course, the version you are working on now is a bit of a moving target but, when it gets released, it ceases to be the version you are working on now and becomes another in the series of versions in a deployed database.

How

So I've shown that the allowable targets of a class of databases should be past and future versions of a released database only because the reality of the most important databases is that they transition from one version to another.  Once you are doing that there are a few options for how you do it.

Define Every Possible Transition

One option is to manage every possible transition in design.  That is, if you've developed three versions of your database design already and you want to add a fourth, you would produce and test a script for how to get from each of those previous versions to the latest.

Aside from the fact that this strategy cannot be depicted without violating the cardinal rule of diagramming design (no crossing lines), this is a lot of work.  You've got to test four paths for your fourth version, five paths for your fifth version, six paths for your sixth... you get the idea.

There's another option: only define the transition from the most recently-released version of your database to the version you intend to release next.  That way, when you want to add a ninety-fifth version, you only need to add to your class and test one set of transition scripts, not ninety-five.

Only Define the Next Transition

How do you get from an older version to the latest?  By applying each of the intervening transitions in the correct order.  Of course, to ensure this is done consistently, you have to build a little bit of infrastructure but that infrastructure is a one time investment and costs almost nothing to write in the first place.  It usually pays for itself in months, not years.

On top of all that, I could make the same argument for how a class of database grows as I did for what its outputs can be.

The most important, valuable, and rigid database in your life probably transitions from to design to design by a sequence of transformations applied to it.  The linear nature of its development is nothing more than a reflection of the linear nature of time (as we are able to interact with it).  You made a series of changes in a particular order so there is a series of changes to applied in a particular order.

Do It

Unless you are in the two percent of people for whom this way of doing things actually doesn't make sense, you should start managing your database designs this way: a linear chain of deltas allowing you to move between discrete versions of design and managed by a lightweight infrastructure.

My Journey to a Class of Databases

If you know anything about me or you read my article on InformIT, Ten Tips for Constructing an Agile Database Development Environment that Works, you know that I think the foundation of database TDD lies in creating a class of database.  That means transferring focus from the design of individual databases onto something that makes databases.  You want the ability to create or upgrade as many instances as you like and know that they all have exactly the same design.

Exactly the same design.

There are many possible ways to implement a concept like that, though.  When I began developing these concepts nearly a decade ago, before I even knew that I wanted a class of databases, I was just focused on controlling database creation.  I experimented with a lot of different kinds of infrastructure and a bunch of different patterns of database growth.

Since I started being a professional software developer, I've tried a bunch of different ways to control how databases grow and transform.  I think a lot of people have had similar thoughts and various times in their lives.  These are the ways and times I thought about these problems.

The Fool's Errand (pre-2005)

The most naive solution is the idea that you can specify what you want the design to be right now and have some tool that will update an existing database to have the new design.  Sometimes, it is a tool that compares two databases.  Sometimes, it is a diagramming tool that will inspect a database and figure out how to make it comply with a drawing.

magic will transmit design changes!

The problem is that this doesn't work.  It doesn't work for the same reason that you can't unscramble an egg.  There is no way for a software system to look at the current design, look at a new design, and figure out how to get from point A to point B.

At least, it's not possible to do that every time and with current technology.  Maybe, one day when we have computer systems that can infer intent, it will be possible.  Right now, however, that's too complex a task for a computer.

The Installer Fallacy (2005-2006)

Another way of thinking about the problem is the way we imagine installers.  Databases have components.  Components have dependencies.  You ask the installer to make sure the features you want are there and it ensures the dependencies are satisfied.

The problem is that there is always a meltdown.  In this case, I'm using that term a little less figuratively than usual.  In a healthy database design, things are changing.  Tables are splitting and recombining into newer, better shapes all the time.

The features all melt together more quickly than you could imagine.  Pretty soon, it's difficult to tell why you are creating separate features and components at all.  Eventually, all the components blend together and you wonder why you ever divided components in the first place.

Rise of the Versions (2006-2008)

After about my third database "feature" that depended on exactly one feature, which in turn depended on only one feature, I started to get the message.  I realized that the forces in the database world are telling us to organize around time, rather than around features.

It turns out that there is usually at least one database instance that as an extremely linear path of transformation, and it happens to be the absolute most important kind of database there is: a source of record database in production.

Production databases tend to metamorphose over time in a series of discrete transitions from one design to another.  At the same time, production databases are the most indispensable and long-lived databases of all.

Everything else (e.g.: test databases or development databases) tends to have a little more flexibility.  At the very least, nothing else has less flexibility.  So why shouldn't the most important and least flexible kind of database define how all databases of a particular kind are built.

Have a Little Class (2008-present)

When I started formulating these thoughts into something that I could start evangelizing, I realized there was more to this than just regulating the flow of design changes from a development environment out into a production environment.  That's an important feature but it's just an implementation detail of a much more critical shift in mindset.

a path of confidence

What really matters is having uniformity of design between all the different database instances filling the same role.  If you have that, tests executed against one instance allow you to make predictions about how another instance will behave.

That mechanism - that way of thinking - serves as a critical underpinning for test-driven development in the database world.

DataClass 3 - Transparent Data Model

One of my goals with DataClass 3 is to have a transparent data model that describes a class of data store without any coupling to a particular syntax.  This serves a couple purposes.

One reason is to better support automation.  With an API and a clear XML format, people can do all kinds of things.

Another reason is to create a natural and strong division between syntax and compiler.  If I do that, I can develop a few different languages.  That way, people who think curly braces are what slow them down can have a language that uses white space to define blocks and sane people can use curly braces or whatever.

Then there is the main force driving this design decision.  Having a clean, platform-independent way of expressing a database design allows me to write, say, a Java client that can explore a database class definition file and expose its structure to Eclipse.  I can do that about as easily as I could with a Visual Studio extensions.

There is a consequence to this design.  While the XML format is theoretically human-editable, it is not intended to be the primary means of expressing a data class.  I'm still committed to having a domain-specific language for that purpose.  So, whereas DataConstructor and the sample code for my book use XML as the primary format, I have no intention of supporting that in DataClass 3.

Test-Driven Database Development: Unlocking Agility Complete; on to DataClass 3

After a year of work on my book, I'm ready for a break.

Just kidding.  Now that the book is done, it's time to write DataClass 3.  How will it be different from DataClass or DataConstructor?  A lot of ways, actually.  I'm not going to plan everything out right here and now.  That wouldn't be very in character, would it?  However, I do have some high-level goals I want to meet.  Here they are in no particular order.

• Add a transparent data model
• Expose parsing capabilities for automation
• Restore a feature from DataConstructor: support for other kinds of data (XML, objects, etc.)
• Extension point for client platforms
• Extension point for database platforms
• First class support for the obvious structures in a database (tables, views, stored procedures, etc.)
• Inferred design

There are a bunch of other "nice to have" features I intend to implement later but for the initial rewrite, I'll stick with that scope.  Sadly, yes, you read correctly.  Once every two years, I run a little experiment to verify that people who think mocking is bad are crazy.  DataClass 2 was that experiment.  It's easier to rewrite it the right way with these new goals than to try and change directions with that code base.

There are other things I'm changing, too.  Most notably, I'm changing the profit model for DataClass.  The compiler and API will be freeware.  Anyone can download it and use it for whatever purpose they may have.  Then I'm going to write extensions for Visual Studio, Eclipse, and maybe IntelliJ IDEA.

I'll delve more into each topic in a separate entry later as I have time.

Agile Database Development Resources

I've started tracking the list of Agile Database Development Resources I know about on a dedicated page.

The two most recent additions will be my recently published article on InformIT - Ten Tips for Constructing an Agile Database Development Environment that Works - and my related upcoming book.

Breaking Down Barriers to Agile Database Development

Having a cross-functional team that works together to solve a problem is a basic property of an agile development environment.  It's impossible to list all the things that can impede the development of an agile database environment.  There are, however, some particular scenarios worthy of note.

One common case, especially in smaller organizations, is that the person who works on the database design is also a programmer.  In that case, fostering a collaborative environment is probably not too difficult.  Don’t let people catch you talking to yourself too often, though.  They might decide you have cracked under the pressure.

Another scenario that is ever-increasing in its frequency is when you have an agile development team with a “database guy” or a “database gal.”  In this situation it is important for programmers and the “database folk” to work together.  However, it’s still not that hard.

The toughest nut to crack is when you have an organizational boundary between a team of programmers and one or more database people, and those database people serve not only as database developers but also as “gatekeepers” to some production databases.  Oftentimes, the gatekeepers are vehemently against anything agile as well as anything automated touching their jealously guarded database structures.

I’m not exactly known as some great builder of bridges.  I’m actually better known for my “scorched earth” manner of dealing with people who I see as “in the way.”

However, there is one trick that has worked for me on the rare occasions when I was able to take a deep breath and count to a thousand before opening my mouth.  That trick is giving the gatekeeper control over the new processes I want implemented.  Instead of persuading them to let me have the most agile thing I can possibly get, software development teams in control of database design, I settle for the next best thing.  I try to persuade the "database guy" to take on the test-driven development and and object-oriented design activities I want implemented.

An example would be trying to get them involved in things like writing transition tests.

People tend to want to do the right thing and, when they are doing the wrong thing, it’s usually because they don’t understand the negative impact.  The other major reason is fear of no longer being needed.  It’s not so important who implements good practices as it is that they are implemented.

Giving control over certain aspects of an agile database development process to someone who is already in control of the database development process often neutralizes both the major impediments to adoption.  The gatekeeper will be able to make sure things are done his way and can be assured that he will remain relevant while slowly growing more accustomed to the modern ways of doing things.