Tuesday, December 10, 2013

Databases as Monuments

In an earlier post, I mentioned wanting to transfer the focus of database developers from individual databases over to classes of databases.  Focusing on the care and feeding of individual database instances was the starting point for my journey to a class of databases.

I want to drill in to each one of the steps I took in the aforementioned post, including the place where I started.  However, rather than talk about what is wrong with each concept, I'm going to focus on what was right about it.  This is as much to challenge myself as it is to explore the problem in an interesting way.

Thousands of Years Ago, Before the Earth Had Cooled

you change your course for a
monument, not the other way around
Once, long ago, we thought of databases primarily in terms of the ultimately deployed instance.  This corresponds with the Lean concept of a monument - a piece of equipment that is so much a fixture that it controls your process rather than conforming to it.

Often, individuals were given caretaker positions and these caretakers quickly became responsible for the maintenance of both health and design.  The caretakers became gatekeepers in very short order.

Even early on, before agility was formalized, patterns-oriented development was codified, or test-driven development was understood, eye witnesses have told me this was a major impediment to the flow of new features in many software systems.

This is no surprise.  Monuments almost always impede flow.

When I was starting out as a developer, this was still a problem.  When I started capturing knowledge about test-driven database development, nearly a decade ago, this was still a problem.  I observe organizations today where this is still a problem.

The real question should never have been "How do we remove the impediment here?"  It should have been "Why does this problem exist everywhere I look?"

Lifeblood

The answer, as with many of the old ways that we so readily scorn today, is that there was something right about the gate-keeping behavior.  Not that it was completely right - I'm not saying that.  There was, however, one thing right about it: the motivation.

Databases are unlike most other kinds of software.  At least that is true in the case of production databases acting as the source of record for business data, which is what most of us think of when we hear the word "database."

Most kinds of software we write have two properties that are interesting in this post.  First, the designs are very complex.  Despite all our efforts to "keep it simple," we are addressing complex problems that demand sophisticated solutions.  Second, they contain little or no data other than their design.  In the old days there was some weird stuff about storing certain data as resources within binaries but that is mostly gone now.

By contrast, databases tend to have small, simple designs housing large, complex bodies of content.  That content is usually extremely important; like, "lose your job if it goes away" important.  As software becomes a more influential part of business, the contents of databases become more vital.

Cover Your Heart!  Cover Your Heart!

When something is vital, you protect it.  Sometimes, you protect it in a way that is otherwise detrimental to your health.

Imagine you are trapped under water but just barely.  Your wrist is caught in something and it is forcing you to face downward.  If you could just turn over, you can breath but you have to break your arm or dislocate your shoulder to do it.

You might think you're too weak to do anything about it - that you would drown.  That's certainly what we are all taught would happen.  Don't kid yourself: You're an animal.  Somewhere in your brain is a thing that wants to survive even if it hurts.  That thing doesn't care that you heard some words in school telling you that humans are different, special, and weak.

At some point, you'd start thrashing about until something gave and you'd stand a very good chance of flipping over and sucking in some air as a result.  It would hurt.  It would cost you something - maybe even a hand or a thumb - but you would live.

That is how I think of monuments, now.  Databases as monuments create gate keeping caretakers.  Those people are not malicious in their conservation of power.  They are not incompetents maintaining job security.

They are an organization's "lizard brain" (in a good way): the part of an organization that forces it to do anything - bend over backwards or even sacrifice a body part - in order to avoid the devastating effects of losing critical data.

Keeping the Faith

When I started getting good at "regular" TDD, I went through a phase where I believed that traditional testers with no programming skills were no longer needed.  While we may not need as many of them now as we did in the '80s (I hear), I freely admit that I was wrong in believing we didn't need them at all.

Testers were once essential to the software development process, providing vital feedback that ensured a product was at least worth showing a customer.  While we can get a lot of that feedback from automation, now, it is still difficult for a developer to have the same perspective as a real tester.

It's a skill.  As one of my coworkers at the time of this post would say, you have to be able to imagine what an end user might do.  So traditional testers have found a new place in the modern world.  One in which they are extremely effective; maybe even more so than before people did a lot of automation, because now they aren't stuck doing as many menial tasks.

Similarly, as we work to build a new practice of test-driven development in the database world, we should remember the vital function that the traditional DBA has filled.  That role and the perspective that comes with it should not be lost.

Part of learning how to create and test-drive classes of databases will be providing views into those structures that allow traditional DBAs to contribute, not to satisfy some political requirement, but to harvest the valuable knowledge and experience in their heads.