Liberate Wall Street

Are you tired of the Occupy movement?  I am.  It looks to me like a bunch of dirty hippies whining about the fact that they bought houses they can't afford and created an economic meltdown.  How the hell is that Wall Street's fault?

Occupiers: You jackasses committed fraud and got away with it.  Or, at the very least, your mortgage brokers did.  Sure.  There was a little bit of economic fallout from your decisions and the Wall Street people definitely didn't do anything to arrest the downturn you created.  That hardly entitles you guys to bitch about them.  It's not their job to look out for you.

Here's what I think we should do.  We have a long history of liberating pansies who can't seem to take it upon themselves to fight back when occupied; most notably the French.  Why don't we get together as Americans and start liberating Occupied spaces?  We can start where it all began: Wall Street.

Liberate Wall Street!

The Gift that Keeps on Giving

Christmas is approaching as rapidly as ever.  In fact, it's approaching at exactly the same pace it always have and probably always will.  I don't know what everyone wants but I do know what most people desperately need.  They need the thing that helps them get through life, cold and twisted a thing as it is.

Unfortunately, if you're in the ninety-nine percent who don't seem to have this, nobody can give it to you.  That is, nobody can give it to you but you.

This year, give yourself the gift that keeps on giving: a fucking sense of humor.

The Reason for the Season

Hey everybody!  Thanksgiving is coming up and I'm very excited.  Of course, there's the feast but it's also a special day on which we all get together and celebrate the circumstances that afford us this free and comfortable lifestyle to which we have all grown so accustomed.

So this year, invite everyone you love to come over to your place.  Gather around the table.  Join hands.  Pray, if you're the religious sorts.  However you do it give thanks for the thing that allowed us to be us.  Was it Columbus?  No.  Was it the corn and beans thing?  I think not.

I wouldn't waste any precious thanks on those things.  Instead, let's remember how it really happened and let that remind us of who we really are.  We're the people who figured out how to make blankets into a weapon of mass destruction because winning a war with bullets, cannons, superior infrastructure, and trick beads wasn't going fast enough.

This Thanksgiving, instead of saying "Happy Thanksgiving," why not something that rings a little more true:

"They never knew what hit them."

"5 Whys" on the Economic Meltdown

The economy crashed.
Why? Because banks were profiteering off of loans the borrowers couldn't afford.
Why? Because people were borrowing more than they could afford.
Why? Because prices were sky-rocketing.
Why? Because government created artificially low interest rates.
Why? Because the economy crashed.

Occupy the Slippery Slope

"You know what?  Maybe it's not just the top 1% who's the problem.  That second percent is pretty rich and therefore responsible for my lot in life, too."  It's not unlikely that the #Occupy movement will take that turn and this is what it looks like...

1% becomes 2%.
2% becomes 4%.
4% becomes 10%.
10% becomes 25%.
25% becomes 49%.

Nearly half of all Americans would be screwed by them trying to "make things fair."

Then, if they get what they want (read their literature), we'll have full scale communism which screws almost everyone all the time.

So here is my modest call to action.  Go to the places where they are organizing and counter organize.  Help them see that they are their own problem and nobody else is responsible for the choices they make but them.  Be relentless with your reason and maybe we can avert the storm they are gathering.

I think it's time to invade China

Check this out:

http://blogs.hbr.org/cs/2011/08/apple_stores_in_china_the_one.html

I know how we could enforce trademarks over there.  We could threaten to default on our loans to them.  We could place an embargo on them.  We could start a good ol'fashion war with them.  It's about time a World War didn't start in Europe, anyway.

Term Coined

This summary is not available. Please click here to view the post.

My Work is Now Cut out for Me

Okay.  So I let myself get sidetracked on the development of my book as a result of having a lot to do at my current job.  That's the bad news.  The good news is that everything has been "percolating" during this time and I've come up with  a really good mental map of how I'm going to guide the reader through all the problems involved in agile database development.

Here it is...

Basically, the idea is as follows.

Goals:
We have the goal of increasing the speed with which value can be delivered by reducing the "bottleneck effect" that generally surrounds database changes.  However, we cannot do that at the expense of things we've already gotten good at; namely preserving the knowledge stored in a database and maintaining the availability of its service as experienced by a user via an intermediary application.

Obstacles:
There are risks posed to the things we are already good at by increasing the speed of change.  Changing structure, the way we do it now, implies risk to existing knowledge stored in a database.  It also introduces the chance that an application coupled to a database might not have had time to adapt to a change in structure by the time a change is rolled out.

There is also an obstacle to agile database development in that whatever infrastructure is required to mitigate the previously mentioned risks will probably not be available for legacy databases.

(From the Other Angle) Axioms:
Databases store knowledge - that is their primary role.  To acquire knowledge and make use of it, databases absorb and emit information.  Another thing to assume is that people will make mistakes.

Here's the big one: databases are objects.  They are objects that store knowledge on behalf of other entities, like applications.

In any environment, where there is risk or instability, test-driven development mitigates those dangers to the point of rendering them almost non-existent.  This is especially true in a good object-oriented environment where behaviors can be easily isolated and tested.

Solution:
Good object oriented platforms, and really even mediocre ones, provide the notion of a class.  That is: an encapsulated, refactorable unit of design that provides a reliable way to instantiate objects with predictable behavior and easy-to-couple-to interfaces.

Implementation:
We cannot simply take existing mechanisms for defining classes and apply them to databases because the forces around a database are different - they have "inertia" in the form of knowledge that cannot be lost in the course of a design change.

The core of the implementation is to introduce a predictable, highly structured create-or-upgrade path that minimizes errors, protects existing knowledge, and regulates how change is introduced to any given database instance.

The next step is to specify exactly how changes are introduced to a database by test-driving them.  This all-but-eliminates the chance that a database will lose knowledge in the course of changing its design.

Once that is accomplished, provide a strong interface for your database to ensure that application developers are warned of breaking changes as soon as they possibly can be.  You, yourself, can be warned of these changes by using test-driven development to specify the behavior of the current database.

Finally, there must be a way to handle errors.  This is a function of good database development practices - like taking regular backups - and providing a mechanism for rapidly introducing corrections to errors that have already been committed to production.  Once you have that system in place, by definition, you will have a process that can be used to enroll legacy databases into your agile database development process.

Where Does the Buck Stop?

So I just watched another Michael Moore propaganda film and the theme of individual non-accountability continues.

I have a question for anyone who is swayed to believe that homeowners are not responsible for the consequences of their actions or that workers aren't responsible for choosing to work a job where they are underpaid.

If individuals should not be accountable for the consequences of their actions, who should?

The Law of Damocles

People keep talking about the Law of Demeter.  It's a pretty good law - although it seems like a hard one to break in a modern programming environment.

Anyway, I'd like to propose another law based on a classical mythical character: The Law of Damocles: All code exists with a sword poised above its head.

Shit code is better deleted than mutated.  Ensure the behaviors are properly specified with tests, delete it, and start writing good code until the tests are all green again.

The Golden Hammer

I think I've finally turned up something in my search for a compelling reason to use Ruby rather than something else. Here is the first argument in favor of it that includes an actual snippet of code that I have found:

http://wowkhmer.com/2009/01/20/dotnet-coders-should-give-ruby-a-try/

I think people's insistence on Ruby being a worthwhile thing is based on its expressiveness when defining the behavior of an algorithm. Personally, I don't see how the Ruby code in that example is any more expressive than the C# code but that's not the point. For the sake of argument, I will grant that it somehow might be for some people.

It still doesn't matter. People are already able to write and understand very complex algorithms. Making them easier to implement is not a good thing. For one thing, it optimizes the easiest part of what we do. For another, it enables programmers with outdated skills sets by allowing them to do what they are comfortable doing: writing "programs" and wrapping them up in objects.

Defining algorithms is not what good programmers do. What good programmers to is encapsulate variation to such an extent that there is no need to define a complex algorithm. This is very difficult as it requires us to think about things in terms of entities with responsibilities rather than in terms of sequences of steps, some of which are conditionally executed.

So I think that Ruby is a bit of a golden hammer: It's optimized for how it looks rather than how it functions - it focuses on letting people indulge their bad habits rather than providing an incentive to correct their behaviors.

That is... unless there is some way it enables people to define abstractions and encapsulate variation better than other languages. If there is, please send me a link to an example demonstrating.

A compelling reason to learn Ruby, anyone?

I am presently in search of a clear, concise explanation of what advantage Ruby might provide over, say, C# or Java and rock-hard evidence to back up that(those) claim(s). Any suggestions?

I'm tired of hearing how teachers are underpaid

Look at this.  They get full benefits and are paid around $40k/y to work about a thousand hours a year.  That means that, in terms of effort that goes in versus money that goes out, they are paid about like a decent programmer on average.  If my experience was at all typical, most of them do a terrible job and don't even try to do a good one.  Throw into that that there is a mechanism specifically designed to ensure they don't have to do a good job (tenure), and it's a pretty sweet deal.

I would say being paid comparably to one of the most highly paid professions in our country while not being held to any real standard of quality and, in fact, being explicitly shielded from any such standards puts teachers in the category of "overpaid" rather than "underpaid."  If a government wants to cut costs by bringing their pay into alignment with the value they deliver (or at least improving the alignment), then so be it.  Don't like it?  Find a different job.

Most of the people who should be teaching children probably aren't in it for the money anyway.

Walk away, walk away / I'll be a parade

"Modus Operandi"

Walk away, walk away / I'll be a parade

"Modus Operandi"

Why TDD Does Not Replace Compilers

There is a commonly held belief that the presence of good tests obviates the need for a compiler.  It is a myth.  It can easily be shown to be false and I will do so in a few paragraphs.

Conceptually
Before demonstrating in concrete terms why a compiler is still necessary.  Now, perhaps it is true that the single task of adding a class to a system of classes might not greatly benefit from the presence of a compiler.  It also might not be true.  Either way, I'm not going to address it because it's not the hard part of what we do.

Where a compiler comes in handy in a way that the simple presence of tests cannot help us is in the process of changing design, not creating it.  TDD does not replace design, it augments it by attaching executable specifications to elements thereof.

Compilers tell you when elements of your design are inconsistent.  In ways that tests cannot.  At least, in ways that tests cannot do without replicating the purpose and behavior of compilers.  They do this by allowing you to create positive coupling that can be validated at compile time.

Duplication
To really understand the value of a compiler, you have to understand the nature of duplication.  Duplication exists whenever you have to make a single change in multiple places.

Unfortunately, duplication is not actually something we can eliminate altogether.  In fact, all the techniques we have been taught for eliminating duplication do not, in fact, eliminate duplication but actually replace uncheckable duplication with coupling - a form of duplication that can be mathematically checked for consistency.

For instance, the signature of a method and the calls to that method are duplication.  There are cases when changing the signature also requires the calls to be changed or requires some other change like creating an overload.  However, with a strong-typing system such as you often find with a compiler, you have a mechanism to ensure that all calls are at least basically in sync with the methods they are invoking.

Duplication doesn't stop you and rarely even slows you down when you are creating something new.  It's when you are changing things that duplication becomes a killer.  The reason it is a killer because people seldom find all instances of duplication on their own.  I refer you to Shalloway's Law.

Change
Healthy products change a great deal.  The more useful and important your software is, the more you are going to need to change its design.  There's a small irony there, isn't there?  The more your software is used, the more it will change.  The more it changes, the more the design will change.  The more the design changes, the more you will want to keep instances of necessary duplication in sync with one another.

The key to surviving change in any system of a reasonable size is not to eliminate duplication; that is impossible to do one-hundred percent.  Instead, it is to eliminate unnecessary duplication and to comply with Shalloway's Principle (same link as before) in the case of necessary duplication.

Dependencies and Strong Typing
Strong typing, which is typically found in compiled languages, is the only tool we have invented at the time of this writing that combats necessary duplication by linking together all instances of a particular duplication.  It's really the strong typing, not the compilation, but at this time the correlation is so strong that you can practically treat them as the same thing, setting aside bastardizations like VB.Net.

Strongly typed languages tend to do this by allowing programmers to create dependencies that are implied by one entity in a piece of software using another.  For instance, all calls to a method depend on the definition of that method.  You cannot compile things that call a method without basically correct parameters.  Likewise, all references to an instance of a type grant the dependent access to its public methods and allow it to pass the referred-to object around as a parameter to suitable methods.

These dependencies can then be checked for an entire compilation unit in a single pass, guaranteeing a basic of potential correctness before even bothering to produce a program that can be tested.  Note that, if there were a strongly typed, statically checked, non-compiled language, then the programmer would get the same benefits except that it would happen when the script is loaded rather than when the program is compiled.  The difference between those two things would be imperceptible to most, if not all, programmers.

Strong, Weak, Explicit, Implicit, Static, and Dynamic Typing
It is also important to note that I am not arguing for or against most of the traits listed in the title of this subsection.  I am arguing in favor of strong typing and against weak typing.  I don't really care about implicit typing or explicit typing, nor do I care about static or dynamic typing, except to the extent that they may influence a language's ability to be strongly or weakly typed.  Let's go over what these various things mean, in case it is not obvious.

A strongly typed system is one in which types and method calls can be validated prior to the execution of any part of the type system being evaluated.  Dependencies are defined in some way (probably by use) that allows the basic compatibility of design elements to be automatically verified with no work required from a programmer beyond defining said design.

A weakly typed system is one that is not strongly typed.  That is, one in which a developer has to do extra work to validate the consistency of relationships between entities in a system or one in which it is not automatically done before a design element can be used.

An explicitly typed system would be one in which all types, including abstractions, must be explicitly defined. Java and C# are examples of such a system.  If you want to create a polymorphic relationship, you have to create some kind of interface that is implemented by variants of the abstraction.

An implicitly typed system is one in which types, including abstractions, can be inferred by some part of the programming language or environment.  A lot of scripting languages do this: you define the interface for an abstraction by how the caller uses implementations.  What you may or may not have considered is that C++ also had elements of implicit typing: templates created a de facto interface between the template-ized thing and its dependencies.  Note that the dependencies of a C++ template are still checked at compile-time so implicit typing is, by no means, inextricably linked to strong, weak, static, or dynamic typing.

A statically typed system is one in which types cannot be redefined at runtime.  C#, C++, and Java are all examples of such a language.  You define a type and that's that.

Finally, there are dynamically typed languages.  JavaScript would be an example of such a language, where a type could be changed after it has been loaded and even after it has produced an object to be used.

I don't care whether a language has implicit or explicit types.  Nor do I care whether it has static or dynamic typing.  I may have a slight bias toward the former in each category but I don't consider it a big problem if a language or platform decides to go the other way.  It is specifically the strength or weakness of a language's typing system that matters in the context of this blog entry.

Strongly typed systems are preferable to weakly typed ones because they aid in refactoring and extension in ways that cannot easily be done using other tools available to date.  They do this by forcing us to reconcile instances of necessary duplication before we even bother testing a code change.

Practically
Hopefully, you are at least intrigued by the theoretical information provided above.  However, I recognize that abstract arguments aren't always enough to make one's case so here's a real world example.  This is a simple case and, the more complex the case, the more dangerous weakly-typed systems are.

In this example, we have a template method pattern.  The pattern is implemented the classical way (through inheritance) even though I seldom do that in real life.  No point getting embroiled in another debate before this one is settled.  :)

The Setup
In the weakly-typed language we are using (which I'm making up on the spot to avoid getting into a religious war), we have no way to define an abstract method that ensures all inheritors of our base class implement it.

Let's look at the base class now:

class BaseClass
  method DoesSomething(x, y)
    factor = 1
    factor *= ComputeFactorForX(x)
    factor *= ComputeFactorForY(y)

    return factor
  end method
end class

Now, let's consider one inheritor of this class:

class RarelyUsed extends BaseClass
  method ComputeFactorForX(x)
    return 1 + x * x
  end method

  method ComputeFactorForY(y)
    return -1 - (y * y)
  end method
end class

In addition, there are twelve other extensions of BaseClass which are used very frequently.  Let's even go as far as to say that we have a strong suite of acceptance tests that validate the value delivered by the software being developed.  Even with such a thing, however, it is impossible to test every combination of every object.

Now, as I've stated before. TDD is handy for helping us define the behavior of each of these classes. It may even allow us to prescribe the relationship between them.  What it doesn't do automatically is enforce the relationship.

The Twist
Let's imagine that we want to test-drive an extension to the behavior of the base class.  That is, we want to change the relationship between BaseClass and its inheritors.  TDD allows us to define the behavior as we want and ensure it is done right:

class BaseClass
  method DoesSomething(x, y, z)
    factor = 1
    factor *= ComputeFactorForX(x)
    factor *= ComputeFactorForY(y)
    factor *= ComputeFactorForZ(z)
    return factor
  end method end class
end class

What it doesn't do is make sure that all inheritors of BaseClass conform to the contract it demands.  We have to wait until the real application is put together in order to see if we did everything we need to do.  Let's say we just plain forgot the type RarelyUsed.  Why wouldn't we?  It is, after all, rarely used and a member of a family of types that is thirteen-large.

The Punchline
How long do we have to wait for that feedback?  Ironically, the best-case scenario is: "Forever."  In that scenario, the bogus class is never used and will be deleted the next time someone looks at it.  The next-best case scenario would be when your automated acceptance tests run.  Even that creates an irritatingly-long delay.

What if it happens during manual testing?  The cost of finding and fixing the problem is going to be dozens of times what it would have been if the problem were apparent when making the change originally.

What if it happens in production?  The loss of goodwill is possibly irreparable but probably could be smoothed over.  However, the cost of finding and fixing a problem with that kind of delay between introduction and discovery would be astronomical, when compared to the cost of finding and fixing it immediately.  We're talking hours or days when compared to seconds or minutes.

The Rebuttal
One might argue that one's tests are deficient.  For instance, one could have a method that checks for the existence of the required methods on an inheritor of BaseClass.  That method could then be called from a test in the test suite for each inheritor.

I'll not try and claim that isn't true.  It obviously is.  My counter-rebuttal is this:

Yes, you can do that.  You can also write your code in assembly and write tests for each method which ensure it is properly formed - that all code paths lead to an exit or to an infinite loop, that it properly puts things in the right registers, etc.

In fact, you can re-write as many parts of a strong-type-system-checker or a compiler.  However, doing so either requires you to write a compiler and then use it from a whole bunch of tests or to duplicate your design in the form of tests.

Conclusion
I think I've made some strong arguments in favor of using a strongly-typed programming language over a weak one.  You cannot avoid creating duplication so, in addition to keeping to to the minimum amount necessary, you should use automation to enforce as much consistency between duplicates as possible.

I've failed, however, to mention the biggest and simplest argument.  I've done this partially because I'm saving it for last.  Here it is: there's no cost.  It's one-hundred percent free to use a strongly typed language instead of a weakly typed one.

At least, that's how all the proponents of weakly-typed languages seem to make things look.  You see.  I've spent days trying to get people who don't like type-safety to explain one advantage of not having it.  Nobody - not one person I've talked to - has been able to identify a single reason why it even might be better to have a language be weakly typed than it is to have it be strongly-typed.

The closest anyone has come has been something along the lines of "it's more challenging, so I work harder and with more focus."  Languages should not provide the challenge.  There are plenty of real challenges: getting your tests right, creating a good design, and aligning implementation with business value, to name a few.  We don't need made-up challenges coming from languages that represent a big step backward.

So here is my final argument: I've shown that there is an advantage to having a strongly typed system.  I submit that until a single, real advantage of weak typing can be identified, that tips the scales in favor of strong typing.  If someone can come up with a strongly typed language that is also implicitly typed and/or dynamically typed, then that should be considered to be a competitor to real languages like Java, C#, and Go.

TDD allows you to use a language more effectively than you could without it but cannot make up for fundamental deficiencies in that language.  One such deficiency that is presently en vogue is the absence of a strong typing system.  You can mitigate that deficiency by writing numerous additional tests but at a number of great costs.

Just because you write good tests doesn't mean you cannot benefit from a compiler or something that fills a similar role and nobody has been able to show me how you could possibly benefit from not having one.

A mile stone in the long road

DataClass now supports Java.  It took a lot longer than I expected but it's done now.

You might be thinking "that's neat but what does it mean that it 'supports Java'?"

Before I answer that, I'll briefly remind you what DataClass does - just in case you stumbled on this entry by accident and don't already know.  DataClass is a compiler that takes documents that describe a class of database and produces binaries that know how to build and interact with instances of that class.  The point is to give you the same ease of development you would have for any other class of objects; for instance, allowing you to quickly spin up test instances with identical behavior to production instances.

Following is an example of a DataClass source document:

database SomeDB {
  types int as integer;

  version 1.0 : initialized {
    design {
      public table T {
        public column X with DataType = type(int);
      }
    }
    construction { 
      step sql { $[T.Declaration] }
    }
  }

  current version 1.1 : 1.0 {
    design {
      public table T : base.T {
        public column Y with DataType = type(int);
      }
    }
    construction { 
      step sql { ALTER TABLE $[T] ADD $[T.Y.Declaration] }
    }
  }
}

That would create a class called SomeDB that knew about two versions of a database design, knew how to build and/or upgrade databases to either of those versions, and had a body of symbols allowing clients to couple to public aspects of each of SomeDB's various designs (1.0 and 1.1).  If stored procedures were involved, there would be a proxy for each version that allowed clients to call these stored procedures as if they were normal methods on a normal object.

Up until build 20.11.1733, DataClass would only produce .NET assemblies that connected to a database through ADO.NET.  This most recent build, however, added the ability to produce database proxies and constructors in a java JAR file.

This involved a significant amount of refactoring before it could readily be supported.  I had to encapsulate all the variation between Java and C# (not too hard), between the JRE and the CLR (there's a fair amount), and between JDBC and ADO.NET (they work in fundamentally different ways).

The really disappointing thing about this was that I discovered how fundamentally different JDBC and ADO.NET were very late in the game.  In ADO.NET, access to schema information and binding by name are both easy and reliable.  In JDBC it appears to be expected that you will bind by position - something I thought we left behind two decades ago - because binding by name is possible for certain providers but not guaranteed.

Anyway, that was a problem I could work around pretty easily.  DataClass uses the logical position of a parameter or column to infer its physical position.  There are, I'm sure, times that this will not suffice.  In such cases, developers can use the PhysicalPosition attribute to dictate the binding position of a parameter or a column.  That might look something like the following:

public procedure MyProcedure {
  // Inferred: PhysicalPosition = 1
  public parameter F;
  // Inferred: PhysicalPosition = 2
  public parameter G;
  // Explicitly: PhysicalPosition = 0
  public parameter H with PhysicalPosition = 0;
}

Another "gotcha" for Java programmers is the fact that DataClass expects physical positions to be zero-based and adjusts accordingly for the platform.  So, even though the first parameter is parameter 1 when binding via JDBC, you must specify it as parameter 0.  I guess the point is this: don't take JDBC's implementation details into account as DataClass hides them.

Do not trust CodeMaid

She steals CodeSilverware.

Seriously, though: CodeMaid appears to randomly delete using statements.  I've done a little work to uncover the root cause and identify a fix: it is not ready for production use; uninstall it.

DataClass now available

It's been a long haul but I think it was well worth it.  DataClass is now available for download, trial, or purchase.

Conference Submissions: What to Do with the Rest?

I was looking through the Agile 2011 submissions and I noticed something.  A lot of them are pretty interesting, well thought out ideas.  I do not envy the chairs of the various stages their task.  They are going to have to make tough decisions about which good talks to include in the conference and who they tell "great idea but there's just no time."

It occurred to me that one mitigation for this might be if conferences like this started soliciting papers from the talks that didn't make it and collected them into a book (or collection of booklets).  That way, all the neat ideas that didn't make the cut could still find their way out into the world.  If done right, the conference could make a little profit off of coordinating their release.

Agile Database Development: From Requirements to Delivery

I've received, signed, and returned a contract for my upcoming book, Agile Database Development: From Requirements to Delivery, scheduled for release in 2012. We'll see whether the "From Requirements to Delivery" part of the title survives the book development process. Regardless, the scope of the book will run that gamut.

Something I've noticed - and I really noticed this when I got my first publication contract for Transition Testing: Cornerstone of Database Agility - is that there is nothing like a contract to focus your attention on a project. It's not even the deadline, which I feel was extremely generous on the part of my publisher. It's those words that rattle around in the back of your mind...

This is really happening. The ball is in your court, now.

There is nothing like it to motivate someone. At least, there is nothing like it to motivate me.

Submissions for Agile 2011 Entered

Hi Everyone,

I've made three entries into the Agile 2011 submission system:

1. Fundamentals of Agile Database Development
   
2. Goad Testing: Guaranteeing that Tests Make Distinctions
3. A New Perspective on Automated Tests: Tests as Increments of Process Improvement

I'd love any review and feedback that anyone may offer.