Your Continuous Integration Builds, They're Slow, Right?

Ok, so this has been an ongoing topic for me for a while:

"Most people create tests that set up data in the database, stimulate some part of the application, then check that the database changed correctly. It's slow. ... It's also brittle and prone to breakage ..." [more]

"You have to let go of the integration tests. Figure out how to replace end-to-end integration tests with unit tests."


"If you're creating end-to-end tests rather than unit tests ... [then] you can get away with all kinds of sloppy, tightly-coupled design."

"Fast builds with awesome tests will change your life. For programmers, they're the best thing you can do for your codebase, your productivity, and your enjoyment of work." [more]

Updated: Places to Work Downtown

I finally got around to updating the map. I added: Fios, Regence, OpenSourcery, Sensory, Coaxis, Kavi, and Vidoop. Let me know if any of these shouldn't be on the list, or if there are others you know of that fit the tag line:

"Places you might want to work for as a software developer within 2 miles of down-town Portland."

Build System Complexity

The Test Early folks show how VizAnt can be used to graph your Ant build system dependencies. "Does your build file look like this? If it does, than it is time to refactor the build files heavily. [followed by a horribly complex diagram]".

Here's an indicator for an overly complex Ant build (that doesn't require a tool):

How many Ant targets are executable?

If it's more than just a few, then think about refactoring the build heavily.

Or, even more cheekily:

Is your build system is complex enough that you need a visualisation tool to make sense of it?

ECF Rocks You Say?

This is a google translation of a Spanish posting by José Manuel Beas.

I've shown the edits that were necessary to fully understand what is being said. Things have come a long way since I last looked at translation tools! I lost all the page links along the way, so you'll need to refer to the source text for those.

---

Title: ECF Rocks!

Many thanks to Christian Lopez and Jhon Oviedo for helping to prove [me try] the Eclipse Collaboration Framework as I commented in my previous entry.

The experience has been somewhat bittersweet (probably because we had placed high expectations on him). The case is that it works (no one has misled us), but has problemillas the following:

1. You can only share a file each time [one file at a time].
2. There seems to be sharing all kinds of files [It looks like you can't share all types of files] (not very clear what I have, but I could not share a pom.xml and I suspect it is something related to the type of editor-publisher that is different to that used for. Java).
3. The file sharing it really is not much that is, [that great]. [A file can be] transferred to the workspace of [a] "receiver"[.] but which [The person who] has all the capabilities to compile [etc] and others is [the one] that [initiated] the collaboration began. Therefore, there are things like options and other similar refactor[ings] that can not be used "as if we were [sitting] together."
4. We have also tested the "collaborative workspace" and [it] has proved unsatisfactory because it seems rather [like] to an IRC that lets you send files and screenshots relatively easily, but is not a "shared workspace".

Anyway, I think the open road is simply "stellar" [future looks bright] (though [they do] not currently appear to have no [any] plan[s] to further develop [the product]).

---

We're still a while away from being able to easily read foreign blog posting in English. This is progress though. No?

PS Has anyone come up with a Firefox plugin to strip out all the annoying onmouseover and onmouseout that make it a pain to copy and paste the translated text?

Management Tools: Preaching to a Convert

James Shore on Version Control vs Software Configuration Management:

"Real management is about hiring good people, giving them the resources and context they need, and trusting them to do their job."

Yes, I hate heavy handed management tools too. Segwaying smoothly into: Down with ClearCase! Long live Subversion!

The Saviour of J2EE?

This is in response to Gidley's Gossipings "Tapestry 5 - Saviour of J2EE":

I agree on the majority of your points. I don't agree that Java is necessarily the way forward (I think Groovy is much more the way to go) but like you said, that's a different topic for debate.

Java does have an untapped potential for effectiveness, you're right. Tapestry 5 is a good example of a framework taking the pain for us so that we can have an easier time as developers (a counter example being JSF). For example a lot of effort has gone into sidestepping the typical Java trappings of inheritance. More importantly an intelligent set of conventions goes a long way towards making life easy for us. I do think that this all works up to a point however.

Ruby on Rails, for instance, has a very rich set of powerful declarative constructs that would be very very painful indeed to implement in Java. You can only annotate a class so much before it becomes obvious that you're using the wrong tool. Also, trying to replacate (G)Rails methodMissing framework capabilities using Java byte code manipulation would be an impossible if not extreme exercise is self-punishment.

You're also not comparing like with like I think. Both RoR & Grails are complete technology stacks from UI through to persistence, including builds, testing, database migration etc. From what I know of Tapestry 5 the edge of the framework is more or less the dependency injection beneath the UI layer. [Edit: ok, so I looked and there's also test support plus hibernate integration.]

That's really all I have to comment on. As I said, I pretty much agree with you on all points.

PS I'd like to hear what you don't like about Tapestry 5. It can't be all roses.

Update: [more info]

Recommend Me a Book!

I used to read a lot more technology books than I do these days. I've been finding other topics far more interesting and rewarding. Can you recommend me any that might help me become a better team member, communicator, or software developer?

To give you a better idea of what I like, I got a lot out of these:

• The Back of a Napkin
• People Skills
• The Five Dysfunctions of a Team
• Becoming a Technical Leader

Software Development books that I've gained from:

• The Art of Agile Development
• Work-in-progress: Domain Specific Languages
• Work-in-progress: Growing Object-Oriented Software

Stories Over Reasoning

No. this isn't about XP. This is a thread from Presentation Zen on communicating messages more effectively:

'Even if you do persuade through argument ... this is not good enough because "...people are not inspired to act on reason alone." The key, then, is to aim to unite an idea with an emotion, which is best done through story.' [more & more]

'... as a storyteller, you want to position the problems in the foreground and then show how you've overcome them.'



'... spin and a glossy, rosy picture actually work against you ... [whereas] overcoming the negative powers is interesting, engaging, and memorable. Stories like this are more convincing.'

GIMP: Know Your Tools

Now that I'm starting to use GIMP to draw images I thought it would make sense to learn how to use it. There are some good tutorials.

TDD: Growing Object-Oriented Sofware

If you have experience with TDD or are interested in TDD then you'll want to give Growing Object-Oriented Software a look. It's a book being written by Nat Pryce and Steve Freeman of MockObjects. The table of contents and the first chapter are online, and you can discuss the content on the Yahoo forum.

Contents (as of July 18th): [source]

I. Introduction

1. What's the point of Test Driven Development?

2. Test-Driven Development with Objects

3. Kick-Starting the Test-Driven Cycle

4. An Introduction to the Tools

5. Review

II. A Worked Example

6. Commissioning an Auction Sniper

7. The Walking Skeleton

8. Passing the First Test

9. Getting ready to bid

10. The Sniper makes a bid

11. The Sniper wins the Auction

12. Towards a real user interface

13. Sniping for Multiple Items

III. Style Notes

14. On Test-Driven Development

15. On Object-Oriented Design

IV. TDD with Mock Objects In Depth

16. The Readability of Tests

17. Constructing Complex Test Data

18. Designing to Fail: the Importance of Diagnostics

19. Brittle and Flexible Tests

20. What Not to Mock

21. Listening to the Tests

22. Time and Testing

23. Unit Testing Concurrent Code with Mock Objects

24. Testing Asynchronous Code

25. Testing Random Behaviour

26. Test-Driven Debugging

V. Appendices

A. Mock Objects from Scratch

B. Case Study. TDD with mocks in Embedded C

C. jMock 2 Cheat Sheet

D. A Comparison of Mock Object Frameworks

Ubuntu Bamboo Fun & Gimp Setup

I just set up my Bamboo Fun graphics tablet in Ubuntu 8.04 by following neko18's very thorough guide. The guide starts by saying "Make sure you have a couple hours available." This was overly cautious. The whole process took me less than ten minutes including the reboot.

Edit: for pressure sensitivity setup in GIMP you will need to edit your input device preferences for Stylus, Eraser, and Cursor to set their mode to Screen. [details]

Many Hits Imply What?

The number of hits I've had on my On Returning To Java post in one day dwarf that of any other post to date by a wide margin. I guess, going by dzone, people are curious as to what I'll have to say and then turned off by the content:

easyb in Action?

I wanted to quickly get a better feel for easyb. So, I thought the best place to look for actual easyb tests might be easyb's repository.

This example follows the story format (when ... given ... then) and feels like non-toy usage.

However, most of their tests follow the rspec-inspired specification format. This closely mimics the familiar xUnit layout. Here's an example.

There's also an alternate narrative format (section "Further documenting stories") that I saw mentioned in the documentation but couldn't find an example of.

Are you a reader that's using easyb?

I'm curious. Have you found that you do in fact get involvement from non-developers now that your tests can be jointly sketched out? What did you get in exchange for trading in all your test tool support? Was it worth it?

And just from a technical stand point, what are the niggles? What's with all the try catch fails? Is it just style, or is there an issue with overly long stack trace output or something? There doesn't seem to be an after to match the before. How do you tear down fixtures?

Bottom-line: do you think I should be using easyb?

Ranking Metrics

This blog comes in at number five on Agile Daily. Hmm. Whilst I'd love for my blog to be that significant I think this highlights that their ranking system needs a lot of work.

Book Work-In-Progress: Growing Object-Oriented Software

Nat Pryce and Steve Freeman of MockObjects have started opening up some of the material for their book on TDD. This is great news. :D

Grails Domain Diagram Including Subclasses

This proved to be a little more tricky than I'd thought. I first needed to understand that DefaultGrailsDomainClass is a wrapper for your actual domain classes. This class is very good about telling you about subclasses, but not about superclasses. That fact, coupled with associations of the parent class being echoed in the associations of the subclasses confused things a bit. Knowing the superclass you need to go back to the DefaultGrailsApplication to look it up.

So, here it is. A longer DotController that outputs an domain digram for your Grails domain objects:

class DotController {

  def domain = {
    render( generateDot() )
  }
    
  def domainDiagram = { 
    // NOTE dot must be on the path
    Process p = Runtime.getRuntime().exec("dot -Tjpg")
    p.outputStream.withStream { stream ->
      stream << generateDot()
    }
    p.waitFor()    
      
    def imageBuffer = new ByteArrayOutputStream()
    imageBuffer << p.inputStream
    byte[] image = imageBuffer.toByteArray()

    response.contentLength = image.length
    response.contentType = 'image/jpeg'
    response.outputStream << image   
  }
  
  private def generateDot = {
    def dotBuffer = new StringWriter()
    def out = new PrintWriter(dotBuffer)
    
    out.println """digraph { size="6,6"; node [shape=rectangle];"""
    grailsApplication.domainClasses.each { domainClass ->
      out.println """"$domainClass.name";"""
      
      getAssociations(domainClass).each { 
        def property = it.key
        def target = it.value.name
        def isOneToMany = domainClass.isOneToMany(property)
        def type = isOneToMany ? "crow" : "tee"
        out.println """"$domainClass.name" -> "$target" [arrowhead = $type];"""
      }
      
      domainClass.subClasses.each { subClass ->
        out.println """"$subClass.name" -> "$domainClass.name" [arrowhead = onormal];"""
      }
    }
    out.println "}"
    
    return dotBuffer.toString()
  }
  
  private getAssociations(domainClass) {
    // super class associations are duplicated in for sub classes so we strip them here
    def associations = domainClass.associationMap
    if (!domainClass.isRoot()) {
      def superClass = grailsApplication.getDomainClass(domainClass.clazz.superclass.name)
      associations = associations.findAll { !superClass.associationMap.containsKey(it.key) }
    }
    
    return associations
  }
}

A key source in getting this all to work (which I neglected to reference in the last post) was the documentation on the dynamic grailsApplication methods.

Grails Domain Diagram Using Graphviz

Thanks to whoever came to my site after googling "grails graphviz domain". What a great idea! (Thanks Visitor, thanks feedburner). It's easily implemented:

class DomainMapController {
  def index = { 
    def dotBuffer = new StringWriter()
    def out = new PrintWriter(dotBuffer)
    
    out.println """digraph unix { size="6,6"; node [shape=rectangle];"""
    grailsApplication.domainClasses.each { domainClass ->
      out.println """"$domainClass.name";"""
      domainClass.associationMap.each { map ->
        map.each { 
          def property = it.key
          def target = it.value.name
          def isOneToMany = domainClass.isOneToMany(property)
          def type = isOneToMany ? "crow" : "tee"
          out.println """"$domainClass.name" -> "$target" [arrowhead = $type];"""
        }
      }
    }
    out.println "}"
      
    // NOTE dot must be on the path
    Process p = Runtime.getRuntime().exec("dot -Tjpg")
    p.outputStream.withStream { stream ->
      stream << dotBuffer.toString()
    }
    p.waitFor()    
      
    def imageBuffer = new ByteArrayOutputStream()
    imageBuffer << p.inputStream
    byte[] image = imageBuffer.toByteArray()
    
    response.contentLength = image.length
    response.contentType = 'image/jpeg'
    response.outputStream << image
  }
}

If I hit http://localhost:8080/challenge/domainMap/index I get this output for my Erlang Challenge work-in-progress:


Is this plug-in worthy?

Edit: just to make clearer what this is doing, this is the intermediary dot file:

digraph unix { size="6,6"; node [shape=rectangle]; 
  "Topic";
  "Topic" -> "Puzzle" [arrowhead = crow];
  "User";
  "User" -> "UserPuzzleRating" [arrowhead = crow];
  "UserPuzzleRating";
  "Puzzle";
  "Puzzle" -> "UserPuzzleRating" [arrowhead = crow];
}

Six Sigma: Can You Help Categorise It?

I've heard that Six Sigma is a Lean-inspired process. Is it Lean? Is it an adaptable process? Is it a light-weight process? I don't know much about it. Please share your experience with me.

Please comment or view the poll.

Reviewing April 2007 through June 2007

In April 2007 I start blogging about things I still find interesting:

• I start tinkering with Grails
• NFJS and time spent talking with speakers yields a number of stimulating topics. Of those remain: Erlang and Tapestry 5.
• Bart Skondin starts the ball rolling for what resulted in the places to work downtown map.
• I catch-up on browser tools. I'm still actively using FireBug and Selenium IDE.
• I fail to admit that some problems are too broken to fix.
• I talk about reviving XPDX which has later, faltering, success. Diana Larsen is now doing well with the Agile User Group lunchtime meetings [more].

Lean Facts?

Toyota is hailed as the shinning example of Lean thinking. It's interesting then to read the "Lead Toyota Engineer Dies of Overwork" thread on the LeanDevelopment discussion group. As part of that thread David Carlton quoted yesterday:

"During full-production periods, when the plant is running 24-7, employees work incredible amounts of overtime" [indirect quote source]

and went on to say:

"...it colors how I look at parts of lean that I _do_ want to emulate. For example, if I'm remembering correctly, the Poppendieck's books talk about how Toyota always hits their product release dates ... I suspect that both set-based design and (perhaps seriously) overworking people are factors in hitting their dates". [more]

Playing With Grails: OpenID Integration

I'm toying with the idea of turning the Erlang Challenge into a real website. Why not using Grails? I thought the first thing I would need is a user login. Now, I hate logging into websites with a vengeance. Take Facebook for example. Maybe it's an incredible website but because I can't find out without signing up I haven't signed up.

Wow. That was a total aside. Anyway, OpenID seems to be a happy compromise. It's easy to use and it doesn't feel risky. I already have an OpenID via my Blogger account. Great.

So how easy is it to setup a Grails app supporting OpenID?

1. grails create-app challenge
2. cd challenge
3. grails install-plugin openid
4. grails create-controller User
5. Copy and paste the sample login.gsp into grails-app/views/user
6. Edit my UserController:

class UserController {

 def login = { }
 
 def loggedin = { redirect(uri:'/') }

}

7. grails run-app
8. Navigate to http://localhost:8080/user/login and it works

Granted, this isn't doing anything interesting in the loggedin action, there are no filters to enforce a login, and I'm not linking a login to a User object.

It is nice to be able to do something quickly though isn't it?

Footnote: this is the first image that I've drawn for my blog using my (now not so new) Bamboo graphics tablet.

On Returning To Java...

Ow. The pain. After a month of a diet of Groovy, coming back to Java is painful. Interestingly, my Java coding style has now changed. Before you read on, it's important to consider whether the ways in which we're hurting are due to Java's very nature, or due to the conventions we impose on ourselves as Java developers. There's a fair share of both.

Just coding 10s of lines of Java code yields a rich supply of pain points for me:

• Visual clutter - generics, general code overhead, interfaces, casting
• Static typing - a false sense of security, I write a test and it shows me that my runtime casting was wrong.
• Autoboxing - it's frustratingly incomplete. I can do new SomeObject(5) if it takes an Integer as an argument, but not assertEquals(5, someObject.getId()).
• Bloat - def list = [] or def map = [:] is readable whereas List<SomeObject> list = new ArrayList<SomeObject>() or Map<String, SomeObject> map = new HashMap<String, SomeObject>() really isn't.

I hurry back to Groovy at the earliest oppertunity...

Footnote: I read the much-referenced Code's Worst Enemy a while ago and I really didn't understand what he was talking about. Now it makes a lot more sense to me. "[People] view code bases much the way construction workers view dirt: they want great big machines that can move the dirt this way and that." [more]

Kanban Development

Corey Ladas has written a long article with some great tools for explaining what Scrum might look like if you gradually introduce ideas from the Lean perspective.

He starts by getting you think of your development process as a stream from start to finish for each individual unit of work. This is presented as a white board in with cards representing the work, moving from across the board as they move into the different stages of the pipeline.

I like the way he achieves this shift. It's subtle, and the effect is to change the way you think about what's important in your development process: "Average lead time and cycle time will become the primary focus of performance."

His key argument is:

"A task card without a limit is not a kanban in the same way that a photocopy of a dollar bill is not money.

... control the number of cards in circulation. If all of the available cards are already in circulation, then the next person who comes looking for one is just going to have to wait until one returns. This is the very purpose of the kanban system." [more]

So because cards are in limited supply the act of creating a new one is a significant event. If there is an unlimited supply then there are no constraints on creating a new one. The later is the arrangement that most people are familiar with.

As I said, it's a long article. It is aimed at converting Scrum-believers into Kanban-believers. This doesn't detract at all from the value of the points he makes. Go read it. It's worth sticking through to the end. [Scrum-ban]

If Grails Then Why?

I've gotten into Grails through being taken by Groovy. I've only implemented a few toy Grails apps so I'm not that well informed. However, these are my main reasons for Grails as an app framework in order of impact:

• Rich domain objects - the conventions will lead you towards a layer of rich domain objects
• DRY constraints - domain object constraints (not just database ones) are all in one place
• UI Scaffolding - you can quickly get a basic CRUD setup
• Persistence helpers - you get a whole host of dynamically generated helper methods
• Environment configs - different database setups for dev / test / prod e.g. hsqldb / mysql / oracle
• Bootstrap setup - bootstrap convention for app startup
• Plugins - a whole host of plugins

Oh, and:

• It's all in Groovy - which can be very close to Java if you want it to be
• You can drop in any existing Java code or APIs you have already

Data Migration

You know that the "blog-o-sphere" is small when two of your 15 RSS subscriptions show up on the same topic with one referencing the other.

Data Migration is a hot topic for me. Mostly because it can easily be done in a way that really cripples you. Yet, finding a solution that supports your production environment at the same time as your lightweight developer testing environment is a thorny issue.

As Martin Fowler points out: "I'm sure I can see the wistful smiles [on the faces of my readers]. Data migration often looks easy from the safety of whiteboard abstractions, but is usually full of nasty details to trip you up."

However I can't say that I've experienced this for myself: "Incremental migration made a significant improvement in communication with the domain experts." [more]

Incidentally the other blog post was from the LiquiBase blog. They're preaching to the converted from my perspective: "Data migration is easy to push off or forget, and can often kill an otherwise great release" [more]

Grails Has A LiquiBase Plugin

So I'm making an effort to read the Grails Framework Reference from start to finish. Along way I see that there's a LiquiBase plugin! Cool stuff.