Migrating ClearQuest 2003.06.xx to TFS 2010 with the TFS Integration Platform and a hammer (part 1)

Can an ALM n00b (me) successfully migrate a really old version of ClearQuest, 2003.06.x over DB2, into TFS 2010?  It took some twists and turns and a lot of screencaps, but I got it working!

Chapter 1: How Hard Can It Be?

The week before Christmas, I got pulled in to help with an interesting problem… you know the kind.  I was asked to shadow and assist Shad with a TFS 2010 migration.  My first migration ever, but Shad’s a ninja, so how hard could it be?

Our customer currently uses IBM Rational ClearQuest, which, OK, people do that.  But it turns out they’ve got a fossil: 2003.06.xx is old.  Really old.  It’s built on DB2.  Not some newfangled recent version of DB2, either.  But we’re optimists, apparently, so we’d told the customer we would try, just to see whether we could get this migration going.

Shad and I followed the normal migration checklist:

  • Our customer installed TFS 2010 and the ClearQuest client
  • We got remote access to the app tier and data tier servers and to ClearQuest
  • On the app tier server, we installed the TFS Integration Platform from CodePlex
  • We created a destination Team Project in TFS for testing
  • The ClearQuest migration adapter points at a query, so we browsed the existing Public Queries (OMG slow) and selected one that would return a small record count (17)
  • We fired up the TFS IP and configured a ClearQuest migration (OMG slow)
  • We clicked “Start”

Chapter 2: Success?

Stuff happened.  But not enough stuff.

Figure: initial, um, success, but nothing migrated

The final log message wasn’t so helpful:

[1/10/2012 4:24:14 PM] TfsMigrationShell.exe Warning: 0 : WorkItemTracking: Unable to record sync point for migration source f9fa297c-07b8-45ac-bb66-613b56e3a29b of session cfb9c695-d401-4c71-b349-42b15e4f5534 because lastMigratedTargetItem.ItemId is null or empty

Quick googling suggested it’s a fall-thru message meaning “something else went wrong earlier”.  OK, earlier.  Shad found a bunch of these:

[1/10/2012 4:22:21 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: Workaround for XML save exception : String was not recognized as a valid DateTime.
[1/10/2012 4:22:21 PM] TfsMigrationShell.exe Warning: 0 : WorkItemTracking: String was not recognized as a valid DateTime.
[1/10/2012 4:22:21 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: System.FormatException: String was not recognized as a valid DateTime.

At this point, Shad felt this was the problem.  I wasn’t convinced.  These were Information- and Warning-level messages and not terminal.  They seemed to me to be known/handled, especially because they said “Workaround for XML save exception” right in them; this really looked intentional to me.  (Spoiler alert: Shad was right.)

Either way, though, the logs weren’t giving us enough information about the issue.  Some kind of date isn’t being parsed, but we can’t see what the starting value is or what the parser doesn’t like about it.  Google turned up some interesting trivia about changes from local time to UTC time between CQ versions 2003.06.xx and versions 7.x+, but that didn’t seem like it should cause a format exception.  We didn’t know what version of DB2 we were running, and I couldn’t find a clear answer from IBM or anyone else about what exactly to expect a datetime to look like in it.

We decided that I should download the TFS IP source and try compiling it on one of our own servers for closer inspection.  This, too, didn’t go well.  It takes a lot to get this solution up and running:

  • You have to have Visual Studio 2008 installed to compile the C++ core components.  Not some framework or library, but a full install of VS 2008, just sitting there, even though you’re actually using VS 2010 to do the work in.  This was a big deal for us, because it meant we couldn’t even try to compile the solution on our customer’s server at first.
  • The source ships with various .bat files you have to run to extract binaries from the GAC and deposit them where the solution expects them to be.  Or something.
  • There’s more, but I didn’t get that far.

I fiddled with it quite a bit, making slow or no progress; eventually Shad disappeared with the source for half a day and turned up again with a compiled solution on our server.  Great, but then he got called to another customer site and I was ninjaless!

Chapter 3: Flailing

With trepidation, I dug into the source code on our server and promptly found myself way in over my head, I thought, with a really complex-seeming architecture and massive abstraction.  I had no idea even where to start.

Then Martin had a few days in-office; Martin’s written his own custom Test Track Pro adapter for the TFS IP, so he definitely knows his way around this code.  As is typical of all my colleagues around here, he was ready to race full speed in a completely different direction.  Sometimes that’s exactly what’s needed and other times it just spins me around, you know?

Steven asked me about the general state of this migration and I told him I’d been talking with Martin about a custom adapter.  “But it took Martin,” he exclaimed, “six weeks to build that adapter!  All we’re trying to do here is a little feasibility assessment!  That’s it—I’ve heard enough.  Don’t worry about this any more.”  And off he went, to tell our customer happy holidays and your migration looks seriously unlikely to happen.

Chapter 4: Defeat

I felt about an inch tall and not at all like the “developer” I claim to be.  Fortunately, it was Friday afternoon before the holiday and we all went down the pub and then adjourned for the weekend and I did my best to forget I’d ever heard of migrating anything.

Chapter 5: … Not

It turned out Steven had no intention of letting it go: he sent an appeal to something called the champs list, a group of Visual Studio ALM Rangers who are responsible for the TFS IP and other solutions.  Emails flowed in with offers of help, each seemingly pointing in a different direction.

The TFS IP works by extracting the source data and pushing it into a holding database, then extracting it again from the holding database and pushing it into TFS.  Previously, we’d gotten all the records from the ClearQuest query into memory but the data never made it into the holding database.  With this new error, we weren’t even getting that far – the ClearQuest query got modified by the settings change and it was now failing.  Interestingly, I could see that the original query had a perfectly valid and recognizable date format wrapped in normal single-quotes and it was working fine that way.  Back to the drawing board.

  • CustomSetting CQQueryDateTimeFormat – tried this, didn’t do anything
  • Create an Analysis AddIn using built-in extension points – didn’t try this
  • Turn up the verbosity using TfsIntegrationPlatformTraceSwitch in the MigrationToolServers.config file – didn’t try this
  • Attach .NET Reflector and debug into the TFS IP executable – now we’re talking

All this time, what I really wanted to know was, what’s causing this error? On my last team, my Project Manager spent nearly two years beating it into my head to be more methodical, more systematic, locate and verify the cause of the bug, then try one thing at a time instead of throwing 87 things at the wall blindly and hoping any of them sticks.  She was right then, and she’d be telling me the same thing now, and she’d still be right.  .NET Reflector it is.

Chapter 6: Diagnosis

I had sufficient perms to download and install .NET Reflector on our customer’s TFS app tier server, and our champs list expert had given me step-by-step instructions, which turned out to be flawlessly accurate, for attaching it and pointing it at the necessary .dlls and .exe.

But now I’m trying to debug a process which, as you’ll recall, is not failing.  This means I’m going to have to set a breakpoint somewhere.  How do I know where to do that?

Back in the logs, I found something Shad had obviously known about all along, that I had overlooked, immediately following the warning messages I’d been so sure were just red herrings:

[1/10/2012 4:22:21 PM] at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)
[1/10/2012 4:22:21 PM] at Microsoft.TeamFoundation.Migration.ClearQuestAdapter.MigrationItem.ClearQuestRecordItem.FindLastRevDtls(OAdEntity record, String& lastAuthor, DateTime& lastChangeDate)
[1/10/2012 4:22:21 PM] at Microsoft.TeamFoundation.Migration.ClearQuestAdapter.MigrationItem.ClearQuestRecordItem.CreateRecordDesc(OAdEntity record, String versionStr, ClearQuestMigrationContext migrationContext, Boolean isLastRevOfThisSyncCycle)
[1/10/2012 4:22:21 PM] at Microsoft.TeamFoundation.Migration.ClearQuestAdapter.MigrationItem.ClearQuestRecordItem.CreateChangeGroup(ChangeGroupService changeGroupService, ClearQuestMigrationContext migrationContext, Boolean isLastRevOfThisSyncCycle)
[1/10/2012 4:22:21 PM] at Microsoft.TeamFoundation.Migration.ClearQuestAdapter.ClearQuestAnalysisProvider.ComputeDeltaPerRecord(OAdEntity aRecord)
[1/10/2012 4:22:21 PM] at Microsoft.TeamFoundation.Migration.ClearQuestAdapter.ClearQuestAnalysisProvider.ComputeDeltaPerRecordType(CQRecordFilter filter, String hwmTime)

Figure: Goll-lee!  Don’t that look just like a stack trace?
So I went back to our solution on our own server and did an Entire-Solution search for one distinctive-looking bit of text: “Workaround for XML save exception”.  Bingo.  It’s found in exactly one place: ClearQuestAdapter.ClearQuestAnalysisProvider.ComputeDeltaPerRecordType, the same method as the stack trace.

I got there and found this.  I am not making this up.  I did not add these comments.

foreach (OAdEntity record in recordQuery)
    // HACK HACK
    if (record != null) // this if check is HACK
        catch (Exception ex) // eating exception is HACK
            TraceManager.TraceInformation(string.Format("Workaround for XML save exception : {0}", ex.Message));
    // HACK HACK

Figure: those comments have a funny smell

This fits what Shad observed in the log: it seems to be a generic exception-catcher that outputs a benign Information-level trace to the logs and continues on.  (I can’t judge; I’ve written my share of these, and my share of “we’ll regret this later” comments, too.)

Of course, the problem with generic exception-catchers is much like commercial tuna fishing: occasionally you snag a dolphin.  Or in my case, this poor defenseless creature, in ClearQuestRecordItem.FindLastRevDtls:

DateTime.Parse(cqHistory.Date, CultureInfo.CurrentCulture);

Now what?

Coming next time: I Has a Breakpoint; What I Do With It?

About the Author:

One Comment

  1. Links–01/26/2012 » ALM Rocks! January 26, 2012 at 3:10 pm

    […] Migrating ClearQuest 2003.06.xx to TFS 2010 with the TFS Integration Platform and a hammer (Part 1) […]

Leave A Comment