Public Class GeoffAppleby

String Performance Part 2

I thought I'd talk a little bit more about a post from a few days ago, String Performance Over Different Languages.

Actually, I think this Part 3, since Bill McCarthy mentioned some more string stuff as a result. He was talking about detecting white space and null strings, and wrote a method that went and determined if Trim would return a 0 length string without actually trimming it. Actually, the next day he then wrote a sequel, trying to outperform his own code (and using GoTo's as a result, go chime in on your opinion of this on his blog :), so perhaps this post is Part 4. I also tried to outperform him, and I sorta did (depending on the string being tested of course) but it took about 8 gotos instead of his 2.

Oh, and don't forget Scott, who went and took it to the ultimate level and reported on the assembler that the JITter spat out for the tests I'd written. (Hey, is it Scott or K. Scott or what? :)

OK, so this is Part 5. Anyone else want to step in here?

Anyway, Joel Pobar left a comment on my first post, which somehow got me thinking - how does it come up in Whidbey?

So I opened my solution up in my trusty copy of Beta2, and asked it to convert my solution. It did, and it totally screwed it up. VB in VS2005 now creates debug and release folders like C# does, but the dll references were to the old bin folders, so whenever I compiled the app it linked against against the 1.1 framework versions from the upgrade.

I threw it away and created a whole new solution, and just copy and pasted the relevant bits of code. This worked much better :)

I was mostly interested in two specific tests:

1. When testing equality in strings in VB, what's the hit to Microsoft.VisualBasic.dll cost us now?
2. When calling string1.Equals(string2), has the C# compiler improved the IL?

Well, there's some interesting results (yes, interesting. Not interesting in the way that the growth on my inner looks, interesting in the way that all the other posts on this subject have been interesting. Pure, geeky, pointless fun. The value in it is about as high as building your own induction circuit with a coil of fuse wire and magnet, just to watch the needle on your multimeter move. But just as cool :)

I can not stress again how very crude my test app is. I'm looping millions of times, comparing the time when i started the loop to the time when i finish, and showing the difference in ticks. However, it's got two buttons, one for a string = test, and the one for a stringvar.Equals() test. Here they are, side by side for screen capture purposes. You can tell which is which by the framework label on the form title bar:

They're very pretty aren't they? The box up the top with 10000000 in it is the number of iterations it will do each test run.

Anyway, so first up is Test1, which compares the two strings using = (or == depending on the language).

The numbers in red my way of highlighting the winner. VB still takes the same amount of time, which doesn't really surprise me, since it still incurs a hit on the Microsoft.VisualBasic dll. I was hoping they'd made some major improvements, but I guess not (this is based of course on my test strings, which aren't good test strings I admit. string1 is "foo" and string 2 is "bar".

But C# has now got slower! Weird huh? I can think of many reasons for this happening, the biggest of which is that this is a beta build of Whidbey, so I wouldn't be surprised if some optimisations are switched off. However, when we fall down to the IL, the thing that strikes me is that the between the two framework versions, the IL is identical. So we're not looking at a compiler problem, we're looking at the JITter. I guess it's time for Scott to take over and compare it at the assembler level :)

Also of note is the changes in Microsoft.VisualBasic. Only one line is different in the IL for the VB versions of the test:

1.1 framework:
L_000f: call int32 [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.StringType::StrCmp(string, string, bool)

2.0 framework:
L_000f: call int32 [Microsoft.VisualBasic]Microsoft.VisualBasic.CompilerServices.Operators::CompareString(string, string, bool)

These both end up doing the same sort of stuff, although the 2.0 version looks better optimised when dealing with null strings. The first ends up in String.CompareOrdinal which forwards onto String.nativeCompareOrdinal. It's declared internal and stuff and reflector doesn't show any source. For the 2.0 framework, it also eventually ends up in String.CompareOrdinal, but this time it forwards onto String.CompareOrdinalHelper, and I _can_ see the source for that - lots of fantastic pointers and stuff!

So now to Test2. Remember this one? This is the one where the C# compiler optimised out the declaration and use of the local Boolean, and as a result ended up taking longer to run.

You'll be pleased to know the fault all lies with the JITter, and it seems it's been improved.

VB is the same, which is good :) C# has been sped up - and it's now finally caught up with VB. Geez, first they get E&C, and now their strings are just as fast. It's just not fair! Again, I'd be interested to know what the assembler looks like. Scott, dude, if I send you my test code again, can you run it through the debugger for me?

So what does all this tell us? Well, it tells me two things mainly.

1. C# in whidbey has been sped up in some places, and slowed down in others. As a result, you'll probably find that on average your code runs the same, but it will still take you longer to write than it will for me in VB, because I don't have to type all those semi colons and curly brackets.
2. VB is a nice constant force that keeps on chuffing along nicely, but I know I'm certainly able to code EVEN faster in whidbey :)

I guess also it tells me that all my decisions are now based on two string - "foo" and "bar" - a couple of ways to test equality, and some dicky red text. All my judgements in these posts are pretty much solely based on this, and therefore should not be applied to the whole of the new .net framework, or even performance when testing with better test data :)