Exact Magic Blog
TestMatrix Updated
Uploaded the 2008.2.0.2177 release this morning, and tidied up the version numbers a bit. Now the last number is tied into the Subversion build repository of the release branch.
Get this from the Check for Updates…
Main new features are saving the call trace to XML. The coverage window will now automatically open when coverage is collected, and you can turn this off in Tools | Options (thanks for David Veeneman for the screenshot!). Main bug fix is for coverage, where if the last instruction of a method was an explicit throw, it was previously possible to get an InvalidProgramException.
In the call trace, there are two new icons ‘blocked’ and ‘exception’.
- Blocked: the method time is taking 1000ms more than the CPU time, indicating some kind of lock or I/O is blocking the call
- Exception: the method exited with an exception, this is useful to quickly see how far up the stack an exception carries
| Permalink | Comments[0] |
Fancy new website!
We’ve just completed a major update to the look and feel of the website. Please let us know if we’ve broken something. We’ll be adding additional content
about the new releases after the holidays.
| Permalink | Comments[0] |
External Method Calls
To make the profiler even more useful, the latest experimental builds now show external calls. External in the case of the profile really means anything not in your solution.
This approach lets you focus on your code, and see at least one more level into referenced libraries as well as the .NET BCL. It keeps performance high by not deeply tracing into assemblies that are part of the framework that you can’t really modify. This is a performance boosting compromise with the goal of letting you pinpoint parts of your code to tune, without cluttering up your view with a lot of .NET framework internal call stacks.
External methods have a little arrow over the icon, and are grayed out indicating they aren’t in your solution …
Remember to get tracing, you’ll need to enable coverage (TestMatrix | Collect Coverage Data)
You can get this feature from the (Experimental) updates starting… now!
| Permalink | Comments[0] |
Profiling and Tracing
We’ve been working on our profiling and tracing system rather a lot in this release. The main innovation being a system of instrumentation that doesn’t require and C++, or COM, or even a profiler DLL. One of the benefits of this new approach is sheer speed at runtime and the ability to collect some really clever data during the test runs.
The most exciting thing is the Call Trace. When you are running with coverage enabled (TestMatrix | Collect Coverage Data), this call trace is populated along with the line level code coverage.
Think of this as showing you the Call Stack that you see while debugging, but after your tests are finished. This lets you review what was called, which can often be surprising to see just how many methods are involved in a test case.
This also lest you see the time and percentages of time in a method, separating out the actual clock time (Time in Method) from the CPU elapsed time (CPU in Method). Gaps in those two numbers indicate some kind of block or wait, such as I/O, Sleep, or a database call.
One thing to keep in mind, in order to be very fast, and to focus on code you can really do something about, the trace only collects data from your classes and methods, not from the base class library. Very specifically, the times for calls nested ‘under’ another call don’t sum up when you call out to the framework as those methods just aren’t in the call trace shown.
Coming soon on the online updates, we’re planning on enhancing this feature with some additional data:
- Calls out the framework from your methods, one level deep so it stays fast but so that the times ‘add up’
- Allocations of your solution’s objects, count and size - but not cluttering up the data with huge numbers of allocations of string, which is the usual problem with memory profiling.
- File, Network, and Registry I/O similar to procmon/filemon/regmon, but inline in the call trace
- Exceptional Exits - marking on the call trace when a method exited via a a thrown exception rather than returning normally
| Permalink | Comments[0] |
Tracking Exceptions while Testing
Tracking exceptions up the Call Trace lets you see where a test failure starts and to visually see how far an exception ‘leaks’ up your call stack before being handled.
The little indicates which calls exited via an exception, as opposed to exiting normally out the bottom of the function, or through a return.
You can double click on any call in the Trace and navigate there in the editor. And — you’ll see the error fingers in the gutter if this exception exited all the way out of your test case.
You can right click on the fingers and navigate up and down the exception stack trace. This feature is available even without profiling being on, as it intercepts exceptions coming out of the test engines and saves that metadata along with the test. Very handy for tracking down why a test failed.
| Permalink | Comments[0] |
Command Line Support
This has been one of the more requested features, and we’ve got it in the latest (Experimental) update downloads. The basic idea is that you can run your Visual Studio solution’s tests with any supported test engine (NUnit, CsUnit, and MbUnit currently — more on the way), as well as collect coverage.
This command line program runs the same sequence as the Visual Studio integrated test engine, but with one exception:
Build Your Solution First
the command line runner looks at the solution and project files in order to determine the location of the output assemblies in order to run tests. In terms of a NAnt script, just build your solution, then execute TestMatrix.exe as a command line task. Here is a sample run, using our self-test testing solution.
C:ExactMagictrunk>TestMatrix.exe /engine:nunit /solution:Test.sln /coverage:coverage.xml 100.00 % Pass: 24 Fail: 2 Not Run: 1 Testing complete in 00:07.032 |
The actual program is located under your user profile — where the hot updates extract. On Vista, this is generally:
C:\Users\[your username]\AppData\Local\ExactMagic\addins\TestMatrix
Copy the entire contents of this folder to wherever you’d like on your build server, or reference it directly, it has all the files you’d require. The guid-named subdirectories are the actual different test engine plugins. Take a peek at the framework.xml and you’ll get a sense as to how the system works.
This command line version doesn’t currently check for a license, so you don’t need to worry about installing Visual Studio on your build server.
| Permalink | Comments[0] |
Support Forums now Online
Finally, the much requested support forums are online. Of course until you get in there and post something there’s not going to be much content now is there.
| Permalink | Comments[0] |
Update on the “Vegas” beta…
We’re putting the finishing touches on the Vegas beta and hope to ship it in the next few weeks. It is working with the final release of Visual Studio 2008 and we’ve managed to reduce the memory footprint significantly. Please give it a go and tell us how it works for you.
| Permalink | Comments[0] |
Improvements and fixes to “Vegas” beta
There’s been a lot of improvements over the weekend, and it should improve coverage compatibility significantly, especially those of you who put your tests in their own assemblies. We’re almost ready to extend the beta to some more folks. Stay tuned to blog for details. Here’s a summary of the changes…
Bug fixes
- Hide things with 0 code lines in coverage explorer
- Test context menus now work ON the test attributes themselves
- Windows should now remember their placement between launches of Visual Studio
- Fixed missing icon for assemblies in coverage result
- Coverage on/off icon not in sync between all windows
- Coverage now works when your tests are in a different DLL then your application
- Cleanup debug shim between runs - each run gets a fresh AppDomain to clean up garbage and reset static members, as well as a guaranteed fresh run-thread to deal with a similar but distinct case of thread local storage.
- Shadow copies are now off, which should fix some tests that are referencing things relative to the assembly DLL
Improvements
- disable selection in the coverage results
- Dropped code lines / hit lines columns
- Added gridlines to coverage results
- GoTo: Methods now include their parameters
- GoTo: Show owning class for member variables
- GoTo and Open File gadgets now centered on same monitor as mouse
New Features
- New hotkey, Run Solution Tests, Ctrl+R, Ctrl+S
- New hotkey, Run Project Tests, Ctrl+R, Ctrl+J
- Added Run Test menu items to TestMatrix menu
- Added a Run Again button to the Test Result window
| Permalink | Comments[0] |
Coming Soon: “Vegas”
We are happy to announce the first details about the next release of TestMatrix for Visual Studio, code named “Vegas”. We’ve gotten lots of great feedback from our customers, and think you will all find this release very exciting. We’ve got a new platform with a lot of great features, here are some of the highlights, stay tuned to the blog for more updates and specifics.
New Platform with Plug-in Support
We have rewritten our platform to improve speed and reliability, and have introduced a new plug-in based architecture. With plug-ins, you’ll be able to install just the features you wish, and leave out those you don’t need. This will give us much better compatibility with other Visual Studio add ons, and allow us to add new features more quickly. A built in auto-update feature will keep everything up to date.
Support for Visual Studio 2008 (Orcas)
Microsoft should be releasing its next version of Visual Studio around the end of the year, and we plan to be right there with them. Many of you brave souls are already running the betas, and have asked for Orcas support. Contact us if you want to try out an early build.
Multi-Framework Unit Testing
The next release will add support for a number of additional unit testing frameworks, including MbUnit, MSTest, csUnit, and NUnit. Support will be fully transparent, and you can use several frameworks within the same project without difficulty. Great for those of you transitioning from one framework for another.
Super Fast Unit Tests
The fastest by far! We’ve added a number of features specifically designed to help speed unit test execution time. We’ve added real time code analysis so that unit tests are identified even before you have compiled and advanced dependency tracking to reduce test startup time in solutions with lots of projects. Fast, pain-free unit testing is what it’s all about.
And Much More
Lots of other enhancements and new features will be included. Improved unit coverage and reports, more flexible spell checking, and enhanced navigation tools.
Availability and Pricing
We are really excited about the new platform, and want all our users to have the best possible experience. Therefore, we’ve decided that this will be a free upgrade to all registered users. TestRunner users who upgrade to TestMatrix 1.0 before Vegas ships will also receive a free upgrade. So, buy your copy today and you’ll get the next release for free!
The final release won’t be available until the end of the year, but we will soon begin a private beta period to be followed by a public beta. We’ll begin posting more details to the blog as well, so stay tuned for more information.
If you want to get an early look at things, and are not afraid of bugs, contact us and tell us you’d like to participate in the private beta. Tell us the platform, language, and version of Visual Studio you typically use. We’ll contact you within the next couple of weeks and begin making builds available.
| Permalink | Comments[1] |