ViEmu/VS and Coderush compatibility week
I’ve spent the past few days tracing code in assembly like crazy to address several issues. Some of them caused by my recent acquisition of a new laptop with the flashy Vista (I will probably post about the horrible experience at http://blog.ngedit.com, including some amazing workarounds I had to devise). But some of them were in order to solve some compatibility issues with DevExpress’s Coderush with ViEmu for Visual Studio (as reported by a user here in the ViEmu forums). The bad news is that I almost got crazy in the process. The good news is that I was able to find a work around and release a new build of ViEmu/VS (2.1.20):
http://www.viemu.com/ViEmuVS-2.1.20.msi
It’s necessary to manually uninstall the previous version of ViEmu/VS before installing this one.
The bug was reported as Visual Studio 2008 crashing if some specific Coderush refactoring was used, a certain specific way of exiting the refactoring-mode was used, and then you used ViEmu’s ‘u’ to undo the action. It took me a couple of days just to reliaby reproduce the bug, mainly because Coderush refused to work on my new Vista laptop. I googled for solutions to the problem, and sure enough I found someone else reporting it on the Coderush forums, but the DevExpress developers had been unable to reproduce or fix it. I finally found a blog post by emad providing a solution, so I was able to install Coderush on my laptop and reproduce the problem. I tried to post the link to the Devexpress support forum, but they almost wanted me to provide my mothers maiden name to post, so I ended up not contributing the solution (hint hint).
After reproducing it, and debugging an instance of VS 2008 with a debug build of ViEmu and Coderush install, I found out that the call that ViEmu does to perform the undo (IOleUndoManager.UndoTo(pUndoTo)) triggered a crash, with the following call stack:
msenv.dll ! NEnvMsenvTextMgr::CGlobalUndoCoordinator::SavePendingTextBuffers() + 0x384
msenv.dll ! CMultiDocUndoParent::Do() + 0x22d
msenv.dll ! COleUndoManager::UndoTo() + 0x7b
ViEmu.dll ! VSBufAccess::Undo()
ViEmu.dll ! ViOpControl<>::Undo()
ViEmu.dll ! nglib::ViCommandControl<>::Undo()
ViEmu.dll ! nglib::ViCommandsImpl<>::CmdUndo(nglib::ViTrackCommand<> * pTrack=0x0012d724)
ViEmu.dll ! nglib::PerformCommand<>(nglib::ViCommandControl<> & control={...}, nglib::ViCookedCommand<> Command={...}, bool doing_dot=false)
ViEmu.dll ! nglib::ViCore<ViCoreControl>::ExecCommand<>()
ViEmu.dll ! nglib::ViCore<ViCoreControl>::CoreProcessKey<>()
ViEmu.dll ! nglib::ViCore<ViCoreControl>::ProcessKey<>(nglib::key_win32w k={...}, ViOpControl<> & OpControl={...}, VSCaretOps<> & CaretOps={...})
ViEmu.dll ! TInterceptor::ProcessKey(nglib::key_win32w key={...})
ViEmu.dll ! TInterceptor::SubClassProc(HWND__ * hwnd=0x00270a36, unsigned int msg=258, unsigned int wParam=117, long lParam=1441793)
user32.dll ! _InternalCallWinProc@20() + 0x28
...
What this means is that ViEmu’s call to undo was triggering some internal pending text buffer save, which was crashing. I was also able to reproduce it with VS2005, so the issue had been lurking in there for some time. I tried all sorts of workarounds to try to force a ‘flush’ of the pending state before doing my call, but in the end I found that the only way to avoid it was to do the ‘undo’s by calling DTE.ExecuteCommand(“Edit.Undo”), rather than the internal IOleUndoManager.UndoTo(). The problem is that using the DTE method doesn’t allow me to specify which or how many actions to undo, which is key to proper vim-like undo emulation (a command like “cwhello<esc>” has to undo both the typing of ‘hello’ and the deletion of the previous word). In the end, I could do a manual inspection of the stack combined with a DTE call for each undo step, which worked around the issue reliably.
This is one of those cases where I like to think that it’s not a ViEmu bug, but one which was hiding inside Visual Studio and only triggered by the pretty sophisticated use of VS that ViEmu has to do. I could have probably recreated the bug by generating a new add-in with one button and 3 lines of code that called the internal UndoTo(). I’ve done that kind of thing in the past to report issues to Microsoft. But since neither MS or DevExpress are likely to release timely fixes, I just went straight to finding a workaround.
Anyway, the new versions is up there for download to interested parties, and I’ll be testing it myself intensively to verify that the new way to do undo grouping doesn’t cause any unwanted issues, and promote this to the official 2.1 build in the near future. Meanwhile, I’ve found a few issues between VS2008, Coderush and Codekana that I need to address, and someone dropped me a line that ViEmu’s Resharper compatibility measures are not doing very well with the latest Resharper 3.0… thus is the life of a plug-in the developer.
Happy New Year everyone, and I hope 2008 will be full of success for all of you and your own projects and void of any new compatibility problems for ViEmu! 🙂
