DanBaumbach

I'm working on a debugger using the ICorDebug interfaces. I'vem only just started and I'm trying to get a proof of concept for my boss. I'm able to launch or attach to managed apps and I'm getting all the proper DebugManagedCallback calls for attach, create process, load module, etc.


Things get wierd when I add the unmanaged callback and set the flags for interop debugging.

My unmanaged callback gets called and I call ICorDebug::Continue with the value of the out of band flag that's passed into the function. Everything works until I get the initial int 3 call. In that case the out of band flag is FALSE but whether I call Continue with TRUE or FALSE, the app is never fully launched and I don't get any of the managed callbacks.

I've tried even using the Win32 call ContinueDebugEvent() but still my app is never fully launched or attached to. Is there something I'm missing here

Thanks in advance.

- Dan.


Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

For a managed-only debugger, you should definitely check out the Mdbg sample regarding consuming ICorDebug.

Writing an interop-debugger is very very complicated, and unfortunately, we don't have it well documented. This will be a very bumpy process. If a managed-only debugger is sufficient for your needs, you'll probably want to stick with that.

In this case, I'm guessing that you're not clearing the exception (ICDProcess::ClearCurrentException) and so the int3 is going unhandled and tearing down the process.






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

Thanks for the quick answer, Mike. I don't know if I would have even gotten this far without your blog.


When I'm getting the initial exception, I added code to call ICorDebugProcess::ClearCurrentException() with the threadId in the DEBUG_EVENT struct. ClearCurrentException() returns S_OK but the app is still never fully loaded. You can see it in Task Manager under processes but not under applications.

I never got a Windows message telling me that I had an unhandled exception even when I didn't clear the exception.

The reason I'm looking at interop debugging because I know that we'll be wanting to be able to do mixed mode debugging where you can trace into unmanaged code from managed code. The other unmanaged debugging that I expect that we'll want to be doing is listing unmanaged threads, modules and call stack. Would be be better off using SOS

- Dan.







Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

I'm not sure what problem you're hitting with the initial int3. After the Clear + Continue, what happens

Note that there are several different things here:
1. Unmanaged breakpoints: you really do need unmanaged debugging for this.
2. viewing unmanaged threads and modules: you can actually do this from managed-only debugging by using non-invasive inspection. For example, kernel32!CreateToolHelp32Snapshot should still work when you're managed-only debugging.
3. Get a mixed callstack: you can actually do this from managed-only debugging too! If unmanaged code calls managed code; you can still get the native frames.

You may be able to get what you want without having to go through the pain of implementing an interop-debugger






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

I'm running my debugger in Visual Studio 8. I have a breakpoint set on EXCEPTION_DEBUG_EVENT of the unmanaged callback function as well as breakpoints set on the create thread and load module managed callback functions.


I hit the breakpoint in the unmanaged callback. I call ClearCurrentException() which returns S_OK. I then call Continue() and finally press F5 for Go in VS. The managed callback breakpoints are never hit and I never see the app i'm debugging on the screen. I can see it in the process list of Task Manager but not the application list. So, I'm assuming it's hung somewhere.


RE interop debugging:: You can debug a C# app in VS and trace into an unmanaged C++ dll. I need to be able to do that too.

Thanks, as always in advance. The docs on ICorDebug are so few and the docs posted on MSDN are already out of date. Without your blog and help, I don't know how I'd do it.











Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

Sorry for my delay in answering. My response is slower because I don't know the answer here . I can't piece things together from your description. There's too many potential questions such as: is this attach or launch are you getting an out-of-band events what exceptions are you getting (just the startup int3 or anything else) What other debug events are you getting What OS are you running on

I'm thinking the best approach here may be to see if I can provide a small MDbg extension that uses interop-debugging to print native debug events. Although that wouldn't be a full interop-debugger, it would show how to use the basic functionality. You could then step through that as a working example which would hopefully answer lots of questions.

"You can debug a C# app in VS and trace into an unmanaged C++ dll. I need to be able to do that too."
Yeah, that requires interop-debugging.

"docs posted on MSDN are already out of date"
True for http://msdn.microsoft.com. The dos on MSDN2 (ICorDebug at http://msdn2.microsoft.com/en-us/library/ms230588.aspx ) should be up to date, although still sparse.






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

Thanks, Mike. I'm having the same problem whether I attach or launch. If I don't have the interop flags in CreateProcess() set, or TRUE in DebugActiveProcess(), everything proceeds as expected. Thanks to MDbg and Mike Peligrino's old MSDN article I've been making good progress. I'll look forward to see how you accomplish Interop debugging in MDbg.

Unfornately I have a deadline on another aspect of this project so I'm going to have to put aside the managed portion of the debugger for a while and work on the purely unmanaged part. Hopefully I'll get back to it before the end of the month. It's fun learning this stuff and learning more about managed code and C# in the process. I did a lot of C# in my previous job and I'd like to feel as comfortable with it as C++.

- Dan.




Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

What thread are you calling ICorDebugProcess::Continue on
Which version of the CLR is running in the debuggee






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

>> What thread are you calling ICorDebugProcess::Continue on

All ICorDebugProcess::Continue() takes is a BOOL for out of band. The ICorDebugProcess interface I'm using is the one returned from DebugActiveProcess() or AttachToProcess()

Here's my implementation of DebugUnmanagedCallback::DebugEvent()

HRESULT STDMETHODCALLTYPE CDebugUnmanagedCallback::DebugEvent(LPDEBUG_EVENT pDebugEvent, BOOL fOutOfBand)

{

switch (pDebugEvent->dwDebugEventCode)

{

    case CREATE_PROCESS_DEBUG_EVENT:

        CloseHandle(pDebugEvent->u.CreateProcessInfo.hFile);

    break;

    case EXCEPTION_DEBUG_EVENT:

        m_pSymInterface->ClearCurrentException(pDebugEvent->dwThreadId);

        break;

    case LOAD_DLL_DEBUG_EVENT:

    break;

    case UNLOAD_DLL_DEBUG_EVENT:

    break;

    case CREATE_THREAD_DEBUG_EVENT:

    break;

    case EXIT_PROCESS_DEBUG_EVENT:

    break;

    }

    return m_pSymInterface->ContinueFromCallback(fOutOfBand);

}

>> Which version of the CLR is running in the debuggee

I'm using the CLR version returned from GetRequestedRuntime() for creating a process and GetVersionFromProcess() for DebugActiveProcess()

- Dan.






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

Which thread in the debugger is the one calling Continue ICorDebug does have a rather frustrating restriction that you can't call Continue(OutOfBand=false) on the thread that dispatches the ICorDebugUnmanaged::DebugEvent callback. This looks like the issue you're hitting above. In .Net 2.0, I'd expect you get a CORDBG_E_CANT_CALL_ON_THIS_THREAD failure on Continue if you tried.


Random trivia re your handler:
- It's dangerous to clear all exception events because that will interefer with exceptions that the debuggee expects to have thrown. A simple adjustment would be to check the exception code and only clear Breakpoints.
- I think your LoadDll needs to close a file handle in the debug event.

FWIW, the latest MDbg sample has managed C# wrappers for the native debugging API, which you may find useful.

"I'm using the CLR version returned from GetRequestedRuntime() for creating a process and GetVersionFromProcess() for DebugActiveProcess()"
What exact version string is that "v1.0.3705" "v1.1.4322" "v2.0.50727"






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

>> Which thread in the debugger is the one calling Continue ICorDebug does have a rather frustrating restriction that you can't call Continue(OutOfBand=false) on the thread that dispatches the ICorDebugUnmanaged::DebugEvent callback. This looks like the issue you're hitting above. In .Net 2.0, I'd expect you get a CORDBG_E_CANT_CALL_ON_THIS_THREAD failure on Continue if you tried.

<<

Are you saying that I could create a new thread to call continue

- Dan.






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

Mike Stall - MSFT

Not only that you can, but you have to. As a very quick prototype, you can have your Continue just spin up a new thread each time, and that threadProc just calls ICorDebugProcess.Continue(oob=false). Obviously, having a new thread each time is not efficient, but it will let you quickly check whether this is the problem.






Re: Building Development and Diagnostic Tools for .Net Interop Debugging with ICorDebug

DanBaumbach

That fixed it. THANK YOU VERY MUCH!!!!!

- Dan.


case EXCEPTION_DEBUG_EVENT:
if (EXCEPTION_BREAKPOINT == pDebugEvent->u.Exception.ExceptionRecord.ExceptionCode)
{
m_pCorDebugSession->ClearCurrentException(pDebugEvent->dwThreadId);
m_dwThreadId = pDebugEvent->dwThreadId;

DWORD dwId;

HANDLE hThread = CreateThread(NULL, NULL,
(LPTHREAD_START_ROUTINE)UnmanagedExceptionContinueProc, this, 0, &dwId);
CloseHandle(hThread);
}
break;