Waiting for return from modal dialog
A modal dialog is one which requires user input before it leaves the screen. The user is unable to do anything else with the application until the modal dialog is dealt with. There are various types, the Common Dialog - such as the Open File or Save File dialog, or one defined by the program itself. For this test the simplest of all the modal dialogs is used, the message box created by calling the API MessageBox.
Switch off single-step message break and use the breakpoint MODAL_DIALOG and run Testbug's modal dialog test. When the breakpoint is reached press F6 (single-step but jump over) until the eip pointer is at MessageBox then press F6 again. The pointer will remain at MessageBox and not move on. This is because the API has been called but not returned. It has not returned because the message box has been made by the system but not been closed. You can see the message box if it has not already appeared, by clicking the appropriate button on the taskbar. You can move the message box about too. This shows that although as far as the debugger is concerned the debuggee is single-stepping, it is in fact running within the system code. This is because you pressed F6 which is "jump over". This caused GoBug to release the trap flag and set a breakpoint at the return from the call from the API. Hence all the functions of the API are properly carried out.
Before you click on "OK" in the message box. See what happens when you press F5 or F6 again. Now you will see a GoBug message box appears reminding you that the debuggee is still in the API and has not yet returned. What this means in effect is that GoBug has lost control of the debuggee until the return from the API.
Pressing F7, F8 or F9 will work, since GoBug simply sets the correct action when the thread returns to the debug loop. Note that the panes go grey, showing that GoBug has little knowledge of what is happening to the thread.
Now press F6 again. GoBug does not permit a return to single-stepping even from F7 in this instance, because it cannot regain control until the return from the API.
You will get the same results if you try to change the action of the visible thread using the traffic light button on the toolbar.
Waiting for return from GetMessage
GetMessage is called by the message loop and it will
not return until there is a queued message which the system wants dealt
with. Queued message are those messages caused either by user input,
(eg. menu command, mouse or key action) or which need not be dealt with
as priorities nor in a strict sequence (eg. WM_TIMER or WM_PAINT).
So the main thread of your application will appear to spend most of
its time in GetMessage.
This is deceptive. While in GetMessage the main thread can do a
lot of work on behalf of the system. For example the system will call
the window procedures using the thread on numerous occasions as part
of its window management and signalling procedures. If not at "START", restart Testbug using the menu item, "file, restart debuggee". Press F8 from "START",
wait till the main window is made, then click on the Windows taskbar or use the Windows start menu and start some other program to de-activate Testbug. View the log to see all the messages this caused. Meanwhile the main thread is still in GetMessage (or at least has come back to it)! You can see the last known eip for the thread by clicking on the thread control button (on GoBug's toolbar).
To view the execution of code in the window procedure you need to set breakpoints or enable the single-step message break. Or you can press F7 and watch it all in the log.
Visibly the same thing happens in GoBug's panes when waiting for a
return from GetMessage as happens when waiting for a return from a
modal dialog. GoBug works in the same way in each case, waiting for
a return from the call.
Waiting for return from an internal procedure
GoBug works in a similar way with internal calls
which take a long time to return.
In this test the following code is used to simulate an internal
procedure which takes at least a minute to finish:-
IP55:
MOV ECX,60
L4:
PUSH ECX
PUSH 1000
CALL Sleep ;just over a second in all
POP ECX
LOOP L4
RET
Use the breakpoint INTERNAL_PROCEDURE. When this is reached
you will see the call to the loop itself (IP55). Once in there
the code will not come out for over a minute (unless you change the
registers!).
Press F6 to single-step but jump over the call to IP55. When you
now press F5 or F6 you will be warned that the thread is still in the call.
[Note if you press F7 and then press F6 you will get different results. This is because F7 sets the trap and watches all the instructions in IP55, whereas F6 releases the trap and just waits for IP55 to return].