You have deployed your service to Azure and discover that it isn’t working correctly.  After going through the troubleshooting series you have identified where the problem is and you think you know how to fix the issue (or you want to add more logging to narrow down the problem in your code), but the prospect of having to delete the existing deployment, rebuild the entire solution, redeploy to Azure, and wait for the VMs to startup is not very appealing.  It would be much easier to just put the fix into the current deployment for testing purposes.

WARNING: The following is only for troubleshooting and testing purposes.  The nature of stateless Azure cloud services means that any changes you make to an Azure VM could be lost at any time so you should absolutely not make any changes to a production service.  If you use these techniques during your troubleshooting or testing phase you must make sure that you incorporate those changes into a new package and redeploy the new package to Azure.

The basic steps to modify a live service are:

1. RDP to the Azure VM
2. Copy the changes (ie. the new DLL) into a temporary location such as the C: drive
3. Terminate the WaHostBootstrapper process from task manager
4. Wait for a few seconds for the role host process (WaWorkerHost or WaIISHost) to terminate
5. Copy the changes (new DLL) into the appropriate folder in the E:/F: drive
6. Wait for the Azure guest agent to restart WaHostBootstrapper
7. Test the changes

Details

2. Copy the changes (ie. the new DLL) into a temporary location such as the C: drive

Shortly after the WaHostBootstrapper process is terminated the guest agent will detect this and begin the process of recycling the role, which includes restarting WaHostBootstrapper.  You will typically have less than a minute between terminating WaHostBootstrapper and the guest agent restarting the process.  Because of this you need to be quick in replacing the updated DLL and you typically won’t have time to do a copy/paste across the RDP connection.  This is why you should first copy the DLL to a temporary location on the VM, then have 2 Windows Explorer windows open (or a cmd prompt with the ‘copy’ command ready to go) so you can quickly copy/paste the updated DLL into the E:/F: drive.

3. Terminate the WaHostBootstrapper process from task manager

From the Role Architecture diagram we know that the WaHostBootsrapper process is the parent process for your startup tasks and role entry point (WaIISHost/WaWorkerHost).  Terminating the host bootstrapper process will also cause all of it’s child processes to terminate.  The guest agent does heartbeats and will detect that WaHostBootstrapper is no longer running and begin the process of recycling the role.  It will first start WaHostBootstrapper, and then WaHostBootstrapper will go through the normal role startup process which includes running startup tasks, configuring IIS, and running your role entry point.

4. Wait for a few seconds for the role host process (WaWorkerHost or WaIISHost) to terminate

It will take a few seconds for Windows to terminate the child processes of WaHostBootstrapper.  If this does not happen automatically you can manually terminate the role host process.

5. Copy the changes (new DLL) into the appropriate folder in the E:/F: drive

Your cspkg contents (role entry point DLL, startup tasks, etc) are extracted to and executed from E:\approot\bin (see http://blogs.msdn.com/b/kwill/archive/2012/10/05/windows-azure-disk-partition-preservation.aspx for scenarios where it will change to the F: drive).  Prior to copying the changed file into the E: drive I would recommend making a backup copy of the file already on the E: drive so you can easily revert back to it as needed.  After you have killed WaHostBootstrapper (and before the new WaHostBootstrapper starts) you need to copy your changes into the appropriate location in the E: drive.  Note that you can change pretty much anything you want – startup tasks, website content, DLLs, etc.

6. Wait for the Azure guest agent to restart WaHostBootstrapper

One the guest agent restarts WaHostBootstrapper your role will startup again and you it will load the updated binary that you just copied into the E: drive.  You can now test to make sure that your fixes resolve the problem (or check your logs for additional trace data, etc).

Changing website content or binaries

The above steps are primarily intended to replace files which are being used directly by Azure (ie. startup tasks or role entry point DLLs).  If you just need to replace website content you can simply terminate w3wp.exe and replace the files in E:\sitesroot\0 (if you have multiple <sites> elements in your csdef then you will need to find the correct folder under E:\Sitesroot).  The next time the site is accessed w3wp.exe will start up and load the updated content.

Note that if you are trying to change IIS configuration settings you cannot simply execute iisreset on an Azure VM.  iisreset also stops the WAS and W3SVC services and then restarts the WAS service.  On an Azure VM the W3SVC service is set to Manual startup so it won’t automatically restart.  If you want to do an iisreset on an Azure VM you need to also run ‘net start w3svc’ (or start the World Wide Web Publishing Service from services.msc).

Preventing WaHostBootstrapper from automatically restarting

If you need more time to make your changes before the azure guest agent automatically restart WaHostBootstrapper then you can break into WindowsAzureGuestAgent.exe with a debugger.  To do this:

1. Download AzureTools from http://blogs.msdn.com/b/kwill/archive/2013/08/26/azuretools-the-diagnostic-utility-used-by-the-windows-azure-developer-support-team.aspx
2. Within AzureTools download the X64 Debuggers And Tools-x64_en-us tool
3. Launch WinDBG.exe (should be at C:\tools\Debuggers\x64\WinDBG.exe)
4. Within WinDBG.exe go to File –> Attach to a Process
5. Select WindowsAzureGuestAgent.exe.  Select No when prompted to save information for workspace.
6. WindowsAzureGuestAgent is now broken into by the debugger and it will stop performing heartbeats and will not automatically restart WaHostBootstrapper.
7. Terminate WaHostBootstrapper and follow the steps above to modify the service.  Note that you will need to manually terminate the role host process.
8. Within WinDBG execute ‘.detach’ (without the quotes).  The WinDBG prompt will change to “NoTarget>”.
9. Close WinDBG.  WindowsAzureGuestAgent will proceed with heartbeats and automatic recovery of the role.

Caution: Do not remain broken into WindowsAzureGuestAgent for 10 minutes.  The Azure host agent process does regular heartbeats to the guest agent process and if it detects no response for 10 minutes it will reboot the VM.