Troubleshooting VMM and supporting technologies

This recipe will take you through the process of troubleshooting VMM and its supporting technologies for a successful VMM deployment.

Getting ready

Having an understanding of the core technologies that VMM depends on to work correctly is the initial step to troubleshooting VMM:

• WS Management (WinRM)
• WMI
• BITS
• DCOM
• WCF

Troubleshooting is never an easy task, but VMM 2016 provides tools and ways to help you find and remediate an issue.

How to do it...

Following are some techniques you can use to troubleshoot:

• Event logs

A good starting point is to look at the event logs. Look for OS- and VMM-related errors or failures. A problem with the operating system (OS) or one of its core services could result or lead to a problem in VMM.

For example, if you are running SQL Server on the same server and it did not start, VMM management service will not start either and VMM operations will fail as a direct result of this. You can easily find this by looking for errors in the system or application logs, errors that would indicate, in this example, that the service is not running (for this example, you can also check Services.msc).

• VM manager log

When looking for VMM errors, it is recommended that you to look at the VMM log as well. To do so, perform the following steps on the VMM running Windows Server 2016:

1. 1. On the Server Manager window, click on Tools
   2. Select Event Viewer, expand Applications and Services logs, then go to the Microsoft and select the VirtualMachineManager log

• VMM installation-related troubleshooting logs

VMM records information about the VMM agent installation. However, if the installation logging is not sufficient to determine the cause of failure, you can enable tracing by using the VMM MPS Reports tool and then restart the installation.

• VMM server setup logging

Installation logs are written, by default, to the C:\ProgramData\VMMLogs
hidden folder

• VMM agent installation logging

When installing an MSI package, such as installing the VMM agent manually, you can enable logging using the following syntax:

        msiexec /I MSIPackageName.msi  /L*V path\logfilename.log 

For example, using the syntax, we can come up with something like the
following command:

        msiexec /I "C:\setup\vmmAgent.msi" /L*V vmmagent.log 

The local agent installation information is logged in the C:\ProgramData\VMMLogs hidden folders.

Look for the logfile vmmAgent.msi_m-d-yyy_hh-mm-dss.log.

In logs, it is common to see errors shown as Carmine errors. Carmine was a VMM project code name during its development process.

• Troubleshooting WinRM

To check if WinRM has remote access, check if:

• • The SID in RootSDDL maps to the VMM Servers local group on each Hyper-V host
  • The local group contains the account that VMM management service runs as a service

How it works...

A good understanding of what a successful installation log contains from a POC or a pilot environment is important to identify possible issues, especially if it appears when deploying VMM on a production environment, as you can then compare both logs.

There's more...

Run the following command on the Hyper-V host:

winrm id 

This should produce an output similar to the following output:

IdentifyResponse
ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor = Microsoft Corporation
ProductVersion = OS: 10.0.14393 SP: 0.0 Stack: 3.0
SecurityProfiles
SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos

If the result shows an error, run the following command for a quick configuration of WinRM:

winrm qc 

If prompted, answer Yes. You will receive a response like the following:

WinRM already is set up to receive requests on this machine. 
WinRM is not set up to allow remote access to this machine for management. 
The following changes must be made: 
Enable the WinRM firewall exception 
Make these changes [y/n]? 
WinRM has been updated for remote management. 
WinRM firewall exception enabled. 
WinRM can now be tested again by typing 'winrm id' as before 

Now check the listener:

winrm enum winrm/config/listener 

Run the following command on the VMM management server:

winrm id -r:http://HyperVHost.yourdomain.local:5985 -u:YOURDOMAIN\AdminUser 

The result will be similar to the following:

IdentifyResponse
ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor = Microsoft Corporation
ProductVersion = OS: 10.0.14393 SP: 0.0 Stack: 3.0

Otherwise you will receive the following error:

Error number: -2144108526 0x80338012 
The Client cannot connect to the destination specified in the request 

This could indicate communication issues, so check your network, host firewall, and connectivity.

Most WinRM-related events appear in the system or application event logs. The Service Control Manager often contains the error, as the WinRM service has terminated or restarted for some reason.

To avoid this scenario, conduct the following checks:

• Make sure you installed all of the prerequisites
• Check the firewall rules and make sure the ports are configured correctly
• Open the command prompt (Run as Administrator) and type the following command:

      winrm qc -q 
      winrm set winrm/config/service/auth @{CredSSP="True"} 
      winrm set winrm/config/winrs @{AllowRemoteShellAccess="True"} 
      winrm set winrm/config/winrs @{MaxMemoryPerShellMB="2048"} 

Verifying WMI providers

You can check if the WinRM can communicate with OS WMI providers by running the
following command:

winrm enum wmi/root/cimv2/Win32_ComputerSystem -r :http://servername:5985 [-u:YOURDOMAIN\AdminUser] 

By running the following command, you can check if the WinRM can communicate with Hyper-V WMI providers:

winrm enum wmi/root/virtualization/v2/msvm_computersystem  
  -r:http://servername:5985 [-u:YOURDOMAIN\AdminUser] 

Also, to check if the WinRM can communicate with the VMM agent WMI provider, run the following command:

winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:servername [-u:YOURDOMAIN\AdminUser] @{} 
 

If you are using VMM services, do not remove and re-add the host. Instead, evacuate the host before removing or, on the host, uninstall and then re-install the agent manually, then reassociate it in VMM.

Troubleshooting tools

Following are the troubleshooting tools available for use:

• Windows Management Instrumentation Tester (wbemtest.exe)
  • The wbemtest.exe gives you the ability to query WMI namespaces on local or remote servers.
  • Connecting to a namespace locally indicates that it is properly registered and accessible via the WMI service. By connecting to a remote server additionally, it also indicates that WMI connectivity between the two machines is working.
• WMI Service Control Utility
  • This tool configures and controls the WMI service, allowing namespace permissions to be modified.
  • To open this tool, in the command prompt type the following:

              wmimgmt.msc

• • Then perform the following steps:
    1. Right-click on WMI Control (Local)
    2. Select Properties
    3. Click on the Security tab and then select Root
    4. Click on the security button to check the permissions

Background Intelligent Transfer Service (BITS) troubleshooting

BITS transfers files between machines, providing information about the operation's progress. The transfer can be asynchronous.

In VMM, BITS is used for encrypted data transfer between managed computers. Encryption is done by using a self-signed certificate generated when the Hyper-V host is added to VMM.

You can use BITSadmin to verify that BITS is working properly outside of VMM.

BITSadmin is available in Windows Server. Some useful examples of BITSadmin are described at: http://msdn.microsoft.com/en-us/library/aa362812(VS.85).aspx.

BTSadmin is deprecated in Windows Server 2016. Administrative tools for the BITS service are now provided by the PowerShell module BitsTransfer:

• Add-BitsFile
• Complete-BitsTransfer
• Get-BitsTransfer
• Remove-BitsTransfer
• Resume-BitsTransfer
• Set-BitsTransfer
• Start-BitsTransfer
• Suspend-BitsTransfer

Data collection tools

The following tools are used to collect data surrounding VMM issues:

• VMM tracing tools: VMM tracing tools provide the ability to manage, collect, and view various traces and diagnostic information in a VMM environment:
• Gathering trace information: When you face an issue and need to report it to Microsoft, you can gather the trace by performing the following steps:
  1. In the VMM server, open the command prompt with administrative rights and type the following command:

logman create trace VMMDebug -v mmddhhmm -o
$env:SystemDrive\VMMlogs\DebugTrace_$env:computername.ETL 
-cnf 01:00:00 -p Microsoft- VirtualMachineManager-Debug                           
-nb 10 250 -bs16 -max 512

• 2. Start the trace collection by executing the following command:

logman start VMMDebug 

1. 3. Next, try to reproduce the issue, and at the end stop the trace collection by executing the following command:

              logman stop VMMDebug  

1. 4. Send the ETL file located in %SystemDrive%\VMMlogs\DebugTrace_%computername%.ETL to Microsoft.
   5. Delete the debug information by executing the following command:

              logman delete VMMDebug

After gathering the trace, you can use netsh trace convert command on the traces. This command converts the ETL binary trace logs into TXT files that can be viewed in any text editor.
To convert the ETL file:

1. Run CMD or PowerShell with administrative privileges.
2. Start conversions by executing the following command:

      nets  trace convert C:\VMMlogs\yourfilename.etl

You will receive a response like the following:

       Input file: C:\VMMlogs\VMMLog_VMM01_09242130.etl
       Dump file: C:\VMMlogs\VMMLog_VMM01_09242130.txt
       Dump format: TXT
       Report file: -
       Generating dump ... done

3. Open the generated TXT-file in Notepad by using the following command:

      notepad c:\VMMlogs\yourfilename.txt

See also

• Configuring ports and protocols on the host firewall for each SCVMM component recipe in Chapter 3, Installing VMM 2016