A comprehensive reference of log files

You should always configure your log files properly. Log files are the best way to get invaluable information in the detection of a problem and in troubleshooting issues. In the upcoming sections, you will find a comprehensive reference about your VMware vCenter and vSphere host infrastructure components.

vSphere log files – vSphere host 5.1 and later

When you troubleshoot different issues within the virtual environment, vSphere log files become the most important troubleshooting tool one can have. Not only can you use the log files to find and fix the problems, but you can also use them to avoid these problems from occurring in the future.

Logs for vSphere host 5.1 or later are grouped according to the source component (for more detailed information, please visit the VMware Knowledgebase):

vsphere Log files	Description
/var/log/auth.log	vSphere host shell authentication success and failure
/var/log/esxupdate.log	vSphere host patch and update installation logs
/var/log/hostd-probe.log	Host management service responsiveness checker
/var/log/usb.log:	USB device arbitration events, such as discovery and pass-through to virtual machines
/var/log/vmkernel.log	Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup
/var/log/lacp.log	Link aggregation control protocol logs
/var/log/hostd.log	Host management service logs, including virtual machine and host task and events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections
/var/log/Xorg.log	Video acceleration
/var/log/dhclient.log	DHCP client service, including discovery, address lease requests, and renewals.
/var/log/rhttpproxy.log	HTTP connections proxied on behalf of other VSphere host webservices
/var/log/shell.log	vSphere host shell usage logs, including enable/disable and every command entered
/var/log/sysboot.log	Early VMkernel startup and module loading
/var/log/boot.gz	A compressed file that contains boot log information and can be read using zcat /var/log/boot.gz|more
/var/log/syslog.log	Management service initialization, watchdogs, scheduled tasks, and DCUI use
/var/log/vobd.log	VMkernel observation events, similar to vob.component.event
/var/log/vmkwarning.log	A summary of warning and alert log messages excerpted from the VMkernel logs
/var/log/vmksummary.log	A summary of VSphere host startup and shutdown, and an hourly heartbeat with uptime, the number of virtual machines running, and service resource consumption

Note

For information about sending logs to another location (such as a datastore or remote syslog server), see Configuring syslog on ESXi 5.0 (2003322).

Logs from vCenter Server components on vSphere host 5.1, 5.5, and 6.0

When vSphere host 5.1/5.5/6.0 is managed by vCenter Server 5.1, 5.5, and 6.0, two components are installed, each with their own logs, as described in the following table:

vsphere related vcenter Log Files	Description
/var/log/vpxa.log	vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent
/var/log/fdm.log	vSphere host high availability logs, produced by the fdm service

vCenter log files

The vCenter Server logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. Sometimes, logs are pointed to store in a drive other than the system drive. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

vcenter Log Files	Description
%ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\	vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003
C:\ProgramData\VMware\VMware VirtualCenter\Logs\	vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008
/var/log/vmware/vpx/	vCenter Server Appliance 5.x
/var/log/vmware/vami	vCenter Server Appliance 5.x UI
C:\ProgramData\VMware\VMware VirtualCenter\Logs\ vpxd.log	Main vCenter Server logs, consisting of all vSphere Client and WebServices connections, internal tasks and events, and communication with the vCenter Server Agent (vpxa) on managed ESX/VSphere hosts
C:\ProgramData\VMware\VMware VirtualCenter\Logs\ vpxd-profiler.log C:\ProgramData\VMware\VMware VirtualCenter\Logs\profiler.log C:\ProgramData\VMware\VMware VirtualCenter\Logs\scoreboard.log	Profiled metrics for operations performed in vCenter Server; used by VPX Operational Dashboard (VOD) accessible at https://VCHostnameOrIPAddress/vod/index.html
C:\ProgramData\VMware\VMware VirtualCenter\Logs\ vpxd-alert.log	Non-fatal information logged about the vpxd process
C:\ProgramData\VMware\VMware VirtualCenter\Logs\cim-diag.log C:\ProgramData\VMware\VMware VirtualCenter\Logs\vws.log	CIM monitoring information, including communication between vCenter Server and managed hosts' CIM interface
C:\ProgramData\VMware\VMware VirtualCenter\Logs\drmdump\cluster.xxx\proposeAction.dump.gz	Actions proposed and taken by VMware Distributed Resource Scheduler (DRS), grouped by the DRS-enabled cluster managed by vCenter Server—these logs are compressed and are inside the cluster folder named proposeAction.dump.gz
C:\ProgramData\VMware\VMware VirtualCenter\Logs\ls.log	Health reports for the Licensing Services extension and connectivity logs to vCenter Server
C:\ProgramData\VMware\VMware VirtualCenter\Logs\vimtool.log	Dump of string used during the installation of vCenter Server with hashed information for DNS, username and output for JDBC creation
C:\ProgramData\VMware\VMware VirtualCenter\Logs\stats.log	Information about the historical performance data collection from the ESXi/ESX hosts
C:\ProgramData\VMware\VMware VirtualCenter\Logs\sms.log	Health reports for the Storage Monitoring Service extension, connectivity logs to vCenter Server, the vCenter Server database, and the xDB for vCenter Inventory Service
C:\ProgramData\VMware\VMware VirtualCenter\Logs\eam.log	Health reports for the ESX Agent Monitor extension, connectivity logs to vCenter Server
C:\ProgramData\VMware\VMware VirtualCenter\Logs\catalina.<date>.log C:\ProgramData\VMware\VMware VirtualCenter\Logs\localhost.<date>.log	The connectivity information and status of the VMware Web Management Services
C:\ProgramData\VMware\VMware VirtualCenter\Logs\jointool.log	The health status of the VMwareVCMSDS service and individual ADAM database objects, internal tasks and events, and replication logs between linked-mode vCenter Servers
C:\ProgramData\VMware\VMware VirtualCenter\Logs\Additional log files: manager.<date>.log C:\ProgramData\VMware\VMware VirtualCenter\Logs\host-manager.<date>.log	Additional log files for the VMware vCenter Server

Tip

If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

vCenter inventory service log files

The vCenter inventory service logs are placed in a different directory on a disk depending on the vCenter Server version and the deployed platform. For vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:

 %ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Inventory Service\Logs 

The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/inventoryservice.

Note

If the vCenter Server inventory service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

To collect the vSphere 5.1 vCenter Server inventory service logs, navigate to Start | All Programs | VMware | Generate Inventory Service log bundle.

Following are the log file locations for vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:

vcenter Log Files	Description
C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\ds.log	The main vCenter Inventory Service logs, consisting of all vCenter Server and Single Sign-On connections, internal tasks and events, and information about the xDB
C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\vim-is-install.log	Information about the installation of Inventory Service including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system
C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\ wrapper.log	Information about the status of the Java runtime environment

vSphere Profile-Driven Storage log files

The vSphere Profile-Driven Storage logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:

%ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Profile-Driven Storage\Logs 

The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/sps.

Note

If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

vSphere Profile-Driven Storage logs are grouped by component and purpose. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.

You cannot view vSphere Profile-Driven Storage logs using vSphere Client or vSphere Web Client. To export these logs, see Collecting Diagnostic Information in the upcoming part of the chapter.

vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:

vcenter Log Files	Description
C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\ sps.log	The main Profile-Driven Storage logs, consisting of all vCenter Server and Management Webservices connections, internal tasks and events, and information about the storage profile integrity
C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\vim-sps-install.log	Information about the installation of Profile-Driven Storage including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system
C:\ProgramData\VMware\Infrastructure\Profile-Driven Storage\Logs\wrapper.log	Information about the state of the Java runtime environment

Configuring logs and collecting logs

There are different ways to collect logs from vCenter Server and vSphere hosts. You should configure logs using CLI or vMA, or you can use Host Profiles to configure syslog functionality for a cluster or for similar hosts. (We are not covering Setting up Host Profiles to enable logging). You can use the following ways to gather all the diagnostic information from your VMware infrastructure:

• vSphere Client
• vSphere Web Client
• PowerCLI
• vm-support
• vm-support from vMA

Using vSphere Client

vSphere host 5.x diagnostic information can be gathered using the vSphere Client connected to the VSphere host or to vCenter Server. To gather diagnostic data using the VMware vSphere Client, follow these steps:

1. Open the vSphere Client and connect to vCenter Server or directly to an vSphere host 5.x.
2. Log in using an account with administrative privileges or with the Global.Diagnostics permission.
3. Select a vSphere host, cluster, or datacenter in the inventory.
4. Navigate to File | Export | Export System Logs.
5. If a group of vSphere hosts is available in the selected context, select the host or group of hosts from the source list.
6. Click Next.
7. In the System Logs pane, select the components for which the diagnostic information must be obtained. To collect diagnostic information for all the components, click Select All.
8. If required, select the Gather performance data option and specify a duration and interval.
9. Specify the download location in the next screen to download the logs, and then click Next, and on the last screen click Finish to finish the wizard. On the last screen, you will be presented with a summary of logs you are going to download.
10. Once the wizard completes collecting logs, it will store them in the location you have specified. The log bundle is named with the current date and time, for example, [e-mail].

Using vSphere Web Client

You can gather the diagnostic information of vSphere host 5.0 and higher version by using the vSphere Web Client. The following steps need to be performed to collect diagnostic data:

1. Log in to the vSphere Web Client using your credentials.
2. Click on the Hosts and Cluster.
3. Select a vSphere host, cluster, or datacenter you want to collect the diagnostic information.
4. Click on Actions.
5. Choose All vCenter Actions and then click on the Export System Logs.
6. Click Next.
7. You can select the different components of diagnostic logs to be exported in the System Logs pane. You can choose Select All if you want to export all the logs.
8. You can also select the Gather performance data option and specify a duration and interval.
9. Then click on Generate Log Bundle and then Download Log Bundle.

Using the vm-support tool

VMware has provided a useful tool in order to collect diagnostics information and troubleshooting problems. It is a script-based tool that can also collect the state information of the virtual machines. You can run the vm-support in vSphere hosts or from the vMA appliance. Some of the logs that can be collected using vm-support are vmkernel, host, CIM, virtual machines, security, vpxa, cronjobs, dmesg, update logs, configuration information of the NICs, switches, storage adapters, NAS mounts, multi-path setup.

Tip

Follow the link for more information http://www.vmware.com/files/pdf/techpaper/VMware-Customizing-vm-support-vSphere50.pdf.

Running vm-support in a console session on vSphere hosts

Let's use vm-support tool to generate log bundle in VSphere hosts:

1. Log in to the vSphere host using a Secure Shell Session (SSH).
2. Type the following command and hit Enter:

   vm-support
   

3. The vm-support command will generate a compressed bundle of logs and will save it in a file with a .tgz extension in one of the following locations:
   • /var/tmp/
   • /var/log/
   • The current working directory

Once this is done, you can download the logs.

Tip

To export the log bundle to a shared vmfs datastore, use this command: vm-support -w /vmfs/volumes/VMFS_DATASTORE_NAME.

Generating logs on stdout

You can also use vm-support to output logs on standard output stream. For example, you can get all the logs over an SSH connection without saving anything locally on the host.

1. From a Linux machine, log in to the vSphere host and type the following command:

   ssh root@vSphereHostname Or IPAddress vm-support -s > logbundle.tgz
   

   The –s flag tells vm-support utility to stream the logs on STDOUT, following which you can specify the path and the file name you would like to save your logs in. The vm-support utility will generate the logs and store everything into the specified file.

   Note

   This requires entry of the password for the root account, and cannot be used with the lockdown mode.

2. Let's do that to generate log bundle and store into a data store:

   ssh root@vSphereHostnameOrIPAddress 'vm-support -s > /vmfs/volumes/datastore001/logbundle.tgz'
   

Using vm-support in vMA to collect logs

VMware has provided a nicely written BASH script to collect vSphere host logs from the vMA appliance. You can download the script from the following link: http://goo.gl/e1R4Zu.

The script uses vi-logger that is now deprecated to collect the logs from vSphere hosts.

Follow these steps to collect the logs from vMA 4.0 or vMA 4.1:

1. Once 1035911_vMA-vilogger-gatherer.txt is downloaded to your vMA appliance, rename it as log_collector.sh by typing the following command:

   mv 1035911_vMA-vilogger-gatherer.sh log_collector.sh
   

2. Give it executable permission:

   chmod +x log_collector.sh
   

3. Run this command to execute the script:

   /vMA-vilogger-gatherer.sh
   

   An archive of log files is created within the current folder.

We can also collect logs from vMA without using the preceding script:

1. Create a directory to store the logs:

   DIRNAME=log_collector 
   

2. Now manually copy the log files by running the cp command:

   cp -r /var/log/vmware/* log_collector/
   

3. You can compress the log files by running the following command:

   tar cvzf log_collector_bundle.tgz log_collector
   

For the 4.x version, please visit http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024122to configure vMA to collect logs.

Using PowerCLI to collect the log bundle

PowerCLI is a very powerful and easy-to-use command-line tool to manage vSphere infrastructure. It lets you have complete control by automating, monitoring, and managing your vSphere infrastructure.

Collecting log bundles from vCenter Server

We will now use PowerCLI to collect diagnostic log bundles from vCenter Server. It is very simple and a one-line command to execute. Run the following in your PowerCLI shell:

Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs

The output appears similar to the following:

Data
----
D:\vCenter_logs\logs\vc-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz

Collecting log bundles from a vSphere host

Let's download a vm-support diagnostic log bundle from a vSphere host managed by vCenter Server. Enter the following command:

Get-VMHost HostNameOrIP | Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs

The output appears similar to the following:

Data
----
D:\vCenter_logs\logs\vm-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz

Collecting log bundles from the vSphere log browser

You can access the log browser directly from the VMware vSphere 5.1 web. The client.Log browser is a plugin that works directly with vSphere 5.1. The log browser can be very beneficial and easy-to-use tool when you are troubleshooting problems. Following are some of the benefits of log browser:

• Log comparison
• Searching logs
• Finding and highlighting keywords

The log browser offers a friendly GUI that you can use to take a snapshot of specified host/vCenter logs. Once you retrieve the logs, you can search, compare, highlight, and categorize the logs based on some specific keywords. You can refresh the snapshots of retrieved and stored logs. The search term then can be highlighted for fast navigation. The admin gets a fast UI with the possibility to have the searched word to appear in color. Once you are done, you can export the logs to a file or as a VMware log bundle.

In the troubleshooting lifecycle of VMware infrastructure, log browser is a very handy and useful tool that you can directly access from the vSphere Client. It installs when you install vSphere, and you do not need to look for any other methods to collect the logs. The only problem is that the log browser is only available in vSphere 5.1.

Let's have a quick walkthrough of the log browser:

1. Log in to your vSphere 5.1 web client using administrator credentials.
2. On the left pane, click on the Log Browser option and in the View pane on your right, click on the Select Object option. This will open up a new window from where you can either choose vSphere hosts or any vCenter in your environment.
3. Once the host is selected, you can choose the log file you would like to view from the Type dropdown menu.
4. From here, you can browse different log files from different objects.
5. You can also click on Refresh for the latest logs.

Exporting logs

You can export log files using the log browser by performing the following steps:

1. Navigate to the log browser and select an object to browse.
2. Select Action and then click on Export.
3. Select the type of file that you would like to export.
4. Once it is done, click on Export. When a new browser window appears, browse a location to save the log bundle.