Fixing Windows Time Problems for Splunk

I’ve just been bitten. We all do, eventually. The case of the dreaded time sync problem. I had a Universal Forwarder sending my Indexer a whole bunch of data. But my searches were not seeing the data because I had a time synchronization error – my Universal Forwarder was a little in advance of my indexer – enough that it was a problem.

Of course, tracking this down is difficult, and there are various techniques you can use. My favorite is using the metrics.log file on the universal forwarder to see if data is being sent. You might also use the “All Time” approach, although I don’t recommend that if you have a lot of data.

So, how do you fix time sync issues. The short version – NTP is your friend. Let’s go through all the bits you have to do.

Firstly, if your host is running on a virtual machine, you need to turn off time sync. Your host is probably getting the time from the underlying VM. Install NTP on your hypervisor and/or turn off time sync within the guest. If you have VMware, you can do this with the following on Windows.

'C:\Program Files\VMware\WMware Tools\VMwareToolboxCmd.exe' timesync disable

If you are on a Hyper-V based hypervisor, you need to open up VM Settings, then go to Management and then Integration Services. Uncheck the Time Synchronization box.

Now that you have the hypervisor behaving and not setting the time for you, you need to set up NTP on all your hosts. On Linux this is relatively easy – edit /etc/ntp.conf to include a server list and then run ntpdate with the new server list. For example, I use pool.ntp.org as my time source, so I have “server pool.ntp.org” in my /etc/ntp.conf file and then I do the following:

ntpdate pool.ntp.org

On Windows there is a little more work to do. Open up a PowerShell window as Administrator (right-click and use Run As Administrator) and enter the following commands:

w32tm /config /manualpeerlist:pool.ntp.org /syncfromflags:MANUAL
restart-service w32time
w32tm /resync

W32tm is a Microsoft tool for managing the ntp peer list from the commandline. Your clock in the system tray will update within a few seconds. Make sure you use the same server list as your Linux boxes. If you need to specify multiple servers, you can space-separate them and use quotes, like this:

w32tm /config /manualpeerlist:"0.pool.ntp.org 1.pool.ntp.org" /syncfromflags:MANUAL

Don’t forget to restart any Splunk searches you were running after you have set the time. With time all set and properly synchronized, you can go about your merry splunking.