Transport time versus log time

I had a situation where data was being placed using date patterns in the filename and/or paths in HDFS didn't match the contents of the directories. The expectation was that data in 2013/03/29 contained all the data for March 29, 2013. But the reality was that the date was being pulled from the transport. It turns out that the version of syslog we were using was rewriting the header, including the date portion, causing the data to take on the transport time and not reflect the original time of the record. Usually the offsets were tiny—just a second or two—so nobody really took notice. But then one day one of the relay servers died and when the data, which had got stuck on upstream servers, was finally sent it had the current time. In this case it was shifted by a couple of days. What a mess.

Be sure this isn't happening to you if you are placing data by date. Check the date edge cases to see that they are what you expect, and make sure you test your outage scenarios...