Timestamps & Daylight Savings off-by-1 (hour) issues, also TDMS

[Kevin P]

I discovered a little quirk that has been discussed before, but I haven't got a handle on exactly what *I* need to do in my app. It has some relation to how Timestamps and Daylight Savings interact, but I'm not sure I complete understand the rules.

Background:

I have released code whose data shows a quirk that I never noticed during development. In question are 2 bits of code that run in parallel in my app. Both retrieve Time information from the same "Get Date/Time in seconds" primitive. Both write that Time information to a TDMS file, but with different methods. When I read that time info back on a different computer, the sets of timestamps differ by 1 hour. I strongly suspect this is tied in with the handling of Daylight savings. The computer that creates the TDMS file is offsite and I don't know its Daylight savings settings. I do know that the file was created in mid-May (when Daylight savings is eligible to be in effect). I'm reading the file now in August on a different PC.

One bit of code is a loop that stores a timestamp and an array of temperatures at a regular rate. These timestamps are stored as a timestamp datatype to a TDMS channel. They accumulate as a 1D array of timestamps that correlate to a 2D array of temperatures.

The other bit of code is an event logger that runs at irregular intervals, on-demand whenever parts of the app code call it. In my app, my events are a cluster containing a bunch of potentially relevant information. One cluster element is a timestamp datatype. However, TDMS doesn't allow me to treat a cluster as a channel. My workaround was to create converter subvi's that would convert back and forth between my cluster and an array of strings. I then wrote this 1D array of strings as a TDMS channel. My converter functions are "Format to String" and "Scan from String" and both use the '%T' timestamp type specifier.

Finally, I'm mostly but not 100% entirely sure that these 1 hour discrepancies didn't show up during development, when I created and read the TDMS files on the same PC. I am quite certain that the Event times agreed with the PC clock though.

Timestamp <--> String & Daylight Savings Questions:

1. Exactly when and how does Daylight Savings influence the timestamp? How can I store and recover this time info in a consistent way so that the times don't have these 1-hour discrepancies?

Thus far, all the data in question has been both written and read during EDT. But in general, I need to handle writing in both EST and EDT and then reading on a different PC in both EST and EDT. I'd prefer a solution that can accomodate the way the data is currently written so I can avoid rolling a rev of the code. (The internal verification procedures for a code rev are far from trivial.)

2. Specifically, let's first consider converting timestamp --> string. I wire a timestamp datatype into "Format to String" using a '%T' type specifier. Let's consider the time portion of the resulting string. Will it depend on what today's date is (whether eligible for DST)? Will it depend on what date is represented by the timestamp? Or will it only depend whether one is during DST and the other isn't? For all questions, will it further matter whether Windows is set to auto-adjust for DST? (Sorry, can't test. I'm administratively locked out of changing date/time settings on my PC). Will it matter whether I'm running XP Home or XP Pro?

3. Now let's consider converting string --> timestamp. I wire the string result from the operation above into "Scan from String" using a '%T' type specifier. What things will affect the raw bits stored in the timestamp datatype? PC date within DST? Time string date within DST. Differing DST status for PC and time string date? Windows DST auto-adjust setting?

Timestamp <--> TDMS & Daylight Savings questions:

Same kinds of questions as above, but in the context of wiring a timestamp datatype as an element of "channel" data into TDMS Write, and how those timestamp bits get interpreted on subsequent TDMS Reads, possibly on a different PC, possibly in a different timezone, possibly on a date whose DST status is different than the DST status of the timestamp itself or the PC that wrote it to the TDMS file.

Finally, if I do have to rev code, my main idea for workaround is to treat the event timestamps like I treat the temperature timestamps. I'll create a separate TDMS channel named "event_timestamp" as part of the "Event" group, and store the times as raw timestamp datatypes. All the other event cluster elements will convert back and forth to a 1D string array named "event_info". Anyone have any better suggestions for storing a moderately complex cluster as a TDMS channel?

-Kevin P.

[Ton Plomp]

QUOTE(Kevin P @ Aug 10 2007, 07:17 PM)

The other bit of code is an event logger that runs at irregular intervals, on-demand whenever parts of the app code call it. In my app, my events are a cluster containing a bunch of potentially relevant information. One cluster element is a timestamp datatype. However, TDMS doesn't allow me to treat a cluster as a channel. My workaround was to create converter subvi's that would convert back and forth between my cluster and an array of strings. I then wrote this 1D array of strings as a TDMS channel. My converter functions are "Format to String" and "Scan from String" and both use the '%T' timestamp type specifier.

use %^T (at least in 8.2), I'm not sure about 8.0 but in 8.2 this forces to use UTC time and won't be influenced by local settings

QUOTE

Finally, I'm mostly but not 100% entirely sure that these 1 hour discrepancies didn't show up during development, when I created and read the TDMS files on the same PC. I am quite certain that the Event times agreed with the PC clock though.

As far as I'm aware there's no difference between developement and deployment, but you might want to check the app's ini file

QUOTE

Timestamp <--> String & Daylight Savings Questions:

1. Exactly when and how does Daylight Savings influence the timestamp? How can I store and recover this time info in a consistent way so that the times don't have these 1-hour discrepancies?

Thus far, all the data in question has been both written and read during EDT. But in general, I need to handle writing in both EST and EDT and then reading on a different PC in both EST and EDT. I'd prefer a solution that can accomodate the way the data is currently written so I can avoid rolling a rev of the code. (The internal verification procedures for a code rev are far from trivial.)

EDT? :question:

QUOTE

2. Specifically, let's first consider converting timestamp --> string. I wire a timestamp datatype into "Format to String" using a '%T' type specifier. Let's consider the time portion of the resulting string. Will it depend on what today's date is (whether eligible for DST)? Will it depend on what date is represented by the timestamp? Or will it only depend whether one is during DST and the other isn't? For all questions, will it further matter whether Windows is set to auto-adjust for DST? (Sorry, can't test. I'm administratively locked out of changing date/time settings on my PC). Will it matter whether I'm running XP Home or XP Pro?

Study the formatting codes of the timestamp! You can get every representation!

I believe LabVIEW only refreshes the time data on startup.

QUOTE

3. Now let's consider converting string --> timestamp. I wire the string result from the operation above into "Scan from String" using a '%T' type specifier. What things will affect the raw bits stored in the timestamp datatype? PC date within DST? Time string date within DST. Differing DST status for PC and time string date? Windows DST auto-adjust setting?

Be aware you are always influenced by the representation (probes/controls whatever). Normally they will interpret the time in the local timezone with the local rules for DST

QUOTE

Timestamp <--> TDMS & Daylight Savings questions:

Same kinds of questions as above, but in the context of wiring a timestamp datatype as an element of "channel" data into TDMS Write, and how those timestamp bits get interpreted on subsequent TDMS Reads, possibly on a different PC, possibly in a different timezone, possibly on a date whose DST status is different than the DST status of the timestamp itself or the PC that wrote it to the TDMS file.

Finally, if I do have to rev code, my main idea for workaround is to treat the event timestamps like I treat the temperature timestamps. I'll create a separate TDMS channel named "event_timestamp" as part of the "Event" group, and store the times as raw timestamp datatypes. All the other event cluster elements will convert back and forth to a 1D string array named "event_info". Anyone have any better suggestions for storing a moderately complex cluster as a TDMS channel?

-Kevin P.

I'm not sure but I believe the DIADem timestamp is just 64 bit and not 128 bit. The best way to store and restore a LabVIEW 128 bit timestamp is using two I64, one for the second and one for the partial second.

There's quite a lengthy discussion on the NI forums. Discussing invalid timestamps and the differences between LabVIEW versions.

Ton

EDIT: Just saw you use 7.1 this version has some DST issues (see the NI discussion)

[Kevin P]

Apologies in advance for my thick-headedness about timestamps and time string formats. I just haven't dealt with them much before. Further comments below not meant to be argumentative, just trying to continue the conversation.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

use %^T (at least in 8.2), I'm not sure about 8.0 but in 8.2 this forces to use UTC time and won't be influenced by local settings

Still wrapping my head around this UTC stuff. The NI forum link was interesting but pretty confusing. Not sure I learned what I should have learned, and not sure how much what I should have learned can help. Regardless, anything involving a change to the "Format Into String" type specifier will be a software revision & release cycle. And I *think* that if it comes to making a software rev, my planned workaround of writing the time to TDMS as a raw timestamp datatype is what I'll probably prefer. The format into string was done only because TDMS doesn't support the ability to accumulate an array of custom clusters.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

As far as I'm aware there's no difference between developement and deployment, but you might want to check the app's ini file

The difference, if any, might be based on other characteristics of the different PCs. During development, I wrote and read on the same PC. I don't recall seeing the 1-hour discrepancy between times written as timestamp datatypes and times converted to and from strings with the '%T' specifier. During deployment, I'm reading on a completely different PC that uses XP pro rather than XP home and may have different Daylight Savings settings. And I'm seeing a 1-hour discrepancy. I just don't know if that discrepancy is irreversibly encoded in the file, or whether there's someway to avoid it or undo it after the fact when I read.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

EDT? :question:

Eastern Daylight Time (and EST = Eastern Standard Time).

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

Study the

formatting codes of the timestamp! You can get every representation!

Should have done it before, but even now it isn't at all clear to me how to use that info to predict the possible interactions with Daylight Savings settings.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

Be aware you are always influenced by the representation (probes/controls whatever). Normally they will interpret the time in the local timezone with the local rules for DST

Ok, I'm starting to grasp this a little better. In my case, I'm not concerned about probes or front panel controls. My specific need is to be able to read a TDMS file and export data and analysis results to an ASCII file. The TDMS file contains 2 kinds of time info. Some of the time info was converted to ASCII strings before writing, some was stored in binary as a timestamp datatypes. Now I read the file on a different PC, which might have different Daylight Savings settings than the PC that wrote the file. I need to export that time info to an ASCII file. I'd like both types of time info to turn into ASCII strings that agree about the hour. I'm still not sure whether I have a chance to do so reliably or whether my initial "Format Into String" produced an irreversible ambiguity.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

I'm not sure but I believe the DIADem timestamp is just 64 bit and not 128 bit. The best way to store and restore a LabVIEW 128 bit timestamp is using two I64, one for the second and one for the partial second.

All I know is that TDMS accepts timestamp datatype directly with no coercion dots.

QUOTE(tcplomp @ Aug 10 2007, 02:04 PM)

EDIT: Just saw you use 7.1 this version has some DST issues (see the NI discussion)

Sorry -- actually using 8.20 Upgraded this spring for TDMS on the app in question. Got errors when I tried to update my profile just now.

-Kevin

[jdunham]

Only time for a quick note. LabVIEW only stores timestamps in UTC. In older versions of LabVIEW this was great EXCEPT that there was no easy way to get UTC out of the timestamp; it would always convert it back to your local PC time zone before showing it to you. As far as I remember, if your machine had changed itself between daylight and standard, there was a lot of potential for an hour of error in your timestamps.

Newer versions of LabVIEW (for sure in 8.2) have mostly fixed this, subject to limitations of the OS. You can choose to convert the timestamps to either local or UTC representations, which is extremely helpful (if a bit late in coming).

If you have discrepancies in your files, I would recommend trying to use the LV Timestamp as much as possible. You may be able to do some tricks like check the first or last times recorded as timestamps versus ascii date/times and see if there is an approximate 1-hour difference. You can also check the OS file timestamp and see whether it matches the last timestamps you recorded in the file.

Good luck, and try to use the timestamps as much as possible.