absolute timestamps with ms accuracy

[giopper]

In my data acquisition system

I need absolute time references

with about 1 ms accuracy.

I first developed some code under Linux Fedora7

and the LV built-in time function was accurate enough

for my application.

When I use the same code under WinXP32 (running on the

same hardware!) however I do not get better than about 15 ms.

I did try LV 7.1, 8.0 and 8.2: same result.

Is it a Win32 API problem?

Is there any known way to get absolute timestamps

under WinXP32 with 1 ms accuracy?

The LV timer seems accurate enough:

is there an accurate and reliable way to convert

timer ticks to absolute time?

And what about the use of the Windows

API function "QueryPerformanceCounter"

to get even more accurate ticks?

Thanks

G

[mross]

This is a complicated subject. How absolute do you want absolute to be?

a) NIST traceable?

b) How about better than 1ms relative to the original start time and and start time is nailed down and tracable?

c)Or maybe just really precise with respect to the start time, but the start time could be off a few ms?

Each of these offers some solace from the fact that a PC is really crap for keeping "good" time. Whatever good means. heh heh

A not too difficult relative but precise method [c)] would be to get the timestamp and start a counter timer, getting the timer count for the events of interest. The timer can run 20MHZ accurately so to the extent that you trust the first time stamp and its relationship to the starting of the timer you have a very good idea when each of the events occured and can easily add them to the time stamp for a pseudo absolute time. I love that, "pseudo absolute."

Really, any DAQ card acquisition is keeping pretty good relative time, it is just a matter of, when exactly did it start? All you have to do is digitize the signal and calculate the time from the start. It is a very good idea to be sure this isn't good enough for your purposes before getting involved in the alternatives. Some of the alternatives are a pain in the butt.

If you explain more the brains here can do a better job of explaing the trickier ways to do this.

[giopper]

Thanks Michael for your reply

QUOTE (mross @ Aug 21 2008, 01:06 AM)

...

If you explain more the brains here can do a better job of explaing the trickier ways to do this.

OK, I'll try to better focus on what I actually need

QUOTE (mross @ Aug 21 2008, 01:06 AM)

...

b) How about better than 1ms relative to the original start time and and start time is nailed down and tracable?

...

The b) scenario corresponds to my case.

My data stream must be synchronized to some other data,

provided by another piece of hardware, and the sync

is done by matching the two timelines.

The data stream is not continuous but in a sequence of

data packages, triggered by an external hardware device.

Some numbers: 120 AI sampled at 500 Hz, 220 ms package,

and about 40 ms pause between packages.

For each package I need at least one absolute timestamp:

the other 109 timestamps are at 2 ms regular interval.

QUOTE (mross @ Aug 21 2008, 01:06 AM)

...

you trust the first time stamp and its relationship to the starting of the timer you have a very good idea when each of the events occured and can easily add them to the time stamp for a pseudo absolute time. I love that, "pseudo absolute."

...

Exactly: I trust the clock of my DAQ hardware,

I know the sample rate therefore I have no problem

to know the relative timestamps with an accuracy

that is enough for my application.

The problem is to have at least 1 absolute accurate (1 ms)

time reference with a well known (again 1 ms accuracy)

relationship to my time series.

In the past, I did use IRIG-B time code given by the station GPS

clock (decoded by a PCI board) to get the absolute time.

However, I found that under Linux Fedora7 + LV 8.2 + NI-DAQmx 8.0

I can use the LV built-in time function to get a "pseudo-absolute"

timestamp which is "absolute" enough for my application

(as long as the ntpd server is running and properly working.)

For some other reason, I was forced to move to WinXP32 and

surprisingly I do not get the same accuracy anymore (at least

a factor 10 worse) although using exactly the same hardware.

I really would like to get rid of that IRIG-B board

and under Linux it was actually possible.

Questions:

a) how to produce 1 ms accuracy absolute timestamps under WinXP32

without using additional/specific hardware?

b) why is the LV built-in time function a factor of 10 worse under WinXP32?

Thanks

G

[Yair]

The resolution of the get time primitive in Windows is ~16 ms. I'm not sure why, but I don't think you can change that.

If you want to get ms values, you can use the tick count primitive, which returns the number of ms since the OS was started (U32, so it wraps after ~49 days). You can wrap it in a functional global, so it would remember the timestamp when it started and would do the calculation itself.

If you want higher accuracy, you can call the queryperformancecounter API function for a microsecond resolution, but unless you can sync it correctly, I doubt it would help you.

[mross]

OK. I think you cannot do this without hardware in your PC to get the tracable timestamp. Then you have to synchronize it with the triggering of the acquisition. This is not so easy as far as I know. It would be better if you could take all channels of data using one system or the other.

If you use a buffered (not continuous) acquisistion this can be triggered externally (like an osciloscope). If you know the timestamp of the trigger pulse then you can count on the timestamp (to better than 1ms resolution) being the start time of the acquisition. Even if your two different signals and sources have different starting tmestamps you can still synchronize the data after the acquisitions are complete. Even if the acquisition rates are different you can use and xy graoh to show them together.

I have seen discussions of this business with the absolute timestamp, but I have never done it myself. It is a difficult complication that I have tried hard to avoid.

PCs are bad about timing that is why there is some much function on the DAQ cards themselves. If more than one DAQ card or system is used and close synchronization is needed it takes quite a bit of extra care.

mike

[giopper]

Here I am again,

Yair, Michael, thanks for your replies.

QUOTE (Yair @ Aug 21 2008, 03:07 PM)

The resolution of the get time primitive in Windows is ~16 ms. I'm not sure why, but I don't think you can change that.

...

No way to change that, I know.

I also tried calling directly the Win API GetSystemTime

but I get exactly the same accuracy (~16 ms), I guess it is a Win limitation.

I can tell you that under Fedora 7 I didn't have problems to get ~1 ms accuracy,

same code, same LV functions.

QUOTE (Yair @ Aug 21 2008, 03:07 PM)

If you want to get ms values, you can use the tick count primitive

...

Do you know an easy way to get absolute time from the tick counter?

I did it in the following (definitely not easy) way:

1) in the initialization part of my code

I use a loop to collect absolute timestamps and timer counts,

(hopefully) simultaneously, for a few seconds at a few ms intervals:

it produces two arrays of some-1000 elements, one has the timer ticks,

the other one the corresponding absolute timestamps.

2) I then use the linear fit function to define the linear relationship

between the two arrays. Although the absolute values have 10-20 ms accuracy,

on the average over a few seconds they well represent

the actual absolute time sequence.

3) in the main part of the code, then, I only use the tick counter and

the previously defined linear relatioship to calculate the

absolute time corresponding to timer ticks.

I get absolute timestamps, with ~1 ms accuracy,

very close to be correct, although I found a small offset (<2 ms)

which is constant during execution but changes from one

run to the other ... In principle can be compensated...

QUOTE (Yair @ Aug 21 2008, 03:07 PM)

If you want higher accuracy, you can call the queryperformancecounter API function for a microsecond resolution, but unless you can sync it correctly, I doubt it would help you.

I already wrote some code to use the QPC API, it works great.

I'm going to try it with the same synchronization procedure described above for the LV timer.

QUOTE (mross @ Aug 21 2008, 04:29 PM)

OK. I think you cannot do this without hardware in your PC to get the tracable timestamp.

Then you have to synchronize it with the triggering of the acquisition.

This is not so easy as far as I know.

...

That's the way I did it in the past, with the help of

this PCI module

which needs to be conected to the station GPS clock and uses the

external hardware trigger to latch the event time with 100 ns resolution.

Kind of increases the complexity of the system :thumbdown:

I'd be happy to get rid of it...

QUOTE (mross @ Aug 21 2008, 04:29 PM)

I have seen discussions of this business with the absolute timestamp, but I have never done it myself. It is a difficult complication that I have tried hard to avoid.

you clever guy

I'm going to do some more tests of the synchronized QPC,

then I'll post here some code to show you how it works,

get your feedback and possibly improve it.

Thanks again for your attention,

G

[Yair]

QUOTE (giopper @ Aug 24 2008, 03:47 AM)

Do you know an easy way to get absolute time from the tick counter?

I did it in the following (definitely not easy) way:

The attached example (8.0) has simpler logic, but you should note that it does not handle the U32 rollover. In any case, you can't expect to get ms accuracy in a desktop OS. While the CPU is certainly capable of it, the OS is not designed to guarantee it. If you expect to get synchronization in more than one machine, I think you will have to use some external triggering or even a real time OS.

[giopper]

QUOTE (Yair @ Aug 24 2008, 02:59 PM)

The attached example (8.0) has simpler logic, but you should note that it does not handle the U32 rollover.

...

Hi Yair, thank you very much for your contribution.

Unfortunately, your VI is a bit too simple: :question:

what I need is a time generator with about 1 ms accuracy

AND ALSO <1 ms absolute error from a traceable source.

Your VI is the simplest way to merge

absolute time with accurate timer ticks.

However, we know that the absolute time comes

with a 10-20 ms error, therefore the following time stream,

although produced using the timer (1 ms relative accuracy)

will be affected by that error and the error will change

from one call to another (inside a +/- 15.6 ms interval.)

This makes the time series unreliable (not in sync with

external absolute time sources like GPS time.)

QUOTE (Yair @ Aug 24 2008, 02:59 PM)

In any case, you can't expect to get ms accuracy in a desktop OS.

While the CPU is certainly capable of it, the OS is not designed to guarantee it.

...

I agree with you as long as we only talk about Win32.

I can tell you that under Linux + ntpd server properly running

you can get better than 1 ms simply using the LV built-in time function.

What I actually wonder is if there is any (simple) way

to get 1 ms accurate AND reliable timstamps under Win32

as I get under Linux (from exactly the same hardware!)

I attach my last test VI (8.0) which gives about 1 ms accuracy

absolute time, using the QPC timer and converting it to absolute

time by using an ad-hoc relationship (does not handle the rollover.)

[

Main:

test_QPC_absolute_time.vi

SubVIs:

qpc_timer.vi

qpc_absolute_timer.vi

]

It's not elegant but I think it does the job:

in the end we get accurate+absolute timestamps.

Obviously, it's a starting point and can be improved,

please let me know your opinion and any comment/suggestion.

Thanks,

G

[giopper]

QUOTE (giopper @ Aug 30 2008, 05:50 AM)

...

[

Main:

test_QPC_absolute_time.vi

SubVIs:

qpc_timer.vi

qpc_absolute_timer.vi

]

...

Forgot to attach "processor_speed.vi" subVI

[

Main:

test_QPC_absolute_time.vi

SubVIs:

processor_speed.vi

qpc_timer.vi

qpc_absolute_timer.vi

]

G

[Yair]

I will start by saying that I have zero knowledge about this, so everything below may be wrong, but since you asked:

QUOTE (giopper @ Aug 30 2008, 06:50 AM)

AND ALSO <1 ms absolute error from a traceable source.

I'm not sure if you can do that reliably without dedicated hardware, even on Linux.

QUOTE

However, we know that the absolute time comes with a 10-20 ms error

Maybe YOU know it. I only see that the displayed resolution is ~16 ms. I know nothing about the accuracy (maybe it's synchronized with sub-ms accuracy?).

QUOTE

I can tell you that under Linux + ntpd server properly running you can get better than 1 ms simply using the

LV

built-in time function.

But can you trust it to be synchornized?

Regarding your code, I can't say much, since I didn't dive into it, but I can comment on a couple of things:

The local is unnecessary and causes a race condition. It can be read before its indicator is written to. You can just get rid of it and wire the data straight into the loop.

Building arrays in loops is a big no-no as it requires repeated calls for memory allocation and can hurt performance and statistics, especially when trying to time code.

When you want to distribute code, it's better if you do a save as and create an LLB, as it's then a single file.

[giopper]

Hi Yair,

thanks for your answer

QUOTE (Yair @ Aug 31 2008, 08:27 PM)

I'm not sure if you can do that reliably without dedicated hardware, even on Linux.

...

But can you trust it to be synchornized?

I did it, I can tell you that when the ntpd is properly running

(not only correctly configured but also well in sync with the external

ntp server) the LV built-in time function gives accurate absolute timestamps,

at least accurate enough for my application (better than 1 ms.)

QUOTE (Yair @ Aug 31 2008, 08:27 PM)

...

Maybe YOU know it. I only see that the displayed resolution is ~16 ms. I know nothing about the accuracy (maybe it's synchronized with sub-ms accuracy?).

Ops, you're completely right, I didn't think about it,

the two output (absolue time and timer) could actually be

internally synchronized, I don't know either.

My worry is only that the time series produced using the timer will

be shifted by the amount of the error of the first absolute timestamp.

QUOTE (Yair @ Aug 31 2008, 08:27 PM)

...

The local is unnecessary...

wire the data straight into the loop.

...

Yes, of course, it's only an example in a test program.

I did use the local only to show that the following

main loop (where the new absolute timestamps are generated)

can be completely disconnected from the first part.

It only needs a few numbers that can be passed using a local.

QUOTE (Yair @ Aug 31 2008, 08:27 PM)

Building arrays in loops is a big no-no as it requires repeated calls for memory allocation and can hurt performance and statistics, especially when trying to time code.

I see your point, what do you suggest, then?

QUOTE (Yair @ Aug 31 2008, 08:27 PM)

When you want to distribute code, it's better if you do a save as and create an LLB, as it's then a single file.

Sorry, I am the only direct user of my applications,

I'm not used to distribute my code.

Next time I'll use a LLB.

Anyways, I have some bad news.

I found that there is always a drift between the absolute

time generated by the LV built-in function and the absolute time

generated with my code.

My explanation is that the error in the determination of the parameters

of the linear relationship, although very small, on the long run can

produce a large time error (I got 50 ms over one hour.)

Another explanation is that only the PC clock is drifting

(after all, 50 ms/h = 1.2 s/day only)

but at the moment I have no way to verify this.

In other words, I will probably give up and go back to the

hardware solution, using the IRIG-B code from the local GPS clock.

G

[crelf]

QUOTE (Yair @ Aug 31 2008, 02:27 PM)

When you want to distribute code, it's better if you do a save as and create an LLB, as it's then a single file.

:!: Thread Hijack Warning: I don't use LLBs at all anymore - there's no real use for them IMHO (I'd be happy to discuss, but if you want to hear more, start a new thred and I'll respond there). Better option: zip your files.

[Yair]

QUOTE (giopper @ Sep 1 2008, 12:32 AM)

I see your point, what do you suggest, then?

The common solution is usually to allocate the full size array and then replace elements in it using a shift register. You can see an example implentation of a circular buffer I posted here.

QUOTE

In other words, I will probably give up and go back to the

hardware solution, using the IRIG-B code from the local GPS clock.

Sounds like a reasonable solution to me.

QUOTE (crelf @ Sep 1 2008, 07:05 PM)

I don't use LLBs at all anymore - there's no real use for them

I don't use 8.x (yet), so I don't need lvproj or other files. Using an LLB has the advantage that you can open the file directly from the browser (without needing to use whichever program reads your zip files) and that you can specify a top level VI. IMO, that makes them useful for posting code online. I find both methods to be valid.

[Phillip Brooks]

QUOTE (Yair @ Sep 2 2008, 12:16 PM)

I don't use 8.x (yet), so I don't need lvproj or other files. Using an LLB has the advantage that you can open the file directly from the browser (without needing to use whichever program reads your zip files) and that you can specify a top level VI. IMO, that makes them useful for posting code online. I find both methods to be valid.

You may be sad when you switch to 8.5 / 8.6

LV 8.6 Release Notes, Page 14:

Opening LLBs in LabVIEW

The Enable Windows Explorer for LLB files option on the Environment page of the Options dialog box no longer exists. LabVIEW opens LLBs in the LLB Manager window. Refer to the National Instruments Web site at ni.com/info and enter the info code http://digital.ni.com/public.nsf/websearch/8209AD1184FF4851862572270054427A?OpenDocument' target="_blank">exvfc5 for more information about opening LLBs.

[Yair]

QUOTE (Phillip Brooks @ Sep 2 2008, 11:16 PM)

You may be sad when you switch to 8.5 / 8.6

I've had quite a few opportunities to use all versions of 8.x, so that option does not surprise me. I never used the Windows Explorer option, so the fact that the top level VIs do not open automatically is annoying, but it's still easier than having to deal with the zip file.

If anything will make me sad, it's the fact that 8.x seems to crash a lot more than 7.0 and be slow, especially when using all the new niceties like projects and LVOOP, which I will be. Also, 8.6 broke the way Hebrew works, so that's annoying if you try to create a UI in that language. If anything, that will be the reason for me not to really use it and I'm disappointed I didn't think to check it during the beta.

[Rolf Kalbermatter]

QUOTE (Yair @ Aug 21 2008, 08:07 AM)

The resolution of the get time primitive in Windows is ~16 ms. I'm not sure why, but I don't think you can change that.

The reason is that this time is actually generated and maintained by the good ol' 55 Hz timer tick interrupt counter from ancient DOS times. On startup some memory is initilialized with the time and date from the real time clock and this timer tick then continously maintains the increment of this memory. In regular intervals (not sure but usually once per 24 hours) the memory location is synchronized with the real time clock and/or an external time server (Windows Domain server or internet time server).

Direct access to the real time clock is minimized since it is accessed through IO space and that is an extremely slow operation in comparison to nowadays computer components.

In Windows 3.1 days there was a system.ini setting that allowed to make the system time being updated with the multi media timer instead which could have a 1ms resolution. But on heavy interrupt loaded systems (network traffic, IO boards such as DAQ and Video) this could lead to an overall degradation of the entire system to the point where normal application execution was almost impossible.

Even on nowadays computers and with the default 55 Hz timer tick you usually see a significant drifting of the system time in between synchronization points when heavy interrupts occur such as with intense network traffic or a fast non DMA DAQ operation.

Rolf Kalbermatter

QUOTE (giopper @ Aug 31 2008, 04:32 PM)

My explanation is that the error in the determination of the parameters

of the linear relationship, although very small, on the long run can

produce a large time error (I got 50 ms over one hour.)

Another explanation is that only the PC clock is drifting

(after all, 50 ms/h = 1.2 s/day only)

but at the moment I have no way to verify this.

As said the system time on Windows which is used by LabVIEW for its absilute time is really only synchronized on regular intervals with the real-time clock in the PC or an external time server. For the rest it is a purely interrupt driven software timer tick that can and usually does get some drift from other interrupt sources in the system.

But even the real time clock, despite its name is anything but real time. It is a timer driven by a local quartz based oscillator whose quality is not to bad but definitly nothing to write home about either. 100ppm clock deviation is certainly something that should be already considered rather good in that respect so that would be in the worst case already up to 10 seconds a day. In reality the deviation of that real time clock is actually more in the range of a few seconds a day but it definitly is not real time.

You can however setup Windows to regulary synchronize with an internet time server including NTP and should then get a more accurate timing. Still the default resolution of ~16ms will remain even then, although I thought this was at some point 10ms for NT based OSes.

Rolf Kalbermatter