allocating extremely large arrays

[psychomanu]

Hi,

we have an application that needs an extremely large array of singles in memory (array sizes up to 1.6 GB, preferably more, size varies)

With LabVIEW we can fill the memory only up to 1 GB (appr. 750 MB array, rest = OS) even though 4 GB RAM is present.

With C we can go up to the Windows defined limit of 2 GB (appr. 1.8 GB array, rest = OS), so RAM fragmentation is not the cause of the 1 GB limit.

Question 1 : does anyone know why the 1 GB limit exists in LabVIEW?

Question 2 : does anyone know a way around this

Question 3 : could we do the allocation in C and then let LabVIEW access this memory space?

Thanks in advance,

Manu.

[Rolf Kalbermatter]

Hi,

we have an application that needs an extremely large array of singles in memory (array sizes up to 1.6 GB, preferably more, size varies)

With LabVIEW we can fill the memory only up to 1 GB (appr. 750 MB array, rest = OS) even though 4 GB RAM is present.

With C we can go up to the Windows defined limit of 2 GB (appr. 1.8 GB array, rest = OS), so RAM fragmentation is not the cause of the 1 GB limit.

Question 1 : does anyone know why the 1 GB limit exists in LabVIEW

Question 2 : does anyone know a way around this

Question 3 : could we do the allocation in C and then let LabVIEW access this memory space?

Thanks in advance,

Manu.

4581[/snapback]

1) I don't think LabVIEW has a hard 1GB limit. But it has its own memory manager layer above the OS memory functions and works with two so called memory zones from which it allocates memory. The DS (data space) zone is the memory LabVIEW uses for all diagram data and the AZ (application zone) is used for internal structures and variables. What is most probably the problem here is that the available memory is split into these two zones at startup and when you try to create your array LabVIEWs memory manager can't find a big enough block of free continous memory in the DS heap for this array.

2) There is probably not much you can do to influence the way LabVIEW manages its internal memory allocation.

3) This is a possibilty but you can't allocate memory in a DLL using standard OS memory allocaters and hand that memory to LabVIEW to work on it as if it would be its own data. LabVIEW can only work with memory allocated through its DSNewPtr/Handle functions. What you could do though is to not only implement the allocation and deallocation in your external DLL but also some accessor fucntions. Still what you want to do is basically at the outmost limit of what Windows can allow any application to do, and that assumes a machine which is not doing anything else, does not have hundreds of services running, no background tasks, a lean OS as much as possible (no unnecessary drivers and such) and last but not least a very detailed control of your memory, something LabVIEW is not really giving you easily.

Even if your application would be entirely written in C and you would be an expert in memory handling you would basically scratch the absolute limits of 32bit Windows.

Under 32bit Linux you would have a little more leeway as there you can configure the kernel to allow an application to have control over up to 3GB memory, but not every application can handle that (Those that assume 32bit integers for their address offsets instead of unsigned integers and I have reasons to believe that LabVIEW might trip over this too).

I do think that the 1.8 GB limit you see in your C application is actually not limited by the OS itself. The OS is mapped into the the upper 2GB address space for an application, if I remember correctly. But your C application also needs some space for management and such of itself.

Rolf Kalbermatter

[psychomanu]

Thanks for this elaborate answer.

We realise we are at the edge, but we had hoped to be able to use at least the 2 GB. With a /3GB flag in your boot.ini you can tell Windows to allow a single application to grab up to 3 GB (we use a third party program that actually does this), but with LabVIEW we still get stuck around 1 GB.

I guess we'll have to work in chunks of about 700 MB. That said, their is a function described in the code interface reference manual, called DSMaxMem (and AZMaxMem) that is supposed to return "the largest contiguous block of memory allocatable". However, it always returns a zero (we made no error in datatype or something). Do you happen to know why, or maybe another way of finding the largest possible memory block (without resorting to virtual memory which is way too slow for what we are doing)?

Thanks again,

Manu.

[didierj]

Do you happen to know why, or maybe another way of finding the largest possible memory block (without resorting to virtual memory which is way too slow for what we are doing)?

4599[/snapback]

As you have a computer that has plenty of memory, and if you use this computer just for this (and similar) application, don't bother about the virtual memory and just disable it. Windows and Linux still work smoothly without virtual memory. The only thing you have to take care of, if your os get's to the memory limit, your app that want's to allocate further memory might crash.

Didier

[Rolf Kalbermatter]

That said, their is a function described in the code interface reference manual, called DSMaxMem (and AZMaxMem) that is supposed to return "the largest contiguous block of memory allocatable". However, it always returns a zero (we made no error in datatype or something).

4599[/snapback]

As far as I can say, these functions do absolutely nothing in current LabVIEW versions. I'm not sure if they are remainders of the old style memory manager in Windows 3.1 or if they were plans to add functionality which never made it into the code. Basically many of the memory management ideas in LabVIEW are not so interesting with modern OSes but were absolutely mandatory when LabVIEW was running on old MacOS and 32bit Windows DOS extenders for 16bit Windows 3.1.

While some of this got removed later on as support for those platforms was dropped the fundamental architecture couldn't be changed withoug breaking lots and lots of already existing applications.

Rolf Kalbermatter