Realloc in CWI Filter

[R.D. Silverman]

For those who have worked with the CWI filter code.

It is fairly common that the code attempts to make a realloc call
to obtain more space. This call frequently fails.

I have done some poking/prodding into the code, but can't find the
CAUSE of the problem.

I am filtering data for 2,1454L. Filter started up, grew to use about
1.2 Gbytes in size, tried to do a realloc by a factor of roughly 18/14
(28% more) and failed. But the system has 4Gbytes of real memory(!!)
and nothing else was running except system background tasks.

I can see why the realloc should fail. I know others have encountered the
same problem.

One can work around the problem by increasing filtmin, doing a pass,
then decreasing filtmin and doing another pass, but the problem is damn
annoying.....

[Chris Card]

what operating system are you running on?
Chris

[R.D. Silverman]

Originally Posted by Chris Card

what operating system are you running on?
Chris

Windows. Generally XP, but sometimes Win2000 Pro.

[Chris Card]

Originally Posted by R.D. Silverman

Windows. Generally XP, but sometimes Win2000 Pro.

ok.
There's a 2 Gb limit for a user process under windows, and also the numbers you see on Task Manager are not entirely useful.

Memory used by the process can be either Reserved or Committed, and the total of Reserved + Committed must be < 2 Gb. The VM size shown by Task Manager doesn't include all the reserved memory, so can give you a misleading impression that you can allocate some more, whereas in fact you've already used the 2Gb address space.

There are tools available that give you an accurate picture of the memory usage. I found one on the web: http://www.volweb.cz/pvones/delphi/

Download the binaries and run the ToolHelpViewer. Then select your program on the list, right-click and select Memory List. The status line of the window shows the amount of Committed/Reserved memory.

HTH
Chris

[R.D. Silverman]

Originally Posted by Chris Card

ok.
There's a 2 Gb limit for a user process under windows, and also the numbers you see on Task Manager are not entirely useful.

Memory used by the process can be either Reserved or Committed, and the total of Reserved + Committed must be < 2 Gb. The VM size shown by Task Manager doesn't include all the reserved memory, so can give you a misleading impression that you can allocate some more, whereas in fact you've already used the 2Gb address space.

There are tools available that give you an accurate picture of the memory usage. I found one on the web: http://www.volweb.cz/pvones/delphi/

Download the binaries and run the ToolHelpViewer. Then select your program on the list, right-click and select Memory List. The status line of the window shows the amount of Committed/Reserved memory.

HTH
Chris

Thanks for suggesting these tools.....

Unfortunately, security restrictions do not allow installing software
from the Internet.

[xilman]

Originally Posted by R.D. Silverman

For those who have worked with the CWI filter code.

It is fairly common that the code attempts to make a realloc call
to obtain more space. This call frequently fails.

I have done some poking/prodding into the code, but can't find the
CAUSE of the problem.

I am filtering data for 2,1454L. Filter started up, grew to use about
1.2 Gbytes in size, tried to do a realloc by a factor of roughly 18/14
(28% more) and failed. But the system has 4Gbytes of real memory(!!)
and nothing else was running except system background tasks.

I can see why the realloc should fail. I know others have encountered the
same problem.

One can work around the problem by increasing filtmin, doing a pass,
then decreasing filtmin and doing another pass, but the problem is damn
annoying.....

Can you boot your OS with the /3GB flag? (I think that's what it's called, but a google search will find it for you.

The problem you see is that although you have plenty of memory, you do not have a large enough contiguous virtual address space. System DLLs are loaded just above the 2GB address in virtual memory. Rebooting with the /3GB flag gives you another gigabyte of virtual address space.

I learned this approach while still at MSR and running very large filter jobs.

If I understand Wacky correctly, MacOS may have a similar misfeature, even on 64-bit systems.

If the problem only shows itself with mergelevel==1, another trick is to greatly over-specify -maxrelsinp. The initial allocation will then be large enough for a realloc() not to be required . The approach is to choose a ludicrous -maxrelsinp, say 100M, and reduce it progressively until the filter process just fits into the contiguous VM.

Paul

[Wacky]

Originally Posted by xilman

If I understand Wacky correctly, MacOS may have a similar misfeature, even on 64-bit systems.

Yes, I believe that this is the case that was giving me difficulty on MacOSX. Other than the two techniques already mentioned, I have not yet found any additional "work-arounds"

[smh]

Originally Posted by xilman

Can you boot your OS with the /3GB flag? (I think that's what it's called, but a google search will find it for you.

Yes, thats what it's called.

To make it more clear, you have to add this flag to the boot.ini file.

If you have more than 4GB, you'll need to boot with the /PAE switch. This switch is only available on the advanced/enterprise (or higher) editions of Windows 2000 and 2003

See for a complete list of boot.ini options http://www.sysinternals.com/Information/bootini.html

[alpertron]

Please notice that the application must be changed in order to use 3 GB, even when the 3GB switch on BOOT.INI exists. From Microsoft site:

Originally Posted by http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

Executables that can use the 3-GB address space are required to have the bit IMAGE_FILE_LARGE_ADDRESS_AWARE set in their image header. If you are the developer of the executable, you can specify a linker flag (/LARGEADDRESSAWARE).

To set this bit, you must use Microsoft Visual Studio Version 6.0 or later and the Editbin.exe utility, which has the ability to modify the image header (/LARGEADDRESSAWARE) flag. For more information on setting this flag, see the Microsoft Visual Studio documentation.

[dsouza123]

What about using Windows XP Pro x64 (based on Server 2003 x64) ?

Will it allow a 32-bit exe to run with the full 4 GB ?

Googling answered my own question.

http://www.gamepc.com/labs/view_cont...amd64xp&page=3

Keep in mind, 32-bit programs which are running under Windows XP 64-bit
will still be limited to 4GB of memory for each process.

The difference here is that each process can have a dedicated 4GB
of memory address space, and does not have to share space
between the operating system or other software.

This leads to the realization that today's high-end 32-bit applications
have the chance of running smoother on a 64-bit CPU with a 64-bit OS,
even if the software itself isn't 64-bit native code.


This page mentions the /3GB boot parameter
http://www.microsoft.com/windowsxp/u...loringx64.mspx

[Prime95]

How about adding a command line switch that lets you pick the initial amount of memory to malloc?

If realloc cannot grow the allocation in-place it would have to allocate the larger space and copy the data. During that time, you would have both the smaller and larger spaces allocated. I don't know if you are actually running into this problem.