Re: zmailer memory bloat

[urlichs@smurf.noris.de (Matthias Urlichs)]

In dist.zmailer, article <199404291634.AA11625@deshaw.com>,
  Mark Moraes <Mark-Moraes@deshaw.com> writes:
> 
> 2. My malloc tries to re-coalesce adjacent blocks on free and allocate
> first-fit, a strategy that seems to be contrary to zmailer internal
> allocator.  (there's a couple more layers of allocator layered on top of
> malloc inside ZMailer).  The Caltech malloc (comes with 4.3BSD, Perl;
> possibly rn and emacs) neither coalesces nor splits large blocks, which

Also INN, which is where I last saw that particular problem.
Memory bloat and performance drop under the Ultrix malloc; no problem and
spiffy operation under the Caltech malloc.

> results in less fragementation in certain situations.  Try recompiling
> with it (or -lmalloc) and see if that helps.  If a different malloc
> helps, I'd still like to get some traces from my malloc to see if I can
> fix this problem without introducing others.

This is not exactly ZMailer-specific, but...

Mallocs which walk memory, trying to be "optimal" in some sense, are a
waste of time (IMHO) because the additional memory references cause
additional paging activity which cause the system to be slower and actually
use more real memory.

The Caltech malloc, on the other hand, is optimal in the sense that it
needs to access exactly one memory location (and possibly one call to
sbrk()) to satisfy a memory request, not counting the allocated memory.
Ditto for freeing.

Actual experience with INN, from memory:
Caltech malloc: 13M allocated, 7.5M in-core (stable)
Normal malloc:  10M allocated, 9M   in-core (after one hour)
Normal malloc:  25M allocated, 15M+ in-core (after three days; heavy paging
            caused almost entirely by malloc() and free() -- yes, I checked)

_Any_ allocation strategy which splits free blocks (first/best/last/whatever-
fit) to service a request can be subverted, no matter what. The Caltech
malloc doesn't, so it's immune to these effects.

The only disadvantage to the Caltech malloc is that because it allocates in
powers of two, the average overhead and thus the amount of wasted space in
swap is 25%. This can be changed, eg. by switching to a Fibonacci sequence,
rather easily.

On my systems, the Caltech malloc is the one I'm using in the C library.
-- 
Matthias Urlichs        \ XLink-POP Nürnberg  | EMail: urlichs@smurf.noris.de
Schleiermacherstraße 12  \  Unix+Linux+Mac    | Phone: ...please use email.
90491 Nürnberg (Germany)  \   Consulting+Networking+Programming+etc'ing     42

Click <A HREF="http://smurf.noris.de/~urlichs/finger">here</A>.