Re: es memory requirements

[Donn Cave <donn@u.washington.edu>]

Date: 	Tue, 5 Jan 1993 16:54:25 -0500
From: Paul Haahr <haahr@mv.us.adobe.com>
Subject: Re: es memory requirements

|> I brought up es "in production" today on a fairly large system, and the
|> good news is that it seems to be working fine - have not heard one complaint
|> about anything breaking.  The application in question runs under 400 lines
|> of es code, so we're not talking about a mammoth program.
|
| still, it's the largest extant es program.  i'm glad to hear that it's
| working.
|
| can you say anything about speed relative to rc?  i would expect
| performance to be comparable.

I believe it is.  The OS is DYNIX/ptx (Sequent), which has a "ptime"
program that gets individual statistics for all the child process
hierarchy, so I was able to compare es versus rc in principle as well
as es versus rc in practice (I was able to eliminate some external programs
when I switched to es.)  My recollection is not very specific, but for
the small amount of CPU time consumed es was indeed comparable.

|> The bad news is that it's using 3 to 4 times as much memory as rc did, to
|> do pretty much the same thing.  That puts it almost in a class with perl.
|> Most of the memory is allocated at run time - the compiled binary sizes are
|> pretty comparable with rc.
|
| we haven't yet profiled memory usage in es, and don't even have real
| mechanisms at our fingertips for giving back info on es's dynamic memory
| usage patterns.  my guess is that while we allocate a fair amount of memory,
| working sets are not much  larger than working sets in rc.  more data is
| necessary.  (we should probably add an extra GCINFO compile-time option for
| reporting amounts of live data.)

Total size and resident set size differences were similar - es was larger by
3 or 4 times.

|> I'm probably going to have to pull the es code and restore rc, my colleagues
|> here are getting pretty uptight over the memory (we apparently had some
|> low on memory problems this morning at only around 300 users, and we expect
|> more like 500 when things pick up.)
|
| does sound bad.  when you say low on memory, do you mean low on swap space
| or do  you mean lots of thrashing?  the former is not that surprising
| (300-500 users!!!), the latter is more disturbing.

Well, there were some disturbed guys in my office yesterday.  Fortunately we
can pull some memory from another system that's on its way out - the problem
is the amount of real memory, not swap space.  We may be suffering from poor
swapping and paging algorithms, which would be kind of typical from what I've
seen, but the basic problem is that we suddenly had hundreds of these
processes whose resident set size had increased dramatically.

|> What I'm wondering, since I faintly recall that es uses garbage collection,
|> would it be possible to influence the collection algorithm?  The application
|> tends to create a lot of functions and lists and then delete them all at once,
|> and I'm thinking it might be possible to do something, along with deleting,
|> that would allow the space to be reclaimed for the next batch of functions.  
|
| deleting them does free them up, the next time a collection is done.  the  
| allocator requires free memory to be contiguous (it's a copying collector), so
| knowing that some things are free already really doesn't help.

When would you expect "the next time a collection is done" to happen?  I'm
thinking perhaps, if I could induce a collection at certain points, things
might improve.  As can be the case with any shell, es spends long periods
dormant while the user is running a "real" application, and if I could get
it to compact a little first it would be a great help.

	Donn Cave, University Computing Services, University of Washington
	donn@cac.washington.edu