High loads...

[Bjorn Hellqvist]

On Wed, 14 Mar 2007, Russ Allbery wrote:

> Bjorn Hellqvist <bjhell at telia.net> writes:
>
>> I have a real problems with load...
>
>> last pid:  3764;  load averages: 95.78, 29.30, 11.191
>> up 0+00:35:00  16:08:58
>> 284 processes: 36 running, 247 sleeping, 1 lock
>> CPU states:  0.7% user,  2.8% nice, 92.0% system,  1.7% interrupt,  2.8%
>> idle
>> Mem: 127M Active, 1578M Inact, 174M Wired, 53M Cache, 101M Buf, 70M Free
>> Swap: 4096M Total, 4096M Free
>
>> This is all done from one client that abuses the system. (ie: me). This is
>> still a testsetup, so I am the only one that uses this system.
>
>> What I do is "catching up" in a more abusive way. In old and trusted
>> netscape messanger I go into a heavy poplulated group and hold down 'T' to
>> mark threads read. After awhile the loads gets real high.
>
>> Is it something I have missed?
>
> I'm definitely missing something since no matter what you're making your
> single client do, it shouldn't be talking to more than a handful of nnrpd
> processes at most.  Which doesn't explain what 95 processes are all in the
> run queue on your system.  INN doesn't usually spawn that many processes
> due to its architecture unless you have a bunch of readers, which you say
> that you don't.
>
> What does top say is trying to run?

nnrpd for the most parts...

This is the output of news.notice when its at the top.

"Unable to allocate memory for transaction detail"

What memory are reported to be out of? Is it something I can alter in the 
OS? like mbufs or similar low level memory allocations levels that I can 
get upped?

This when it breaks to too high load:
Mar 14 16:07:36 newsread2 nnrpd[2457]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2457]:  times user 0.008 system 0.062 idle 
0.000 elapsed 3.090
Mar 14 16:07:36 newsread2 nnrpd[2491]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2491]:  times user 0.000 system 0.064 idle 
0.000 elapsed 1.932
Mar 14 16:07:36 newsread2 nnrpd[2468]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2468]:  times user 0.005 system 0.060 idle 
0.000 elapsed 3.424
Mar 14 16:07:36 newsread2 nnrpd[2469]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2469]:  times user 0.000 system 0.050 idle 
0.000 elapsed 3.316
Mar 14 16:07:36 newsread2 nnrpd[2510]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2510]:  times user 0.005 system 0.060 idle 
0.000 elapsed 2.274
Mar 14 16:07:36 newsread2 nnrpd[2451]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2458]: load 17.99 > 16
Mar 14 16:07:36 newsread2 nnrpd[2458]:  times user 0.000 system 0.072 idle 
0.000 elapsed 2.306
Mar 14 16:07:36 newsread2 nnrpd[2451]:  times user 0.008 system 0.074 idle 
0.000 elapsed 2.533
Mar 14 16:07:36 newsread2 nnrpd[2452]: load 17.99 > 16

This it at the top load:
Mar 14 16:08:52 newsread2 nnrpd[3572]:  times user 0.008 system 0.057 idle 
0.000 elapsed 11.528
Mar 14 16:08:52 newsread2 nnrpd[3617]: load 101.95 > 16
Mar 14 16:08:52 newsread2 nnrpd[3617]:  times user 0.008 system 0.058 idle 
0.000 elapsed 8.336
Mar 14 16:08:52 newsread2 nnrpd[3674]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:52 newsread2 nnrpd[3674]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:52 newsread2 nnrpd[3686]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:52 newsread2 nnrpd[3686]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:52 newsread2 nnrpd[3710]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:52 newsread2 nnrpd[3710]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:52 newsread2 nnrpd[3704]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:52 newsread2 nnrpd[3704]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3699]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3699]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 innd: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 innd: OVDB: t_add txn_begin: Successful return: 
0
Mar 14 16:08:53 newsread2 innd: SERVER cant store overview for 
@030A42494E533400000000131DC400000008@
Mar 14 16:08:53 newsread2 nnrpd[3614]: load 101.95 > 16
Mar 14 16:08:53 newsread2 nnrpd[3614]:  times user 0.017 system 0.050 idle 
0.000 elapsed 9.146
Mar 14 16:08:53 newsread2 nnrpd[3680]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3680]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3713]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3713]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3717]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3717]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3668]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3668]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3687]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3687]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3718]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3718]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3716]: OVDB: Unable to allocate memory for 
transaction detail
Mar 14 16:08:53 newsread2 nnrpd[3716]: OVDB: t_open_groupinfo txn_begin: 
Successful return: 0
Mar 14 16:08:53 newsread2 nnrpd[3601]: load 101.95 > 16
Mar 14 16:08:53 newsread2 nnrpd[3601]:  times user 0.013 system 0.029 idle 
0.000 elapsed 10.343

Maybe this is not a real problem on real use. I am currently pushing 20-30 
new connections per second to the system. And are doing this for a couple 
of minutes. ( 16:06:35 - 16:09:09 )

But never the less. I dont want a operational system to die out like that 
from just one abusive customer.

/Bjorn