High loads...

[Bjorn Hellqvist]

On Thu, 15 Mar 2007, Bill Davidsen wrote:

> Russ Allbery wrote:
>> Bjorn Hellqvist <bjhell at telia.net> writes:
>>
>>> On Wed, 14 Mar 2007, Russ Allbery wrote:
>>>
>>
>>
>>>> I'm definitely missing something since no matter what you're making
>>>> your single client do, it shouldn't be talking to more than a handful
>>>> of nnrpd processes at most.  Which doesn't explain what 95 processes
>>>> are all in the run queue on your system.  INN doesn't usually spawn
>>>> that many processes due to its architecture unless you have a bunch of
>>>> readers, which you say that you don't.
>>>>
>>
>>
>>>> What does top say is trying to run?
>>>>
>>
>>
>>> nnrpd for the most parts...
>>>
>>
>>
>>> This is the output of news.notice when its at the top.
>>>
>>
>>
>>> "Unable to allocate memory for transaction detail"
>>>
>>
>>
>>> What memory are reported to be out of?
>>>
>>
>> System memory.  You have enough nnrpds running that you probably ran out
>> of swap.  (The reason why your system is so slow and has such a high load
>> is that you ran the system into swap, most likely.)
>>
>>
>>> Maybe this is not a real problem on real use. I am currently pushing
>>> 20-30 new connections per second to the system. And are doing this for a
>>> couple of minutes. ( 16:06:35 - 16:09:09 )
>>>
>>
>> If you're starting 25 new connections every second for about 150 seconds,
>> at the end of that you have 3,750 nnrpd processes running (assuming that
>> you're not closing any of those connections down).  If each one has 1MB of
>> overhead memory required, you need 3.7GB of system memory to be able to
>> run them all plus whatever innd needs.
>>
>> INN can handle that sort of load given appropriate resources, but the
>> resources may be high.
>>
>>
>>> But never the less. I dont want a operational system to die out like
>>> that from just one abusive customer.
>>>
>>
>> So what you really want to do is prevent someone from opening that many
>> connections that quickly, since nearly 4,000 simultaneous users probably
>> doesn't represent your real production load.  For that, either run innd
>> and nnrpd on separate ports (433 is the recommended choice for the port
>> line in inn.conf) and then run nnrpd under xinetd or some other smart
>> server that can do connection throttling, or look into using the -W, -T,
>> and -X flags to innd (set via innflags in inn.conf and documented in the
>> innd man page).  Just using -X 30 or thereabouts would probably keep this
>> problem under control.
>>
>>
> I half-agree with Russ, running on separate ports is the way to go.
> However, rather than letting xinetd start a separate process for each
> connection, I always ran nnrpd as a daemon (-D) when I ran the servers
> at Prodigy/SBC. This appeared to give a lower impact for process
> initiate, since there was just a fork rather than a fork+exec. I don't
> have the figures any more, I was careful not to take anything with me,
> but memory usage was lower in real memory, presumably due to copy on
> write. That was Linux, YMMV.
>
> All the stuff about flags and options is what you need to study, and
> pick values which keep you machine from melting.

Ok, I'll check that out next week when I have more time on my hands.

But isnt forking a waste of resources? wouldnt it be better to do 
something like apache does? fork a couple of nnrpd that then can do 
threading? like only have 20 forks and each fork can thread 100 processes?

A good configured apache will only allow X amount of processes to be 
executed, then it closes down the fork and start a new one.

/Bjorn