Problem in Stunnel with too many connections
Hi Friends, I've just joined this group. I would take this as an oppurtunity to discuss a problem I've been facing in using Stunnel. I've been using Stunnel for quite some time now. It has worked very well all this while. Recently, I had to shift our webserver onto the Internet and Stunnel accepts the https connections on the server's behalf. The problem is there are about 4.5 million clients out there in the field and there's a client coming up to the server atleast every second, if not less. Stunnel accepts the connections and after a while (like an hour) it just dies ! I see that at a point there are about 950 active connections and thats the maximum I could see before it dies. Could someone help me diagnose the problem (or better to fix it) - Is there any performance related issue thats a known problem and I don't know of ? - What I can look at to see what could be the possible cause ? - I haven't had a chance to look at the logs yet. Probably they might tell me something. - Any other pointers / inputs / suggestions would be heartily welcome. I've not been to my office so I won't be able to attach the necessary information like the version of stunnel, the logs, configuration options etc. But promise to send them as soon as I'm back on Monday after Thanksgiving. Thanks and Regards, Kunal
Hi there, What's the maximum number of open file descriptors for your OS? Limits of 1024 is normal - you might be hitting that. Running ulimit -n should allow you to check and set that value. Dan On 26/11/2006, at 14:51 , ~ Kunal Sharma ~ wrote:
Hi Friends,
I've just joined this group. I would take this as an oppurtunity to discuss a problem I've been facing in using Stunnel.
I've been using Stunnel for quite some time now. It has worked very well all this while. Recently, I had to shift our webserver onto the Internet and Stunnel accepts the https connections on the server's behalf.
The problem is there are about 4.5 million clients out there in the field and there's a client coming up to the server atleast every second, if not less. Stunnel accepts the connections and after a while (like an hour) it just dies ! I see that at a point there are about 950 active connections and thats the maximum I could see before it dies.
Could someone help me diagnose the problem (or better to fix it) - Is there any performance related issue thats a known problem and I don't know of ? - What I can look at to see what could be the possible cause ? - I haven't had a chance to look at the logs yet. Probably they might tell me something. - Any other pointers / inputs / suggestions would be heartily welcome.
I've not been to my office so I won't be able to attach the necessary information like the version of stunnel, the logs, configuration options etc. But promise to send them as soon as I'm back on Monday after Thanksgiving.
Thanks and Regards, Kunal
_______________________________________________ stunnel-users mailing list stunnel-users@mirt.net http://stunnel.mirt.net/mailman/listinfo/stunnel-users
--- Dan Milne d@nmilne.com http://politiwiki.kicks-ass.net/
Hi Dan, Thanks for your input. Here's some more info - - I'm using Stunnel version 4.16 on Windows Server 2003 Standard. I downloaded the binaries. - I had logs at level 7 ON but they're HUGE so I can't attach them here. What I can tell is that when the server exited, there was no message logged to suggest that there was a problem. The server was just going through the negotiations with a client when it died. - Log at startup time says "No limit detected for number of clients" . - There are messages that say "Connection rejected: create_client failed" and "readsocket: Connection reset by peer (WSAECONNRESET) (10054)" but i don't think they're doing anything to shut the server down. I have not been able to locate where to increase the no. of max FDs on windows. Please let me know if you do. I'm going through the Stunnel source code. I came across the function where limits are set. What I could derive is that for a windows based system there is no limit set to FDs and clients (function get-limits in stunnel.c). Does it mean that limits are governed by Windows settings in this case ? Please respond back with whatever you can help !! Thanks n Regards, Kunal On 11/26/06, Dan Milne <d@nmilne.com> wrote:
Hi there, What's the maximum number of open file descriptors for your OS? Limits of 1024 is normal - you might be hitting that.
Running ulimit -n should allow you to check and set that value.
Dan
On 26/11/2006, at 14:51 , ~ Kunal Sharma ~ wrote:
Hi Friends,
I've just joined this group. I would take this as an oppurtunity to discuss a problem I've been facing in using Stunnel.
I've been using Stunnel for quite some time now. It has worked very well all this while. Recently, I had to shift our webserver onto the Internet and Stunnel accepts the https connections on the server's behalf.
The problem is there are about 4.5 million clients out there in the field and there's a client coming up to the server atleast every second, if not less. Stunnel accepts the connections and after a while (like an hour) it just dies ! I see that at a point there are about 950 active connections and thats the maximum I could see before it dies.
Could someone help me diagnose the problem (or better to fix it) - Is there any performance related issue thats a known problem and I don't know of ? - What I can look at to see what could be the possible cause ? - I haven't had a chance to look at the logs yet. Probably they might tell me something. - Any other pointers / inputs / suggestions would be heartily welcome.
I've not been to my office so I won't be able to attach the necessary information like the version of stunnel, the logs, configuration options etc. But promise to send them as soon as I'm back on Monday after Thanksgiving.
Thanks and Regards, Kunal
_______________________________________________ stunnel-users mailing list stunnel-users@mirt.net http://stunnel.mirt.net/mailman/listinfo/stunnel-users
--- Dan Milne d@nmilne.com http://politiwiki.kicks-ass.net/
_______________________________________________ stunnel-users mailing list stunnel-users@mirt.net http://stunnel.mirt.net/mailman/listinfo/stunnel-users
On Tuesday 28 November 2006 16:24, ~ Kunal Sharma ~ wrote:
- I'm using Stunnel version 4.16 on Windows Server 2003 Standard.
The general rule is *not* to report problem with an old release of software. Next time upgrade first, and then check if the problem still exists.
- There are messages that say "Connection rejected: create_client failed" and "readsocket: Connection reset by peer (WSAECONNRESET) (10054)" but i don't think they're doing anything to shut the server down.
Not the server. The *connection*. The error mean that _beginthread() function failed = your Windows Server 2003 Standard was unable to create more threads. There's possibly some internal limit in Windows Server 2003 Standard for the number of concurrent threads.
I have not been able to locate where to increase the no. of max FDs on windows. Please let me know if you do.
It's not the problem with file descriptors. Best regards, Mike
Hi Mike, Thanks for your thoughts ! I'm using 4.16 and I believe thats the latest version. Atleast I see its the latest one vailable on the download page of the website. Or is there some other latest version I know not of ?
- There are messages that say "Connection rejected: create_client failed" and "readsocket: Connection reset by peer (WSAECONNRESET) (10054)" but i don't think they're doing anything to shut the server down.
When I said *server*, I meant the Stunnel application. Sorry for the goofup. Even after these messages the Stunnel continues to run and process new connections. The latest update on the problem is - The m/c has not been shutdown for 6 days now. Now when I run Stunnel it closes down in 15 mins. The "time-to-die" for Stunnel has steadily decreased over the past few days from 1 hour and now its 15 mins. Still looking for the setting to UP the mximum number of concurrent threads on Windows :-( Thanks n Regards, Kunal
On Tuesday 28 November 2006 21:02, ~ Kunal Sharma ~ wrote:
I'm using 4.16 and I believe thats the latest version. Atleast I see its the latest one vailable on the download page of the website. Or is there some other latest version I know not of ?
Are we talking about *the home site*? http://stunnel.mirt.net/
The latest update on the problem is - The m/c has not been shutdown for 6 days now. Now when I run Stunnel it closes down in 15 mins. The "time-to-die" for Stunnel has steadily decreased over the past few days from 1 hour and now its 15 mins.
For a web server I'd try something like: TIMEOUTidle = 30
Still looking for the setting to UP the mximum number of concurrent threads on Windows :-(
I don't think it's a configurable parameter. Best regards, Mike
Hi Mike, I downloaded Stunnel from stunnel.org and had no idea I have a new version at *the home site*. But with the same version, I made the change suggested by you (TIMEOUTidle = 30) and Stunnel has now being working for more than 18 hrs on the trot now !!! Can you please explain to me (only if you have time) what magic this did ? Thanks n Regards, Kunal On 11/28/06, Michal Trojnara <Michal.Trojnara@mobi-com.net> wrote:
On Tuesday 28 November 2006 21:02, ~ Kunal Sharma ~ wrote:
I'm using 4.16 and I believe thats the latest version. Atleast I see its the latest one vailable on the download page of the website. Or is there some other latest version I know not of ?
Are we talking about *the home site*? http://stunnel.mirt.net/
The latest update on the problem is - The m/c has not been shutdown for 6 days now. Now when I run Stunnel it closes down in 15 mins. The "time-to-die" for Stunnel has steadily decreased over the past few days from 1 hour and now its 15 mins.
For a web server I'd try something like: TIMEOUTidle = 30
Still looking for the setting to UP the mximum number of concurrent threads on Windows :-(
I don't think it's a configurable parameter.
Best regards, Mike
_______________________________________________ stunnel-users mailing list stunnel-users@mirt.net http://stunnel.mirt.net/mailman/listinfo/stunnel-users
On Wednesday 29 November 2006 16:56, ~ Kunal Sharma ~ wrote:
I downloaded Stunnel from stunnel.org and had no idea I have a new version at *the home site*.
I see. This page should put some light on it: http://www.stunnel.org/related/ Brian Hatch, the author of www.stunnel.org web page did a great job building the page, writing documentation, patches, etc. Unfortunately he is quite busy nowadays, so the page is becoming obsolete and misleading.
But with the same version, I made the change suggested by you (TIMEOUTidle = 30) and Stunnel has now being working for more than 18 hrs on the trot now !!!
Can you please explain to me (only if you have time) what magic this did ?
By default stunnel tries to keep idle (not transferring any traffic) connections up for 43200 seconds (12 hours). It's generally a good idea (imagine a telnet or an irc session). The drawback is that when a client has disconnected without shutting down or resetting TCP session (like it was turned off with the power switch or the the network cable was pulled off) stunnel server uses server resources (like a cpu thread, memory or tcp sockets) for the next 12 hours. That's not good on a heavy loaded server. Reducing the idle timeout from 43200 to 30 seconds eliminates this problem. Best regards, Mike
Around about 2006-11-29 21:16 +0100, Michal Trojnara offered:
On Wednesday 29 November 2006 16:56, ~ Kunal Sharma ~ wrote:
I downloaded Stunnel from stunnel.org and had no idea I have a new version at *the home site*.
I see. This page should put some light on it: http://www.stunnel.org/related/
Brian Hatch, the author of www.stunnel.org web page did a great job building the page, writing documentation, patches, etc. Unfortunately he is quite busy nowadays, so the page is becoming obsolete and misleading.
Actually, the problem was different. The website was hosted on a box behind a firewall, and the firewall maintainer (not me) blocked off port 22, so the automatic rsyncs were failing, and I hadn't noticed. It should be up to speed again now, including 4.20 which just came out. The remaining points Mike makes are, sadly, true. -- Brian Hatch "Sorry, sorry, Systems and sorry, much apologizings." Security Engineer http://www.ifokr.org/bri/ Every message PGP signed
participants (4)
-
Brian Hatch -
Dan Milne -
Michal Trojnara -
~ Kunal Sharma ~