Inconsistent performance across stunnel and/or OpenSSL versions
Hi All, I am experimenting with stunnel inside a VM on Xenserver 6.02. So this may not be a "common" use case, nonetheless, I would like to point out the issue. It is a basic setup as follows with 3 VMs running "ab" for webclient, stunnel+haproxy combo as the https/http bridge and nginx as the webserver. client VM (ab) <-> stunnel + haproxy VM <-> webserver VM (nginx) Specs on the stunnel VM are 2GB RAM, single-core dedicated 2.5 GHz Xeon E5-2640 vCPU, running Debian Squeeze 2.6.32-5-amd64 (64 bit) kernel. All things unchanged such as stunnel/haproxy/nginx config and taking care of common gotchas such as file descriptor limit, TIMEOUTclose, disabling libwrap, etc. stunnel is built with pthreads. Also stunnel is configured to use a 2K RSA self-signed certificate, and the client is being forced to do TLS1. "ab" is invoked with "-c 4" so as to peg the stunnel VM CPU (any value beyond 4 doesn't seem to matter). nginx serves a tiny 32 byte static html file. With stunnel 4.29 and OpenSSL 0.9.8o, I get 300 requests per second as max throughput. With stunnel 4.56 and OpenSSL 1.0.1e, I get only 40 requests per second as max throughput. tcpdump tells me that the Server Hello sent to the client takes a longer time in the latter case. Tinkering with TCP_NODELAY doesn't change anything. Anyone else seeing this maybe even on physical hardware? Has anything changed in OpenSSL across those versions? Suggestions? Thanks.
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
Hi All,
I am experimenting with stunnel inside a VM on Xenserver 6.02. So this may not be a "common" use case, nonetheless, I would like to point out the issue.
It is a basic setup as follows with 3 VMs running "ab" for webclient, stunnel+haproxy combo as the https/http bridge and nginx as the webserver.
client VM (ab) <-> stunnel + haproxy VM <-> webserver VM (nginx)
Specs on the stunnel VM are 2GB RAM, single-core dedicated 2.5 GHz Xeon E5-2640 vCPU, running Debian Squeeze 2.6.32-5-amd64 (64 bit) kernel.
All things unchanged such as stunnel/haproxy/nginx config and taking care of common gotchas such as file descriptor limit, TIMEOUTclose, disabling libwrap, etc. stunnel is built with pthreads. Also stunnel is configured to use a 2K RSA self-signed certificate, and the client is being forced to do TLS1. "ab" is invoked with "-c 4" so as to peg the stunnel VM CPU (any value beyond 4 doesn't seem to matter). nginx serves a tiny 32 byte static html file.
With stunnel 4.29 and OpenSSL 0.9.8o, I get 300 requests per second as max throughput.
With stunnel 4.56 and OpenSSL 1.0.1e, I get only 40 requests per second as max throughput.
tcpdump tells me that the Server Hello sent to the client takes a longer time in the latter case. Tinkering with TCP_NODELAY doesn't change anything.
Anyone else seeing this maybe even on physical hardware? Has anything changed in OpenSSL across those versions? Suggestions?
That is a strange question;) A lot has changed between OpenSSL 0.9.8 and 1.0.1 and I mean really a _lot_. The single most important change is support for TLS 1.1 and 1.2 with additional cipher modes. Not to mention countless other changes (including performance improvements). If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher. -- Janusz Dziemidowicz
On Wed, Apr 17, 2013 at 12:23 PM, Janusz Dziemidowicz <rraptorr@nails.eu.org
wrote:
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher.
Hi Janusz, As per your suggestions and mea culpa in some stated results. Here is a hopefully complete/better matrix. Making sure that CPU is pegged at 100% and in stunnel.conf (sslVersion = TLSv1) stunnel 4.29, OpenSSL 0.9.8o - ~300 requests per sec stunnel 4.29, OpenSSL 1.0.1e - ~360 requests per sec stunnel 4.56, OpenSSL 0.9.8o - ~100 requests per sec stunnel 4.56, OpenSSL 1.0.1e - ~120 requests per sec Regards.
Another data point after a binary search across versions keeping OpenSSL version identical at 1.0.1e I see this performance regression between stunnel versions 4.39 and 4.40. Regards. On Wed, Apr 17, 2013 at 4:46 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
On Wed, Apr 17, 2013 at 12:23 PM, Janusz Dziemidowicz < rraptorr@nails.eu.org> wrote:
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher.
Hi Janusz,
As per your suggestions and mea culpa in some stated results. Here is a hopefully complete/better matrix. Making sure that CPU is pegged at 100% and in stunnel.conf (sslVersion = TLSv1)
stunnel 4.29, OpenSSL 0.9.8o - ~300 requests per sec stunnel 4.29, OpenSSL 1.0.1e - ~360 requests per sec stunnel 4.56, OpenSSL 0.9.8o - ~100 requests per sec stunnel 4.56, OpenSSL 1.0.1e - ~120 requests per sec
Regards.
It appears including static DH params in the certificate brings the performance back up in 4.40 and onward. Would like to mark this RESOLVED. Regards. On Wed, Apr 17, 2013 at 11:29 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
Another data point after a binary search across versions keeping OpenSSL version identical at 1.0.1e
I see this performance regression between stunnel versions 4.39 and 4.40.
Regards.
On Wed, Apr 17, 2013 at 4:46 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
On Wed, Apr 17, 2013 at 12:23 PM, Janusz Dziemidowicz < rraptorr@nails.eu.org> wrote:
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher.
Hi Janusz,
As per your suggestions and mea culpa in some stated results. Here is a hopefully complete/better matrix. Making sure that CPU is pegged at 100% and in stunnel.conf (sslVersion = TLSv1)
stunnel 4.29, OpenSSL 0.9.8o - ~300 requests per sec stunnel 4.29, OpenSSL 1.0.1e - ~360 requests per sec stunnel 4.56, OpenSSL 0.9.8o - ~100 requests per sec stunnel 4.56, OpenSSL 1.0.1e - ~120 requests per sec
Regards.
2013/4/18 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
It appears including static DH params in the certificate brings the performance back up in 4.40 and onward.
If you need the best performance, then disabling DH key exchange might be a good idea as it is quite time consuming. If you still need forward secrecy then ECDH should be a lot better, especially with OpenSSL 1.0.1. In all cases, it might be worth looking at Google SSL configuration, it is configured for quite a lot of traffic:) https://www.ssllabs.com/ssltest/analyze.html?d=google.com -- Janusz Dziemidowicz
Hi PPingPongBaker, Could you repeat your tests with: ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:!DH:-MEDIUM:RC4:+HIGH and ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:!DH:!ECDH:-MEDIUM:RC4:+HIGH ? It might be interesting to see the performance with DH (and possibly also ECDH) ciphersuites completely disabled. TIA, Mike On 2013-04-18 21:02, PPingPongBaker PPingPongBaker wrote:
It appears including static DH params in the certificate brings the performance back up in 4.40 and onward.
Would like to mark this RESOLVED.
Regards.
On Wed, Apr 17, 2013 at 11:29 PM, PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com <mailto:ppingpongbaker@gmail.com>> wrote:
Another data point after a binary search across versions keeping OpenSSL version identical at 1.0.1e
I see this performance regression between stunnel versions 4.39 and 4.40.
Regards.
On Wed, Apr 17, 2013 at 4:46 PM, PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com <mailto:ppingpongbaker@gmail.com>> wrote:
On Wed, Apr 17, 2013 at 12:23 PM, Janusz Dziemidowicz <rraptorr@nails.eu.org <mailto:rraptorr@nails.eu.org>> wrote:
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com <mailto:ppingpongbaker@gmail.com>>:
If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher.
Hi Janusz,
As per your suggestions and mea culpa in some stated results. Here is a hopefully complete/better matrix. Making sure that CPU is pegged at 100% and in stunnel.conf (sslVersion = TLSv1)
stunnel 4.29, OpenSSL 0.9.8o - ~300 requests per sec stunnel 4.29, OpenSSL 1.0.1e - ~360 requests per sec stunnel 4.56, OpenSSL 0.9.8o - ~100 requests per sec stunnel 4.56, OpenSSL 1.0.1e - ~120 requests per sec
Regards.
_______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
On 19/04/13 07:02, PPingPongBaker PPingPongBaker wrote:
It appears including static DH params in the certificate brings the performance back up in 4.40 and onward.
Does this mean stunnel dynamically generates DH keys if the "openssl dhparam 2048" trick mentioned in the man page isn't done - and that causes an initial pause that impacts the overall throughput? Would that be once at startup, or per connection? I have seen some SSL apps where there is (say) an hourly/daily cronjob that generates new DH keys into a file, and the app uses that instead of doing it dynamically - very similar to the append operation mentioned in the man page. Actually, given how CPU intensive generating a 2048bit DH key is, what is the *real* downside of having a static DH key? Sounds to me there is effectively no downside and some upside for doing that by default? Maybe at least the package maintainers of stunnel (eg for Redhat/Debian) should do that as part of their installation process? I'm sure we're all for better performance if there's no real security downside Jason PS: DH is two-way - so what is the client doing? eg if this was a web browser going to mod_ssl on apache, does the client sit there thinking hard - generating it's DH key? Surely both ends would need to move to a "static" key model to get the performance improvement? PPS: crypto isn't my strongest area, so forgive my naive questions ;-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
* > **On Fri Apr 19 17:10:31 CEST 2013, **Michal Trojnara* Michal.Trojnara at mirt.net <stunnel-users%40stunnel.org?Subject=Re%3A%20%5Bstunnel-users%5D%20Inconsistent%20performance%20across%20stunnel%20and/or%0A%20OpenSSL%20versions&In-Reply-To=%3C51715E67.1000701%40mirt.net%3E> wrote:
Hi PPingPongBaker,
Could you repeat your tests with: ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:!DH:-MEDIUM:RC4:+HIGH and ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:!DH:!ECDH:-MEDIUM:RC4:+HIGH ?
It might be interesting to see the performance with DH (and possibly also ECDH) ciphersuites completely disabled.
Hi Mike, The best compilation of results on this topic that I have seen and agree with are at [1] DHE modular exponentiation really hurts SSL performance; no wonder Google resorted to ECDHE. [1] http://vincent.bernat.im/en/blog/2011-ssl-perfect-forward-secrecy.html On Thu, Apr 18, 2013 at 12:02 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
It appears including static DH params in the certificate brings the performance back up in 4.40 and onward.
Would like to mark this RESOLVED.
Regards.
On Wed, Apr 17, 2013 at 11:29 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
Another data point after a binary search across versions keeping OpenSSL version identical at 1.0.1e
I see this performance regression between stunnel versions 4.39 and 4.40.
Regards.
On Wed, Apr 17, 2013 at 4:46 PM, PPingPongBaker PPingPongBaker < ppingpongbaker@gmail.com> wrote:
On Wed, Apr 17, 2013 at 12:23 PM, Janusz Dziemidowicz < rraptorr@nails.eu.org> wrote:
2013/4/17 PPingPongBaker PPingPongBaker <ppingpongbaker@gmail.com>:
If you want to compare various stunnel versions, then use the same OpenSSL version. If you want to compare OpenSSL... then use the same stunnel version. The configuration you mentioned above doesn't make a lot of sense as it makes it hard to tell where the performance drop comes from. If you really must test such configuration, the best way would be to ensure the same TLS version (1.0, not 1.1 or 1.2, OpenSSL 1.0.1 defaults to 1.2) and the same cipher.
Hi Janusz,
As per your suggestions and mea culpa in some stated results. Here is a hopefully complete/better matrix. Making sure that CPU is pegged at 100% and in stunnel.conf (sslVersion = TLSv1)
stunnel 4.29, OpenSSL 0.9.8o - ~300 requests per sec stunnel 4.29, OpenSSL 1.0.1e - ~360 requests per sec stunnel 4.56, OpenSSL 0.9.8o - ~100 requests per sec stunnel 4.56, OpenSSL 1.0.1e - ~120 requests per sec
Regards.
On 2013-04-23 04:19, PPingPongBaker PPingPongBaker wrote:
The best compilation of results on this topic that I have seen and agree with are at [1] DHE modular exponentiation really hurts SSL performance; no wonder Google resorted to ECDHE. [1] http://vincent.bernat.im/en/blog/2011-ssl-perfect-forward-secrecy.html I'm well aware of this. Stunnel server should only negotiate DH if ECDH is not supported by connecting client. Can you confirm that this is the case for the client you used for your tests?
The reason I asked you to compare your results with various ciphers was to find reasonable defaults for the upcoming stunnel 5.00. 8-) Mike
On *Tue Apr 23 22:23:54 CEST 2013, Michal Trojnara Michal.Trojnara at mirt.net <stunnel-users%40stunnel.org?Subject=Re%3A%20%5Bstunnel-users%5D%20Inconsistent%20performance%20across%20stunnel%20and/or%0A%20OpenSSL%20versions&In-Reply-To=%3C51715E67.1000701%40mirt.net%3E> wrote:*
Can you confirm that this is the case for the client you used for your tests?
Yes, my client doesn't support ECDHE and DHE was negotiated (as confirmed by tcpdump as well).
participants (4)
-
Janusz Dziemidowicz -
Jason Haar -
Michal Trojnara -
PPingPongBaker PPingPongBaker