Possible remote exploit in TLS when using Bouncy Castle provider

David Hook

2014-10-15 23:37:53 UTC

I'd suspect a dud key of the same size could also be used to trigger
something like this. Are you saying a handshake failure caused by a
RuntimeException is enough to bring the server down in this case?

Reviewing the code for engineDoFinal I think we should probably be
throwing IllegalBlockSizeException on engineDoFinal, although
realistically ArrayIndexOutOfBoundsException is also acceptable (it's
the only way to signal something is wrong on engineUpdate, so using RSA,
or any cipher for that matter, without being able to handle a
RuntimeException isn't a good idea in general). The server code should
be able to handle a RuntimeException on a handshake - a malicious client
is at liberty to send anything, and the source for
Handshaker.checkThrown shows that any RuntimeExceptions are rethrown as
such.

Regards,

David

Post by Jason Resch
Hello,
I encountered an interesting stack trace which started when we began
to install BouncyCastleProvider as the highest priority provider. To
reproduce this required pairing a 1024-bit private key together with a
larger 2048-bit public key and using a TLS cipher suite that uses RSA
2014-10-11 09:54:31.430 WARN [New I/O worker #4]
org.cleversafe.protocol.acceptor.netty.AcceptorHandler - Network
Could not generate dummy secret
java.lang.RuntimeException: Could not generate dummy secret
at
sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1287)
~[?:1.7.0_55]
at
sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
~[?:1.7.0_55]
at
sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:793)
~[?:1.7.0_55]
at
sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:761)
~[?:1.7.0_55]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
~[?:1.7.0_55]
at
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1229)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:914)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.7.0.Final.cs.1.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) [?:1.7.0_55]
Caused by: java.lang.RuntimeException: Could not generate dummy secret
at
sun.security.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:163)
~[?:1.7.0_55]
at
sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:190)
~[?:1.7.0_55]
at
sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) ~[?:1.7.0_55]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:808) ~[?:1.7.0_55]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:806) ~[?:1.7.0_55]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_55]
at
sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
~[?:1.7.0_55]
at
org.jboss.netty.handler.ssl.SslHandler$4.run(SslHandler.java:1371)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.ImmediateExecutor.execute(ImmediateExecutor.java:31)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1368)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
~[netty-3.7.0.Final.cs.1.jar:?]
... 13 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: too much data for RSA block
at
org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown
Source) ~[bcprov-jdk15on-1.50.jar:1.50.0]
at javax.crypto.Cipher.doFinal(Cipher.java:1922) ~[?:1.7.0_55]
at
sun.security.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:149)
~[?:1.7.0_55]
at
sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:190)
~[?:1.7.0_55]
at
sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) ~[?:1.7.0_55]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:808) ~[?:1.7.0_55]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:806) ~[?:1.7.0_55]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.7.0_55]
at
sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227)
~[?:1.7.0_55]
at
org.jboss.netty.handler.ssl.SslHandler$4.run(SslHandler.java:1371)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.ImmediateExecutor.execute(ImmediateExecutor.java:31)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1368)
~[netty-3.7.0.Final.cs.1.jar:?]
at
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
~[netty-3.7.0.Final.cs.1.jar:?]
... 13 more
the larger public key means the encrypted master secret will be
2048-bits long, which is longer than any message BouncyCastle allows
to be decrypted with the shorter 1024-bit private key. BouncyCastle's
RSA block cipher does not permit messages larger than one block to be
encrypted or decrypted, which is why the runtime exception was thrown.
However, this raises a more serious question: could a malicious client
purposely send an encrypted master secret larger than the size of the
private key to cause the TLS server to crash? If so, this seems to be
a remote exploit that any server using the JSSE for TLS and BC for RSA
encryption would be vulnerable to.
Jason