I recently had to write a server in java that can speak SSL/TLS. The actual application protocol is handled by classes upstream to the SSL/TL layer. For scalability I decided to use Java NIO - which implied using the javax.net.ssl.SSLEngine.
The SSLEngine api is pretty cool though could possibly be a bit better designed. Essentially the application needs to manage 4 buffers of data, 2 of which are read by and 2 of which are written to by the SSLEngine. Sounds easy, infact it is, if it were not for the Handshake and renegotiation. Calling an api on the SSLEngine can cause it to ask you to call another api which may in turn ask you to call the original ... and so on.
I first wrote the SSL layer using recursion but with upto 5 exit points and special code to take care of stack unwinding and having to manipulate global buffers this was deemed too risky. So I re implemented the SSL engine via state transition. Since I was already playing with NIO and had to use a thread pool to handle concurrent clients, I decided to make the SSL processing asynchronous. Essentially I broke down the SSL processing (including breaking down the handshake and app layer protocol parsing) into various states, the currently executing thread would finish executing 1 state and hand over the "session" to the thread pool where another free thread will finish the next state. Ofcourse I have coded in optimization where an implementation of a state can decide whether to transition over a thread boundary or force the current thread to execute the next state.
Long story short. It works andit fits very nicely into the server's threading model.