Exceptions can block forever when minConnections is set to 1 [JIRA: CLIENTS-501] #523

As reported by @hankipanky on the riak-users mailing list.

I've reported in the past that my application is seeing stange exceptions when running under even the slightest load (riak-users discussion)

The observation is that some threads simply hang when this exception occurs, waiting for ever without understanding what is going on. This unreliability is very bad for my application where a lot of Riak-related tasks are handled in Threads in the background.

I keep trying to find out what is causing this, but I have no real clue. But after playing around with RiakNode.Builder.min/maxConnections, the problem does not occur anymore.

Specifically, I have now set:

RiakNode riakNode = new RiakNode.Builder()
  .withConnectionTimeout(5000)
  .withMinConnections(2)
  .withMaxConnections(16)
  .withIdleTimeout(500)
  .withBlockOnMaxConnections(false)
  .withRemoteAddress(Config.getInstance().getRiakAddress())
  .withRemotePort(Config.getInstance().getRiakPort())
  .build();

If I leave minConnections at 1, the problem occurs, reproducibly. With higher values, it doesn't.

Is there maybe an issue in the code? Are connections reaped too quickly?

Cheers,
Henning