Skip to content
This repository was archived by the owner on Aug 13, 2026. It is now read-only.

The Big Transport Refactor - #179

Merged
seancribbs merged 45 commits into
masterfrom
sdc-transport-cleanup
Jan 31, 2013
Merged

The Big Transport Refactor#179
seancribbs merged 45 commits into
masterfrom
sdc-transport-cleanup

Conversation

@seancribbs

Copy link
Copy Markdown

The internal structure of the client has limited us from doing several important things listed below. This branch is an effort to restructure the internals, while maintaining some amount of API compatibility with the old version, and warning users of deprecated/removed/disabled patterns. The partially-completed items are:

  • Connection manager and monitor are gone. The relationship between the connection manager and the transport implementation was reversed. Instead it has been replaced with the Pool class, one for each transport type. The monitor never worked anyway!
  • Transport implementations are broken up into multiple files for clarity.
  • Streaming operations (list-keys, MapReduce) are supported, implemented as iterators. Example:
for keylist in bucket.stream_keys():
    for key in keylist:
        print key
  • All request operations go through the RiakClient object, which handles the connection pools. Accessing the transport directly is now forbidden.
  • The client can connect to multiple nodes, and each node has an associated error rate (to be used later when implementing error recovery). The node selected for new connections depends on the error rate.
  • Retry/recovery from network failures and down nodes will be automatic. Failed requests will be retried on other nodes and bump the error rate for the failed node.

Sean Cribbs added 23 commits December 27, 2012 10:40
* Connection manager and monitor are gone.
* PBC transport is broken up into multiple files for clarity.
* Outline of streaming operations (currently only PBC).
* All requests go through the RiakClient object, which handles the
  connection pools. Accessing the transport directly is now forbidden.
* The client can connect to multiple nodes, and each node has an
  associated error rate (to be used later when implementing error
  recovery). The node selected for new connections depends on the
  error rate.

TODO:

* Break up HTTP into smaller files and add resource detection (for URL
  generation).
* Add streaming operations to HTTP.
* RiakClient.transport is now RiakClient._transport (via
  RiakClientTransport mixin).
* HTTP and PBC transports define their own version of what are
  retryable exceptions.
@ghost ghost assigned seancribbs Dec 31, 2012
@seancribbs

Copy link
Copy Markdown
Author

@evanmcc @coderoshi @reiddraper @shuhaowu @glassresistor I believe I have resolved all remaining issues on the feature list. Aside from adding a few tests about retries, I believe this is done.

@shuhaowu

Copy link
Copy Markdown
Contributor

Docs needs to be updated as well.

Comment thread riak/client/operations.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we're breaking the API, it might be simpler to collapse put and put_new into the same function.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Seems redundant

@evanmcc

evanmcc commented Jan 31, 2013

Copy link
Copy Markdown
Contributor

Tests all pass on 2.6 and 2.7, code looks fine other than a few quibbles I commented on.

@seancribbs

Copy link
Copy Markdown
Author

I'll fix those bits in the morning, thanks.

Sean Cribbs

On Jan 30, 2013, at 7:31 PM, Evan Vigil-McClanahan notifications@github.com wrote:

Tests all pass on 2.6 and 2.7, code looks fine other than a few quibbles I commented on.


Reply to this email directly or view it on GitHub.

seancribbs pushed a commit that referenced this pull request Jan 31, 2013
Closes #153.
Closes #150.
Obsoletes #151.
Closes #21.
Obsoletes #186.

Conflicts:
	riak/client.py
	riak/riak_object.py
	riak/tests/test_all.py
@seancribbs
seancribbs merged commit 535d0bc into master Jan 31, 2013
@seancribbs
seancribbs deleted the sdc-transport-cleanup branch January 31, 2013 21:06
@shuhaowu

shuhaowu commented Feb 1, 2013

Copy link
Copy Markdown
Contributor

Looking at this retrospectively, should riak_index_entry.py still be there? I feel like that really should be a 2-item tuple.

Edit: should be 2 element tuple instead of 3.

@seancribbs

Copy link
Copy Markdown
Author

@shuhaowu Let's make a new PR for that refactor. Looks like you already made one.

@seancribbs seancribbs mentioned this pull request Feb 6, 2013
@seancribbs seancribbs removed their assignment May 8, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants