Minor name changes and expand documentation.

James William Pye · James William Pye · commit f466eb51ffb2 · 2010-03-25T17:43:29.000-07:00
diff --git a/postgresql/documentation/copyman.txt b/postgresql/documentation/copyman.txt
@@ -7,7 +7,7 @@ Copy Management
 .. warning:: `postgresql.copyman` is a new feature in v1.0.
 
 The `postgresql.copyman` module provides a way to quickly move COPY data coming
-from one connection to many connections. Alternatively, it can also be sourced
+from one connection to many connections. Alternatively, it can be sourced
 by arbitrary iterators and target arbitrary callables.
 
 Statement execution methods offer a way for running COPY operations
@@ -37,13 +37,13 @@ The `postgresql.copyman.CopyManager` class manages the Producer and the
 Receivers involved in a COPY operation. Normally,
 `postgresql.copyman.StatementProducer` and
 `postgresql.copyman.StatementReceiver` instances. Naturally, a Producer is the
-object that produces the COPY data to be given to the manager's Receivers.
+object that produces the COPY data to be given to the Manager's Receivers.
 
-Using a CopyManager directly means that there is a need for more control over
+Using a Manager directly means that there is a need for more control over
 the operation. The Manager is both a context manager and an iterator. The
-context manager interfaces handle initialization and finalization, and the
-iterator provides an event loop emitting information about the amount of
-COPY data copied this cycle. Normal usage takes the form::
+context manager interfaces handle initialization and finalization of the COPY
+state, and the iterator provides an event loop emitting information about the
+amount of COPY data transferred this cycle. Normal usage takes the form::
 
 	>>> from postgresql import copyman
 	>>> send_stmt = source.prepare("COPY (SELECT i FROM generate_series(1, 1000000) AS g(i)) TO STDOUT")
@@ -57,15 +57,14 @@ COPY data copied this cycle. Normal usage takes the form::
 	...   for num_messages, num_bytes in copy:
 	...    update_rate(num_bytes)
 
-The use of the context manager is necessary for ensuring that connection state
-is properly restored at the end of the COPY.
-
 As an alternative to a for-loop inside a with-statement block, the `run` method
 can be called to perform the operation::
 
 	>>> with source.xact(), destination.xact():
 	...  copyman.CopyManager(producer, receiver).run()
 
+However, there is little benefit beyond using the high-level
+`postgresql.copyman.transfer` function.
 
 Manager Interface Points
 ------------------------
@@ -82,30 +81,54 @@ an iterator for controlling the COPY operation.
 
  ``CopyManager.__exit__(typ, val, tb)``
   Finish the COPY operation. Fails in the case of an incomplete
-  COPY, or an untrapped exception.
+  COPY, or an untrapped exception. Either returns `None` or raises the generalized
+  exception, `postgresql.copyman.CopyFail`.
 
  ``CopyManager.__iter__()``
   Returns the CopyManager instance.
 
  ``CopyManager.__next__()``
   Transfer the next chunk of COPY data to the receivers. Yields a tuple
-  consisting of the number of messages and bytes transferred. Raises
-  `StopIteration` when complete.
+  consisting of the number of messages and bytes transferred,
+  ``(num_messages, num_bytes)``. Raises `StopIteration` when complete.
+
+  Raises `postgresql.copyman.ReceiverFault` when a Receiver raises an
+  exception.
+  Raises `postgresql.copyman.ProducerFault` when the Producer raises an
+  exception. The original exception is available via the exception's
+  ``__context__`` attribute.
 
  ``CopyManager.reconcile(faulted_receiver)``
   Reconcile a faulted receiver. When a receiver faults, it will no longer
-  be in the receiver set. This method is used to signal to the manager that the
-  problem has been cleared up, and the receiver is again ready to receive.
+  be in the set of Receivers. This method is used to signal to the manager that the
+  problem has been corrected, and the receiver is again ready to receive.
+
+ ``CopyManager.receivers``
+  The `builtins.set` of Receivers involved in the COPY operation.
+
+ ``CopyManager.producer``
+  The Producer emitting the data to be given to the Receivers.
 
 
 Faults
 ======
 
-The CopyManager generalizes some exceptions that occur during transfer. While
+The CopyManager generalizes any exceptions that occur during transfer. While
 inside the context manager, `postgresql.copyman.Fault` may be raised if a
-Receiver raises an exception. The Manager assumes the Fault is fatal to a
-Receiver, and immediately removes it from the set of target receivers.
-Additionally, if the Fault goes untrapped, the copy will fail.
+Receiver or a Producer raises an exception. A `postgresql.copyman.ProducerFault`
+in the case of the Producer, and `postgresql.copyman.ReceiverFault` in the case
+of the Receivers.
+
+.. note::
+ Faults are only raised by `postgresql.copyman.CopyManager.__next__`. The
+ ``run()`` method will always raise `postgresql.copyman.CopyFail`.
+
+Receiver Faults
+---------------
+
+The Manager assumes the Fault is fatal to a Receiver, and immediately removes
+it from the set of target receivers. Additionally, if the Fault goes untrapped,
+the copy will ultimately fail.
 
 The Fault exception references the Manager that raised the exception, and the
 actual exceptions that occurred, associated with the Receiver that caused them::
@@ -123,36 +146,38 @@ actual exceptions that occurred, associated with the Receiver that caused them::
 	...    try:
 	...     for num_messages, num_bytes in copy:
 	...      update_rate(num_bytes)
-	...    except copyman.Fault as cf:
+	...    except copyman.ReceiverFault as cf:
+	...     # Access the original exception using the receiver as the key.
 	...     original_exception = cf.faults[receiver]
 	...     if unknown_failure(original_exception):
 	...      ...
 	...      raise
 
 
-Fault Properties
-----------------
+ReceiverFault Properties
+~~~~~~~~~~~~~~~~~~~~~~~~
 
-The following attributes exist on `postgresql.copyman.Fault` instances:
+The following attributes exist on `postgresql.copyman.ReceiverFault` instances:
 
- ``Fault.manager``
-  The `postgresql.copyman.CopyManager` instance that raised the exception; the
-  same manager that caught the fault.
+ ``ReceiverFault.manager``
+  The subject `postgresql.copyman.CopyManager` instance.
 
- ``Fault.faults``
-  A dictionary mapping the Receiver to the exception that occurred. The Manager
-  will give processing to every Receiver, so only one Fault will occur per
-  transfer cycle.
+ ``ReceiverFault.faults``
+  A dictionary mapping the Receiver to the exception raised by that Receiver.
+  The Manager will give processing time to every Receiver, so only *one* Fault will
+  occur per transfer cycle, each iteration.
 
-Reconciliation
---------------
 
-When a Fault occurs, it is possible that it was not fatal. In such cases the
-`postgresql.copyman.CopyManager.reconcile` method can be used to reintroduce the
-Receiver to the Manager's set. That is, when a Fault occurs, the Manager
-immediately removes the Receiver so that the COPY operation can continue.
+Reconciliation
+~~~~~~~~~~~~~~
 
-Faults should be trapped from within the Manager's context::
+When a `postgresql.copyman.ReceiverFault` is raised, the Manager immediately
+removes the Receiver so that the COPY operation can continue. Continuation of
+the COPY can occur by trapping the exception and continuing the iteration of the
+Manager. However, if the fault is recoverable, the
+`postgresql.copyman.CopyManager.reconcile` method must be used to reintroduce the
+Receiver into the Manager's set. Faults should be trapped from within the
+Manager's context::
 
 	>>> import socket
 	>>> from postgresql import copyman
@@ -168,7 +193,7 @@ Faults should be trapped from within the Manager's context::
 	...    try:
 	...     for num_messages, num_bytes in copy:
 	...      update_rate(num_bytes)
-	...    except copyman.Fault as cf:
+	...    except copyman.ReceiverFault as cf:
 	...     if isinstance(cf.faults[receiver], socket.timeout):
 	...      copy.reconcile(receiver)
 	...     else:
@@ -179,6 +204,82 @@ so, often, it's best to avoid conditions in which reconciliable Faults may
 occur.
 
 
+Producer Faults
+---------------
+
+Producer faults are normally fatal to the COPY operation and should rarely be
+trapped. However, the Manager makes no state changes when a Producer faults,
+so, unlike Receiver Faults, no reconciliation process is necessary; rather,
+if it's safe to continue, the Manager's iterator should continue to be
+processed.
+
+ProducerFault Properties
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following attributes exist on `postgresql.copyman.ProducerFault` instances:
+
+ ``ReceiverFault.manager``
+  The subject `postgresql.copyman.CopyManager`.
+
+ ``ReceiverFault.__context__``
+  The original exception raised by the Producer.
+
+
+Failures
+========
+
+When a COPY operation is aborted, either by an exception or by the iterator
+being broken, a `postgresql.copyman.CopyFail` exception will be raised,
+generalizing the failure. When a failure occurs, the Manager will *attempt* to
+recover and realign the Producer and the Receivers. Regardless of the success of
+the recovery process, a `postgresql.copyman.CopyFail` exception will be raised.
+
+The `postgresql.copyman.CopyFail` offers to record any exceptions that occur
+during the exit of the context manager.
+
+
+CopyFail Properties
+-------------------
+
+The following properties exist on `postgresql.copyman.CopyFail` exceptions:
+
+ ``CopyFail.manager``
+  The Manager whose COPY operation failed.
+
+ ``CopyFail.receiver_faults``
+  A dictionary mapping a `postgresql.copyman.Receiver` to the exception raised
+  by that Receiver's ``__exit__``. `None` if no exceptions were raised by the
+  Receivers.
+
+ ``CopyFail.producer_fault``
+  The exception Raised by the `postgresql.copyman.Producer`. `None` if none.
+
+
+Producers
+=========
+
+The following Producers are available:
+
+ ``postgresql.copyman.StatementProducer(postgresql.api.Statement)``
+  Given a Statement producing COPY data, construct a Producer.
+
+ ``postgresql.copyman.IteratorProducer(collections.Iterator)``
+  Given an Iterator producing *chunks* of COPY lines, construct a Producer to
+  manage the data coming from the iterator.
+
+
+Receivers
+=========
+
+ ``postgresql.copyman.StatementReceiver(postgresql.api.Statement)``
+  Given a Statement producing COPY data, construct a Producer.
+
+ ``postgresql.copyman.CallReceiver(callable)``
+  Given a callable, construct a Receiver that will transmit COPY data in chunks
+  of lines. That is, the callable will be given a list of COPY lines for each
+  transfer cycle.
+
+
 Terminology
 ===========
 
@@ -207,13 +308,13 @@ processes of the `postgresql.copyman` module:
   necessary steps for a Receiver's reintroduction into the COPY operation after
   a Fault.
 
- Realignment
-  The process of providing compensating data to the receivers so that the
-  connection will be on a message boundary. Occurs when the COPY operation
-  fails.
-
  Failed Copy
   A failed copy is an aborted COPY operation. This occurs in situations of
   untrapped exceptions or an incomplete COPY. Specifically, the COPY will be
   noted as failed in cases where the Manager's iterator is *not* ran until
   exhaustion.
+
+ Realignment
+  The process of providing compensating data to the receivers so that the
+  connection will be on a message boundary. Occurs when the COPY operation
+  fails.
diff --git a/postgresql/test/test_copyman.py b/postgresql/test/test_copyman.py
@@ -328,7 +328,7 @@ def testCopyWithFailure(self):
 					else:
 						# Done with copy.
 						break
-				except copyman.ReceiverFaults as cf:
+				except copyman.ReceiverFault as cf:
 					if sr2 not in cf.faults:
 						raise
 		self.failUnless(done)
@@ -396,7 +396,7 @@ def testNoReceivers(self):
 								dst.pq.socket.close()
 							else:
 								self.fail("failed to detect dead socket")
-					except copyman.ReceiverFaults as cf:
+					except copyman.ReceiverFault as cf:
 						self.failUnless(sr1 in cf.faults)
 						# Don't reconcile. Let the manager drop the receiver.
 		except copyman.CopyFail:
@@ -436,7 +436,7 @@ def failed_write(*args):
 					else:
 						# Done with COPY, break out of while copy.receivers.
 						break
-				except copyman.ReceiverFaults as cf:
+				except copyman.ReceiverFault as cf:
 					if isinstance(cf.faults[sr], RecoverableError):
 						if done is True:
 							self.fail("failed_write was called twice?")
@@ -485,7 +485,7 @@ def failed_write(*args):
 					else:
 						# Done with COPY, break out of while copy.receivers.
 						break
-				except copyman.ReceiverFaults as cf:
+				except copyman.ReceiverFault as cf:
 					self.failUnless(isinstance(cf.faults[sr2], TheCause))
 					if done is True:
 						self.fail("failed_write was called twice?")
@@ -528,7 +528,8 @@ def testProducerFailure(self):
 					self.failUnless(pf.__context__ is not None)
 			self.fail('expected CopyManager to raise CopyFail')
 		except copyman.CopyFail as cf:
-			pass
+			# Expecting to see CopyFail
+			self.failUnless(True)
 		self.failUnless(done)
 		self.failUnlessRaises(Exception, sqlexec, 'select 1')
 		self.failUnlessEqual(dst.prepare(dstcount).first(), 0)
