feat: Batched Uploads and Green Threads by william-silversmith · Pull Request #14 · seung-lab/python-task-queue

william-silversmith · 2019-02-15T05:14:25Z

This contains a few features.

Send messages to AWS SQS using the send_message_batch (via tq.insert_all) which allows sending 10 messages at a time. This results in a 10x speedup.
Add GreenTaskQueue as an option. Users need to manually monkeypatch their application to make it work. This can be useful in NUMA or vCPU environments.
Added delay_seconds to insert and insert_all which allows you to delay a message's visibility in the queue.

Breaks localtaskqueue. We can just have slow green threads instead.

nkemnitz

I can test this change along with my next test run once the tests are passing.

Splitting the features into separate PR's would help me speed up review and tests.

nkemnitz · 2019-02-15T16:26:41Z

+  try:
+    iter(obj)
+    if type(obj) is dict:
+      return [ obj ]


list(obj) does what you want, [obj] just puts the dictionary itself in a list.

That's actually what I wanted for this case.

Got it. If it's not a "general purpose" function, you could try to hide it a bit from the user - maybe name it _to_iterable. Not crucial, though.

nkemnitz · 2019-02-15T16:30:54Z

 from .secrets import aws_credentials

+
+def toiter(obj):


Would change the name to to_iterable - you are not returning an iterator.

nkemnitz · 2019-02-15T16:55:56Z

+    AWS_BATCH_SIZE = 10 
+
+    resps = []
+    for i in range(0, len(tasks), AWS_BATCH_SIZE):


Iterables are not guaranteed to have len (generators, for example)

This is true and a headache. I should call it "tolist".

nkemnitz · 2019-02-15T16:58:11Z

+  def check_monkey_patch_status(self):
+    import gevent.monkey
+    if not gevent.monkey.is_module_patched("socket"):
+      print(yellow("""


yellow is not defined

william-silversmith · 2019-02-17T23:38:58Z

@nkemnitz latest changes include allowing the use of generators instead of lists in insert_all for both GreenTaskQueue and TaskQueue. Additionally, we do not monkey patch automatically. Instead, if the socket standard library is not monkey patched and you are using GreenTaskQueue, we issue a warning containing instructions on how to monkey patch your application. TaskQueue continues to work.

Generators are necessary for two reasons when the number of tasks being generated becomes large. Firstly, they allow the commencement of upload nearly instantly. Previously, while batching was an option, typically people would create lists of hundreds of thousands or millions of elements before submitting them for upload. This process took many seconds of time to complete, resulting in substantial latency. Secondly, these lists of millions of items would often result in MemoryErrors, requiring more complex batching logic. Now, so long as the user structures their submission logic as a generator, both issues are disposed of completely.

william-silversmith · 2019-02-17T23:45:57Z


 import taskqueue
-from taskqueue import RegisteredTask, TaskQueue, MockTask, PrintTask, LocalTaskQueue, MockTaskQueue
+from taskqueue import RegisteredTask, GreenTaskQueue, TaskQueue, MockTask, PrintTask, LocalTaskQueue, MockTaskQueue


No monkey patching here, so GreenTaskQueue will simply behave as a single threaded TaskQueue. monkey patching was disrupting LocalTaskQueue.

approved in person

william-silversmith added 2 commits February 4, 2019 17:46

wip: green threads

b3c32ce

feat: green threads and regular threads coexist

941be4d

william-silversmith added feature refactor labels Feb 15, 2019

william-silversmith self-assigned this Feb 15, 2019

william-silversmith requested a review from nkemnitz February 15, 2019 05:14

william-silversmith added 3 commits February 15, 2019 00:25

fix: add scheduler to git and delay_seconds to insert

955cd4a

fix: set delay_seconds default to 0

54b8916

fix: remove monkeypatching from test

fd3a906

Breaks localtaskqueue. We can just have slow green threads instead.

nkemnitz previously requested changes Feb 15, 2019

View reviewed changes

william-silversmith added 5 commits February 15, 2019 16:06

fix: import "yellow"

b8c1f64

fix: aws insert method accepts generators and iterators

2c57f03

feat: GreenTaskQueue accepts generators for insert_all

1f92d45

feat: TaskQueue (threaded) works with generators

282e6a6

chore: drop py34 support add py37

b61a1e9

william-silversmith force-pushed the wms_green_threads branch from e4c68fb to b61a1e9 Compare February 17, 2019 23:30

william-silversmith commented Feb 17, 2019

View reviewed changes

william-silversmith force-pushed the wms_green_threads branch from a840860 to 8d2482e Compare February 18, 2019 02:34

docs: improved narrative, updated examples, how to get high throughput

f450e39

william-silversmith force-pushed the wms_green_threads branch from 8d2482e to f450e39 Compare February 18, 2019 02:35

william-silversmith added 4 commits February 17, 2019 21:41

fix: MockTaskQueue and LocalTaskQueue support insert_all

4b17cab

fix: LocalTaskQueue should process when .wait() is called

eb74559

docs: more nuanced note on passing generators

fead475

feat: LocalTaskQueue at parallel=1 same as MockTaskQueue

c29f2c5

william-silversmith force-pushed the wms_green_threads branch 3 times, most recently from df58d58 to eb688c7 Compare February 18, 2019 04:09

docs: using Iterators instead of Generators

3d752d8

william-silversmith force-pushed the wms_green_threads branch from eb688c7 to 3d752d8 Compare February 18, 2019 04:11

william-silversmith added 2 commits February 17, 2019 23:18

docs: fix wording

72cb34a

docs: generic queue name in listing 7

2091f86

william-silversmith force-pushed the wms_green_threads branch 3 times, most recently from 8384f8a to e06192c Compare February 18, 2019 04:26

docs: show how to use total argument for generators

4adf1f0

william-silversmith force-pushed the wms_green_threads branch from e06192c to 4adf1f0 Compare February 18, 2019 04:27

william-silversmith added 2 commits February 18, 2019 00:01

docs: amended motiviation section

3780ffd

docs: used slightly more precise language

4190fb9

william-silversmith merged commit d88fcdf into master Feb 19, 2019

william-silversmith deleted the wms_green_threads branch February 19, 2019 01:16

Navigation Menu

Repository navigation

feat: Batched Uploads and Green Threads - #14

feat: Batched Uploads and Green Threads#14
william-silversmith merged 21 commits into
masterfrom
wms_green_threads

william-silversmith commented Feb 15, 2019 •

edited

Loading

Uh oh!

nkemnitz left a comment

Uh oh!

nkemnitz Feb 15, 2019

Uh oh!

william-silversmith Feb 15, 2019

Uh oh!

nkemnitz Feb 15, 2019

Uh oh!

nkemnitz Feb 15, 2019

Uh oh!

william-silversmith Feb 17, 2019

Uh oh!

nkemnitz Feb 15, 2019

Uh oh!

william-silversmith Feb 15, 2019

Uh oh!

nkemnitz Feb 15, 2019

Uh oh!

william-silversmith Feb 15, 2019

Uh oh!

william-silversmith commented Feb 17, 2019 •

edited

Loading

Uh oh!

william-silversmith Feb 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

william-silversmith commented Feb 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nkemnitz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

william-silversmith commented Feb 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

william-silversmith commented Feb 15, 2019 •

edited

Loading

william-silversmith commented Feb 17, 2019 •

edited

Loading