forked from taskflow/taskflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathDataParallelPipeline.html
More file actions
264 lines (253 loc) · 32.1 KB
/
Copy pathDataParallelPipeline.html
File metadata and controls
264 lines (253 loc) · 32.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Taskflow Algorithms » Data-parallel Pipeline | Taskflow QuickStart</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600" />
<link rel="stylesheet" href="m-dark+documentation.compiled.css" />
<link rel="icon" href="favicon.ico" type="image/vnd.microsoft.icon" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="theme-color" content="#22272e" />
</head>
<body>
<header><nav id="navigation">
<div class="m-container">
<div class="m-row">
<span id="m-navbar-brand" class="m-col-t-8 m-col-m-none m-left-m">
<a href="https://taskflow.github.io"><img src="taskflow_logo.png" alt="" />Taskflow</a> <span class="m-breadcrumb">|</span> <a href="index.html" class="m-thin">QuickStart</a>
</span>
<div class="m-col-t-4 m-hide-m m-text-right m-nopadr">
<a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
<path id="m-doc-search-icon-path" d="m6 0c-3.31 0-6 2.69-6 6 0 3.31 2.69 6 6 6 1.49 0 2.85-0.541 3.89-1.44-0.0164 0.338 0.147 0.759 0.5 1.15l3.22 3.79c0.552 0.614 1.45 0.665 2 0.115 0.55-0.55 0.499-1.45-0.115-2l-3.79-3.22c-0.392-0.353-0.812-0.515-1.15-0.5 0.895-1.05 1.44-2.41 1.44-3.89 0-3.31-2.69-6-6-6zm0 1.56a4.44 4.44 0 0 1 4.44 4.44 4.44 4.44 0 0 1-4.44 4.44 4.44 4.44 0 0 1-4.44-4.44 4.44 4.44 0 0 1 4.44-4.44z"/>
</svg></a>
<a id="m-navbar-show" href="#navigation" title="Show navigation"></a>
<a id="m-navbar-hide" href="#" title="Hide navigation"></a>
</div>
<div id="m-navbar-collapse" class="m-col-t-12 m-show-m m-col-m-none m-right-m">
<div class="m-row">
<ol class="m-col-t-6 m-col-m-none">
<li><a href="pages.html">Handbook</a></li>
<li><a href="namespaces.html">Namespaces</a></li>
</ol>
<ol class="m-col-t-6 m-col-m-none" start="3">
<li><a href="annotated.html">Classes</a></li>
<li><a href="files.html">Files</a></li>
<li class="m-show-m"><a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
<use href="#m-doc-search-icon-path" />
</svg></a></li>
</ol>
</div>
</div>
</div>
</div>
</nav></header>
<main><article>
<div class="m-container m-container-inflatable">
<div class="m-row">
<div class="m-col-l-10 m-push-l-1">
<h1>
<span class="m-breadcrumb"><a href="Algorithms.html">Taskflow Algorithms</a> »</span>
Data-parallel Pipeline
</h1>
<nav class="m-block m-default">
<h3>Contents</h3>
<ul>
<li><a href="#ParallelDataPipelineIncludeHeaderFile">Include the Header</a></li>
<li><a href="#CreateADataPipelineModuleTask">Create a Data Pipeline Module Task</a></li>
<li><a href="#UnderstandInternalDataStorage">Understand Internal Data Storage</a></li>
<li><a href="#DataParallelPipelineLearnMore">Learn More about Taskflow Pipeline</a></li>
</ul>
</nav>
<p>Taskflow provides another variant, <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a>, on top of <a href="classtf_1_1Pipeline.html" class="m-doc">tf::<wbr />Pipeline</a> (see <a href="TaskParallelPipeline.html" class="m-doc">Task-parallel Pipeline</a>) to help you implement data-parallel pipeline algorithms while leaving data management to Taskflow. We recommend you finishing reading TaskParallelPipeline first before learning <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a>.</p><section id="ParallelDataPipelineIncludeHeaderFile"><h2><a href="#ParallelDataPipelineIncludeHeaderFile">Include the Header</a></h2><p>You need to include the header file, <code>taskflow/algorithm/data_pipeline.hpp</code>, for implementing data-parallel pipeline algorithms.</p><pre class="m-code"><span class="cp">#include</span><span class="w"> </span><span class="cpf"><taskflow/algorithm/data_pipeline.hpp></span><span class="cp"></span></pre></section><section id="CreateADataPipelineModuleTask"><h2><a href="#CreateADataPipelineModuleTask">Create a Data Pipeline Module Task</a></h2><p>Similar to creating a task-parallel pipeline (<a href="classtf_1_1Pipeline.html" class="m-doc">tf::<wbr />Pipeline</a>), there are three steps to create a data-parallel pipeline application:</p><ol><li>Define the pipeline structure (e.g., pipe type, pipe callable, stopping rule, line count)</li><li>Define the data storage and layout, if needed for the application</li><li>Define the pipeline taskflow graph using composition</li></ol><p>The following example creates a data-parallel pipeline that generates a total of five dataflow tokens from <code>void</code> to <code>int</code> at the first stage, from <code>int</code> to <code>std::string</code> at the second stage, from <code>std::string</code> to <code>float</code> at the third stage, and <code>float</code> to <code>void</code> at the final stage. Data storage between stages is automatically managed by <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a>.</p><pre class="m-code"><span class="cp">#include</span><span class="w"> </span><span class="cpf"><taskflow/taskflow.hpp></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><taskflow/algorithm/data_pipeline.hpp></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// data flow => void -> int -> std::string -> float -> void </span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Taskflow</span><span class="w"> </span><span class="n">taskflow</span><span class="p">(</span><span class="s">"pipeline"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Executor</span><span class="w"> </span><span class="n">executor</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">size_t</span><span class="w"> </span><span class="n">num_lines</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// create a pipeline graph</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">DataPipeline</span><span class="w"> </span><span class="n">pl</span><span class="p">(</span><span class="n">num_lines</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="kt">void</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="o">></span><span class="p">(</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kt">int</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">stop</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"first pipe returns %lu</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}),</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span><span class="p">(</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[](</span><span class="kt">int</span><span class="o">&</span><span class="w"> </span><span class="n">input</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"second pipe returns a strong of %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">100</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">to_string</span><span class="p">(</span><span class="n">input</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">100</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}),</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="kt">void</span><span class="o">></span><span class="p">(</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[](</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&</span><span class="w"> </span><span class="n">input</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"third pipe receives the input string %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="n">c_str</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// build the pipeline graph using composition</span>
<span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">composed_of</span><span class="p">(</span><span class="n">pl</span><span class="p">).</span><span class="n">name</span><span class="p">(</span><span class="s">"pipeline"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// dump the pipeline graph structure (with composition)</span>
<span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// run the pipeline</span>
<span class="w"> </span><span class="n">executor</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">taskflow</span><span class="p">).</span><span class="n">wait</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span></pre><p>The interface of <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a> is very similar to <a href="classtf_1_1Pipeline.html" class="m-doc">tf::<wbr />Pipeline</a>, except that the library transparently manages the dataflow between pipes. To create a stage in a data-parallel pipeline, you should always use the helper function <a href="namespacetf.html#a8975fa5762088789adb0b60f38208309" class="m-doc">tf::<wbr />make_data_pipe</a>:</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="p">[](</span><span class="kt">int</span><span class="o">&</span><span class="w"> </span><span class="n">input</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">to_string</span><span class="p">(</span><span class="n">input</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">100</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">);</span><span class="w"></span></pre><p>The helper function starts with a pair of an input and an output types in its template arguments. Both types will always be decayed to their original form using <a href="http://en.cppreference.com/w/cpp/types/decay.html" class="m-doc-external">std::<wbr />decay</a> (e.g., <code>const int&</code> becomes <code>int</code>) for storage purpose. In terms of function arguments, the first argument specifies the direction of this data pipe, which can be either <a href="namespacetf.html#abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" class="m-doc">tf::<wbr />PipeType::<wbr />SERIAL</a> or <a href="namespacetf.html#abb7a11e41fd457f69e7ff45d4c769564adf13a99b035d6f0bce4f44ab18eec8eb" class="m-doc">tf::<wbr />PipeType::<wbr />PARALLEL</a>, and the second argument is a callable to invoke by the pipeline scheduler. The callable must take the input data type in its first argument and returns a value of the output data type. Additionally, the callable can take a <a href="classtf_1_1Pipeflow.html" class="m-doc">tf::<wbr />Pipeflow</a> reference in its second argument which allows you to query the runtime information of a stage task, such as its line number and token number.</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="kt">int</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="p">[](</span><span class="kt">int</span><span class="o">&</span><span class="w"> </span><span class="n">input</span><span class="p">,</span><span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"token=%lu, line=%lu</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">(),</span><span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">to_string</span><span class="p">(</span><span class="n">input</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">100</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">)</span><span class="w"></span></pre><aside class="m-note m-info"><h4>Note</h4><p>By default, <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a> passes the data in reference to your callable at which you can take it in copy or in reference depending on application needs.</p></aside><p>For the first pipe, the input type should always be <code>void</code> and the callable must take a <a href="classtf_1_1Pipeflow.html" class="m-doc">tf::<wbr />Pipeflow</a> reference in its argument. In this example, we will stop the pipeline when processing five tokens.</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="kt">void</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="o">></span><span class="p">(</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="kt">int</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">5</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">stop</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="c1">// returns a dummy value</span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}),</span><span class="w"></span></pre><p>Similarly, the output type of the last pipe should be <code>void</code> as no more data will go out of the final pipe.</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">make_data_pipe</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="kt">void</span><span class="o">></span><span class="p">(</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[](</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&</span><span class="w"> </span><span class="n">input</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span><span class="w"></span>
<span class="p">})</span><span class="w"></span></pre><p>Finally, you need to compose the pipeline graph by creating a module task (i.e., tf::Taskflow::compoased_of).</p><pre class="m-code"><span class="c1">// build the pipeline graph using composition</span>
<span class="n">taskflow</span><span class="p">.</span><span class="n">composed_of</span><span class="p">(</span><span class="n">pl</span><span class="p">).</span><span class="n">name</span><span class="p">(</span><span class="s">"pipeline"</span><span class="p">);</span><span class="w"></span>
<span class="c1">// dump the pipeline graph structure (with composition)</span>
<span class="n">taskflow</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="p">);</span><span class="w"></span>
<span class="c1">// run the pipeline</span>
<span class="n">executor</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">taskflow</span><span class="p">).</span><span class="n">wait</span><span class="p">();</span><span class="w"></span></pre><div class="m-graph"><svg style="width: 41.600rem; height: 21.600rem;" viewBox="0.00 0.00 416.00 216.00">
<g transform="scale(1 1) rotate(0) translate(4 212)">
<title>Taskflow</title>
<g class="m-cluster">
<title>cluster_p0x7ffc47e53358</title>
<polygon points="8,-129 8,-200 106,-200 106,-129 8,-129"/>
<text text-anchor="middle" x="57" y="-188" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Taskflow</text>
</g>
<g class="m-cluster">
<title>cluster_p0x7ffc47e53220</title>
<polygon points="114,-8 114,-200 400,-200 400,-8 114,-8"/>
<text text-anchor="middle" x="257" y="-188" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">m1</text>
</g>
<g class="m-node m-flat">
<title>p0x1878a88</title>
<polygon points="98,-173 20,-173 16,-169 16,-137 94,-137 98,-141 98,-173"/>
<polyline points="94,-169 16,-169 "/>
<polyline points="94,-169 94,-137 "/>
<polyline points="94,-169 98,-173 "/>
<text text-anchor="middle" x="57" y="-152.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">pipeline [m1]</text>
</g>
<g class="m-node">
<title>p0x1878600</title>
<polygon points="257,-173 223.8992,-155 257,-137 290.1008,-155 257,-173"/>
<text text-anchor="middle" x="257" y="-152.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cond</text>
</g>
<g class="m-node m-flat">
<title>p0x18786e8</title>
<polygon points="176,-52 122,-52 122,-48 118,-48 118,-44 122,-44 122,-24 118,-24 118,-20 122,-20 122,-16 176,-16 176,-52"/>
<polyline points="122,-48 126,-48 126,-44 122,-44 "/>
<polyline points="122,-24 126,-24 126,-20 122,-20 "/>
<text text-anchor="middle" x="149" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt-0</text>
</g>
<g class="m-edge">
<title>p0x1878600->p0x18786e8</title>
<path stroke-dasharray="5,2" d="M245.8489,-142.5066C228.4613,-123.0261 194.5317,-85.0123 171.8111,-59.5569"/>
<polygon points="174.4206,-57.2243 165.1504,-52.0945 169.1982,-61.8856 174.4206,-57.2243"/>
<text text-anchor="middle" x="209.5" y="-92" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">0</text>
</g>
<g class="m-node m-flat">
<title>p0x18787d0</title>
<polygon points="248,-52 194,-52 194,-48 190,-48 190,-44 194,-44 194,-24 190,-24 190,-20 194,-20 194,-16 248,-16 248,-52"/>
<polyline points="194,-48 198,-48 198,-44 194,-44 "/>
<polyline points="194,-24 198,-24 198,-20 194,-20 "/>
<text text-anchor="middle" x="221" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt-1</text>
</g>
<g class="m-edge">
<title>p0x1878600->p0x18787d0</title>
<path stroke-dasharray="5,2" d="M252.2936,-139.1814C246.4568,-119.5631 236.4064,-85.7825 229.2713,-61.8008"/>
<polygon points="232.6191,-60.7793 226.4127,-52.1927 225.9098,-62.7756 232.6191,-60.7793"/>
<text text-anchor="middle" x="243.5" y="-92" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
</g>
<g class="m-node m-flat">
<title>p0x18788b8</title>
<polygon points="320,-52 266,-52 266,-48 262,-48 262,-44 266,-44 266,-24 262,-24 262,-20 266,-20 266,-16 320,-16 320,-52"/>
<polyline points="266,-48 270,-48 270,-44 266,-44 "/>
<polyline points="266,-24 270,-24 270,-20 266,-20 "/>
<text text-anchor="middle" x="293" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt-2</text>
</g>
<g class="m-edge">
<title>p0x1878600->p0x18788b8</title>
<path stroke-dasharray="5,2" d="M261.7064,-139.1814C267.5432,-119.5631 277.5936,-85.7825 284.7287,-61.8008"/>
<polygon points="288.0902,-62.7756 287.5873,-52.1927 281.3809,-60.7793 288.0902,-62.7756"/>
<text text-anchor="middle" x="279.5" y="-92" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">2</text>
</g>
<g class="m-node m-flat">
<title>p0x18789a0</title>
<polygon points="392,-52 338,-52 338,-48 334,-48 334,-44 338,-44 338,-24 334,-24 334,-20 338,-20 338,-16 392,-16 392,-52"/>
<polyline points="338,-48 342,-48 342,-44 338,-44 "/>
<polyline points="338,-24 342,-24 342,-20 338,-20 "/>
<text text-anchor="middle" x="365" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt-3</text>
</g>
<g class="m-edge">
<title>p0x1878600->p0x18789a0</title>
<path stroke-dasharray="5,2" d="M268.1511,-142.5066C285.5387,-123.0261 319.4683,-85.0123 342.1889,-59.5569"/>
<polygon points="344.8018,-61.8856 348.8496,-52.0945 339.5794,-57.2243 344.8018,-61.8856"/>
<text text-anchor="middle" x="317.5" y="-92" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">3</text>
</g>
</g>
</svg>
</div></section><section id="UnderstandInternalDataStorage"><h2><a href="#UnderstandInternalDataStorage">Understand Internal Data Storage</a></h2><p>By default, <a href="classtf_1_1DataPipeline.html" class="m-doc">tf::<wbr />DataPipeline</a> uses <a href="https://en.cppreference.com/w/cpp/utility/variant">std::<wbr />variant</a> to store a type-safe union of all input and output data types extracted from the given data pipes. To avoid false sharing, each line keeps a variant that is aligned with the cacheline size. When invoking a pipe callable, the input data is acquired in reference from the variant using <a href="https://en.cppreference.com/w/cpp/utility/variant/get">std::<wbr />get</a>. When returning from a pipe callable, the output data is stored back to the variant using assignment operator.</p></section><section id="DataParallelPipelineLearnMore"><h2><a href="#DataParallelPipelineLearnMore">Learn More about Taskflow Pipeline</a></h2><p>Visit the following pages to learn more about pipeline:</p><ol><li><a href="TaskParallelPipeline.html" class="m-doc">Task-parallel Pipeline</a></li><li><a href="TaskParallelScalablePipeline.html" class="m-doc">Task-parallel Scalable Pipeline</a></li><li><a href="TextProcessingPipeline.html" class="m-doc">Text Processing Pipeline</a></li><li><a href="GraphProcessingPipeline.html" class="m-doc">Graph Processing Pipeline</a></li><li><a href="TaskflowProcessingPipeline.html" class="m-doc">Taskflow Processing Pipeline</a></li></ol></section>
</div>
</div>
</div>
</article></main>
<div class="m-doc-search" id="search">
<a href="#!" onclick="return hideSearch()"></a>
<div class="m-container">
<div class="m-row">
<div class="m-col-m-8 m-push-m-2">
<div class="m-doc-search-header m-text m-small">
<div><span class="m-label m-default">Tab</span> / <span class="m-label m-default">T</span> to search, <span class="m-label m-default">Esc</span> to close</div>
<div id="search-symbolcount">…</div>
</div>
<div class="m-doc-search-content">
<form>
<input type="search" name="q" id="search-input" placeholder="Loading …" disabled="disabled" autofocus="autofocus" autocomplete="off" spellcheck="false" />
</form>
<noscript class="m-text m-danger m-text-center">Unlike everything else in the docs, the search functionality <em>requires</em> JavaScript.</noscript>
<div id="search-help" class="m-text m-dim m-text-center">
<p class="m-noindent">Search for symbols, directories, files, pages or
modules. You can omit any prefix from the symbol or file path; adding a
<code>:</code> or <code>/</code> suffix lists all members of given symbol or
directory.</p>
<p class="m-noindent">Use <span class="m-label m-dim">↓</span>
/ <span class="m-label m-dim">↑</span> to navigate through the list,
<span class="m-label m-dim">Enter</span> to go.
<span class="m-label m-dim">Tab</span> autocompletes common prefix, you can
copy a link to the result using <span class="m-label m-dim">⌘</span>
<span class="m-label m-dim">L</span> while <span class="m-label m-dim">⌘</span>
<span class="m-label m-dim">M</span> produces a Markdown link.</p>
</div>
<div id="search-notfound" class="m-text m-warning m-text-center">Sorry, nothing was found.</div>
<ul id="search-results"></ul>
</div>
</div>
</div>
</div>
</div>
<script src="search-v2.js"></script>
<script src="searchdata-v2.js" async="async"></script>
<footer><nav>
<div class="m-container">
<div class="m-row">
<div class="m-col-l-10 m-push-l-1">
<p>Taskflow handbook is part of the <a href="https://taskflow.github.io">Taskflow project</a>, copyright © <a href="https://tsung-wei-huang.github.io/">Dr. Tsung-Wei Huang</a>, 2018–2023.<br />Generated by <a href="https://doxygen.org/">Doxygen</a> 1.8.14 and <a href="https://mcss.mosra.cz/">m.css</a>.</p>
</div>
</div>
</div>
</nav></footer>
</body>
</html>