forked from taskflow/taskflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathDataParallelPipeline.xml
More file actions
153 lines (153 loc) · 23.2 KB
/
Copy pathDataParallelPipeline.xml
File metadata and controls
153 lines (153 loc) · 23.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
<?xml version='1.0' encoding='UTF-8' standalone='no'?>
<doxygen xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="compound.xsd" version="1.12.0" xml:lang="en-US">
<compounddef id="DataParallelPipeline" kind="page">
<compoundname>DataParallelPipeline</compoundname>
<title>Data-parallel Pipeline</title>
<tableofcontents>
<tocsect>
<name>Include the Header</name>
<reference>DataParallelPipeline_1ParallelDataPipelineIncludeHeaderFile</reference>
</tocsect>
<tocsect>
<name>Create a Data Pipeline Module Task</name>
<reference>DataParallelPipeline_1CreateADataPipelineModuleTask</reference>
</tocsect>
<tocsect>
<name>Understand Internal Data Storage</name>
<reference>DataParallelPipeline_1UnderstandInternalDataStorage</reference>
</tocsect>
<tocsect>
<name>Learn More about Taskflow Pipeline</name>
<reference>DataParallelPipeline_1DataParallelPipelineLearnMore</reference>
</tocsect>
</tableofcontents>
<briefdescription>
</briefdescription>
<detaileddescription>
<para>Taskflow provides another variant, <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref>, on top of <ref refid="classtf_1_1Pipeline" kindref="compound">tf::Pipeline</ref> (see <ref refid="TaskParallelPipeline" kindref="compound">Task-parallel Pipeline</ref>) to help you implement data-parallel pipeline algorithms while leaving data management to Taskflow. We recommend you finishing reading TaskParallelPipeline first before learning <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref>.</para>
<sect1 id="DataParallelPipeline_1ParallelDataPipelineIncludeHeaderFile">
<title>Include the Header</title><para>You need to include the header file, <computeroutput>taskflow/algorithm/data_pipeline.hpp</computeroutput>, for implementing data-parallel pipeline algorithms.</para>
<para><programlisting filename=".cpp"><codeline><highlight class="preprocessor">#include<sp/><taskflow/algorithm/data_pipeline.hpp></highlight></codeline>
</programlisting></para>
</sect1>
<sect1 id="DataParallelPipeline_1CreateADataPipelineModuleTask">
<title>Create a Data Pipeline Module Task</title><para>Similar to creating a task-parallel pipeline (<ref refid="classtf_1_1Pipeline" kindref="compound">tf::Pipeline</ref>), there are three steps to create a data-parallel pipeline application:</para>
<para><orderedlist>
<listitem><para>Define the pipeline structure (e.g., pipe type, pipe callable, stopping rule, line count)</para>
</listitem><listitem><para>Define the data storage and layout, if needed for the application</para>
</listitem><listitem><para>Define the pipeline taskflow graph using composition</para>
</listitem></orderedlist>
</para>
<para>The following example creates a data-parallel pipeline that generates a total of five dataflow tokens from <computeroutput>void</computeroutput> to <computeroutput>int</computeroutput> at the first stage, from <computeroutput>int</computeroutput> to <computeroutput>std::string</computeroutput> at the second stage, and <computeroutput>std::string</computeroutput> to <computeroutput>void</computeroutput> at the final stage. Data storage between stages is automatically managed by <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref>.</para>
<para><programlisting filename=".cpp"><codeline><highlight class="preprocessor">#include<sp/><<ref refid="taskflow_8hpp" kindref="compound">taskflow/taskflow.hpp</ref>></highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"></highlight><highlight class="preprocessor">#include<sp/><taskflow/algorithm/data_pipeline.hpp></highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"></highlight><highlight class="keywordtype">int</highlight><highlight class="normal"><sp/>main()<sp/>{</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="comment">//<sp/>data<sp/>flow<sp/>=><sp/>void<sp/>-><sp/>int<sp/>-><sp/>std::string<sp/>-><sp/>void<sp/></highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="classtf_1_1Taskflow" kindref="compound">tf::Taskflow</ref><sp/>taskflow(</highlight><highlight class="stringliteral">"pipeline"</highlight><highlight class="normal">);</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="classtf_1_1Executor" kindref="compound">tf::Executor</ref><sp/>executor;</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="keyword">const</highlight><highlight class="normal"><sp/></highlight><highlight class="keywordtype">size_t</highlight><highlight class="normal"><sp/>num_lines<sp/>=<sp/>4;</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="comment">//<sp/>create<sp/>a<sp/>pipeline<sp/>graph</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref><sp/>pl(num_lines,</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<void, int></ref>(<ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/>[&](<ref refid="classtf_1_1Pipeflow" kindref="compound">tf::Pipeflow</ref>&<sp/>pf)<sp/>-><sp/></highlight><highlight class="keywordtype">int</highlight><highlight class="normal">{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">if</highlight><highlight class="normal">(pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>()<sp/>==<sp/>5)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><sp/><sp/>pf.<ref refid="classtf_1_1Pipeflow_1a830b7f204cb87fff17e8d424918d9453" kindref="member">stop</ref>();</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/>0;</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">else</highlight><highlight class="normal"><sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><sp/><sp/>printf(</highlight><highlight class="stringliteral">"first<sp/>pipe<sp/>returns<sp/>%lu\n"</highlight><highlight class="normal">,<sp/>pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>());</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/>pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>();</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/>}),</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<int, std::string></ref>(<ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/>[](</highlight><highlight class="keywordtype">int</highlight><highlight class="normal">&<sp/>input)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><ref refid="cpp/io/c/fprintf" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">printf</ref>(</highlight><highlight class="stringliteral">"second<sp/>pipe<sp/>returns<sp/>a<sp/>string<sp/>of<sp/>%d\n"</highlight><highlight class="normal">,<sp/>input<sp/>+<sp/>100);</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/><ref refid="cpp/string/basic_string/to_string" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::to_string</ref>(input<sp/>+<sp/>100);</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/>}),</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<std::string, void></ref>(<ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/>[](<ref refid="cpp/string/basic_string" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::string</ref>&<sp/>input)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/><sp/><sp/><ref refid="cpp/io/c/fprintf" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">printf</ref>(</highlight><highlight class="stringliteral">"third<sp/>pipe<sp/>receives<sp/>the<sp/>input<sp/>string<sp/>%s\n"</highlight><highlight class="normal">,<sp/>input.c_str());</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/>})</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>);</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="comment">//<sp/>build<sp/>the<sp/>pipeline<sp/>graph<sp/>using<sp/>composition</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>taskflow.composed_of(pl).name(</highlight><highlight class="stringliteral">"pipeline"</highlight><highlight class="normal">);</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="comment">//<sp/>dump<sp/>the<sp/>pipeline<sp/>graph<sp/>structure<sp/>(with<sp/>composition)</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>taskflow.dump(<ref refid="cpp/io/basic_ostream" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::cout</ref>);</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="comment">//<sp/>run<sp/>the<sp/>pipeline</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>executor.<ref refid="classtf_1_1Executor_1a519777f5783981d534e9e53b99712069" kindref="member">run</ref>(taskflow).wait();</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/>0;</highlight></codeline>
<codeline><highlight class="normal">}</highlight></codeline>
</programlisting></para>
<para>The interface of <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref> is very similar to <ref refid="classtf_1_1Pipeline" kindref="compound">tf::Pipeline</ref>, except that the library transparently manages the dataflow between pipes. To create a stage in a data-parallel pipeline, you should always use the helper function <ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe</ref>:</para>
<para><programlisting filename=".cpp"><codeline><highlight class="normal"><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<int, std::string></ref>(</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>[](</highlight><highlight class="keywordtype">int</highlight><highlight class="normal">&<sp/>input)<sp/>{<sp/></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/><ref refid="cpp/string/basic_string/to_string" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::to_string</ref>(input<sp/>+<sp/>100);</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal">);</highlight></codeline>
</programlisting></para>
<para>The helper function starts with a pair of an input and an output types in its template arguments. Both types will always be decayed to their original form using <ref refid="cpp/types/decay" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::decay</ref> (e.g., <computeroutput>const int&</computeroutput> becomes <computeroutput>int</computeroutput>) for storage purpose. In terms of function arguments, the first argument specifies the direction of this data pipe, which can be either <ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref> or <ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564adf13a99b035d6f0bce4f44ab18eec8eb" kindref="member">tf::PipeType::PARALLEL</ref>, and the second argument is a callable to invoke by the pipeline scheduler. The callable must take the input data type in its first argument and returns a value of the output data type. Additionally, the callable can take a <ref refid="classtf_1_1Pipeflow" kindref="compound">tf::Pipeflow</ref> reference in its second argument which allows you to query the runtime information of a stage task, such as its line number and token number.</para>
<para><programlisting filename=".cpp"><codeline><highlight class="normal"><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<int, std::string></ref>(</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>[](</highlight><highlight class="keywordtype">int</highlight><highlight class="normal">&<sp/>input,<sp/><ref refid="classtf_1_1Pipeflow" kindref="compound">tf::Pipeflow</ref>&<sp/>pf)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/>printf(</highlight><highlight class="stringliteral">"token=%lu,<sp/>line=%lu\n"</highlight><highlight class="normal">,<sp/>pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>(),<sp/>pf.<ref refid="classtf_1_1Pipeflow_1afee054e6a99965d4b3e36ff903227e6c" kindref="member">line</ref>());</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/><ref refid="cpp/string/basic_string/to_string" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::to_string</ref>(input<sp/>+<sp/>100);</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal">)</highlight></codeline>
</programlisting></para>
<para><simplesect kind="attention"><para>By default, <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref> passes the data in reference to your callable at which you can take it in copy or in reference depending on application needs.</para>
</simplesect>
For the first pipe, the input type should always be <computeroutput>void</computeroutput> and the callable must take a <ref refid="classtf_1_1Pipeflow" kindref="compound">tf::Pipeflow</ref> reference in its argument. In this example, we will stop the pipeline when processing five tokens.</para>
<para><programlisting filename=".cpp"><codeline><highlight class="normal"><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<void, int></ref>(<ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/>[](<ref refid="classtf_1_1Pipeflow" kindref="compound">tf::Pipeflow</ref>&<sp/>pf)<sp/>-><sp/></highlight><highlight class="keywordtype">int</highlight><highlight class="normal">{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="keywordflow">if</highlight><highlight class="normal">(pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>()<sp/>==<sp/>5)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/>pf.<ref refid="classtf_1_1Pipeflow_1a830b7f204cb87fff17e8d424918d9453" kindref="member">stop</ref>();</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/>0;<sp/><sp/><sp/><sp/></highlight><highlight class="comment">//<sp/>returns<sp/>a<sp/>dummy<sp/>value</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/></highlight><highlight class="keywordflow">else</highlight><highlight class="normal"><sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><sp/><sp/></highlight><highlight class="keywordflow">return</highlight><highlight class="normal"><sp/>pf.<ref refid="classtf_1_1Pipeflow_1a295e5d884665c076f4ef5d78139f7c51" kindref="member">token</ref>();</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/>}</highlight></codeline>
<codeline><highlight class="normal">}),</highlight></codeline>
</programlisting></para>
<para>Similarly, the output type of the last pipe should be <computeroutput>void</computeroutput> as no more data will go out of the final pipe.</para>
<para><programlisting filename=".cpp"><codeline><highlight class="normal"><ref refid="namespacetf_1a8975fa5762088789adb0b60f38208309" kindref="member">tf::make_data_pipe<std::string, void></ref>(<ref refid="namespacetf_1abb7a11e41fd457f69e7ff45d4c769564a7b804a28d6154ab8007287532037f1d0" kindref="member">tf::PipeType::SERIAL</ref>,<sp/>[](<ref refid="cpp/string/basic_string" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::string</ref>&<sp/>input)<sp/>{</highlight></codeline>
<codeline><highlight class="normal"><sp/><sp/><ref refid="cpp/io/basic_ostream" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::cout</ref><sp/><<<sp/>input<sp/><<<sp/><ref refid="cpp/io/manip/endl" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::endl</ref>;</highlight></codeline>
<codeline><highlight class="normal">})</highlight></codeline>
</programlisting></para>
<para>Finally, you need to compose the pipeline graph by creating a module task (i.e., tf::Taskflow::compoased_of).</para>
<para><programlisting filename=".cpp"><codeline><highlight class="comment">//<sp/>build<sp/>the<sp/>pipeline<sp/>graph<sp/>using<sp/>composition</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal">taskflow.composed_of(pl).name(</highlight><highlight class="stringliteral">"pipeline"</highlight><highlight class="normal">);</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"></highlight><highlight class="comment">//<sp/>dump<sp/>the<sp/>pipeline<sp/>graph<sp/>structure<sp/>(with<sp/>composition)</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal">taskflow.dump(<ref refid="cpp/io/basic_ostream" kindref="compound" external="/Users/twhuang/Code/taskflow/doxygen/cppreference-doxygen-web.tag.xml">std::cout</ref>);</highlight></codeline>
<codeline><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal"></highlight><highlight class="comment">//<sp/>run<sp/>the<sp/>pipeline</highlight><highlight class="normal"></highlight></codeline>
<codeline><highlight class="normal">executor.<ref refid="classtf_1_1Executor_1a519777f5783981d534e9e53b99712069" kindref="member">run</ref>(taskflow).wait();</highlight></codeline>
</programlisting></para>
<para><dotfile name="pipeline_basic_dependency_graph.dot"></dotfile>
</para>
</sect1>
<sect1 id="DataParallelPipeline_1UnderstandInternalDataStorage">
<title>Understand Internal Data Storage</title><para>By default, <ref refid="classtf_1_1DataPipeline" kindref="compound">tf::DataPipeline</ref> uses <ulink url="https://en.cppreference.com/w/cpp/utility/variant">std::variant</ulink> to store a type-safe union of all input and output data types extracted from the given data pipes. To avoid false sharing, each line keeps a variant that is aligned with the cacheline size. When invoking a pipe callable, the input data is acquired in reference from the variant using <ulink url="https://en.cppreference.com/w/cpp/utility/variant/get">std::get</ulink>. When returning from a pipe callable, the output data is stored back to the variant using assignment operator.</para>
</sect1>
<sect1 id="DataParallelPipeline_1DataParallelPipelineLearnMore">
<title>Learn More about Taskflow Pipeline</title><para>Visit the following pages to learn more about pipeline:</para>
<para><orderedlist>
<listitem><para><ref refid="TaskParallelPipeline" kindref="compound">Task-parallel Pipeline</ref></para>
</listitem><listitem><para><ref refid="TaskParallelScalablePipeline" kindref="compound">Task-parallel Scalable Pipeline</ref></para>
</listitem><listitem><para><ref refid="TextProcessingPipeline" kindref="compound">Text Processing Pipeline</ref></para>
</listitem><listitem><para><ref refid="GraphProcessingPipeline" kindref="compound">Graph Processing Pipeline</ref></para>
</listitem><listitem><para><ref refid="TaskflowProcessingPipeline" kindref="compound">Taskflow Processing Pipeline</ref> </para>
</listitem></orderedlist>
</para>
</sect1>
</detaileddescription>
<location file="doxygen/algorithms/data_pipeline.dox"/>
</compounddef>
</doxygen>