---

title: Title

keywords: fastai

sidebar: home_sidebar

summary: "summary"

description: "summary"

---

<!--

#################################################

### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###

#################################################

# file to edit: 06_Distributed_ML.ipynb

# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="o">%</span><span class="k">load_ext</span> autotime

<span class="o">%</span><span class="k">load_ext</span> autoreload

<span class="o">%</span><span class="k">autoreload</span> 2

</pre></div>

</div>

</div>

</div>

</div>

{% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">

<div class="text_cell_render border-box-sizing rendered_html">

<h1 id="Distributed-Machine-Learning">Distributed Machine Learning<a class="anchor-link" href="#Distributed-Machine-Learning"> </a></h1><p>Eines der grössten Schwierigkeiten ist es passende ML Algorithmen zu finden. Die meisten Algorithmen werden aus Performancegründen in C, C++ oder ähnlichne Sprachen geschrieben. R und Python sind für die Entwicklung von Algorithmen ungeeignet. Um dennoch aufzuzigen, dass Federeated Learning einen Mehrwert bringt werden wir ein Cluster von Docker Containern aufbauen. Dieser Cluster besteht aus 2 Worker Nodes und einem Master Node. Gesteuert werden diese mit <a href="https://www.open-mpi.org/">Open MPI</a> (Open Source High Performance Computing - Library). MPI wird von XGBoost als auch von Microsofts LightGBM unterstützt. Letzteres verwenden wir an dieser Stelle, da die Implementierung sich weniger Komplex gestaltet.</p>

</div>

</div>

</div>

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">

<div class="text_cell_render border-box-sizing rendered_html">

<h2 id="LightGBM">LightGBM<a class="anchor-link" href="#LightGBM"> </a></h2><p><a href="https://github.com/microsoft/LightGBM/blob/master/docs/Parallel-Learning-Guide.rst">LigthGBM Distributed Learning</a></p>

</div>

</div>

</div>

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">

<div class="text_cell_render border-box-sizing rendered_html">

<h2 id="Data-Preparation">Data Preparation<a class="anchor-link" href="#Data-Preparation"> </a></h2>

</div>

</div>

</div>

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">data</span> <span class="o">=</span> <span class="n">get_two_region_data</span><span class="p">()</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>load data

prepare data

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">data</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_text output_subarea output_execute_result">

<pre>dict_keys([&#39;product_dict&#39;, &#39;product_reverse_dict&#39;, &#39;feature_cols&#39;, &#39;target_cols&#39;, &#39;train&#39;, &#39;test&#39;, &#39;train_south&#39;, &#39;train_north&#39;, &#39;train_X_south&#39;, &#39;train_y_south&#39;, &#39;train_X_north&#39;, &#39;train_y_north&#39;, &#39;train_X&#39;, &#39;train_y&#39;, &#39;test_south&#39;, &#39;test_north&#39;])</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#we prepare the data in one csv file where y is the label column</span>

<span class="n">export_cols</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s1">&#39;feature_cols&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>

<span class="n">export_cols</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">&#39;y&#39;</span><span class="p">]</span>

<span class="n">train</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s1">&#39;train&#39;</span><span class="p">]</span>

<span class="n">train_south</span> <span class="o">=</span> <span class="n">train</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">train</span><span class="o">.</span><span class="n">region</span> <span class="o">==</span> <span class="s1">&#39;south&#39;</span><span class="p">,]</span>

<span class="n">train_north</span> <span class="o">=</span> <span class="n">train</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="n">train</span><span class="o">.</span><span class="n">region</span> <span class="o">==</span> <span class="s1">&#39;north&#39;</span><span class="p">,]</span>

<span class="n">train</span><span class="p">[</span><span class="n">export_cols</span><span class="p">]</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s1">&#39;data/final/data_lightgbm/train.csv&#39;</span><span class="p">,</span> <span class="n">index</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span>

<span class="n">train_south</span><span class="p">[</span><span class="n">export_cols</span><span class="p">]</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s1">&#39;data/final/data_lightgbm/train_south.csv&#39;</span><span class="p">,</span> <span class="n">index</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span>

<span class="n">train_north</span><span class="p">[</span><span class="n">export_cols</span><span class="p">]</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s1">&#39;data/final/data_lightgbm/train_north.csv&#39;</span><span class="p">,</span> <span class="n">index</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>time: 3.57 s

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">valid</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s1">&#39;data/final/data_lightgbm/validation.csv&#39;</span><span class="p">)</span>

<span class="n">valid</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">encode_products</span><span class="p">(</span><span class="n">valid</span><span class="p">)</span>

<span class="n">valid</span><span class="p">[</span><span class="n">export_cols</span><span class="p">]</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s1">&#39;data/final/data_lightgbm/validation_enc.csv&#39;</span><span class="p">,</span> <span class="n">index</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>time: 176 ms

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">

<div class="text_cell_render border-box-sizing rendered_html">

<h2 id="train_community.conf">train_community.conf<a class="anchor-link" href="#train_community.conf"> </a></h2>

<pre><code>task = train

valid_data = validation_enc.csv

boosting_type = gbdt

objective = multiclass

num_class = 22

eta = 0.05

min_child_weight = 1

#subsample = 0.7

subsample = 1

colsample_bytree = 0.7

max_depth = 8

metric = multi_logloss

metric_freq = 1

is_training_metric = true

num_trees = 50

data = train.csv

header = true

label_column = name:y

machine_list_file = mlist.txt

num_machines = 2

tree_learner = data

seed = 0

#num_threads = 1

output_model = model.txt</code></pre>

<h2 id="run-cluster-training">run cluster training<a class="anchor-link" href="#run-cluster-training"> </a></h2>

<pre><code>mpiexec --machinefile mlist.txt -npernode 1 ./lightgbm config=train.conf</code></pre>

</div>

</div>

</div>

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">train_org</span><span class="p">,</span> <span class="n">test</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">load_data</span><span class="p">()</span>

<span class="n">product_dict</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">get_product_dict</span><span class="p">(</span><span class="n">train_org</span><span class="p">)</span>

<span class="n">product_reverse_dict</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">get_product_reverse_dict</span><span class="p">(</span><span class="n">train_org</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">data</span> <span class="o">=</span> <span class="n">get_two_region_data</span><span class="p">()</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>load data

prepare data

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">bst</span> <span class="o">=</span> <span class="n">lgb</span><span class="o">.</span><span class="n">Booster</span><span class="p">(</span><span class="n">model_file</span><span class="o">=</span><span class="s1">&#39;data/final/data_lightgbm/model.txt&#39;</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">bst</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">&#39;test&#39;</span><span class="p">][</span><span class="n">base_model</span><span class="o">.</span><span class="n">feature_cols</span><span class="p">])</span>

</pre></div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argsort</span><span class="p">(</span><span class="n">preds</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">fliplr</span><span class="p">(</span><span class="n">preds</span><span class="p">)[:,:</span><span class="mi">7</span><span class="p">]</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">preds</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">applymap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">product_reverse_dict</span><span class="p">[</span><span class="n">x</span><span class="p">])</span>

<span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span>

<span class="n">test_data</span> <span class="o">=</span> <span class="n">test</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>

<span class="n">target_cols</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">target_cols</span>

<span class="n">test_data</span><span class="o">.</span><span class="n">reset_index</span><span class="p">(</span><span class="n">inplace</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="p">[</span><span class="n">target_cols</span><span class="p">]</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">compress</span><span class="p">(</span><span class="n">target_cols</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">)),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">base_model</span><span class="o">.</span><span class="n">apk</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]),</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;mean average precision single node all data = </span><span class="si">{</span><span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>mean average precision single node all data = 0.022738471706301475

time: 1min 2s

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">bst</span> <span class="o">=</span> <span class="n">lgb</span><span class="o">.</span><span class="n">Booster</span><span class="p">(</span><span class="n">model_file</span><span class="o">=</span><span class="s1">&#39;data/final/data_lightgbm/model_north.txt&#39;</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">bst</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">test</span><span class="p">[</span><span class="n">base_model</span><span class="o">.</span><span class="n">feature_cols</span><span class="p">])</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argsort</span><span class="p">(</span><span class="n">preds</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">fliplr</span><span class="p">(</span><span class="n">preds</span><span class="p">)[:,:</span><span class="mi">7</span><span class="p">]</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">preds</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">applymap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">product_reverse_dict</span><span class="p">[</span><span class="n">x</span><span class="p">])</span>

<span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span>

<span class="n">test_data</span> <span class="o">=</span> <span class="n">test</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>

<span class="n">target_cols</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">target_cols</span>

<span class="n">test_data</span><span class="o">.</span><span class="n">reset_index</span><span class="p">(</span><span class="n">inplace</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="p">[</span><span class="n">target_cols</span><span class="p">]</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">compress</span><span class="p">(</span><span class="n">target_cols</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">)),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">base_model</span><span class="o">.</span><span class="n">apk</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]),</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;mean average precision single node north = </span><span class="si">{</span><span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>mean average precision single node north = 0.022714324920308386

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">bst</span> <span class="o">=</span> <span class="n">lgb</span><span class="o">.</span><span class="n">Booster</span><span class="p">(</span><span class="n">model_file</span><span class="o">=</span><span class="s1">&#39;data/final/data_lightgbm/model_south.txt&#39;</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">bst</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">test</span><span class="p">[</span><span class="n">base_model</span><span class="o">.</span><span class="n">feature_cols</span><span class="p">])</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">argsort</span><span class="p">(</span><span class="n">preds</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">fliplr</span><span class="p">(</span><span class="n">preds</span><span class="p">)[:,:</span><span class="mi">7</span><span class="p">]</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">preds</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">applymap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">product_reverse_dict</span><span class="p">[</span><span class="n">x</span><span class="p">])</span>

<span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">preds</span> <span class="o">=</span> <span class="n">preds</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span>

<span class="n">test_data</span> <span class="o">=</span> <span class="n">test</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>

<span class="n">target_cols</span> <span class="o">=</span> <span class="n">base_model</span><span class="o">.</span><span class="n">target_cols</span>

<span class="n">test_data</span><span class="o">.</span><span class="n">reset_index</span><span class="p">(</span><span class="n">inplace</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">preds</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="p">[</span><span class="n">target_cols</span><span class="p">]</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">list</span><span class="p">(</span><span class="n">compress</span><span class="p">(</span><span class="n">target_cols</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">values</span><span class="p">)),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">test_data</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">base_model</span><span class="o">.</span><span class="n">apk</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s1">&#39;truth_list&#39;</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="s1">&#39;added_products&#39;</span><span class="p">]),</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;mean average precision single node south = </span><span class="si">{</span><span class="n">test_data</span><span class="p">[</span><span class="s1">&#39;apk&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stdout output_text">

<pre>mean average precision single node south = 0.02279485233961093

</pre>

</div>

</div>

</div>

</div>

</div>

{% endraw %}

{% raw %}

<div class="cell border-box-sizing code_cell rendered">

<div class="input">

<div class="inner_cell">

<div class="input_area">

<div class=" highlight hl-ipython3"><pre><span></span><span class="n">data</span>

</pre></div>

</div>

</div>

</div>

<div class="output_wrapper">

<div class="output">

<div class="output_area">

<div class="output_text output_subarea output_execute_result">

<pre>{&#39;product_dict&#39;: {&#39;ind_cco_fin_ult1&#39;: 0,

&#39;ind_cder_fin_ult1&#39;: 1,

&#39;ind_cno_fin_ult1&#39;: 2,

&#39;ind_ctju_fin_ult1&#39;: 3,

&#39;ind_ctma_fin_ult1&#39;: 4,

&#39;ind_ctop_fin_ult1&#39;: 5,

&#39;ind_ctpp_fin_ult1&#39;: 6,

&#39;ind_deco_fin_ult1&#39;: 7,

&#39;ind_dela_fin_ult1&#39;: 8,

&#39;ind_deme_fin_ult1&#39;: 9,

&#39;ind_ecue_fin_ult1&#39;: 10,

&#39;ind_fond_fin_ult1&#39;: 11,

&#39;ind_hip_fin_ult1&#39;: 12,

&#39;ind_nom_pens_ult1&#39;: 13,

&#39;ind_nomina_ult1&#39;: 14,

&#39;ind_plan_fin_ult1&#39;: 15,

&#39;ind_pres_fin_ult1&#39;: 16,

&#39;ind_reca_fin_ult1&#39;: 17,

&#39;ind_recibo_ult1&#39;: 18,

&#39;ind_tjcr_fin_ult1&#39;: 19,

&#39;ind_valo_fin_ult1&#39;: 20,

&#39;ind_viv_fin_ult1&#39;: 21},

&#39;product_reverse_dict&#39;: {0: &#39;ind_cco_fin_ult1&#39;,

1: &#39;ind_cder_fin_ult1&#39;,

2: &#39;ind_cno_fin_ult1&#39;,

3: &#39;ind_ctju_fin_ult1&#39;,

4: &#39;ind_ctma_fin_ult1&#39;,

5: &#39;ind_ctop_fin_ult1&#39;,

6: &#39;ind_ctpp_fin_ult1&#39;,

7: &#39;ind_deco_fin_ult1&#39;,

8: &#39;ind_dela_fin_ult1&#39;,

9: &#39;ind_deme_fin_ult1&#39;,

10: &#39;ind_ecue_fin_ult1&#39;,

11: &#39;ind_fond_fin_ult1&#39;,

12: &#39;ind_hip_fin_ult1&#39;,

13: &#39;ind_nom_pens_ult1&#39;,

14: &#39;ind_nomina_ult1&#39;,

15: &#39;ind_plan_fin_ult1&#39;,

16: &#39;ind_pres_fin_ult1&#39;,

17: &#39;ind_reca_fin_ult1&#39;,

18: &#39;ind_recibo_ult1&#39;,

19: &#39;ind_tjcr_fin_ult1&#39;,

20: &#39;ind_valo_fin_ult1&#39;,

21: &#39;ind_viv_fin_ult1&#39;},

&#39;feature_cols&#39;: [&#39;ind_empleado&#39;,

&#39;sexo&#39;,

&#39;age&#39;,

&#39;renta&#39;,

&#39;ind_nuevo&#39;,

&#39;indrel&#39;,

&#39;indrel_1mes&#39;,

&#39;tiprel_1mes&#39;,

&#39;indresi&#39;,

&#39;indext&#39;,

&#39;conyuemp&#39;,

&#39;indfall&#39;,

&#39;tipodom&#39;,

&#39;ind_actividad_cliente&#39;,

&#39;segmento&#39;,

&#39;antiguedad&#39;,

&#39;pais_residencia&#39;,

&#39;canal_entrada&#39;,

&#39;ind_cco_fin_ult1_s&#39;,

&#39;ind_cder_fin_ult1_s&#39;,

&#39;ind_cno_fin_ult1_s&#39;,

&#39;ind_ctju_fin_ult1_s&#39;,

&#39;ind_ctma_fin_ult1_s&#39;,

&#39;ind_ctop_fin_ult1_s&#39;,

&#39;ind_ctpp_fin_ult1_s&#39;,

&#39;ind_deco_fin_ult1_s&#39;,

&#39;ind_deme_fin_ult1_s&#39;,

&#39;ind_dela_fin_ult1_s&#39;,

&#39;ind_ecue_fin_ult1_s&#39;,

&#39;ind_fond_fin_ult1_s&#39;,

&#39;ind_hip_fin_ult1_s&#39;,

&#39;ind_plan_fin_ult1_s&#39;,

&#39;ind_pres_fin_ult1_s&#39;,

&#39;ind_reca_fin_ult1_s&#39;,

&#39;ind_tjcr_fin_ult1_s&#39;,

&#39;ind_valo_fin_ult1_s&#39;,

&#39;ind_viv_fin_ult1_s&#39;,

&#39;ind_nomina_ult1_s&#39;,

&#39;ind_nom_pens_ult1_s&#39;,

&#39;ind_recibo_ult1_s&#39;],

&#39;target_cols&#39;: [&#39;ind_ahor_fin_ult1&#39;,

&#39;ind_aval_fin_ult1&#39;,

&#39;ind_cco_fin_ult1&#39;,

&#39;ind_cder_fin_ult1&#39;,

&#39;ind_cno_fin_ult1&#39;,

&#39;ind_ctju_fin_ult1&#39;,

&#39;ind_ctma_fin_ult1&#39;,

&#39;ind_ctop_fin_ult1&#39;,

&#39;ind_ctpp_fin_ult1&#39;,

&#39;ind_deco_fin_ult1&#39;,

&#39;ind_deme_fin_ult1&#39;,

&#39;ind_dela_fin_ult1&#39;,

&#39;ind_ecue_fin_ult1&#39;,

&#39;ind_fond_fin_ult1&#39;,

&#39;ind_hip_fin_ult1&#39;,

&#39;ind_plan_fin_ult1&#39;,

&#39;ind_pres_fin_ult1&#39;,

&#39;ind_reca_fin_ult1&#39;,

&#39;ind_tjcr_fin_ult1&#39;,

&#39;ind_valo_fin_ult1&#39;,

&#39;ind_viv_fin_ult1&#39;,

&#39;ind_nomina_ult1&#39;,

&#39;ind_nom_pens_ult1&#39;,

&#39;ind_recibo_ult1&#39;],

&#39;train&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

0 3 0 56 326124 0 1 1

1 3 0 56 326124 0 1 1

2 3 0 56 326124 0 1 1

3 3 1 61 430477 0 1 1

4 3 1 61 430477 0 1 1

... ... ... ... ... ... ... ...

421949 0 0 45 75445 1 1 1

421950 0 0 45 75445 1 1 1

421951 0 0 45 75445 1 1 1

421952 0 0 45 75445 1 1 1

421953 0 0 45 75445 1 1 1

tiprel_1mes indresi indext ... ind_valo_fin_ult1_s \

0 0 1 0 ... 1

1 0 1 0 ... 1

2 0 1 0 ... 1

3 0 1 0 ... 1

4 0 1 0 ... 1

... ... ... ... ... ...

421949 0 1 0 ... 0

421950 0 1 0 ... 0

421951 0 1 0 ... 0

421952 0 1 0 ... 0

421953 0 1 0 ... 0

ind_viv_fin_ult1_s ind_nomina_ult1_s ind_nom_pens_ult1_s \

0 0 0 0

1 0 0 0

2 0 0 0

3 0 0 0

4 0 0 0

... ... ... ...

421949 0 0 0

421950 0 0 0

421951 0 0 0

421952 0 0 0

421953 0 0 0

ind_recibo_ult1_s month_int id y nomprov region

0 0 5 15889 19 MADRID south

1 0 12 15889 19 MADRID south

2 0 15 15889 19 MADRID south

3 1 6 15892 0 MADRID south

4 1 9 15892 8 MADRID south

... ... ... ... .. ... ...

421949 0 11 1454615 2 BARCELONA north

421950 0 12 1454615 14 BARCELONA north

421951 0 12 1454615 13 BARCELONA north

421952 0 14 1454615 14 BARCELONA north

421953 0 14 1454615 13 BARCELONA north

[421954 rows x 45 columns],

&#39;test&#39;: id ind_empleado pais_residencia sexo age ind_nuevo \

0 15889 3 0 0 56 0

1 15890 1 0 0 63 0

2 15892 3 0 1 62 0

3 15893 0 0 0 63 0

4 15894 1 0 0 60 0

... ... ... ... ... ... ...

702430 1454615 0 0 0 46 0

702431 1454616 0 0 1 21 0

702432 1454617 0 0 1 21 0

702433 1454618 0 0 0 20 0

702434 1454620 0 0 0 20 0

antiguedad indrel indrel_1mes tiprel_1mes ... \

0 255 1 1 0 ...

1 256 1 1 0 ...

2 256 1 1 0 ...

3 256 1 1 0 ...

4 256 1 1 0 ...

... ... ... ... ... ...

702430 8 1 1 0 ...

702431 8 1 1 0 ...

702432 8 1 1 0 ...

702433 8 1 1 1 ...

702434 8 1 1 0 ...

ind_pres_fin_ult1_s ind_reca_fin_ult1_s ind_tjcr_fin_ult1_s \

0 0 0 0

1 0 0 1

2 0 1 1

3 0 0 0

4 0 1 1

... ... ... ...

702430 0 0 0

702431 0 0 0

702432 0 0 0

702433 0 0 0

702434 0 0 0

ind_valo_fin_ult1_s ind_viv_fin_ult1_s ind_nomina_ult1_s \

0 1 0 0

1 0 0 1

2 1 0 0

3 1 0 0

4 1 0 1

... ... ... ...

702430 0 0 0

702431 0 0 0

702432 0 0 0

702433 0 0 0

702434 0 0 0

ind_nom_pens_ult1_s ind_recibo_ult1_s nomprov region

0 0 0 MADRID south

1 1 1 MADRID south

2 0 1 MADRID south

3 0 0 MADRID south

4 1 1 MADRID south

... ... ... ... ...

702430 0 0 BARCELONA north

702431 0 0 BURGOS north

702432 0 0 CADIZ south

702433 0 0 MADRID south

702434 0 0 CACERES north

[702435 rows x 81 columns],

&#39;train_south&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

0 3 0 56 326124 0 1 1

1 3 0 56 326124 0 1 1

2 3 0 56 326124 0 1 1

3 3 1 61 430477 0 1 1

4 3 1 61 430477 0 1 1

... ... ... ... ... ... ... ...

421937 0 1 28 118461 1 1 1

421938 0 1 28 118461 1 1 1

421939 0 0 41 52101 1 1 1

421942 0 0 60 37612 1 1 1

421947 0 0 20 175851 1 1 1

tiprel_1mes indresi indext ... ind_valo_fin_ult1_s \

0 0 1 0 ... 1

1 0 1 0 ... 1

2 0 1 0 ... 1

3 0 1 0 ... 1

4 0 1 0 ... 1

... ... ... ... ... ...

421937 0 1 1 ... 0

421938 0 1 1 ... 0

421939 0 1 0 ... 0

421942 0 1 0 ... 0

421947 0 1 0 ... 0

ind_viv_fin_ult1_s ind_nomina_ult1_s ind_nom_pens_ult1_s \

0 0 0 0

1 0 0 0

2 0 0 0

3 0 0 0

4 0 0 0

... ... ... ...

421937 0 0 0

421938 0 0 0

421939 0 0 0

421942 0 0 0

421947 0 0 0

ind_recibo_ult1_s month_int id y nomprov region

0 0 5 15889 19 MADRID south

1 0 12 15889 19 MADRID south

2 0 15 15889 19 MADRID south

3 1 6 15892 0 MADRID south

4 1 9 15892 8 MADRID south

... ... ... ... .. ... ...

421937 1 14 1454555 14 MADRID south

421938 1 14 1454555 13 MADRID south

421939 0 11 1454583 18 MURCIA south

421942 0 10 1454606 18 SEVILLA south

421947 0 10 1454608 0 MADRID south

[278334 rows x 45 columns],

&#39;train_north&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

40 2 0 52 130903 0 1 1

41 2 0 52 130903 0 1 1

186 0 0 57 215210 0 1 1

188 0 1 68 195763 0 1 1

189 0 1 68 195763 0 1 1

... ... ... ... ... ... ... ...

421949 0 0 45 75445 1 1 1

421950 0 0 45 75445 1 1 1

421951 0 0 45 75445 1 1 1

421952 0 0 45 75445 1 1 1

421953 0 0 45 75445 1 1 1

tiprel_1mes indresi indext ... ind_valo_fin_ult1_s \

40 0 1 0 ... 0

41 0 1 0 ... 0

186 0 1 0 ... 0

188 0 1 0 ... 1

189 0 1 0 ... 1

... ... ... ... ... ...

421949 0 1 0 ... 0

421950 0 1 0 ... 0

421951 0 1 0 ... 0

421952 0 1 0 ... 0

421953 0 1 0 ... 0

ind_viv_fin_ult1_s ind_nomina_ult1_s ind_nom_pens_ult1_s \

40 0 0 0

41 0 0 0

186 0 0 0

188 0 0 0

189 0 0 0

... ... ... ...

421949 0 0 0

421950 0 0 0

421951 0 0 0

421952 0 0 0

421953 0 0 0

ind_recibo_ult1_s month_int id y nomprov region

40 1 2 15924 19 PALMAS, LAS north

41 1 7 15924 19 PALMAS, LAS north

186 0 2 16162 0 BARCELONA north

188 1 6 16187 14 BARCELONA north

189 1 6 16187 13 BARCELONA north

... ... ... ... .. ... ...

421949 0 11 1454615 2 BARCELONA north

421950 0 12 1454615 14 BARCELONA north

421951 0 12 1454615 13 BARCELONA north

421952 0 14 1454615 14 BARCELONA north

421953 0 14 1454615 13 BARCELONA north

[143620 rows x 45 columns],

&#39;train_X_south&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

0 3 0 56 326124 0 1 1

1 3 0 56 326124 0 1 1

2 3 0 56 326124 0 1 1

3 3 1 61 430477 0 1 1

4 3 1 61 430477 0 1 1

... ... ... ... ... ... ... ...

421937 0 1 28 118461 1 1 1

421938 0 1 28 118461 1 1 1

421939 0 0 41 52101 1 1 1

421942 0 0 60 37612 1 1 1

421947 0 0 20 175851 1 1 1

tiprel_1mes indresi indext ... ind_tjcr_fin_ult1_s \

0 0 1 0 ... 0

1 0 1 0 ... 0

2 0 1 0 ... 0

3 0 1 0 ... 1

4 0 1 0 ... 1

... ... ... ... ... ...

421937 0 1 1 ... 0

421938 0 1 1 ... 0

421939 0 1 0 ... 0

421942 0 1 0 ... 0

421947 0 1 0 ... 0

ind_valo_fin_ult1_s ind_viv_fin_ult1_s ind_nomina_ult1_s \

0 1 0 0

1 1 0 0

2 1 0 0

3 1 0 0

4 1 0 0

... ... ... ...

421937 0 0 0

421938 0 0 0

421939 0 0 0

421942 0 0 0

421947 0 0 0

ind_nom_pens_ult1_s ind_recibo_ult1_s month_int id nomprov \

0 0 0 5 15889 MADRID

1 0 0 12 15889 MADRID

2 0 0 15 15889 MADRID

3 0 1 6 15892 MADRID

4 0 1 9 15892 MADRID

... ... ... ... ... ...

421937 0 1 14 1454555 MADRID

421938 0 1 14 1454555 MADRID

421939 0 0 11 1454583 MURCIA

421942 0 0 10 1454606 SEVILLA

421947 0 0 10 1454608 MADRID

region

0 south

1 south

2 south

3 south

4 south

... ...

421937 south

421938 south

421939 south

421942 south

421947 south

[278334 rows x 44 columns],

&#39;train_y_south&#39;: 0 19

1 19

2 19

3 0

4 8

..

421937 14

421938 13

421939 18

421942 18

421947 0

Name: y, Length: 278334, dtype: int8,

&#39;train_X_north&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

40 2 0 52 130903 0 1 1

41 2 0 52 130903 0 1 1

186 0 0 57 215210 0 1 1

188 0 1 68 195763 0 1 1

189 0 1 68 195763 0 1 1

... ... ... ... ... ... ... ...

421949 0 0 45 75445 1 1 1

421950 0 0 45 75445 1 1 1

421951 0 0 45 75445 1 1 1

421952 0 0 45 75445 1 1 1

421953 0 0 45 75445 1 1 1

tiprel_1mes indresi indext ... ind_tjcr_fin_ult1_s \

40 0 1 0 ... 0

41 0 1 0 ... 0

186 0 1 0 ... 0

188 0 1 0 ... 1

189 0 1 0 ... 1

... ... ... ... ... ...

421949 0 1 0 ... 0

421950 0 1 0 ... 0

421951 0 1 0 ... 0

421952 0 1 0 ... 0

421953 0 1 0 ... 0

ind_valo_fin_ult1_s ind_viv_fin_ult1_s ind_nomina_ult1_s \

40 0 0 0

41 0 0 0

186 0 0 0

188 1 0 0

189 1 0 0

... ... ... ...

421949 0 0 0

421950 0 0 0

421951 0 0 0

421952 0 0 0

421953 0 0 0

ind_nom_pens_ult1_s ind_recibo_ult1_s month_int id \

40 0 1 2 15924

41 0 1 7 15924

186 0 0 2 16162

188 0 1 6 16187

189 0 1 6 16187

... ... ... ... ...

421949 0 0 11 1454615

421950 0 0 12 1454615

421951 0 0 12 1454615

421952 0 0 14 1454615

421953 0 0 14 1454615

nomprov region

40 PALMAS, LAS north

41 PALMAS, LAS north

186 BARCELONA north

188 BARCELONA north

189 BARCELONA north

... ... ...

421949 BARCELONA north

421950 BARCELONA north

421951 BARCELONA north

421952 BARCELONA north

421953 BARCELONA north

[143620 rows x 44 columns],

&#39;train_y_north&#39;: 40 19

41 19

186 0

188 14

189 13

..

421949 2

421950 14

421951 13

421952 14

421953 13

Name: y, Length: 143620, dtype: int8,

&#39;train_X&#39;: ind_empleado sexo age renta ind_nuevo indrel indrel_1mes \

0 3 0 56 326124 0 1 1

1 3 0 56 326124 0 1 1

2 3 0 56 326124 0 1 1

3 3 1 61 430477 0 1 1

4 3 1 61 430477 0 1 1

... ... ... ... ... ... ... ...

421949 0 0 45 75445 1 1 1

421950 0 0 45 75445 1 1 1

421951 0 0 45 75445 1 1 1

421952 0 0 45 75445 1 1 1

421953 0 0 45 75445 1 1 1

tiprel_1mes indresi indext ... ind_tjcr_fin_ult1_s \

0 0 1 0 ... 0

1 0 1 0 ... 0

2 0 1 0 ... 0

3 0 1 0 ... 1

4 0 1 0 ... 1

... ... ... ... ... ...

421949 0 1 0 ... 0

421950 0 1 0 ... 0

421951 0 1 0 ... 0

421952 0 1 0 ... 0

421953 0 1 0 ... 0

ind_valo_fin_ult1_s ind_viv_fin_ult1_s ind_nomina_ult1_s \

0 1 0 0

1 1 0 0

2 1 0 0

3 1 0 0

4 1 0 0

... ... ... ...

421949 0 0 0

421950 0 0 0

421951 0 0 0

421952 0 0 0

421953 0 0 0

ind_nom_pens_ult1_s ind_recibo_ult1_s month_int id nomprov \