Fix documentation links in README (+other fixes)

benfred · benfred · commit fbad798c6658 · 2020-05-12T16:03:09.000-07:00
* Fix documentation links in introduction (contributing.md / operators.md etc)
* Cleanup the 'tutorials and examples' section in the README
* Switch to use the RTD theme
* rename gpu_benchmark-criteo.ipynb to criteo-example.ipynb
* Add basic requirements.txt / requirement-dev.txt files
* Remove redudant section headings in docs
* flake8/black/isort
diff --git a/README.md b/README.md
@@ -43,17 +43,13 @@ Requirements.yml
 
 ### Examples and Tutorials
 
-A workflow demonstrating the preprocessing and data-loading components of NVTabular can be found in the DeepLearningExamples tutorial on training Facebook's [Deep Learning Recommender Model (DLRM)](https://github.com/facebookresearch/dlrm/) on the [Criteo 1TB dataset](https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/).
+An example demonstrating how to use NVTabular to preprocess the [Criteo 1TB dataset](https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/) can be found in the [criteo example notebook](examples/criteo-example.ipynb). This example also shows how to use NVTabular's data-loaders on the preprocessed data to train Facebook's [Deep Learning Recommender Model (DLRM)](https://github.com/facebookresearch/dlrm/).
 
-[ DLRM Criteo Workflow ](https://developer.nvidia.com/deep-learning-examples#rec-sys)
-
-We also have a simple tutorial that demonstrates similar functionality on a much smaller dataset, providing a pipeline for the [Rossman store sales dataset](https://www.kaggle.com/c/rossmann-store-sales) fed into a [fast.ai tabular data model](https://docs.fast.ai/tabular.html).  
-
-[ Rossman Store Sales ](examples/gpu_benchmark-rossmann.ipynb)
+We also have a [simple tutorial](examples/rossmann-store-sales-example.ipynb) that demonstrates similar functionality on a much smaller dataset, providing a pipeline for the [Rossman store sales dataset](https://www.kaggle.com/c/rossmann-store-sales) fed into a [fast.ai tabular data model](https://docs.fast.ai/tabular.html).  
 
 ### Contributing
 
-If you wish to contribute to the library directly please see [Contributing.md](https://github.com/nvidia/NVTabular/blob/master/CONTRIBUTING.md).  We are in particular interested in contributions or feature requests for feature engineering or preprocessing operations that you have found helpful in your own workflows.
+If you wish to contribute to the library directly please see [Contributing.md](./CONTRIBUTING.md).  We are in particular interested in contributions or feature requests for feature engineering or preprocessing operations that you have found helpful in your own workflows.
 
 ### Learn More
 
diff --git a/docs/source/Operators.md b/docs/source/Operators.md
diff --git a/docs/source/api/index.rst b/docs/source/api/index.rst
@@ -3,7 +3,6 @@ API Documentation
 
 .. toctree::
    :maxdepth: 2
-   :caption: API Documentation:
 
    Workflow <workflow>
    Operators <ops/index>
diff --git a/docs/source/api/ops/index.rst b/docs/source/api/ops/index.rst
@@ -3,7 +3,6 @@ Operators
 
 .. toctree::
    :maxdepth: 2
-   :caption: Operators:
 
    Categorify <categorify>
    FillMissing <fillmissing>
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -13,12 +13,11 @@
 import os
 import sys
 
-sys.path.insert(0, os.path.abspath("../../."))
-
-import recommonmark
-from recommonmark.transform import AutoStructify
+import sphinx
 from recommonmark.parser import CommonMarkParser
 
+sys.path.insert(0, os.path.abspath("../../."))
+
 
 # -- Project information -----------------------------------------------------
 
@@ -36,11 +35,13 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    "sphinx_rtd_theme",
     "recommonmark",
     "nbsphinx",
     "sphinx.ext.autodoc",
     "sphinx.ext.coverage",
     "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -57,7 +58,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "alabaster"
+html_theme = "sphinx_rtd_theme"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
@@ -67,13 +68,31 @@
 source_parsers = {".md": CommonMarkParser}
 source_suffix = [".rst", ".md"]
 
-def setup(app):
-    app.add_config_value('recommonmark_config', {
-            'enable_math': True,
-            'enable_eval_rst': True,
-            'auto_code_block': True,
-            }, True)
-    app.add_transform(AutoStructify)
+nbsphinx_allow_errors = True
+html_show_sourcelink = False
+
+# certain references in the README couldn't be autoresolved here,
+# hack by forcing to the either the correct documentation page (examples)
+# or to a blob on the repo
+_REPO = "https://github.com/NVIDIA/NVTabular/blob/master/"
+_URL_MAP = {
+    "./examples": "examples/index",
+    "examples/rossmann-store-sales-example.ipynb": "examples/rossmann",
+    "examples/criteo-example.ipynb": "examples/criteo",
+    "./CONTRIBUTING": _REPO + "/CONTRIBUTING.md",
+    "./Operators": _REPO + "/Operators.md",
+}
+
+
+class GitHubDomain(sphinx.domains.Domain):
+    def resolve_any_xref(self, env, docname, builder, target, node, contnode):
+        resolved = _URL_MAP.get(target)
+        print("resolver", target, resolved)
+        if resolved:
+            contnode["refuri"] = resolved
+            return [("github:any", contnode)]
+        return []
 
 
-nbsphinx_allow_errors = True
+def setup(app):
+    app.add_domain(GitHubDomain)
diff --git a/docs/source/examples/criteo.ipynb b/docs/source/examples/criteo.ipynb
@@ -1 +1 @@
-../../../examples/gpu_benchmark-criteo.ipynb
+../../../examples/criteo-example.ipynb
diff --git a/docs/source/examples/index.rst b/docs/source/examples/index.rst
@@ -2,8 +2,7 @@ Examples
 ========
 
 .. toctree::
-   :maxdepth: 4
-   :caption: Examples:
+   :maxdepth: 2
 
    Rossmann Example <rossmann>
    Criteo Example <criteo>
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -8,12 +8,11 @@ Welcome to NVTabular's documentation!
 
 .. toctree::
    :maxdepth: 3
-   :caption: Contents:
 
    Introduction <Introduction>
    How it Works <HowItWorks>
-   API Documentation <api/index>
    Examples <examples/index>
+   API Documentation <api/index>
 
 
 Indices and tables
diff --git a/examples/criteo-example.ipynb b/examples/criteo-example.ipynb
diff --git a/examples/rossmann-store-sales-example.ipynb b/examples/rossmann-store-sales-example.ipynb
@@ -264,7 +264,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Tensorflow\n",
+    "### Tensorflow\n",
     "\n",
     "`KerasSequenceDataset` wraps a lightweight iterator around a `dataset` object to handle chunking, shuffling, and application of any workflows (which can be applied online as a preprocessing step). For column names, can use either a list of string names or a list of TensorFlow `feature_columns` that will be used to feed the network"
    ]
@@ -324,8 +324,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### PyTorch\n",
-    "`workflow.ds_to_tensors` maps a symbolic dataset object to `cat_features`, `cont_features`, `labels` PyTorch tenosrs by iterating through the dataset and concatenating the results. Note that this means that the whole of the dataset is _in memory_. For larger than memory datasets, see the example in [gpu_benchmark-criteo.ipynb](./gpu_benchmark-criteo.ipynb) leveraing PyTorch `ChainDataset`s."
+    "### PyTorch\n",
+    "`workflow.ds_to_tensors` maps a symbolic dataset object to `cat_features`, `cont_features`, `labels` PyTorch tenosrs by iterating through the dataset and concatenating the results. Note that this means that the whole of the dataset is _in memory_. For larger than memory datasets, see the example in [criteo-example.ipynb](./criteo-example.ipynb) leveraing PyTorch `ChainDataset`s."
    ]
   },
   {
diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -0,0 +1,9 @@
+# Dependencies required to develop nvtabular
+-r requirements.txt
+
+black>=19
+flake8>=3.7
+nbsphinx>=0.6
+pytest>=5
+recommonmark>=0.6
+Sphinx>=3
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1 @@
+cudf>=0.14
diff --git a/tests/conftest.py b/tests/conftest.py
@@ -1,10 +1,10 @@
-import cudf
-import pytest
-import numpy as np
-import random
 import os
+import random
 from functools import wraps
 
+import cudf
+import numpy as np
+import pytest
 
 allcols_csv = ["timestamp", "id", "label", "name-string", "x", "y", "z"]
 mycols_csv = ["name-string", "id", "label", "x", "y"]
diff --git a/tests/unit/test_dataloader.py b/tests/unit/test_dataloader.py
@@ -338,7 +338,7 @@ def test_tf_gpu_dl(tmpdir, datasets, batch_size, gpu_memory_frac, engine):
     if engine == "parquet":
         cat_names.append("name-cat")
 
-    columns = cont_names+cat_names
+    columns = cont_names + cat_names
 
     processor = nvt.Workflow(
         cat_names=cat_names, cont_names=cont_names, label_name=label_name, to_cpu=True,
@@ -355,7 +355,7 @@ def test_tf_gpu_dl(tmpdir, datasets, batch_size, gpu_memory_frac, engine):
         buffer_size=gpu_memory_frac,
         label_name=label_name[0],
         engine=engine,
-        shuffle=False
+        shuffle=False,
     )
     processor.update_stats(data_itr.nvt_dataset, record_stats=True)
     data_itr.map(processor)
@@ -382,7 +382,7 @@ def test_tf_gpu_dl(tmpdir, datasets, batch_size, gpu_memory_frac, engine):
                 these_cols.remove(column)
             except ValueError:
                 raise AssertionError
-            assert(x.shape[0] == num_samples)
+            assert x.shape[0] == num_samples
         assert len(these_cols) == 0
 
         rows += num_samples
@@ -395,7 +395,7 @@ def test_tf_gpu_dl(tmpdir, datasets, batch_size, gpu_memory_frac, engine):
         assert (x.numpy() == x0.numpy()).all()
     assert len(X0) == 0
 
-    # accounts for incomplete batches at the end of chunks 
+    # accounts for incomplete batches at the end of chunks
     # that dont necesssarily have the full batch_size
     assert (idx + 1) * batch_size >= rows
-    assert rows == (60*24*3 + 1)
+    assert rows == (60 * 24 * 3 + 1)
diff --git a/tests/unit/test_workflow.py b/tests/unit/test_workflow.py

Original file line number	Diff line number	Diff line change
`@@ -1 +1 @@`
`1`		`-../../../examples/gpu_benchmark-criteo.ipynb`
	`1`	`+../../../examples/criteo-example.ipynb`
Original file line number	Diff line number	Diff line change
`@@ -264,7 +264,7 @@`
`264`	`264`	`"cell_type": "markdown",`
`265`	`265`	`"metadata": {},`
`266`	`266`	`"source": [`
`267`		`- "#### Tensorflow\n",`
	`267`	`+ "### Tensorflow\n",`
`268`	`268`	`"\n",`
`269`	`269`	"`KerasSequenceDataset` wraps a lightweight iterator around a `dataset` object to handle chunking, shuffling, and application of any workflows (which can be applied online as a preprocessing step). For column names, can use either a list of string names or a list of TensorFlow `feature_columns` that will be used to feed the network"
`270`	`270`	`]`
`@@ -324,8 +324,8 @@`
`324`	`324`	`"cell_type": "markdown",`
`325`	`325`	`"metadata": {},`
`326`	`326`	`"source": [`
`327`		`- "#### PyTorch\n",`
`328`		- "`workflow.ds_to_tensors` maps a symbolic dataset object to `cat_features`, `cont_features`, `labels` PyTorch tenosrs by iterating through the dataset and concatenating the results. Note that this means that the whole of the dataset is _in memory_. For larger than memory datasets, see the example in [gpu_benchmark-criteo.ipynb](./gpu_benchmark-criteo.ipynb) leveraing PyTorch `ChainDataset`s."
	`327`	`+ "### PyTorch\n",`
	`328`	+ "`workflow.ds_to_tensors` maps a symbolic dataset object to `cat_features`, `cont_features`, `labels` PyTorch tenosrs by iterating through the dataset and concatenating the results. Note that this means that the whole of the dataset is _in memory_. For larger than memory datasets, see the example in [criteo-example.ipynb](./criteo-example.ipynb) leveraing PyTorch `ChainDataset`s."
`329`	`329`	`]`
`330`	`330`	`},`
`331`	`331`	`{`