start of python version

Matt C · Matt C · commit 67524605fe34 · 2020-12-03T16:45:12.000Z
diff --git a/the-predictive-lambda/python/.gitignore b/the-predictive-lambda/python/.gitignore
@@ -0,0 +1,10 @@
+*.swp
+package-lock.json
+__pycache__
+.pytest_cache
+.env
+*.egg-info
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
diff --git a/the-predictive-lambda/python/README.md b/the-predictive-lambda/python/README.md
@@ -0,0 +1,103 @@
+# The Predictive Lambda Pattern
+
+![architecture](img/arch.png)
+
+This is a pattern that uses a container inside Lambda to deploy a custom Python ML model to predict the nearest Chipotle restaurant based on your lat/long.
+
+Some Useful References:
+
+| Author        | Link           |
+| ------------- | ------------- |
+| AWS Blog | [New for AWS Lambda – Container Image Support](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) |
+| AWS Docs | [Lambda now supports container images](https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-now-supports-container-images-as-a-packaging-format/) |
+| Yan Cui | [Package your Lambda function as a container image](https://lumigo.io/blog/package-your-lambda-function-as-a-container-image/) |
+| Scikit Learn Docs | [User Guide](https://scikit-learn.org/stable/user_guide.html) |
+| AWS ECR Gallery | [Python Lambda Image](https://gallery.ecr.aws/lambda/python) |
+| Docker Docs | [CLI Reference](https://docs.docker.com/reference/) |
+
+## What's Included In This Pattern?
+This pattern uses sklearn to create a custom k nearest neighbour model to predict the nearest Chipotle to a given Latitude and Longitude. The model is deployed inside a container attached to AWS Lambda.
+
+### The Data
+If you want to look at the data used for this model you can look at the [jupyter notebook](model/training/Chipotle.ipynb), the raw data came from [kaggle](https://www.kaggle.com/jeffreybraun/chipotle-locations)
+
+### The ML Model
+This is a very simple model to demonstrate the concept (I didn't even check the accuracy because it doesn't change the pattern). It uses [sklearn nearest neighbors](https://scikit-learn.org/stable/modules/neighbors.html) to predict the closest Chipotle location to a given lat/long
+
+### Two Docker Containers
+I use the Lambda image to train the ML model in one container and then I use a separate container for the deployed Lambda Function. The reason I do this is because it means that you know you have pickled your model in the same environment it will be deployed but you can use things that wont be packaged into your deployed function keeping it as lightweight as possible. You will also have a built container image containing the raw data, the training logic and the trained model. These images could be archived to have a history of your model.
+
+### A Lambda Function
+I have this setup with a 15 second timeout and 4GB ram to comfortably run our model
+
+### An API Gateway HTTP API
+Setup as a proxy integration, all requests hit the Lambda Function
+
+## How Do I Test This Pattern?
+
+do "npm run deploy" from the base directory and you will have the url for an API Gateway output into the logs or in the CloudFormation console. Open that url in a browser but add "?lat=39.153198&long=-77.066176" to the end and you should get back a prediction.
+
+## How Does It Work?
+
+Most of the logic for this lives in the model folder. There are two Dockerfiles:
+- Dockerfile - used by Lambda during the deploy
+- TrainingDockerfile - used to spin up the container to train our model
+
+I have added the trained model to version control but if you want to retrain it yourself what you have to do is make sure docker is running and:
+
+```bash
+cd model
+./trainmodel.sh
+```
+
+This uses the Lambda Python image to run the file training/training.py and then copy the chipotle.pkl file out of the container. The requirements.txt is shared between the training container and the deployed container.
+
+The actual logic that runs when we hit our url is in model/deployment/app.py, it unpickles the model, makes a prediction and returns the response as a string.
+
+
+## Useful CDK Commands
+
+To manually create a virtualenv on MacOS and Linux:
+
+```
+$ python3 -m venv .venv
+```
+
+After the init process completes and the virtualenv is created, you can use the following
+step to activate your virtualenv.
+
+```
+$ source .venv/bin/activate
+```
+
+If you are a Windows platform, you would activate the virtualenv like this:
+
+```
+% .venv\Scripts\activate.bat
+```
+
+Once the virtualenv is activated, you can install the required dependencies.
+
+```
+$ pip install -r requirements.txt
+```
+
+At this point you can now synthesize the CloudFormation template for this code.
+
+```
+$ cdk synth
+```
+
+To add additional dependencies, for example other CDK libraries, just add
+them to your `setup.py` file and rerun the `pip install -r requirements.txt`
+command.
+
+## Useful commands
+
+ * `cdk ls`          list all stacks in the app
+ * `cdk synth`       emits the synthesized CloudFormation template
+ * `cdk deploy`      deploy this stack to your default AWS account/region
+ * `cdk diff`        compare deployed stack with current state
+ * `cdk docs`        open CDK documentation
+
+Enjoy!
diff --git a/the-predictive-lambda/python/app.py b/the-predictive-lambda/python/app.py
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+from aws_cdk import core
+
+from the_predictive_lambda.the_predictive_lambda_stack import ThePredictiveLambdaStack
+
+
+app = core.App()
+ThePredictiveLambdaStack(app, "the-predictive-lambda")
+
+app.synth()
diff --git a/the-predictive-lambda/python/cdk.json b/the-predictive-lambda/python/cdk.json
@@ -0,0 +1,9 @@
+{
+  "app": "python3 app.py",
+  "context": {
+    "@aws-cdk/core:enableStackNameDuplicates": "true",
+    "aws-cdk:enableDiffNoFail": "true",
+    "@aws-cdk/core:stackRelativeExports": "true",
+    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true
+  }
+}
diff --git a/the-predictive-lambda/python/img/arch.png b/the-predictive-lambda/python/img/arch.png
diff --git a/the-predictive-lambda/python/requirements.txt b/the-predictive-lambda/python/requirements.txt
@@ -0,0 +1 @@
+-e .
diff --git a/the-predictive-lambda/python/setup.py b/the-predictive-lambda/python/setup.py
@@ -0,0 +1,45 @@
+import setuptools
+
+
+with open("README.md") as fp:
+    long_description = fp.read()
+
+
+setuptools.setup(
+    name="the_predictive_lambda",
+    version="0.0.1",
+
+    description="An empty CDK Python app",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+
+    author="author",
+
+    package_dir={"": "the_predictive_lambda"},
+    packages=setuptools.find_packages(where="the_predictive_lambda"),
+
+    install_requires=[
+        "aws-cdk.core==1.76.0",
+    ],
+
+    python_requires=">=3.6",
+
+    classifiers=[
+        "Development Status :: 4 - Beta",
+
+        "Intended Audience :: Developers",
+
+        "License :: OSI Approved :: Apache Software License",
+
+        "Programming Language :: JavaScript",
+        "Programming Language :: Python :: 3 :: Only",
+        "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+
+        "Topic :: Software Development :: Code Generators",
+        "Topic :: Utilities",
+
+        "Typing :: Typed",
+    ],
+)
diff --git a/the-predictive-lambda/python/source.bat b/the-predictive-lambda/python/source.bat
@@ -0,0 +1,13 @@
+@echo off
+
+rem The sole purpose of this script is to make the command
+rem
+rem     source .venv/bin/activate
+rem
+rem (which activates a Python virtualenv on Linux or Mac OS X) work on Windows.
+rem On Windows, this command just runs this batch file (the argument is ignored).
+rem
+rem Now we don't need to document a Windows command for activating a virtualenv.
+
+echo Executing .venv\Scripts\activate.bat for you
+.venv\Scripts\activate.bat
diff --git a/the-predictive-lambda/python/the_predictive_lambda/__init__.py b/the-predictive-lambda/python/the_predictive_lambda/__init__.py
diff --git a/the-predictive-lambda/python/the_predictive_lambda/the_predictive_lambda_stack.py b/the-predictive-lambda/python/the_predictive_lambda/the_predictive_lambda_stack.py
@@ -0,0 +1,9 @@
+from aws_cdk import core
+
+
+class ThePredictiveLambdaStack(core.Stack):
+
+    def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
+        super().__init__(scope, construct_id, **kwargs)
+
+        # The code that defines your stack goes here
