From d7e1c0e387dc0ab1c3652a854f0cf41db0000ab2 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 20:59:28 -0500 Subject: [PATCH 1/9] Update README.md --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3a2ead2..314a536 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,10 @@ Template for lambda with localstack and sam integration for localtest ## getting the source code - `git clone https://github.com/uujo/lambda_localstack_python_template.git` + `git clone https://github.com/uujo/lambda_localstack_python_templage.git` + + `cd lambda_localstack_python_template` + ## Set up git hooks (pre-commit, pre-push) - This is one time setting. Doesn't have to be repeated every time. @@ -46,7 +49,8 @@ For special cases, if you still need to commit, push the code use _--no-verify_ `source .venv/bin/activate` - activate virtual environment - `pip install -r requirements.txt` - on time setting + `pip install -r requirements.txt` - one time setting (If this command fails with __xcun__ error, run __xcode-select --install__ + * To start local test setup From c2434d287e49aa7754863563f1e7dc1572508214 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 21:04:35 -0500 Subject: [PATCH 2/9] Update README.md --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 314a536..ea9e03e 100644 --- a/README.md +++ b/README.md @@ -49,12 +49,13 @@ For special cases, if you still need to commit, push the code use _--no-verify_ `source .venv/bin/activate` - activate virtual environment - `pip install -r requirements.txt` - one time setting (If this command fails with __xcun__ error, run __xcode-select --install__ + `pip install -r requirements.txt` - one time setting + (If install fails with __xcun__ error, run __xcode-select --install__) * To start local test setup - `bin/start_local_test` + `bin/start_local_test_template.sh` _-i_ option sets interactive mode on localstack instead of running it background, if you want to monitor localstack log use this option. Without this option, it automatically create the sample data in S3 bucket. From 8dbcad7c3cc951d7a5535f0cdac9b4613c5168d6 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 21:25:49 -0500 Subject: [PATCH 3/9] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ea9e03e..516ab5a 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * To start local test setup - `bin/start_local_test_template.sh` + `./start_local_test_template.sh` _-i_ option sets interactive mode on localstack instead of running it background, if you want to monitor localstack log use this option. Without this option, it automatically create the sample data in S3 bucket. @@ -101,7 +101,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * To stop the local test and clean up docker - `bin/stop_local_test` + `./stop_local_test_template.sh` ## Integration Test: To be added From 3d311a5ab8bc138cdc5a7fad7f414a59d1804f22 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 22:41:43 -0500 Subject: [PATCH 4/9] Update README.md --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 516ab5a..a3b4d13 100644 --- a/README.md +++ b/README.md @@ -70,18 +70,18 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * put data in S3 - `aws s3api put-object --endpoint-url=http://localhost:4572 --bucket VARSAP_BUCKET --key annotation/g-snv-7-7578212-G-A/data.json --body tests/s3_test_data.json` + `aws s3api put-object --endpoint-url=http://localhost:4572 --bucket TEST_BUCKET --key test_path/data.json --body tests/s3_test_data.json` * To check S3 object is created - `aws s3api list-objects --bucket=VARSAP_BUCKET --endpoint-url=http://localhost:4572` + `aws s3api list-objects --bucket=TEST_BUCKET --endpoint-url=http://localhost:4572` -* To test annotation lambdas +* To test lambdas * invoke S3 ObjectCreated event - `sam local invoke HandleS3Event -t build/template.yaml --docker-network [repository_name]_local_aws_network -e tests/s3_test_event.json` + `sam local invoke HandleS3Event -t build/template.yaml --docker-network lambda_localstack_python_templage_local_aws_network -e tests/s3_test_event.json` * To check whether table has an entry inserted. @@ -93,7 +93,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * invoke SQS event - `sam local invoke RequestAnnotation -t build/template.yaml --docker-network [repository_name]_local_aws_network -e tests/sqs_test_event.json` + `sam local invoke RequestAnnotation -t build/template.yaml --docker-network lambda_localstack_python_templage_local_aws_network -e tests/sqs_test_event.json` * check S3 Object is updated: (Might take a while to update annotations) - It will store object in _s3_test_out.json_ From c9ed2e1cbb0bd4792a5f09030fbb4f67cafb2849 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 22:46:42 -0500 Subject: [PATCH 5/9] Update README.md --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a3b4d13..c7d6eb1 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# lambda_localstack_python_templage +# lambda_localstack_template Template for lambda with localstack and sam integration for localtest ## Prerequesite (on Mac) @@ -15,7 +15,7 @@ Template for lambda with localstack and sam integration for localtest ## getting the source code - `git clone https://github.com/uujo/lambda_localstack_python_templage.git` + `git clone https://github.com/uujo/lambda_localstack_template.git` `cd lambda_localstack_python_template` @@ -81,7 +81,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * invoke S3 ObjectCreated event - `sam local invoke HandleS3Event -t build/template.yaml --docker-network lambda_localstack_python_templage_local_aws_network -e tests/s3_test_event.json` + `sam local invoke HandleS3Event -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/s3_test_event.json` * To check whether table has an entry inserted. @@ -93,7 +93,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * invoke SQS event - `sam local invoke RequestAnnotation -t build/template.yaml --docker-network lambda_localstack_python_templage_local_aws_network -e tests/sqs_test_event.json` + `sam local invoke RequestAnnotation -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/sqs_test_event.json` * check S3 Object is updated: (Might take a while to update annotations) - It will store object in _s3_test_out.json_ From 7b4926b7c2908e3bd31854af2088a7ff51e0739a Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Tue, 25 Feb 2020 22:56:31 -0500 Subject: [PATCH 6/9] Update README.md --- README.md | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index c7d6eb1..cd7369a 100644 --- a/README.md +++ b/README.md @@ -79,10 +79,10 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * To test lambdas - * invoke S3 ObjectCreated event - - `sam local invoke HandleS3Event -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/s3_test_event.json` + * invoke SQS event + `sam local invoke TestLambda -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/sqs_test_event.json` + * To check whether table has an entry inserted. `aws dynamodb scan --table-name TEST_TABLE --endpoint-url http://localhost:4569` @@ -91,13 +91,6 @@ For special cases, if you still need to commit, push the code use _--no-verify_ `aws sqs receive-message --queue-url http://localhost:4576/queue/TEST_QUEUE --endpoint-url=http://localhost:4576 --max-number-of-messages=10` - * invoke SQS event - - `sam local invoke RequestAnnotation -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/sqs_test_event.json` - - * check S3 Object is updated: (Might take a while to update annotations) - It will store object in _s3_test_out.json_ - - `aws s3api get-object --endpoint-url=http://localhost:4572 --bucket TEST_BUCKET --key [path_to_object]/[name_of_object].json s3_test_out.json` * To stop the local test and clean up docker From e5d9e03adc301a370e3eeb359b1bd471f8b783aa Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Wed, 26 Feb 2020 11:30:14 -0500 Subject: [PATCH 7/9] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index cd7369a..163ec72 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Template for lambda with localstack and sam integration for localtest `git clone https://github.com/uujo/lambda_localstack_template.git` - `cd lambda_localstack_python_template` + `cd lambda_localstack_template` ## Set up git hooks (pre-commit, pre-push) - This is one time setting. Doesn't have to be repeated every time. @@ -41,9 +41,9 @@ For special cases, if you still need to commit, push the code use _--no-verify_ ## Local test set up using aws sam and localstack (with python3 venv) -* set the virtual environment (optional, you can use other virtual env tools or without it) +* set the virtual environment (optional, you can use other virtual env tools or without it) - `cd lambda_localstack_python_template` + `cd lambda_localstack_template` `python3 -m venv .venv` From da0ef6f9d194ba91cac2c006cb991308f86d4f86 Mon Sep 17 00:00:00 2001 From: seangchan-ryu-nih Date: Sat, 23 May 2020 23:04:54 -0400 Subject: [PATCH 8/9] add dynamo test --- .git_templates/hooks/pre-commit | 2 +- .git_templates/hooks/pre-push | 2 +- .travis.yml | 54 +++++++++++++ Makefile | 22 +++++ README.md | 81 ++++++++----------- .../api-gw-template-sam.yaml | 63 +++++++++++++++ .../apigw-dynamo-config/docker-compose.yaml | 51 ++++++++++++ .../api-gw-dynamo-server-sam.yaml | 62 ++++++++++++++ .../s3-sqs-dynamo-config/docker-compose.yaml | 18 ++--- .../s3-sqs-dynamo-config/template-sam.yaml | 24 +----- requirements-travis.txt | 5 ++ {projectname => src}/__init__.py | 0 .../awshelper => src/projectname}/__init__.py | 0 .../projectname/awshelper}/__init__.py | 0 .../projectname}/awshelper/awsenv.py | 68 +++++++++------- src/projectname/awshelper/awswraper.py | 54 +++++++++++++ src/projectname/exceptionhelper/__init__.py | 0 .../exceptionhelper/exception_handler.py | 67 +++++++++++++++ src/projectname/handler/__init__.py | 0 .../handler/dynamo_handler/__init__.py | 0 .../handler/dynamo_handler/delete_handler.py | 21 +++++ .../handler/dynamo_handler/get_handler.py | 21 +++++ .../handler/dynamo_handler/post_handler.py | 25 ++++++ .../handler/dynamo_handler/put_handler.py | 27 +++++++ .../handler/s3_sqs_dynamo_handler/__init__.py | 0 .../lambda_handler_template.py | 23 +++--- start_local_dynamo_server_test.sh | 58 +++++++++++++ start_local_dynamo_test.sh | 49 +++++++++++ start_local_test_template.sh | 29 +++++-- stop_local_dynamo_server_test.sh | 5 ++ stop_local_dynamo_test.sh | 5 ++ stop_local_test_template.sh | 2 +- tests/data/s3_test_data.json | 5 ++ tests/data/sqs_test_event.json | 3 + tests/unittests/projectname_t/__init__.py | 0 .../projectname_t/awshelper_t/__init__.py | 0 .../projectname_t/awshelper_t/test_awsenv.py | 44 ++++++++++ .../projectname_t/handler_t/__init__.py | 0 38 files changed, 760 insertions(+), 130 deletions(-) create mode 100644 .travis.yml create mode 100644 Makefile create mode 100644 local-test-configs/apigw-dynamo-config/api-gw-template-sam.yaml create mode 100644 local-test-configs/apigw-dynamo-config/docker-compose.yaml create mode 100644 local-test-configs/apigw-dynamo-server-config/api-gw-dynamo-server-sam.yaml rename docker-compose.yaml => local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml (81%) rename template-sam.yaml => local-test-configs/s3-sqs-dynamo-config/template-sam.yaml (54%) create mode 100644 requirements-travis.txt rename {projectname => src}/__init__.py (100%) rename {projectname/awshelper => src/projectname}/__init__.py (100%) rename {projectname/handler => src/projectname/awshelper}/__init__.py (100%) rename {projectname => src/projectname}/awshelper/awsenv.py (59%) create mode 100644 src/projectname/awshelper/awswraper.py create mode 100644 src/projectname/exceptionhelper/__init__.py create mode 100644 src/projectname/exceptionhelper/exception_handler.py create mode 100644 src/projectname/handler/__init__.py create mode 100644 src/projectname/handler/dynamo_handler/__init__.py create mode 100644 src/projectname/handler/dynamo_handler/delete_handler.py create mode 100644 src/projectname/handler/dynamo_handler/get_handler.py create mode 100644 src/projectname/handler/dynamo_handler/post_handler.py create mode 100644 src/projectname/handler/dynamo_handler/put_handler.py create mode 100644 src/projectname/handler/s3_sqs_dynamo_handler/__init__.py rename {projectname/handler => src/projectname/handler/s3_sqs_dynamo_handler}/lambda_handler_template.py (70%) create mode 100755 start_local_dynamo_server_test.sh create mode 100755 start_local_dynamo_test.sh create mode 100755 stop_local_dynamo_server_test.sh create mode 100755 stop_local_dynamo_test.sh create mode 100644 tests/data/s3_test_data.json create mode 100644 tests/data/sqs_test_event.json create mode 100644 tests/unittests/projectname_t/__init__.py create mode 100644 tests/unittests/projectname_t/awshelper_t/__init__.py create mode 100644 tests/unittests/projectname_t/awshelper_t/test_awsenv.py create mode 100644 tests/unittests/projectname_t/handler_t/__init__.py diff --git a/.git_templates/hooks/pre-commit b/.git_templates/hooks/pre-commit index 55b0609..5f1c12c 100755 --- a/.git_templates/hooks/pre-commit +++ b/.git_templates/hooks/pre-commit @@ -14,4 +14,4 @@ black --config ./pyproject.toml . # Run pylint and check the score echo "pre-commit: Running lint check" -pylint --rcfile=./pylintrc projectname/ tests/ +pylint --rcfile=./pylintrc src/projectname tests/unittests diff --git a/.git_templates/hooks/pre-push b/.git_templates/hooks/pre-push index ad1b408..7e62cd7 100755 --- a/.git_templates/hooks/pre-push +++ b/.git_templates/hooks/pre-push @@ -8,7 +8,7 @@ # run the unittests if it fails abort the push echo "pre-push: Run unittest" -coverage run --source=projectname -m pytest tests/unittests/ +coverage run --source=src/projectname -m pytest tests/unittests/ # Above command doesn't exit when unittests fails # It needs to be caught and exit non 0 to abort the push diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..96db16f --- /dev/null +++ b/.travis.yml @@ -0,0 +1,54 @@ +dist: xenial +language: python + +branches: + only: master + +sudo: true + +python: 3.7 + +before_install: + - export AUTHOR=`git --no-pager show -s --format='%an <%ae>'` + - export DATE=`TZ=America/New_York date "+%m-%d-%y-%H%M"` + - export DATE_TM=`TZ=America/New_York date "+%m-%d-%y %H:%M"` + - python --version + +env: + global: + - AWS_DEFAULT_REGION=us-east-1 + - PYTHONPATH=$TRAVIS_BUILD_DIR:$PYTHONPATH + +install: + - pip install awscli --upgrade + - pip install -r requirements-travis.txt + - pip install awscli --upgrade + +script: + # run tests + #- py.test -vv -r sxX + - export PYTHONPATH=$PWD/src:$PWD/tests:$PYTHONPATH + - BOTO_CONFIG=none pytest --cov-report xml --cov=src/projectname/ tests/unittests/ + - coverage xml && python-codacy-coverage -r coverage.xml + - ls -alh + + - make + # consult cloud engineer for set up below + #- mv ./dist/lambda.zip ./dist/[lamdba-name]:latest.zip + #- cp ./dist/[lamdba-name]:latest.zip ./dist/[lambda-name]:$DATE.zip + #- cp ./dist/[lamdba-name]:latest.zip ./dist/[lambda-name]:latest.zip + - ls -alh ./dist/ + + +after_success: + + # consult a cloud engineer for set up below + #- aws s3 cp ../dist/[lamdba-name]:$DATE.zip s3://biad-lambda-hub/[ProjectName]/ --acl public-read + # run integration test if exists + #- pytest -vv ./tests/integrationtests/ || exit $? + + # consult a cloud engineer for heartbeat test + +notifications: + # consult a cloud engineer for slack notification + #slack: clinicalbiomed:EphqNgtuf0NzIGaOE43vSFDc \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..f632860 --- /dev/null +++ b/Makefile @@ -0,0 +1,22 @@ +default: clean install copy zip + +install: build_path + pip install -r requirements-lambda.txt -t ./build + +build_path: + mkdir ./build + +build_dist: + mkdir ./dist + +copy: + ls -alh + cp -R src/* ./build/ + ls -alh ./build + +zip: build_dist + cd ./build && zip -r9 ./dist/lambda.zip . 2>&1 >>/dev/null + +clean: + rm -rf ./build + rm -rf ./dist \ No newline at end of file diff --git a/README.md b/README.md index 163ec72..c035541 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,23 @@ -# lambda_localstack_template -Template for lambda with localstack and sam integration for localtest +# nci-aws-lambda-localstack-template + +This provides a templates for aws lambda test in local environment using localstack or local server instances (i.e. dynamoDB). This examples shows only fraction of the services localstack provides. The full list of services are [here](https://github.com/localstack/localstack). This template provides the basic settings for local test which can be used in different projects with minimal configuration changes. Details for the configuration changes is explained `How to use this template` section below. ## Prerequesite (on Mac) * [Python3](https://www.python.org/downloads/) -* [AWS client](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) (pip3 install awscli --upgrade --user) +* [Docker](https://docs.docker.com/docker-for-mac/install/) - need for the local lambda function testing with sam -* [AWS SAM client](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-mac.html#serverless-sam-cli-install-mac-pip) (pip3 install --user aws-sam-cli) - - * Reference: [Adjust path to use sam cli command](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-mac-path.html) +* [AWS client](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) -* [Docker](https://docs.docker.com/docker-for-mac/install/) - need for the local lambda function testing +* [AWS SAM client](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-mac.html) + ## getting the source code - `git clone https://github.com/uujo/lambda_localstack_template.git` + `git clone https://github.com/BIAD/nci-aws-lambda-localstack-template.git` - `cd lambda_localstack_template` + `cd nci-aws-lambda-localstack-template` ## Set up git hooks (pre-commit, pre-push) - This is one time setting. Doesn't have to be repeated every time. @@ -39,11 +39,18 @@ For special cases, if you still need to commit, push the code use _--no-verify_ > git push origin [branch_name] --no-verify ``` -## Local test set up using aws sam and localstack (with python3 venv) +## set PYTHONPATH for unittest + + `export PYTHONPATH=$PWD/src:$PWD/tests:$PYTHONPATH` + + If virtual environment is used below, you can set the above path in __.venv/bin/activate__ file + + +## Local test set up using aws sam and localstack (with python3 venv) - virtual environment setting is optional. * set the virtual environment (optional, you can use other virtual env tools or without it) - `cd lambda_localstack_template` + `cd nci-aws-lambda-localstack-template` `python3 -m venv .venv` @@ -51,55 +58,31 @@ For special cases, if you still need to commit, push the code use _--no-verify_ `pip install -r requirements.txt` - one time setting (If install fails with __xcun__ error, run __xcode-select --install__) - - -* To start local test setup + +* Now it is ready to set up the test below. + +* After all the test is done. get out of virtual environment. - `./start_local_test_template.sh` + `deactivate` - _-i_ option sets interactive mode on localstack instead of running it background, if you want to monitor localstack log use this option. Without this option, it automatically create the sample data in S3 bucket. - +## Tests Examples -* To check the table, sqs and bucket are created (**Use this setting only if you use _-i_ option, without _-i_, these steps runs automatically**) - - `aws dynamodb list-tables --endpoint-url http://localhost:4569` - - `aws sqs list-queues --endpoint-url=http://localhost:4576` - - `aws s3api list-buckets --endpoint-url=http://localhost:4572` - - * put data in S3 - - `aws s3api put-object --endpoint-url=http://localhost:4572 --bucket TEST_BUCKET --key test_path/data.json --body tests/s3_test_data.json` +### [Lambda with dynamo, API gateway testing (Localstack)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-Dynamo-Test) - * To check S3 object is created + +### [Lambda with dynamo, s3, sqs testing (Localstack)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-S3-SQS-Dynamo-Test) - `aws s3api list-objects --bucket=TEST_BUCKET --endpoint-url=http://localhost:4572` - - -* To test lambdas +### [Lambda with dynamo, API gateway testing (local dynamo server)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-Local-Dynamo-Server-Test) - * invoke SQS event - - `sam local invoke TestLambda -t build/template.yaml --docker-network lambda_localstack_template_local_aws_network -e tests/sqs_test_event.json` - * To check whether table has an entry inserted. +## Integration Test: To be added - `aws dynamodb scan --table-name TEST_TABLE --endpoint-url http://localhost:4569` - - * To check sqs messge is queued - - `aws sqs receive-message --queue-url http://localhost:4576/queue/TEST_QUEUE --endpoint-url=http://localhost:4576 --max-number-of-messages=10` - - -* To stop the local test and clean up docker - `./stop_local_test_template.sh` - +## [How to use this template](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/How-to-use-this-template) -## Integration Test: To be added + * goal: With minimal configuation change, testing environment can be set up for different project. + * There are two main configuration - One for localstack and one for sam. If starting and stop scrip need to be changed as well if you want to reuse the script. - ## Reference * [AWS SAM guick start guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-quick-start.html) * [SAM template](https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md) diff --git a/local-test-configs/apigw-dynamo-config/api-gw-template-sam.yaml b/local-test-configs/apigw-dynamo-config/api-gw-template-sam.yaml new file mode 100644 index 0000000..5830a45 --- /dev/null +++ b/local-test-configs/apigw-dynamo-config/api-gw-template-sam.yaml @@ -0,0 +1,63 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 + +Globals: + Function: + Runtime: python3.7 + Timeout: 60 + Environment: + Variables: + # Set localtest env + # Set environment variable so amazon service can be switched in local settings + # This has to be matched with localstack config in docker-compose.yaml + # Current example contains DynamoDB + TEST_ENV: LOCAL + +Resources: + PostTest: + Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.post_handler.post + Events: + Validation: + Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api + Properties: + Path: /api/v1/testapi + Method: post + + GetTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.get_handler.get + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: get + + PutTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.put_handler.put + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: put + + DeleteTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.delete_handler.delete + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: delete \ No newline at end of file diff --git a/local-test-configs/apigw-dynamo-config/docker-compose.yaml b/local-test-configs/apigw-dynamo-config/docker-compose.yaml new file mode 100644 index 0000000..c3486d4 --- /dev/null +++ b/local-test-configs/apigw-dynamo-config/docker-compose.yaml @@ -0,0 +1,51 @@ +version: "3" + +services: + localstack: + image: localstack/localstack:0.11.1 + ports: + - "4566-4599:4566-4599" + - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}" + environment: + - DOCKER_HOST=unix:///var/run/docker.sock + - SERVICES=dynamodb + - DEFAULT_REGION=us-east-1 + - EDGE_PORT=4566 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /private${TMPDIR}:/tmp/localstack + networks: + - local_aws_network + + aws-cli: + image: mesosphere/aws-cli + volumes: + - ./dev_env:/project/dev_env + environment: + - AWS_ACCESS_KEY_ID=1234 + - AWS_SECRET_ACCESS_KEY=1234 + - AWS_DEFAULT_REGION=us-east-1 + entrypoint: /bin/sh -c + command: > + " + # Sleep is needed so all localstack components will startup correctly (There's a better way to do this) + sleep 10; + + # Initialize dynamodb in localstack + aws dynamodb create-table --endpoint-url=http://localstack:4566 \ + --table-name TEST_TABLE \ + --attribute-definitions \ + AttributeName=id,AttributeType=S \ + --key-schema \ + AttributeName=id,KeyType=HASH \ + --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5; + + # you can run more command here + " + networks: + - local_aws_network + depends_on: + - localstack + +networks: + local_aws_network: \ No newline at end of file diff --git a/local-test-configs/apigw-dynamo-server-config/api-gw-dynamo-server-sam.yaml b/local-test-configs/apigw-dynamo-server-config/api-gw-dynamo-server-sam.yaml new file mode 100644 index 0000000..b034dd6 --- /dev/null +++ b/local-test-configs/apigw-dynamo-server-config/api-gw-dynamo-server-sam.yaml @@ -0,0 +1,62 @@ +Transform: AWS::Serverless-2016-10-31 + +Globals: + Function: + Runtime: python3.7 + Timeout: 60 + Environment: + Variables: + # Set localtest env + # Set environment variable so amazon service can be switched in local settings + # This has to be matched with localstack config in docker-compose.yaml + # Current example contains DynamoDB + TEST_ENV: LOCAL_DYNAMO_SERVER + +Resources: + PostTest: + Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.post_handler.post + Events: + Validation: + Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api + Properties: + Path: /api/v1/testapi + Method: post + + GetTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.get_handler.get + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: get + + PutTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.put_handler.put + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: put + + DeleteTest: + Type: AWS::Serverless::Function + Properties: + CodeUri: . + Handler: projectname.handler.dynamo_handler.delete_handler.delete + Events: + Annotation: + Type: Api + Properties: + Path: /api/v1/testapi/{id} + Method: delete \ No newline at end of file diff --git a/docker-compose.yaml b/local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml similarity index 81% rename from docker-compose.yaml rename to local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml index 59ab126..b44747d 100644 --- a/docker-compose.yaml +++ b/local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml @@ -2,14 +2,15 @@ version: "3" services: localstack: - image: localstack/localstack + image: localstack/localstack:0.11.1 ports: - - "4568-4576:4568-4576" + - "4566-4599:4566-4599" - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}" environment: - DOCKER_HOST=unix:///var/run/docker.sock - SERVICES=dynamodb, sqs, s3 - DEFAULT_REGION=us-east-1 + - EDGE_PORT=4566 volumes: - /var/run/docker.sock:/var/run/docker.sock - /private${TMPDIR}:/tmp/localstack @@ -28,26 +29,25 @@ services: command: > " # Sleep is needed so all localstack components will startup correctly (There's a better way to do this) - sleep 10; + sleep 15; # Initialize S3 in localstack - aws s3api create-bucket --endpoint-url=http://localstack:4572 \ - --bucket TEST_BUCKET; + aws s3api create-bucket --endpoint-url=http://localstack:4566 --bucket test-bucket; # Initialize dynamodb in localstack - aws dynamodb create-table --endpoint-url=http://localstack:4569 \ + aws dynamodb create-table --endpoint-url=http://localstack:4566 \ --table-name TEST_TABLE \ --attribute-definitions \ AttributeName=key_id,AttributeType=S \ - AttributeName=updated,AttributeType=N \ + AttributeName=last_updated,AttributeType=N \ --key-schema \ AttributeName=key_id,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \ --global-secondary-indexes \ - IndexName=SourceIndex,KeySchema=[{AttributeName=updated,KeyType=HASH}],Projection={ProjectionType=KEYS_ONLY},ProvisionedThroughput={ReadCapacityUnits=5,WriteCapacityUnits=5}; + IndexName=SourceIndex,KeySchema=[{AttributeName=last_updated,KeyType=HASH}],Projection={ProjectionType=KEYS_ONLY},ProvisionedThroughput={ReadCapacityUnits=5,WriteCapacityUnits=5}; # Initialize sqs in localstack - aws sqs create-queue --endpoint-url=http://localstack:4576 --queue-name TEST_QUEUE; + aws sqs create-queue --endpoint-url=http://localstack:4566 --queue-name TEST_QUEUE; # you can run more command here " networks: diff --git a/template-sam.yaml b/local-test-configs/s3-sqs-dynamo-config/template-sam.yaml similarity index 54% rename from template-sam.yaml rename to local-test-configs/s3-sqs-dynamo-config/template-sam.yaml index 536b949..bd75a2b 100644 --- a/template-sam.yaml +++ b/local-test-configs/s3-sqs-dynamo-config/template-sam.yaml @@ -14,28 +14,8 @@ Globals: TEST_ENV: LOCAL Resources: - RequestAnnotation: + LambdaTemplateTest: Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction Properties: CodeUri: . - Handler: annotation_requestor.insert_annotation - - HandleS3Event: - Type: AWS::Serverless::Function - Properties: - CodeUri: . - Handler: s3_event_handler.trigger_annotation - - AnnotaionDynamoDBTable: - Type: AWS::DynamoDB::Table - Properties: - TableName: ANNOTATION_TABLE - PrimaryKey: - Name: job_id - Type: String - AttributeDefinitions: - AttributeName: job_id - AttributeType: HASH - ProvisionedThroughput: - ReadCapacityUnits: 5 - WriteCapacityUnits: 5 + Handler: projectname.handler.s3_sqs_dynamo_handler.lambda_handler_template.populate_dynamo_from_sqs_s3 diff --git a/requirements-travis.txt b/requirements-travis.txt new file mode 100644 index 0000000..c5545e1 --- /dev/null +++ b/requirements-travis.txt @@ -0,0 +1,5 @@ +boto3 +codacy-coverage +coverage +pytest +pytest-cov diff --git a/projectname/__init__.py b/src/__init__.py similarity index 100% rename from projectname/__init__.py rename to src/__init__.py diff --git a/projectname/awshelper/__init__.py b/src/projectname/__init__.py similarity index 100% rename from projectname/awshelper/__init__.py rename to src/projectname/__init__.py diff --git a/projectname/handler/__init__.py b/src/projectname/awshelper/__init__.py similarity index 100% rename from projectname/handler/__init__.py rename to src/projectname/awshelper/__init__.py diff --git a/projectname/awshelper/awsenv.py b/src/projectname/awshelper/awsenv.py similarity index 59% rename from projectname/awshelper/awsenv.py rename to src/projectname/awshelper/awsenv.py index 21c4640..ed1927e 100644 --- a/projectname/awshelper/awsenv.py +++ b/src/projectname/awshelper/awsenv.py @@ -1,18 +1,31 @@ """ This is just the somple code for accessing parameter store and secrets manager -and how to swich local env to aws env +and how to swich local env to aws env """ import os import json +import logging + import boto3 + +EDGE_PORT = 4566 +LOCALSTACK_ENDPOINT = f"http://localstack:{EDGE_PORT}" + LOCAL_TABLE_NAME = "TEST_TABLE" -LOCAL_DYNAMO_ENDPOINT = "http://localstack:4569" -LOCAL_BUCKET_NAME = "TEST_BUCKET" -LOCAL_S3_ENDPOINT = "http://localstack:4572" +LOCAL_BUCKET_NAME = "test-bucket" # bucket name cannot contain uppercase or _. LOCAL_QUEUE_NAME = "TEST_QUEUE" -LOCAL_SQS_ENDPOINT = "http://localstack:4576" +# using local dynamo server +LOCAL_DYNAMO_SERVER_ENDPOINT = "http://dynamodb:8000" + + +def is_local_dynamo(): + """ + check whether local dynamodb is set up + """ + # This is set on template-samp.yaml + return os.getenv("TEST_ENV") == "LOCAL_DYNAMO_SERVER" def is_local_env(): @@ -22,7 +35,8 @@ def is_local_env(): :return: """ # This is set on template-samp.yaml - return os.getenv("TEST_ENV") == "LOCAL" + logging.info(f"ENVIRONMENT: {os.getenv('TEST_ENV')}") + return os.getenv("TEST_ENV") == "LOCAL" or is_local_dynamo() def get_params_from_ssm(path_to_config=None): @@ -37,6 +51,7 @@ def get_params_from_ssm(path_to_config=None): ssm = boto3.Session(region_name='us-east-1').client('ssm') param = ssm.get_parameter(Name=path_to_config, WithDecryption=True) + param_dict = {} if param and "Parameter" in param: param_dict = json.loads(param['Parameter']['Value']) @@ -56,64 +71,63 @@ def get_params_from_secretsmanager(path_to_config=None): return param_dict -@property def dynamo_endpoint(): """ returns dynamodb endpoint """ - return LOCAL_DYNAMO_ENDPOINT if is_local_env() else None + # check local dynamo first + if is_local_dynamo(): + return LOCAL_DYNAMO_SERVER_ENDPOINT + if is_local_env(): + return LOCALSTACK_ENDPOINT + return None -@property def s3_endpoint(): """ returns s3 endpoint """ - return LOCAL_S3_ENDPOINT if is_local_env() else None + return LOCALSTACK_ENDPOINT if is_local_env() else None -@property def sqs_endpoint(): """ returns sqs endpoint """ - return LOCAL_SQS_ENDPOINT if is_local_env() else None + return LOCALSTACK_ENDPOINT if is_local_env() else None -@property def dynamo_table_name(): """ returns table name from parameter store """ if is_local_env(): return LOCAL_TABLE_NAME - else: - # get data from parameter store with correct key - # table_name = get_params_from_ssm()["CORRECT_KEY"] - return "table_name" + + # get data from parameter store with correct key + # table_name = get_params_from_ssm()["CORRECT_KEY"] + return "table_name" -@property def s3_bucket_name(): """ returns s3 bucket name from parameter store """ if is_local_env(): return LOCAL_BUCKET_NAME - else: - # get data from parameter store with correct key - # bucket_name = get_params_from_ssm()["CORRECT_KEY"] - return "bucket_name" + + # get data from parameter store with correct key + # bucket_name = get_params_from_ssm()["CORRECT_KEY"] + return "bucket_name" -@property def sqs_name(): """ returns sqs name from parameter store """ if is_local_env(): return LOCAL_QUEUE_NAME - else: - # get data from parameter store with correct key - # sqs_name = get_params_from_ssm()["CORRECT_KEY"] - return "sqs_name" + + # get data from parameter store with correct key + # sqs_name = get_params_from_ssm()["CORRECT_KEY"] + return "sqs_name" diff --git a/src/projectname/awshelper/awswraper.py b/src/projectname/awshelper/awswraper.py new file mode 100644 index 0000000..2fbf73c --- /dev/null +++ b/src/projectname/awshelper/awswraper.py @@ -0,0 +1,54 @@ +""" +Wrapper for AWS services +""" +import time +import boto3 +from .awsenv import dynamo_endpoint, dynamo_table_name + +# pylint: disable=no-member + +class DynamoDB: + """ + wrapper class for dynamo db + """ + + def __init__(self): + """ + initializing dynamo table + """ + dynamo = boto3.resource("dynamodb", endpoint_url=dynamo_endpoint()) + self.dynamo_table = dynamo.Table(dynamo_table_name()) + + def retrieve_from_dynamo(self, key): + """ + retrieve data to dynamodb + """ + result = self.dynamo_table.get_item(Key={"id": key}) + return result.get("Item", {}) + + def insert_to_dynamo(self, data): + """ + insert data to dynamodb + """ + + item = {"created": int(time.time()), "last_updated": int(time.time())} + item.update(data) + return self.dynamo_table.put_item(Item=item) + + def delete_from_dynamo(self, key): + """ + delete data from dynamodb + """ + return self.dynamo_table.delete_item(Key={"id": key}) + + def update_to_dynamo(self, key, data): + """ + update data to dynamodb + """ + + result = self.dynamo_table.update_item( + Key={"id": key}, + UpdateExpression="SET test_data = :r, last_updated = :n", + ExpressionAttributeValues={':r': data['test_data'], ':n': int(time.time())}, + ) + return result diff --git a/src/projectname/exceptionhelper/__init__.py b/src/projectname/exceptionhelper/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/projectname/exceptionhelper/exception_handler.py b/src/projectname/exceptionhelper/exception_handler.py new file mode 100644 index 0000000..0d54d76 --- /dev/null +++ b/src/projectname/exceptionhelper/exception_handler.py @@ -0,0 +1,67 @@ +""" + Handles various exceptions for validation and db exceptoin + Convert exception to lamdba response +""" + +import json +import traceback +from functools import wraps + +# disable pylint exception warning +# pylint: disable=broad-except + + +def lambda_exception_handler(lambda_func): + """ + + This decorator will handle the exceptions occurred during the lambda processing + by providing responsed depending on the exceptions handinged + :param lambda_func: aws lambda function + :return: wrapped function + """ + + @wraps(lambda_func) + def wrapper(event, context): + """ + :param event: aws event + :param context: aws context + :return: aws lambda response i.e. {"statusCode": 200, "body": results, "header": {} } + """ + try: + data = lambda_func(event, context) + + response = { + "statusCode": 200, + "body": json.dumps(data, default=float), # assuming we have only Decimal encoding problem + "headers": {"Content-Type": "application/json"}, + } + return response + + except (json.JSONDecodeError, TypeError) as ex: + return { + "statusCode": 400, + "body": json.dumps({"input_error": f"Input JSON error: {str(ex)}: {event['body']}"}), + "headers": {"Content-Type": "application/json"}, + } + + except Exception as ex: + response = {"statusCode": 500, "body": "Server error. {}".format(traceback.format_exc())} + return response + + return wrapper + + +def invalid_input_key_check(key, input_data): + """ + :param key: + :param input_data: + :return: + """ + if key in input_data: + return { + "statusCode": 400, + "body": json.dumps({"input_error": "id field cannot be updated"}), + "headers": {"Content-Type": "application/json"}, + } + + return {} diff --git a/src/projectname/handler/__init__.py b/src/projectname/handler/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/projectname/handler/dynamo_handler/__init__.py b/src/projectname/handler/dynamo_handler/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/projectname/handler/dynamo_handler/delete_handler.py b/src/projectname/handler/dynamo_handler/delete_handler.py new file mode 100644 index 0000000..41f00c9 --- /dev/null +++ b/src/projectname/handler/dynamo_handler/delete_handler.py @@ -0,0 +1,21 @@ +""" +lambda function handling delete +""" + +from ...exceptionhelper.exception_handler import lambda_exception_handler +from ...awshelper.awswraper import DynamoDB + +# disable pylint warning about unused 'context' argument +# pylint: disable=unused-argument + + +@lambda_exception_handler +def delete(event, context): + """ + get data from dynamo + TODO: need proper error handling + """ + + key = event['path'].split("/")[-1] + + return DynamoDB().delete_from_dynamo(key) diff --git a/src/projectname/handler/dynamo_handler/get_handler.py b/src/projectname/handler/dynamo_handler/get_handler.py new file mode 100644 index 0000000..c52f202 --- /dev/null +++ b/src/projectname/handler/dynamo_handler/get_handler.py @@ -0,0 +1,21 @@ +""" +retrieve data from dynamo db +""" + +import logging +from ...exceptionhelper.exception_handler import lambda_exception_handler +from ...awshelper.awswraper import DynamoDB + +# disable pylint warning about unused 'context' argument +# pylint: disable=unused-argument + + +@lambda_exception_handler +def get(event, context): + """ + get data from dynamo + TODO: need proper error handling + """ + logging.info(event["path"]) + key = event['path'].split("/")[-1] + return DynamoDB().retrieve_from_dynamo(key) diff --git a/src/projectname/handler/dynamo_handler/post_handler.py b/src/projectname/handler/dynamo_handler/post_handler.py new file mode 100644 index 0000000..94ecc42 --- /dev/null +++ b/src/projectname/handler/dynamo_handler/post_handler.py @@ -0,0 +1,25 @@ +""" +This is the sample code waiting event from SQS, +then read the data from S3 then insert data to dynamodb +It is just a sample. It doesn't include proper error handling codes + +""" +import json + +from ...exceptionhelper.exception_handler import lambda_exception_handler +from ...awshelper.awswraper import DynamoDB + + +# disable pylint warning about unused 'context' argument +# pylint: disable=unused-argument + + +@lambda_exception_handler +def post(event, context): + """ + inserting the data from REST input to dynamo + """ + + input_data = json.loads(event['body']) + + return DynamoDB().insert_to_dynamo(input_data) diff --git a/src/projectname/handler/dynamo_handler/put_handler.py b/src/projectname/handler/dynamo_handler/put_handler.py new file mode 100644 index 0000000..f1adb00 --- /dev/null +++ b/src/projectname/handler/dynamo_handler/put_handler.py @@ -0,0 +1,27 @@ +""" +update data in dynamodb +""" + +import json + +from ...exceptionhelper.exception_handler import lambda_exception_handler, invalid_input_key_check +from ...awshelper.awswraper import DynamoDB + +# disable pylint warning about unused 'context' argument +# pylint: disable=unused-argument + + +@lambda_exception_handler +def put(event, context): + """ + updating data from REST input to dynamo + """ + + db_id = event['path'].split("/")[-1] + input_data = json.loads(event['body']) + + response = invalid_input_key_check("id", input_data) + if response: + return response + + return DynamoDB().update_to_dynamo(db_id, input_data) diff --git a/src/projectname/handler/s3_sqs_dynamo_handler/__init__.py b/src/projectname/handler/s3_sqs_dynamo_handler/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/projectname/handler/lambda_handler_template.py b/src/projectname/handler/s3_sqs_dynamo_handler/lambda_handler_template.py similarity index 70% rename from projectname/handler/lambda_handler_template.py rename to src/projectname/handler/s3_sqs_dynamo_handler/lambda_handler_template.py index 96a2be2..a980cfa 100644 --- a/projectname/handler/lambda_handler_template.py +++ b/src/projectname/handler/s3_sqs_dynamo_handler/lambda_handler_template.py @@ -10,15 +10,18 @@ import boto3 -from ..awshelper.awsenv import dynamo_endpoint, s3_endpoint, dynamo_table_name, s3_bucket_name +from ...awshelper.awsenv import dynamo_endpoint, s3_endpoint, dynamo_table_name, s3_bucket_name + +# disable pylint warning about unused 'context' argument +# pylint: disable=unused-argument, no-member def get_s3_data(key): """ getting s3 data """ - s3 = boto3.resource("s3", endpoint_url=s3_endpoint) - s3_obj = s3.Object(s3_bucket_name, key=key) + s3 = boto3.resource("s3", endpoint_url=s3_endpoint()) + s3_obj = s3.Object(s3_bucket_name(), key=key) data = s3_obj.get(ResponseContentType="application/json") body = json.loads(data["Body"].read()) return body @@ -31,20 +34,16 @@ def insert_to_dynamo(key, data): :param data: :return: """ - dynamo = boto3.resource("dynamodb", endpoint_url=dynamo_endpoint) - dynamo_table = dynamo.Table(dynamo_table_name) + dynamo = boto3.resource("dynamodb", endpoint_url=dynamo_endpoint()) + dynamo_table = dynamo.Table(dynamo_table_name()) - item = { - "key_id": key, - "data": data, - "created": int(time.time()), - "updated": int(time.time()), - } + item = {"key_id": key, "data": data, "created": int(time.time()), "last_updated": int(time.time())} result = dynamo_table.put_item(Item=item) return result -def populate_dynamo(event, context): +# +def populate_dynamo_from_sqs_s3(event, context): """ :param event: aws event :param context: aws context diff --git a/start_local_dynamo_server_test.sh b/start_local_dynamo_server_test.sh new file mode 100755 index 0000000..54332db --- /dev/null +++ b/start_local_dynamo_server_test.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +while getopts u option ;do + case ${option} in + u) SRC_UPDATE="SET" ;; + *) echo "No option is added -i interactive mode for localstack";; + esac +done + +update_source() { + echo "Replace build source to original source" + + cp -rf $PWD/src/projectname $PWD/build/PostTest + cp -rf $PWD/src/projectname $PWD/build/GetTest + cp -rf $PWD/src/projectname $PWD/build/PutTest + cp -rf $PWD/src/projectname $PWD/build/DeleteTest +} + +aws_status() { + echo "check local dynamodb aws status" + aws dynamodb list-tables --endpoint-url http://localhost:8000 +} + +if [[ ${SRC_UPDATE} = "SET" ]] + then + update_source + exit 0 +fi + +# sam build for the local test +echo "sam build" +sam build -t ./local-test-configs/apigw-dynamo-server-config/api-gw-dynamo-server-sam.yaml -m requirements-lambda.txt -u -b ./build/ -s src/ + +# start local DynamoDB +echo "Start local DynamoDB" + +docker network create lambda-local +docker run -d -v "$PWD":/dynamodb_local_db -p 8000:8000 --network lambda-local --name dynamodb amazon/dynamodb-local + +sleep 10 + +echo "Create a test table" +aws dynamodb create-table --endpoint-url=http://localhost:8000 \ + --table-name TEST_TABLE \ + --attribute-definitions \ + AttributeName=id,AttributeType=S \ + --key-schema \ + AttributeName=id,KeyType=HASH \ + --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 + +aws_status + +# start sam local +echo "Start sam local with api gateway" + +sam local start-api -t build/template.yaml --docker-network lambda-local + + diff --git a/start_local_dynamo_test.sh b/start_local_dynamo_test.sh new file mode 100755 index 0000000..96c15fb --- /dev/null +++ b/start_local_dynamo_test.sh @@ -0,0 +1,49 @@ +#!/bin/bash + +while getopts iu option ;do + case ${option} in + i) INTERACTIVE="SET" ;; + u) SRC_UPDATE="SET" ;; + *) echo "No option is added -i interactive mode for localstack";; + esac +done + +update_source() { + echo "Replace build source to original source" + + cp -rf $PWD/src/projectname $PWD/build/PostTest + cp -rf $PWD/src/projectname $PWD/build/GetTest + cp -rf $PWD/src/projectname $PWD/build/PutTest + cp -rf $PWD/src/projectname $PWD/build/DeleteTest +} + +aws_status() { + echo "check localstack aws status" + aws dynamodb list-tables --endpoint-url http://localhost:4566 +} + +if [[ ${SRC_UPDATE} = "SET" ]] + then + update_source + exit 0 +fi + +# sam build for the local test +echo "sam build" +sam build -t ./local-test-configs/apigw-dynamo-config/api-gw-template-sam.yaml -m requirements-lambda.txt -u -b ./build/ -s src/ + +# start localstat +echo "Start localstack" + +if [[ ${INTERACTIVE} = "SET" ]] + then + docker-compose -f ./local-test-configs/apigw-dynamo-config/docker-compose.yaml up + else + docker-compose -f ./local-test-configs/apigw-dynamo-config/docker-compose.yaml up -d + # wait until local stack is set + sleep 15 + aws_status + echo "Start aws sam local api gateway" + sam local start-api -t build/template.yaml --docker-network apigw-dynamo-config_local_aws_network +fi + diff --git a/start_local_test_template.sh b/start_local_test_template.sh index e7b03ec..d63cba9 100755 --- a/start_local_test_template.sh +++ b/start_local_test_template.sh @@ -1,33 +1,46 @@ #!/bin/bash -while getopts si option ;do +while getopts ui option ;do case ${option} in i) INTERACTIVE="SET" ;; + u) SRC_UPDATE="SET" ;; *) echo "No option is added -i interactive mode for localstack";; esac done +update_source() { + echo "Replace build source to original source" + + cp -rf $PWD/src/projectname $PWD/build/LambdaTemplateTest +} + +if [[ ${SRC_UPDATE} = "SET" ]] + then + update_source + exit 0 +fi + aws_status() { echo "check localstack aws status" - aws dynamodb list-tables --endpoint-url http://localhost:4569 - aws sqs list-queues --endpoint-url=http://localhost:4576 - aws s3api list-buckets --endpoint-url=http://localhost:4572 + aws dynamodb list-tables --endpoint-url http://localhost:4566 + aws sqs list-queues --endpoint-url=http://localhost:4566 + aws s3api list-buckets --endpoint-url=http://localhost:4566 } # sam build for the local test echo "sam build" -sam build -t template-sam.yaml -m requirements-lambda.txt -u -b ./build/ -s projectname/ +sam build -t ./local-test-configs/s3-sqs-dynamo-config/template-sam.yaml -m requirements-lambda.txt -u -b ./build/ -s src/ # start localstat echo "Start localstack" if [[ ${INTERACTIVE} = "SET" ]] then - docker-compose up + docker-compose -f ./local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml up else - docker-compose up -d + docker-compose -f ./local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml up -d # wait until local stack is set - sleep 10 + sleep 18 aws_status fi diff --git a/stop_local_dynamo_server_test.sh b/stop_local_dynamo_server_test.sh new file mode 100755 index 0000000..94ee714 --- /dev/null +++ b/stop_local_dynamo_server_test.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +echo "shutting down local dynamodb" +docker container stop $(docker container ls -aq) +docker system prune \ No newline at end of file diff --git a/stop_local_dynamo_test.sh b/stop_local_dynamo_test.sh new file mode 100755 index 0000000..fa4bf5d --- /dev/null +++ b/stop_local_dynamo_test.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +echo "shutting down localstack" +docker container stop $(docker container ls -aq) +docker-compose -f ./local-test-configs/apigw-dynamo-config/docker-compose.yaml down \ No newline at end of file diff --git a/stop_local_test_template.sh b/stop_local_test_template.sh index 57a8fc1..596a9cb 100755 --- a/stop_local_test_template.sh +++ b/stop_local_test_template.sh @@ -2,4 +2,4 @@ echo "shutting down localstack" docker container stop $(docker container ls -aq) -docker-compose down \ No newline at end of file +docker-compose -f ./local-test-configs/s3-sqs-dynamo-config/docker-compose.yaml down \ No newline at end of file diff --git a/tests/data/s3_test_data.json b/tests/data/s3_test_data.json new file mode 100644 index 0000000..6b76931 --- /dev/null +++ b/tests/data/s3_test_data.json @@ -0,0 +1,5 @@ +{ + "id": "A456", + "test_data": "S3_test", + "test_number": 100 +} \ No newline at end of file diff --git a/tests/data/sqs_test_event.json b/tests/data/sqs_test_event.json new file mode 100644 index 0000000..bb21813 --- /dev/null +++ b/tests/data/sqs_test_event.json @@ -0,0 +1,3 @@ +{"Records": [{"eventSource": "aws:sqs", + "body": "{\"key\": \"test_path/data.json\"}" + }]} \ No newline at end of file diff --git a/tests/unittests/projectname_t/__init__.py b/tests/unittests/projectname_t/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/unittests/projectname_t/awshelper_t/__init__.py b/tests/unittests/projectname_t/awshelper_t/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/unittests/projectname_t/awshelper_t/test_awsenv.py b/tests/unittests/projectname_t/awshelper_t/test_awsenv.py new file mode 100644 index 0000000..302d1e3 --- /dev/null +++ b/tests/unittests/projectname_t/awshelper_t/test_awsenv.py @@ -0,0 +1,44 @@ +""" +Test aws_env module. mocking aws ssm access +""" +import json +from unittest.mock import patch +from projectname.awshelper.awsenv import get_params_from_ssm, get_params_from_secretsmanager + + +@patch("projectname.awshelper.awsenv.boto3.Session") +def test_get_ssm_success(boto3_session): + """ + Test successful case when get_params_from_ssm returns param + :param boto_session: + :return: None + """ + PARAM = {'Parameter': {'Value': json.dumps({"TEST_TABLE": "AnnotationTable"})}} + boto3_session().client().get_parameter.return_value = PARAM + + response = get_params_from_ssm("/test/config") + assert response == json.loads(PARAM['Parameter']['Value']) + + # When param doesn't exists + boto3_session().client().get_parameter.return_value = {} + response = get_params_from_ssm("/test/config") + assert response == {} + + +@patch("projectname.awshelper.awsenv.boto3.Session") +def test_get_param_success(boto3_session): + """ + Test successful case when get_params_from_ssm returns param + :param boto_session: + :return: None + """ + PARAM = {'SecretString': json.dumps({'CosmicSecrete': "abc"})} + boto3_session().client().get_secret_value.return_value = PARAM + + response = get_params_from_secretsmanager("/test/config") + assert response == json.loads(PARAM['SecretString']) + + # When param doesn't exists + boto3_session().client().get_secret_value.return_value = {} + response = get_params_from_secretsmanager("/test/config") + assert response == {} diff --git a/tests/unittests/projectname_t/handler_t/__init__.py b/tests/unittests/projectname_t/handler_t/__init__.py new file mode 100644 index 0000000..e69de29 From 7924bd5cd317cb5ed3bdf270f1cb031362673f89 Mon Sep 17 00:00:00 2001 From: Seangchan Ryu Date: Sat, 23 May 2020 23:12:43 -0400 Subject: [PATCH 9/9] Update README.md --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index c035541..9d5c20e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ This provides a templates for aws lambda test in local environment using localst `git clone https://github.com/BIAD/nci-aws-lambda-localstack-template.git` - `cd nci-aws-lambda-localstack-template` + `cd lambda-localstack-template` ## Set up git hooks (pre-commit, pre-push) - This is one time setting. Doesn't have to be repeated every time. @@ -50,7 +50,7 @@ For special cases, if you still need to commit, push the code use _--no-verify_ * set the virtual environment (optional, you can use other virtual env tools or without it) - `cd nci-aws-lambda-localstack-template` + `cd lambda-localstack-template` `python3 -m venv .venv` @@ -67,18 +67,18 @@ For special cases, if you still need to commit, push the code use _--no-verify_ ## Tests Examples -### [Lambda with dynamo, API gateway testing (Localstack)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-Dynamo-Test) +### [Lambda with dynamo, API gateway testing (Localstack)](https://github.com/uujo/lambda-localstack-template/wiki/Lambda-Dynamo-Test) -### [Lambda with dynamo, s3, sqs testing (Localstack)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-S3-SQS-Dynamo-Test) +### [Lambda with dynamo, s3, sqs testing (Localstack)](https://github.com/uujo/lambda-localstack-template/wiki/Lambda-S3-SQS-Dynamo-Test) -### [Lambda with dynamo, API gateway testing (local dynamo server)](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/Lambda-Local-Dynamo-Server-Test) +### [Lambda with dynamo, API gateway testing (local dynamo server)](https://github.com/BIAD/lambda-localstack-template/wiki/Lambda-Local-Dynamo-Server-Test) ## Integration Test: To be added -## [How to use this template](https://github.com/BIAD/nci-aws-lambda-localstack-template/wiki/How-to-use-this-template) +## [How to use this template](https://github.com/uujo/lambda-localstack-template/wiki/How-to-use-this-template) * goal: With minimal configuation change, testing environment can be set up for different project. * There are two main configuration - One for localstack and one for sam. If starting and stop scrip need to be changed as well if you want to reuse the script.