Unbreakable DevOps Pipeline Tutorial with AWS CodeDeploy, AWS CodePipeline, AWS Lambda, EC2 and Dynatrace.

The goal of this tutorial is having a full end-to-end AWS DevOps Pipeline (Staging, Approval, Production) that is fully monitored with Dynatrace. With Dynatrace injected into the pipeline you get the following features

1. Monitor your Staging Environment

2. Automate Approve/Reject Promotion from Staging to Production based on Performance Data

3. Monitor your Production Environment

4. Automatic Deploy of previous revision in case Dynatrace detected problems in Production

Before we launch the CloudFormation stack which will create all required resources (EC2 Instances, Lambdas, CodeDeploy, CodePipeline, API Gateway) lets make sure we have all pre-requisits covered!

1. You need an AWS account. If you dont have one [get one here](https://aws.amazon.com/)

2. You need a Dynatrace Account. Get your [Free SaaS Trial here!](http://bit.ly/dtsaastrial)

3. You need to clone or copy the content of this GitHub repo to your local disk!

**Amazon**

As we are going to use AWS CodeDeploy, AWS CodePipeline, AWS Lambda, DynamoDB, API Gateway and EC2 make sure the AWS Region you select provides all these services. We have tested this cloud formation on US-West-2a (Oregon) and US-East-2b (Ohio). To be on the safe side we suggest you pick one of these regions!

1. Create an EC2 Key Pair for your AWS Region! Our CloudFormation Template needs an EC2 Key Pair!

1.1. To learn more about Key Pairs and how to connect to EC2 Instances for troubleshooting read [Connect to your Linux Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html)

2. Create a S3 Bucket with the naming scheme: <yourname>-dynatracedevops and enable versioning. See following screenshots for reference

3. Copy the content from the folder "copytos3" to your newly created S3 bucket. This includes the application package, tests, monspec as well as all Lambda functions

**Dynatrace**

We need a couple of things to launch the CloudFormation Template

1. Your *Dynatrace Tenant URL*: For SaaS that would be something like http://<yourtenant>.live.dynatrace.com. For Managed it would be http://<yourserver>/e/<your-env-id>

2. Your *Dynatrace OneAgent for Linux Download URL*: Go to Deploy Dynatrace -> Start Installation -> Linux and copy the URL within the quotes as shown below:

3. A *Dynatrace API Token*: Go to Settings -> Integration -> Dynatrace API and create a new Token

You can download the stack definition from [here](./AWSDevOpsTutorialCloudFormationStack.json) - or simply click on one of the following links which will directly get you to the CloudFormation Wizard for the respective region.

Region | Launch Template

------------ | -------------

**N. Virginia** (us-east-1) | [](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Ohio** (us-east-2) | [](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.jsonn)

**Oregon** (us-west-2) | [](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Ireland** (eu-west-1) | [](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Frankfurt** (eu-central-1) | [](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Tokyo** (ap-northeast-1) | [](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Seoul** (ap-northeast-2) | [](https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Singapore** (ap-southeast-1) | [](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

**Sydney** (ap-southeast-2) | [](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=dynatracedevopsstack&templateURL=https://github.com/Dynatrace/AWSDevOpsTutorial/AWSDevOpsTutorialCloudFormationStack.json)

var port = process.env.PORT || 80,

http = require('http'),

fs = require('fs'),

os = require('os'),

dttags = process.env.DT_TAGS || "<EMPTY>",

dtcustprops = process.env.DT_CUSTOM_PROP || "<EMPTY>",

dtclusterid = process.env.DT_CLUSTER_ID || "<EMPTY>",

html = fs.readFileSync('index.html').toString().replace("HOSTNAME", os.hostname() + " with DT_TAGS=" + dttags + "\nDT_CUSTOM_PROP=" + dtcustprops + "\nDT_CLUSTER_ID=" + dtclusterid);

var buildNumber = 1;

var minSleep = 500;

var requestCount = 0;

var inProduction = false;

var invokeRequestCount = 0;

var failInvokeRequestPercentage = 0;

var init = function(newBuildNumber) {

// CHECK IF WE ARE RUNNING "In Production"

inProduction = process.env.DEPLOYMENT_GROUP_NAME && process.env.DEPLOYMENT_GROUP_NAME.startsWith("Production");

if(inProduction) {

minSleep = 300; // we just simulate that production is a bit faster than staging, e.g: better hardware!

}

// here are some "problems" we simulate for different builds. Builds are identified via Env Variable BUILD_NUMBER;

// Build # | Problem

// 1 | no problem

// 2 | 50% of requests return HTTP 500 Status Code

// 3 | back to normal

// 4 | no problem in staging but problem in prod -> higher sleep time and 10% of requests fail

// X | any other build number will run like 1 & 3

if(newBuildNumber != null) {

buildNumber = parseInt(newBuildNumber);

}

else if(process.env.BUILD_NUMBER && process.env.BUILD_NUMBER != null) {

buildNumber = parseInt(process.env.BUILD_NUMBER);

}

switch(buildNumber) {

case 2:

failInvokeRequestPercentage = 2;

break;

case 4:

if(inProduction) {

minSleep = minSleep * 2;

failInvokeRequestPercentage = 10;

}

break;

default:

// everything normal here

failInvokeRequestPercentage = 0;

break;

}

console.log("Init: " + buildNumber + "/" + failInvokeRequestPercentage);

}

var backgroundColors = ["#EEA53E", "#73A53E", "#FF0000", "#FFFF00", "#777777"]

var getBackgroundColor = function() {

var buildNumberForBackgroundColor = buildNumber;

if(buildNumber == 0 || buildNumber > 4) buildNumberForBackgroundColor = 1;

return backgroundColors[buildNumberForBackgroundColor];

var logstream = fs.createWriteStream('./serviceoutput.log');

var SEVERITY_DEBUG = "Debug";

var SEVERITY_INFO = "Info";

var SEVERITY_WARNING = "Warning";

var SEVERITY_ERROR = "Error";

var log = function(severity, entry) {

// console.log(entry);

if (severity === SEVERITY_DEBUG) {

// dont log debug

} else {

var logEntry = new Date().toISOString() + ' - ' + severity + " - " + entry + '\n';

// fs.appendFileSync('./serviceoutput.log', new Date().toISOString() + ' - ' + severity + " - " + entry + '\n');

logstream.write(logEntry);

}

function sleep(time) {

if(time < minSleep) time = minSleep;

var stop = new Date().getTime();

while(new Date().getTime() < stop + time) {

;

}

var server = http.createServer(function (req, res) {

if (req.method === 'POST') {

var body = '';

req.on('data', function(chunk) {

body += chunk;

});

req.on('end', function() {

if (req.url === '/') {

log(SEVERITY_DEBUG, 'Received message: ' + body);

} else if (req.url = '/scheduled') {

log(SEVERITY_DEBUG, 'Received task ' + req.headers['x-aws-sqsd-taskname'] + ' scheduled at ' + req.headers['x-aws-sqsd-scheduled-at']);

}

res.writeHead(200, 'OK', {'Content-Type': 'text/plain'});

res.end();

});

} else if (req.url.startsWith("/api")) {

var url = require('url').parse(req.url, true);

var closeResponse = true;

// sleep a bit :-)

var sleeptime = parseInt(url.query["sleep"]);

if(sleeptime === 0) sleeptime = minSleep;

log(SEVERITY_DEBUG, "Sleeptime: " + sleeptime);

sleep(sleeptime);

// figure out which API call they want to execute

var status = "Unkown API Call";

if(url.pathname === "/api/sleeptime") {

// Usage: /api/sleeptime?min=1234

var sleepValue = parseInt(url.query["min"]);

if(sleepValue >= 0 && sleepValue <= 10000) minSleep = sleepValue;

status = "Minimum Sleep Time set to " + minSleep;

}

if(url.pathname === "/api/echo") {

// Usage: /api/echo?text=your text to be echoed!

status = "Thanks for saying: " + url.query["text"];

}

if(url.pathname === "/api/login") {

// Usage: /api/login?username=your user name

status = "Welcome " + url.query["username"];

}

if(url.pathname === "/api/invoke") {

// count the invokes for failed requests

var returnStatusCode = 200;

if(failInvokeRequestPercentage > 0) {

invokeRequestCount++;

var failRequest = (invokeRequestCount % failInvokeRequestPercentage);

if(failRequest == 0) {

returnStatusCode = 500;

invokeRequestCount = 0;

}

}

// Usage: /api/invoke?url=http://www.yourdomain.com

var urlRequest = url.query["url"];

status = "Trying to invoke remote call to: " + urlRequest;

var http = null;

if(urlRequest.startsWith("https")) http = require("https");

else http = require("http");

closeResponse = false;

var options = {

host: urlRequest,

path: '/'

};

var result = http.get(urlRequest, function(getResponse) {

log(SEVERITY_DEBUG, 'STATUS: ' + getResponse.statusCode);

log(SEVERITY_DEBUG, 'HEADERS: ' + JSON.stringify(getResponse.headers));

// Buffer the body entirely for processing as a whole.

var bodyChunks = [];

getResponse.on('data', function(chunk) {

bodyChunks.push(chunk);

}).on('end', function() {

var body = Buffer.concat(bodyChunks);

log(SEVERITY_DEBUG, 'BODY: ' + body);

status = "Request to '" + url.query["url"] + "' returned with HTTP Status: " + getResponse.statusCode + " and response body length: " + body.length;

res.writeHead(returnStatusCode, returnStatusCode == 200 ? 'OK' : 'ERROR', {'Content-Type': 'text/plain'});

res.write(status);

res.end();

}).on('error', function(error) {

status = "Request to '" + url.query["url"] + "' returned in an error: " + error;

res.writeHead(returnStatusCode, returnStatusCode == 200 ? 'OK' : 'ERROR', {'Content-Type': 'text/plain'});

res.write(status);

res.end();

log(SEVERITY_INFO, status);

})

});

}

if(url.pathname === "/api/version") {

if (url.query["newBuildNumber"] && url.query["newBuildNumber"] != null) {

init(url.query["newBuildNumber"]);

}

// usage: /api/version

// simply returns the build number as defined in BUILD_NUMBER env-variable which is specified

status = "Running build number: " + buildNumber;

}

if(url.pathname === "/api/causeerror") {

log(SEVERITY_ERROR, "somebody called /api/causeerror");

status = "We just caused an error log entry"

}

// only close response handler if we are done with work!

if(closeResponse) {

res.writeHead(200, 'OK', {'Content-Type': 'text/plain'});

res.write(status);

res.end();

}

}

else

{

res.writeHead(200, 'OK', {'Content-Type': 'text/html'});

// replace buildnumber and background color

var finalHtml = html.replace("BACKGROUND-COLOR", getBackgroundColor()).replace("BUILD_NUMBER", buildNumber);

res.write(finalHtml);

res.end();

}

requestCount++;

if(requestCount >= 100) {

log(SEVERITY_INFO, "Just served another 100 requests!");

requestCount = 0;

}

console.log('Server running at http://127.0.0.1:' + port + '/');

log(SEVERITY_INFO, "Service is up and running - feed me with data!");

version: 0.0

os: linux

files:

- source: index.html

destination: /home/ec2-user/

- source: app.js

destination: /home/ec2-user/

- source: package.json

destination: /home/ec2-user/

permissions:

- object: /home/ec2-user/

mode: 777

type:

- file

hooks:

BeforeInstall:

- location: scripts/stop_server.sh

timeout: 60

runas: root

ApplicationStart:

- location: scripts/start_server.sh

timeout: 60

runas: root

ApplicationStop:

- location: scripts/stop_server.sh

timeout: 60

runas: root

version: 0.2

phases:

install:

commands:

- echo Installing Mocha...

pre_build:

commands:

- echo Installing source NPM dependencies...

build:

commands:

- echo Build started on `date`

post_build:

commands:

- echo Build completed on `date`

artifacts:

files:

- app.js

- index.html

- package.json