Repository navigation
Resource timed out waiting for creation of physical resource #210
Description
Hello,
Thanks for releasing it. Hopefully it will prove useful to a lot of people. I'm trialling it by transferring an existing custom resource to this new system. I'm trying to utilize the chaining functionality but I've hit a bit of a wall. When I utilize the new resource Cloudformation fails with Resource timed out waiting for creation of physical resource. Interestingly, after this occurs I can see that the handler does run the stabilization part of the chain and complete (but Cloudformation has already failed by this part). I've copied the create handler below. Any obvious issues you can see?
public class CreateHandler extends BaseHandler<CallbackContext> {
@Override
public ProgressEvent<ResourceModel, CallbackContext> handleRequest(
final AmazonWebServicesClientProxy proxy,
final ResourceHandlerRequest<ResourceModel> request,
CallbackContext callbackContext,
final Logger logger) {
return proxy.initiate("ecs:CreateService", proxy.newProxy(EcsClient::create), request.getDesiredResourceState(), callbackContext == null ? new CallbackContext() : callbackContext)
.request(model -> ResourceModelConverter.toCreateServiceRequest(model, request.getDesiredResourceTags()))
.retry(Constant.of().delay(Duration.ofSeconds(20)).timeout(Duration.ofMinutes(60)).build())
.call((createServiceRequest, client) ->
client.injectCredentialsAndInvokeV2(createServiceRequest, client.client()::createService))
.stabilize((createServiceRequest, createServiceResponse, client, model, context) -> {
logger.log("Running stabilization test");
String expectedPrimaryId = createServiceResponse.service().deployments().stream().filter(
deployment -> deployment.status().equals("PRIMARY")
).findAny()
.map(Deployment::id)
.orElseThrow(() -> new CfnGeneralServiceException("Should have one primary deployment in service"));
DescribeServicesRequest describeServicesRequest = DescribeServicesRequest.builder()
.services(createServiceResponse.service().serviceArn())
.cluster(createServiceResponse.service().clusterArn())
.build();
List<Service> services = client.injectCredentialsAndInvokeV2(describeServicesRequest, client.client()::describeServices).services();
if (services.size() != 1) {
throw new CfnGeneralServiceException("Could not find service which was created");
}
List<Deployment> updatedDeployments = services.get(0).deployments();
Deployment primaryLatest = updatedDeployments.stream()
.filter(deployment -> deployment.status().equals("PRIMARY"))
.filter(deployment -> deployment.id().equals(expectedPrimaryId))
.findAny()
.orElseThrow(() -> new CfnGeneralServiceException("Service has been modified from expected configuration"));
boolean notInProgress = updatedDeployments.stream()
.allMatch(deployment -> deployment.status().equals("PRIMARY") || deployment.status().equals("INACTIVE"));
if (!notInProgress) {
throw new CfnGeneralServiceException("Deployment has been modified concurrently");
}
boolean done = primaryLatest.runningCount().equals(primaryLatest.desiredCount());
if (!done) {
Integer maximumFailures = model.getAllowableStartupFailures();
if (maximumFailures == null) {
maximumFailures = 0;
}
if (BadCreateDetection.badExits(client, expectedPrimaryId, createServiceResponse.service().clusterArn()) > maximumFailures) {
throw new CfnGeneralServiceException("Some containers failed during startup");
}
}
return primaryLatest.runningCount().equals(primaryLatest.desiredCount());
}).done((createServiceRequest, createServiceResponse, ecsClient, resourceModel, context) -> ProgressEvent.success(
ResourceModelConverter.merge(createServiceResponse.service(), resourceModel),
context
));
}
}
Activity
dchakrav-github commentedon Nov 20, 2019
thanks for raising the issue. Looking into it right now.
samgurtman-zz commentedon Nov 20, 2019
My primary identifier is the Arn of the ECS Service. I've fixed that particular problem by setting it during the stabilization process. Now I'm getting a different error in cloudwatch logs:
My model for reference:
Basically what I'm doing with this resource is automating rollback of an ECS service as the built in ECS resource is quite limited in this regard.
rjlohan commentedon Nov 20, 2019
Seems like you've hit an issue with our logic for scheduling re-invocation through CloudWatchEvents: https://github.com/aws-cloudformation/aws-cloudformation-rpdk-java-plugin/blob/1dd595a016f2071180203ab299aa415c6f17b04d/src/main/java/software/amazon/cloudformation/scheduler/CloudWatchScheduler.java#L115
We're hitting this limit: https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_Target.html#eventbridge-Type-Target-Input
We'll have to make some changes to fix this issue. Thanks for picking this up!
dchakrav-github commentedon Nov 22, 2019
Can you attach a redacted template with just the resource usage and CreateHandler code here? That would be very helpful.
samgurtman-zz commentedon Nov 22, 2019
samgurtman-zz commentedon Nov 27, 2019
Will leave open for re-invocation bug. Otherwise the original issue is resolved.
dchakrav-github commentedon Dec 2, 2019
Whenever you do get time, please do have samples attached to the issue. I have used your existing definition for the time being,
samgurtman-zz commentedon Dec 9, 2019
fargate-service.tar.gz
This is a version that still has the issue with the cloudwatch events bug. My production version has a workaround for it (involving trimming down the initial AWS response).
ammokhov commentedon Sep 23, 2020
this issue has been solved by this #255 and #279