Provide default callback implementations for various commands by marcuslinke · Pull Request #281 · docker-java/docker-java

marcuslinke · 2015-07-24T20:55:31Z

No description provided.

Provide default callback implementations for various commands

KostyaSha · 2015-07-24T21:02:04Z

please put spaces after if, hard to read glued text

KostyaSha · 2015-07-24T21:06:46Z

Hi @marcuslinke i think you can provide some time window or call for review other interested people before merge :) Imho it should add some quality degree in this library

marcuslinke · 2015-07-24T21:11:16Z

@KostyaSha Will do so next time. Thanks for reviewing. I will push a fixing commit soon.

KostyaSha · 2015-08-18T19:15:11Z

@marcuslinke what is the right way to print build output into some destination during build?
on 1.4.0 i'm trying to use

                            try (
                                FrameReader frameReader = new FrameReader(
                                        getClient().buildImageCmd(f)
                                                .withTag(tag)
                                                .exec()))
                            {

                                Frame frame;
                                while ((frame = frameReader.readFrame()) != null){
                                    if (frame.getStreamType() == StreamType.STDERR) {
                                        listener.error(frame.toString());
                                    } else {
                                        listener.getLogger().print(frame.toString());
                                    }
                                }
                            } catch (Exception ex) {}

but it didn't work.

marcuslinke · 2015-08-18T19:36:19Z

@KostyaSha On v1.4.0 the build command exec() returns a Response object that has method getItems(). See https://github.com/docker-java/docker-java/blob/docker-java-1.4.0/src/test/java/com/github/dockerjava/core/command/BuildImageCmdImplTest.java#L213. FrameReader was for processing log command response only. But you should really upgrade to v2.0.0

KostyaSha · 2015-08-18T19:39:37Z

@marcuslinke i can update to 2.0.0 docker-plugin if it has no some bug blockers for usage ;)

marcuslinke · 2015-08-18T19:41:51Z

OK, OK your point ;-) I'm working on it...

KostyaSha · 2015-08-18T19:44:54Z

@marcuslinke i see fixes in master, maybe do some 2.0.1 and i will try migrate to it?

KostyaSha · 2015-08-18T21:24:53Z

@marcuslinke getItems() not returned in real-time?

marcuslinke · 2015-08-19T06:32:51Z

@KostyaSha For this you need v2.0.0 of docker-java hence the API is callback-driven there. There is only one issue left that should be fixed in the upcoming v2.0.1 bugfix release (#286).

KostyaSha · 2015-08-28T02:13:50Z

HI, trying buildimageCallback to be progressive

    BuildImageResultCallback resultCallback = new BuildImageResultCallback() {
                                    public void onNext(BuildResponseItem item) {
                                        String text = item.getStream();
                                        if (text != null) {
                                            listener.getLogger().append(text);
                                        }
                                        super.onNext(item);
                                    }
                                };

                                imageId = getClient().buildImageCmd(f)
                                    .withTag(tag)
                                    .exec(resultCallback)
                                    .awaitImageId();

But console output printed only after image fully build...

marcuslinke · 2015-08-28T05:13:25Z

Strange... This is almost the same implementation that is used in the BuildImageCmdImplTest where it works. Maybe the logger is buffering?

KostyaSha · 2015-08-28T11:20:26Z

Interesting idea, will try print() instead (it PrintStream).

KostyaSha · 2015-08-28T11:34:17Z

Docker Build : building tag some:image
Step 0 : FROM centos:centos7
 ---> 7322fbe74aa5
Step 1 : MAINTAINER kanstantsin.sha@gmail.com
 ---> Running in df31f0ab250a
 ---> 5a4c9abbc61f
Removing intermediate container df31f0ab250a
Step 2 : RUN echo Hello world
 ---> Running in fa5e5268d0e5
Hello world
 ---> 579d22f99190
Removing intermediate container fa5e5268d0e5
Step 3 : RUN sleep 30
 ---> Running in 5986f1f6584c
Could not build image: null
com.github.dockerjava.api.DockerClientException: Could not build image: null
    at com.github.dockerjava.core.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:68)
    at com.github.dockerjava.core.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:45)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run$1.invoke(DockerBuilderPublisher.java:250)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run$1.invoke(DockerBuilderPublisher.java:226)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:328)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Yes, you was right, changed to .print() and it fails now in this place :)

KostyaSha · 2015-08-28T12:42:44Z

readTimeout influences, it should be longer than any command output delay... Is it possible to have no timeout for read until tcp connection wouldn't be timeouted?

KostyaSha · 2015-08-28T14:06:51Z

seems i need review what is default and what was enforced in docker-plugin from previous times...

marcuslinke · 2015-08-28T16:12:07Z

Why you suspect a read timeout?

KostyaSha · 2015-08-28T16:19:21Z

@marcuslinke in jaxrs it 0 = infinity by default, probably it was bad idea set-in it.

stefanleh · 2015-10-21T07:35:18Z

Hi, i ran into same problem with (jenkins) docker-plugin 0.15.0 which uses docker-java 2.1.1:

Docker Build
Docker Build: building image at path /home/jenkins/workspace/hybris-docker
Step 0 : FROM isuper/java-oracle
 ---> bbe65a9e7047
Step 1 : MAINTAINER Stefan Lehmann <me@domain.com>
 ---> Using cache
 ---> d9150a636fd3
Step 2 : RUN useradd -d /home/hybris -u 1000 -m -s /bin/bash hybris
 ---> Using cache
 ---> 70126f9b9c50
Step 3 : RUN echo "hybris:hybris" | chpasswd
 ---> Using cache
 ---> d6e90d375689
Step 4 : USER hybris
 ---> Running in 255cfb1b729c
 ---> c5920ff633dd
Removing intermediate container 255cfb1b729c
Step 5 : COPY hybris /home/hybris
ERROR: Build step failed with exception
com.github.dockerjava.api.DockerClientException: Could not build image: null
    at com.github.dockerjava.core.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:68)
    at com.github.dockerjava.core.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:45)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run$1.invoke(DockerBuilderPublisher.java:242)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run$1.invoke(DockerBuilderPublisher.java:228)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2691)
    at hudson.remoting.UserRequest.perform(UserRequest.java:121)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to docker-2086e807f1de(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
    at hudson.remoting.Channel.call(Channel.java:778)
    at hudson.FilePath.act(FilePath.java:980)
    at hudson.FilePath.act(FilePath.java:969)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run.buildImage(DockerBuilderPublisher.java:228)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher$Run.run(DockerBuilderPublisher.java:186)
    at com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher.perform(DockerBuilderPublisher.java:281)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.Build$BuildExecution.build(Build.java:205)
    at hudson.model.Build$BuildExecution.doRun(Build.java:162)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Build step 'Build / Publish Docker Containers' marked build as failure
Finished: FAILURE

Do you know if the issue you was talking about above is fixed in that 2.1.1 version?

Tnx in advance,
Stefan

marcuslinke · 2015-10-21T07:59:54Z

@stefanleh Could you provide logs when doing the very same build from docker CLI please?

stefanleh · 2015-10-21T08:11:45Z

I think its related to the size of the data i try to copy. There i hit some timeout.

    [root@docker-host-fkb playground]# docker build .
Sending build context to Docker daemon 2.398 GB
Step 0 : FROM isuper/java-oracle
 ---> bbe65a9e7047
Step 1 : MAINTAINER Stefan Lehmann <me@domain.com>
 ---> Using cache
 ---> d9150a636fd3
Step 2 : RUN useradd -d /home/hybris -u 1000 -m -s /bin/bash hybris
 ---> Using cache
 ---> 70126f9b9c50
Step 3 : RUN echo "hybris:hybris" | chpasswd
 ---> Using cache
 ---> d6e90d375689
Step 4 : USER hybris
 ---> Using cache
 ---> c5920ff633dd
Step 5 : COPY hybris /home/hybris
 ---> 945866cccfd6
Removing intermediate container a805b4b57e48
Step 6 : EXPOSE 9002
 ---> Running in e166297244ea
 ---> 6b834287b1fe
Removing intermediate container e166297244ea
Step 7 : EXPOSE 9001
 ---> Running in 4c36532dfd58
 ---> f13c2000b56e
Removing intermediate container 4c36532dfd58
Step 8 : CMD /bin/bash /home/hybris/hybris/bin/platform/setantenv.sh && /home/hybris/hybris/bin/platform/hybrisserver.sh
 ---> Running in 3e29772d6bf6
 ---> 89aa17c2b953
Removing intermediate container 3e29772d6bf6
Successfully built 89aa17c2b953
[root@docker-host-fkb playground]#

marcuslinke · 2015-10-21T08:38:56Z

@KostyaSha Any ideas?

stefanleh · 2015-10-21T09:45:36Z

Just had a look into the plugin code. I think the client gets a timeout set somewhere.

in com.nirima.jenkins.plugins.docker.builder.DockerBuilderPublisher.Run.buildImage() they call ...

String imageId = getClient().buildImageCmd(f)
.exec(resultCallback)
.awaitImageId();

for running the callback.

But i cant find where the connection is established and where the timeout is set.

marcuslinke · 2015-10-21T09:49:07Z

@stefanleh Read timeout for docker-java is set as shown here: https://github.com/docker-java/docker-java/wiki#intialize-docker-client-advanced

stefanleh · 2015-10-21T10:33:49Z

Sorry guys, it was my fault. Just found the timeout configuration in jenkins.

For big containers these have to be set according the time needed to transfer the data to the docker host daemon.

marcuslinke · 2015-10-21T11:15:27Z

Good to know :)

Marcus Linke added 2 commits July 24, 2015 22:40

Provide default callback implementations for various commands

58720dc

Remove comments

d54c521

marcuslinke added a commit that referenced this pull request Jul 24, 2015

Merge pull request #281 from docker-java/refact-streaming

f0666ec

Provide default callback implementations for various commands

marcuslinke merged commit f0666ec into master Jul 24, 2015

KostyaSha reviewed Jul 24, 2015
View reviewed changes

marcuslinke deleted the refact-streaming branch August 28, 2015 19:30

Conversation

marcuslinke commented Jul 24, 2015

Uh oh!

KostyaSha Jul 24, 2015

Choose a reason for hiding this comment

Uh oh!

KostyaSha commented Jul 24, 2015

Uh oh!

marcuslinke commented Jul 24, 2015

Uh oh!

KostyaSha commented Aug 18, 2015

Uh oh!

marcuslinke commented Aug 18, 2015

Uh oh!

KostyaSha commented Aug 18, 2015

Uh oh!

marcuslinke commented Aug 18, 2015

Uh oh!

KostyaSha commented Aug 18, 2015

Uh oh!

KostyaSha commented Aug 18, 2015

Uh oh!

marcuslinke commented Aug 19, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

marcuslinke commented Aug 28, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

marcuslinke commented Aug 28, 2015

Uh oh!

KostyaSha commented Aug 28, 2015

Uh oh!

stefanleh commented Oct 21, 2015

Uh oh!

marcuslinke commented Oct 21, 2015

Uh oh!

stefanleh commented Oct 21, 2015

Uh oh!

marcuslinke commented Oct 21, 2015

Uh oh!

stefanleh commented Oct 21, 2015

Uh oh!

marcuslinke commented Oct 21, 2015

Uh oh!

stefanleh commented Oct 21, 2015

Uh oh!

marcuslinke commented Oct 21, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants