Docker Swarm Service Container : Capability prefixed with CAP_ cannot be deserialized (InvalidFormatException)

Docker: version 20.10.18, build b40c2f6
docker-java:3.2.8

Description
We have a service deployed to a Docker Swarm, using docker-compose to set the IPC_LOCK capability, as follows:

services:
  myService:
    image: myImage
    cap_add:
      - IPC_LOCK

Calling the docker-java InspectContainerCmdImpl to inspect the container on the node to which the service task has been deployed fails and produces the following exception:

2022-10-11T14:40:41.800756150Z java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "CAP_IPC_LOCK": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
2022-10-11T14:40:41.800761129Z  at [Source: (com.github.dockerjava.core.DefaultInvocationBuilder$2); line: 1, column: 2062] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapAdd"]->java.lang.Object[][0])
2022-10-11T14:40:41.800784357Z 	at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:77) ~[docker-java-core-3.2.8.jar!/:na]
2022-10-11T14:40:41.800787901Z 	at com.github.dockerjava.core.exec.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:31) ~[docker-java-core-3.2.8.jar!/:na]
2022-10-11T14:40:41.800792420Z 	at com.github.dockerjava.core.exec.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:13) ~[docker-java-core-3.2.8.jar!/:na]
2022-10-11T14:40:41.800795370Z 	at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.8.jar!/:na]
2022-10-11T14:40:41.800798829Z 	at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.8.jar!/:na]
2022-10-11T14:40:41.800802249Z 	at com.github.dockerjava.core.command.InspectContainerCmdImpl.exec(InspectContainerCmdImpl.java:52) ~[docker-java-core-3.2.8.jar!/:na]

Based on the documentation, Docker uses a capability name with/without the CAP_ prefix interchangeably:
https://docs.docker.com/engine/reference/run/

The --cap-add and --cap-drop flags accept capabilities to be specified with a CAP_ prefix. The following examples are therefore equivalent:
docker run --cap-add=SYS_ADMIN ...
docker run --cap-add=CAP_SYS_ADMIN ...

This does not appear to be a Docker / Docker swarm bug. Can the CAP_ prefix be automatically stripped to prevent the deserialization issue?

Thanks

there is another issue related to missing values and as you can see those were addressed. The report didn't mention any CAP_ prefix. I understand docker makes the translation itself when using SYS_ADMIN and that's why CAP_SYS_ADMIN is also accepted. However, given the issue I mentioned and this one looks like there is an inconsistency. I am guessing that the previous report is not using docker swarm.

Docker uses a capability name with/without the CAP_ prefix interchangeably

This is related to the data being sent to docker server for which docker-java doesn't use the prefix.

I would like to understand the behavior first between the swarm usage reported and without it. Not saying is a bug on docker swarm but I think some consistency would be appreciated :)

@eddumelendez

Thanks for the quick response @eddumelendez I will take a look at the swarm settings to see if there is an option to change the inspect response, clearly there does appear to be an inconsistency as you stated. When the container is started locally, we don't see the problem, it is only when it is deployed as a swarm service that the exception occurs.

Here is the docker container inspect response for the container deployed through the Docker Swarm service orchestration

[
    {
        "Id": "fe527e7a952e1413479f2ec909c833fb510881c8d4ab4288bddfceb54f95fbe4",
        "Created": "2022-10-06T20:20:38.343440528Z",
        "Path": "/bin/sh",
...
        "HostConfig": {
            "CapAdd": [
                "CAP_IPC_LOCK"
            ],
...

Here is the docker container inspect response for the container running locally:

[
    {
        "Id": "a504f0adda2be1c0fd52bfbfdb366bf5084cd7d240057844f13a559e7b416ba4",
        "Created": "2022-10-06T16:07:34.62992082Z",
        "Path": "/bin/sh",
...
        "HostConfig": {
            "CapAdd": [
                "IPC_LOCK"
            ],

I have temporarily worked around this issue in my application code-base by implementing a wrapper that implements the DockerClientConfig interface and delegates to the actual DefaultDockerClientConfig object; this pattern allows the application to create its own ObjectMapper mapper and control its deserializer configuration.

There are three solutions that I considered (listed below). For the record, I selected option 2. mainly because of its simplicity and the fact that our application doesn't need the Capability value, so replacing it with a null is OK:

Enable the ObjectMapper feature : DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL
This is the crudest approach that prevents the exception, it applies to all deserialized enum classes.

objectMapper.enable(DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL);

Add a DeserializationProblemHandler, to the ObjectMapper.
The handler only acts on the Capability.class, where it replaces unknown enum values's with a null.

        objectMapper.addHandler(new DeserializationProblemHandler() {
            @Override
            public Object handleWeirdStringValue(DeserializationContext ctxt,
                                                 Class<?> targetType, String valueToConvert,
                                                 String failureMsg)
                    throws IOException {
                if (targetType.isAssignableFrom(Capability.class)) {
                    log.info("Failed to assign Capability with value {}", valueToConvert);
                    return null;
                }
                return NOT_HANDLED;

            }
        });

Register a custom deserializer for the Capability class.
This approach strips the CAP_ prefix during deserialization, essentially mapping the swarm's capability value to the matching enum.

     SimpleModule module = new SimpleModule("capability-override-module");
     module.addDeserializer(Capability.class, new CapabilityDeserializer(Capability.class));
     objectMapper.registerModule(module);

and

    static private class CapabilityDeserializer extends StdDeserializer<Capability> {
        private static final String CAP_PREFIX = "CAP_";
        private final int CAP_PREFIX_LEN = CAP_PREFIX.length();

        public CapabilityDeserializer() {
            this(null);
        }

        public CapabilityDeserializer(Class<?> vc) {
            super(vc);
        }

        @Override
        public Capability deserialize(JsonParser jp, DeserializationContext ctxt)
                throws IOException, JsonProcessingException {
            JsonNode node = jp.getCodec().readTree(jp);
            if (node != null) {
                String textValue = node.textValue();
                if (textValue != null && textValue.startsWith(CAP_PREFIX)) {
				    // Strip the CAP_ prefix
                    textValue = textValue.substring(CAP_PREFIX_LEN);
                }
                return Capability.valueOf(textValue);
            } else {
                return null;
            }
        }
    }

I think one of these solutions should be baked into docker-java code-base.
-Graham

@armband

@armband have you considered opening an issue on https://github.com/moby/moby ?

This does not appear to be a Docker / Docker swarm bug. Can the CAP_ prefix be automatically stripped to prevent the deserialization issue?

btw, docker daemon only validates that the capability is the right one. It doesn't remove or add anything. So, if SYS_ADMIN is sent it will add CAP_ verify that's valid and that's all. When using CAP_SYS_ADMIN it will skip adding the prefix. Not sure why swarm behaves differently.

@armband

I have also looked into this; it might be that compose (which version are you using @armband) behaves differently for swarm and local mode and specifically adds CAP_ when in swarm mode.

However, since Docker will transparently set CAP_ prefixed capabilities when creating containers locally as with

docker run -d --cap-add="CAP_SYS_ADMIN" httpd

"CapAdd": [
  "CAP_SYS_ADMIN"
],

as well, I think docker-java must be able to handle those, since docker-java has no control over how a container has been created if it was not created through docker-java.

@armband

@armband are you willing to contribute with option 3?

@kiview

@kiview

which version are you using?

The docker-compose version we are using is fairly recent : v2.11.2

@eddumelendez

have you considered opening an issue on https://github.com/moby/moby ?

I have not, but can. It may be an inconsistency they can address, but I feel the behavior is acceptable based on the wording of their documentation.

are you willing to contribute with option 3?

Of course, I would be happy to contribute.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@armband

@armband Did you have any time to look into contributing the change?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This is is now very relevant for Docker version 28 (released a few hours ago), which includes this PR: moby/moby#48551
So now capabilities always start with CAP_ (Edit: when using the docker CLI), even if you start the container with --cap-add="SYS_ADMIN" it'll show as CAP_SYS_ADMIN in the docker API.

Can this get some priority considering how simple the fix is? There is already a pending PR since quite some time. This lib is basically broken since Docker version 28.

Fixed via #2418

Navigation Menu

Repository navigation

Docker Swarm Service Container : Capability prefixed with CAP_ cannot be deserialized (InvalidFormatException) #1980

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Docker Swarm Service Container : Capability prefixed with CAP_ cannot be deserialized (InvalidFormatException) #1980

Description

Activity

eddumelendez commented on Oct 13, 2022

armband commented on Oct 13, 2022

armband commented on Nov 30, 2022

eddumelendez commented on Nov 30, 2022

eddumelendez commented on Nov 30, 2022

kiview commented on Nov 30, 2022

eddumelendez commented on Nov 30, 2022

armband commented on Nov 30, 2022

stale commented on Mar 23, 2023

kiview commented on Apr 13, 2023

stale commented on Aug 12, 2023

Zinurist commented on Feb 20, 2025

MatthiasGrandl commented on May 12, 2025

eddumelendez commented on May 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions