Skip to content

[wip] CLOUDSTACK-8708: gson update and ArrayTypeAdaptor rewrite#932

Closed
DaanHoogland wants to merge 15 commits into
apache:masterfrom
DaanHoogland:gson-upgrade
Closed

[wip] CLOUDSTACK-8708: gson update and ArrayTypeAdaptor rewrite#932
DaanHoogland wants to merge 15 commits into
apache:masterfrom
DaanHoogland:gson-upgrade

Conversation

@DaanHoogland
Copy link
Copy Markdown
Contributor

unit test for RequestTest is passing in eclipse. On macosx 10.11 using java 1.8 or 1.7 it fails from the command line.

Comment thread pom.xml Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our standard currently is to pin to a specific version, in this case you depend on any 2.4.X version. Although I understand your reasons, is this the PR to change that behavior?

It makes testing harder when deps change as long as we don't have proper testing in place.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the moment this list is a mess. not all deps are in it and the list is unsorted making it very hardd to maintain so I would prefer an automatic update like above.

@DaanHoogland
Copy link
Copy Markdown
Contributor Author

So far so good, all test passed in CI. This touches every command and answer that is serialized, however. please all test

@wido
Copy link
Copy Markdown
Contributor

wido commented Oct 20, 2015

So, this PR is hard to test. For now I've verified that it compiles on my Ubuntu desktop with Java 8 and all the Unit Tests succeed, so that's good.

Asking @borisroman if we can spin up a KVM cluster and verify the Agent communcation.

This PR is a blocker for #625 since that PR also requires the new Gson.

@borisroman
Copy link
Copy Markdown
Contributor

@wido Will do!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you think about using "return (Answer[])Array.newInstance(Answer.class, size);"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is the same and the optimizer will treat it the same but this is easier to debug

@rafaelweingartner
Copy link
Copy Markdown
Member

@ DaanHoogland,
I ran the tests from my Windows environment, after I did some work arounds everything started passing the tests. The fixes I did I created a PR into your branch.

I used JDK 1.8.66, running from eclipse into a Windows 8 environment.

@DaanHoogland
Copy link
Copy Markdown
Contributor Author

Thanks @rafaelweingartner , i'll look at your bit and merge it in.

@borisroman
Copy link
Copy Markdown
Contributor

@DaanHoogland I tested this PR again today. 👎

When using the listSystemVms API call:

INFO  [c.c.a.ApiServer] (1975058800@qtp-1945752215-4:ctx-c6162e68 ctx-1705495f) class 
org.apache.cloudstack.api.response.SystemVmResponse declares multiple JSON fields named jobid

And in general when collecting host stats:

ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-a786a412) Error trying to retrieve storage stats
java.lang.ArrayIndexOutOfBoundsException: 0
    at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:365)
    at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:125)
    at com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:725)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
WARN  [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-887e9d12) Exception while sending
java.lang.ArrayIndexOutOfBoundsException: 0
    at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:365)
    at com.cloud.agent.manager.AgentManagerImpl.easySend(AgentManagerImpl.java:938)
    at com.cloud.resource.ResourceManagerImpl.getHostStatistics(ResourceManagerImpl.java:2485)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy158.getHostStatistics(Unknown Source)
    at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:380)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThradPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
WARN  [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-887e9d12) Unable to obtain host 1 statistics. 

So it doesn't work in my setup. Could you have a look again? And also run the vm lifecycle integration tests and post the results, please.

Ping @wido

@borisroman
Copy link
Copy Markdown
Contributor

@DaanHoogland Could you please have a look again?

WARN  [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-94b6fe0e) Exception while sending
java.lang.ArrayIndexOutOfBoundsException: 0
    at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:365)
    at com.cloud.agent.manager.AgentManagerImpl.easySend(AgentManagerImpl.java:938)
    at com.cloud.resource.ResourceManagerImpl.getHostStatistics(ResourceManagerImpl.java:2485)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy159.getHostStatistics(Unknown Source)
    at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:380)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
WARN  [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-94b6fe0e) Unable to obtain host 7 statistics. 
WARN  [c.c.s.StatsCollector] (StatsCollector-1:ctx-94b6fe0e) Received invalid host stats for host: 7
WARN  [c.c.a.m.AgentManagerImpl] (StatsCollector-1:ctx-d81486fe) Exception while sending
java.lang.ArrayIndexOutOfBoundsException: 0
    at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:365)
    at com.cloud.agent.manager.AgentManagerImpl.easySend(AgentManagerImpl.java:938)
    at com.cloud.resource.ResourceManagerImpl.getHostStatistics(ResourceManagerImpl.java:2485)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy159.getHostStatistics(Unknown Source)
    at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:380)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
WARN  [c.c.r.ResourceManagerImpl] (StatsCollector-1:ctx-d81486fe) Unable to obtain host 7 statistics. 
WARN  [c.c.s.StatsCollector] (StatsCollector-1:ctx-d81486fe) Received invalid host stats for host: 7
WARN  [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (1660072634@qtp-1397971223-12:ctx-2c1dc089 ctx-28ffad24 ctx-73d05ad9) Unable to establish a connection between Host[-7-Routing] and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4b8ae89a
java.lang.NullPointerException
    at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:91)
    at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:954)
    at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:414)
    at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:708)
    at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:191)
    at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
    at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
    at com.cloud.api.ApiServer.queueCommand(ApiServer.java:703)
    at com.cloud.api.ApiServer.handleRequest(ApiServer.java:528)
    at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:296)
    at com.cloud.api.ApiServlet$1.run(ApiServlet.java:127)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:124)
    at com.cloud.api.ApiServlet.doGet(ApiServlet.java:86)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
WARN  [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (1660072634@qtp-1397971223-12:ctx-2c1dc089 ctx-28ffad24 ctx-73d05ad9) No host can access storage pool org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4b8ae89a on cluster 7
INFO  [c.c.a.ApiServer] (1660072634@qtp-1397971223-12:ctx-2c1dc089 ctx-28ffad24 ctx-73d05ad9) Failed to add data store: Failed to access storage pool

Adding primary storage doesn't function :(

@remibergsma
Copy link
Copy Markdown
Contributor

@DaanHoogland I have the same result as @borisroman reports:

==== Deploy DC Started ====
Exception Occurred: ['Traceback (most recent call last):\n', '  File "/data/git/cs1/cloudstack/tools/marvin/marvin/deployDataCenter.py", line 254, in createPrimaryStorages\n    ret = self.__apiClient.createStoragePool(primarycmd)\n', '  File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 1727, in createStoragePool\n    response = self.connection.marvinRequest(command, response_type=response, method=method)\n', '  File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest\n    raise e\n', 'CloudstackAPIException: Execute cmd: createstoragepool failed, due to: errorCode: 530, errorText:Failed to add data store: Failed to access storage pool\n']

@DaanHoogland
Copy link
Copy Markdown
Contributor Author

This is far from done and i'll change the title to be prefixed with [wip] it is a crosscutting change the touches all internal communication of cloudstack. It is still at a 'what unit tests do we need to define?' state. Even if we pass all integration tests we can not be completely sure of this one, I'm afraid

@DaanHoogland DaanHoogland changed the title CLOUDSTACK-8708: gson update and ArrayTypeAdaptor rewrite [wip] CLOUDSTACK-8708: gson update and ArrayTypeAdaptor rewrite Oct 23, 2015
@DaanHoogland
Copy link
Copy Markdown
Contributor Author

@remibergsma @borisroman @wido I just pushed a 650 file commit. I fair the future of this change.

@borisroman
Copy link
Copy Markdown
Contributor

Well...

We could have expected it...

https://sites.google.com/site/gson/gson-roadmap

After version 2.0 Primtive Types weren't allowed anymore.

If it works and is properly tested, then the future is bright :)

@DaanHoogland DaanHoogland force-pushed the gson-upgrade branch 2 times, most recently from 2b1b556 to 8576aa7 Compare October 27, 2015 12:58
@DaanHoogland
Copy link
Copy Markdown
Contributor Author

get #987 working first then rethink this code.

@DaanHoogland DaanHoogland deleted the gson-upgrade branch August 16, 2023 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants