Answers to: Deployment of iar is very slow

Comment by Reguel Wermelinger on khanh11166's answer

Reguel Wermelinger — Wed, 29 Apr 2020 03:05:17 -0400

Also on linux you can opt in for an Oracle JDK 1.8 and you will be able to use flight recorder.

I do not see something special. Seems like the normal annotation processing work is going on (to detect REST endpoints).

Comment by khanh11166 on khanh11166's answer

khanh11166 — Tue, 28 Apr 2020 04:57:53 -0400

Thanks for your recommend to use Ivy 8.0, it will be in the future near.

I just finished recorded the CPU sampling (using Java VisualVM - can not use JMC because we are using OpenJDK and OpenJDK not include the FlightRecorder feature). I see that on the slow engine, deployment process run some methods as the picture. https://prnt.sc/s792xy

On the fast engine, the deployment process did'n run these methods. Do you have some idea?

Comment by Reguel Wermelinger on khanh11166's answer

Reguel Wermelinger — Mon, 27 Apr 2020 02:29:33 -0400

Thanks for the update @khanh11166

The 'DeploymentTool' can of course be a cause of inefficient deployments. As stated before, it's not an official platform tool. Has not been written by members of the ivy-core and is therefore neither supported nor promoted to be used. So if this DeploymentTools is slow - we can't support you in fixing it. Even with the JMC records.

What I still recommend is to migrate to Ivy 8.0 and benefit of all the new cool deployment features natively provided by the platform - without using any unsupported third parties.

Comment by khanh11166 on khanh11166's answer

khanh11166 — Mon, 27 Apr 2020 00:14:00 -0400

Thanks Reguel, I will record the JMC and analyze it. Btw, I also tried something below: - Delete all data in iwa_businessdata and iwa.tasks -> still slow - Check the deployment log, seem like the slow part is here: https://prnt.sc/s6i2vz

I will back to you later with the JMC recorded

Comment by Reguel Wermelinger on khanh11166's answer

Reguel Wermelinger — Wed, 22 Apr 2020 05:56:29 -0400

That sounds interesting. But I couldn't quickly see any IWA_BusinessData interactions when looking at our deployment process.

However, it should be really simple to analyze what the java process is doing in these 10 mins:

visual vm inspection: thread dumps, gc activity screenshots https://developer.axonivy.com/doc/latest/engine-guide/monitoring/visualvm.html
or even better: flight recorder records taken during these deployments https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/about.htm#JFRUH172

Answer by khanh11166

khanh11166 — Wed, 22 Apr 2020 05:14:42 -0400

Hi Reguel, We have changed to use linux for the engine. At the very clean engine (I meant after finish set it up and running), the deploy process is very fast (about maximum 3 minutes). But it keep slower and slower. Now, the same deploy process took around 10 minutes.

We have another engine (running the same physical server with the one above), still have fast deploy.

When I compared the 2 databases:

The slow one: public.iwa_businessdata ~ 6000 records, about 6.5GB
The fast one: public.iwa_businessdata ~ 300 records, about 150MB (because it use for just a small group of people)

Question is: Is there any step in the deployment process of IVY engine related to public.iwa_businessdata table or other table?

Thanks in advance,

Comment by Reguel Wermelinger on Reguel Wermelinger's answer

Reguel Wermelinger — Thu, 05 Dec 2019 03:03:03 -0500

a glimpse towards 8.0 LTS also shows that deployment scenarios supported by the platform were remarkably extended: https://developer.axonivy.com/doc/8.0.0/engine-guide/administration/deployment.html

... one benefit is for instance, that IAR projects being deployed no longer need to be unzipped at all. They will run in packed format similar as JAR files do in the JRE.

Answer by Reguel Wermelinger

Reguel Wermelinger — Thu, 05 Dec 2019 02:51:28 -0500

Since unzipping is slow I assume that your operating system slows it down: I'd look for involved virus scanners or a crypted filesystem.

Nevertheless, I'd recommend to measure deployment also with a standard deployment approach which is actually supported by the ivy.core platform: https://developer.axonivy.com/doc/7.0.latest/EngineGuideHtml/administration.html#administration-deployment-directory

Comment by Reguel Wermelinger on khanh11166's question

Reguel Wermelinger — Thu, 05 Dec 2019 02:48:02 -0500

the screenshot shows that CPU is blocked while using a third-party for deploying ('soreco deployer'). this is not part of the axonivy.core and may has its own well known quality issues. Moreover, its feature-set is covered natively by the platform. So I'd advise you to switch to a standard deployment approach: https://developer.axonivy.com/doc/7.0.latest/EngineGuideHtml/administration.html#administration-deployment-directory

Comment by khanh11166 on khanh11166's question

khanh11166 — Thu, 05 Dec 2019 00:00:58 -0500

I tried to record by JMC, and found out that https://prnt.sc/q6etrn when unzip the iar file, it took too long....

more details: I'm, using window server

Comment by Reguel Wermelinger on khanh11166's question

Reguel Wermelinger — Thu, 28 Nov 2019 05:31:05 -0500

the screenshot shows that the CPU is high. but still we have no facts what is happening. What would help to nail the issue:

logs from the engine during that time
visual vm inspection: thread dumps, gc activity screenshots https://developer.axonivy.com/doc/latest/engine-guide/monitoring/visualvm.html (yes, maybe the heap is an issue, but how should we know without jvm inspection?)
or even better: flight recorder records taken during these deployments https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/about.htm#JFRUH172

Comment by khanh11166 on khanh11166's question

khanh11166 — Thu, 28 Nov 2019 05:00:34 -0500

Sorry for late reply, today we faced the problem again, and it consume almost all CPU. Could you take a look on the picture and share me some idea?

One question, could I increase the heap will help?!

Comment by Reguel Wermelinger on khanh11166's question

Reguel Wermelinger — Mon, 11 Nov 2019 03:08:17 -0500

ok then, but as said there is no known performance issue at deploy time. so, it's your turn to provide more insights on your actual deployment. hard to advise anything with the facts currently available

Comment by khanh11166 on khanh11166's question

khanh11166 — Sun, 10 Nov 2019 23:08:15 -0500

The file size is depend on module I deploy, sometime the file is just small (<1MB), some other is bigger (~50 MB). But the time is the same same with both 2 of them.

The CPU and RAM is not always 100% at the deploy time. So I'm not sure it related to the issue.

Comment by Reguel Wermelinger on khanh11166's question

Reguel Wermelinger — Fri, 08 Nov 2019 05:24:47 -0500

Hard to say what causes the delays. But the deployment process definitively shouldn't take that long unless your diskspace is almost full or CPU is under constant stress. Maybe the log outputs would help to see which stage takes that long (validation or a certain deployment step?) Can you share the size of the IAR you are trying to deploy as well?