HP-UX上一次高CPU占用問題的分析

一、背景：

本周，另一項目組的一個應用系統更換伺服器，由原來的HP-UX11.11遷移到HP-UX11.23上，考慮到是作業系統都是HP-UX，是以直接從源伺服器把整個weblogic安裝目錄（包括域目錄）直接複制到目标伺服器上，能正常啟動和運作。但是發現在執行某些操作的時候，CPU占用很高（weblogic所在的java程序會100%占用4~5顆CPU），應用系統的表現就是反應很慢。

想登入Admin Console看看線程情況也登入不進去。

二、分析過程：

1、根據CPU占用極高的情況來判斷，可能是線程很忙，而無法登入Admin Console可能是因為Admin在與出問題的Managed Server進行通訊時發生阻塞所緻；如果應用系統中出現死循環，可能會導緻這種現象出現，但遷移前後應用系統沒有改動，是以死循環這個原因可能性不大；

2、為了檢視線程使用情況，用kill -3 <pid>作ThreadDump，先後作了7次，然後對線程情況進行分析，發現有很多線程在讀寫資料庫：

"ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d5f0 nid=21 lwp_id=428827 runnable [0x29f41000..0x29f40530]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at oracle.net.ns.Packet.receive(Unknown Source)
	at oracle.net.ns.DataPacket.receive(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteDescribe(TTC7Protocol.java:850)
	- locked <439f1b58> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2599)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a91e7bb8> (a oracle.jdbc.driver.OracleStatement)
	- locked <439f1bc0> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:295)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:266)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:449)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:359)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:80)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:87)
	at weblogic.jdbc.pool.Driver.connect(Driver.java:148)
	at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:507)
	at weblogic.jdbc.jts.Driver.connect(Driver.java:139)
	at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:305)
	at com.ursa.j2ee.J2eeDataSource.getConnection(J2eeDataSource.java:121)
	at com.ursa.acf.component.datasource.impl.StrutsDSManager.getConnection(StrutsDSManager.java:107)
	at com.ursa.acf.component.datasource.DatabaseHelper.getConnection(DatabaseHelper.java:40)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:217)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d1b8 nid=18 lwp_id=428824 runnable [0x2a181000..0x2a180530]
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at oracle.net.ns.DataPacket.send(Unknown Source)
	- locked <53bfdb28> (a java.net.SocketOutputStream)
	at oracle.net.ns.NetOutputStream.flush(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:1231)
	- locked <528ca5a8> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2616)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a9200028> (a oracle.jdbc.driver.OracleStatement)
	- locked <52850b28> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.releaseResource(ResourcePoolImpl.java:517)
	at weblogic.jdbc.common.internal.ConnectionPool.release(ConnectionPool.java:604)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.release(ConnectionPoolManager.java:105)
	at weblogic.jdbc.wrapper.PoolConnection.doClose(PoolConnection.java:164)
	at weblogic.jdbc.wrapper.PoolConnection.close(PoolConnection.java:124)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:229)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

還有一些是：

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=00074508 nid=100 lwp_id=428908 runnable [0x26400000..0x26400530]
	at weblogic.socket.PosixSocketMuxer.poll(Native Method)
	at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
	- locked <3ba3f728> (a java.lang.String)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

也有一些在輸出日志：

"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e568 nid=32 lwp_id=428840 runnable [0x29701000..0x29700530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:110)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.write(PrintStream.java:258)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c12d840> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c12d700> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)
	at com.ursa.acf.util.SQLUtils.getContents(SQLUtils.java:398)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.getAttrs(DbProcessAdmin.java:40)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.setID(DbProcessAdmin.java:56)
	at com.ursa.acf.plugins.workflow.WorkflowHelper.getAllProcessAdmin(WorkflowHelper.java:148)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:220)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

剩下的多數就是在waitForRequest：

"ExecuteThread: '61' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00072078 nid=74 lwp_id=428882 in Object.wait() [0x27780000..0x27780530]
	at java.lang.Object.wait(Native Method)
	- waiting on <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
	- locked <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)

從以上情況，首先看到有比較多的資料庫操作，考慮會不會是資料庫出現問題或應用程式中資料庫操作部分效率不高，但是本次遷移沒有動資料庫，應用程式也沒改動，是以這種可能性也不是很大；

再綜合這些線程來看，這些線程都在處理I/O相關的事情（除去那些“waitForRequest”的線程），會不會是I/O效率低導緻的問題呢？我以前曾經遇到過因為weblogic上的“Native I/O”選擇沒有打開而導緻性能低下的案例，這次會不會也是這個問題呢？登入到Admin Console（伺服器已經重新開機過了），看到出問題伺服器的“Native I/O”選擇是選上的，看來也不是“Native I/O”的問題了。

在ThreadDump日志裡，最嚴重的一個地方是一個線程在寫日志：

"ExecuteThread: '39' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00070188 nid=52 lwp_id=428860 runnable [0x28802000..0x28800530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.flush(PrintStream.java:136)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c13bb50> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)

            ……

，另外有55個線程在等待：

"ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e9a0 nid=35 lwp_id=428843 waiting for monitor entry [0x294c2000..0x294c0530]
	at java.util.logging.StreamHandler.publish(StreamHandler.java:173)
	- waiting to lock <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:87)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java
                ……

從這些現象綜合來看，應該還是I/O效率的問題。

我再次到網上搜尋，找到Weblogic的老家關于I/O調優的部分：

http://download.oracle.com/docs/cd/E13222_01/wls/docs100/perform/WLSTuning.html#wp1152246

裡面講到了“Performance Packs”的問題，根據該文訓示的方向：

http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html

，我核對了一下作業系統與weblogic版本的對應的Performance Packs，發現在“HP-UX 11i v2 on PA-RISC”上，并不支援“weblogic SP3”的“Performance Packs”（http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html）。

最後，我分析的結論是：

在遷移時weblogic是直接Copy到目标機上的，複制過去的Performance Packs(

lib/hpux11/libmuxer.sl

)不适合作業系統的版本“HP-UX 11i v2 on PA-RISC”，導緻I/O效率低下，進而導緻線程池繁忙，占用CPU高。

解決辦法是：重新安裝weblogic8.1 SP4以上版本，啟用“Native I/O”。

=============================

另外，有同僚發現weblogic運作在Development Mode，這也可能是影響性能的因素之一，已改成Produectio Mode，正在觀察效果。

HP-UX上一次高CPU占用問題的分析

繼續閱讀

nginx location中斜線的位置的重要性

27 Best Free Eclipse Plug-ins for Java Developer to be ProductiveCode Quality PluginsText Editor PluginsDependency ManagementVersion Control Integration PluginsFramework Development Continuous Integration Related PluginsOther Utility Plugins

Java String.format方法的簡單使用

neo4j之cypher使用文檔

GitHub連夜封殺！這份阿裡 10W 字内部 Java 字面試手冊到底有多強？

spark/scala關于【資源檔案】加載方法概述外部檔案加載方案測試資源檔案打包入jar包中小結

mybatis_入門程式Mybatis入門

AOP程式設計_Android優雅權限架構(1)概念基礎，2021金三銀四前言正文大綱正文

Effective Java 8:通用程式設計

OOM三種類型

工廠模式-三種類型

【遞歸】高效率求2的n次幂

win10本地scala和spark安裝安裝scala安裝spark

JBoss,Geronimo和Glassfish初窺

Oracle 批量查詢傳入List 傳回List

scala (3) Function 和 Method