HP-UX上一次高CPU占用问题的分析

一、背景：

本周，另一项目组的一个应用系统更换服务器，由原来的HP-UX11.11迁移到HP-UX11.23上，考虑到是操作系统都是HP-UX，所以直接从源服务器把整个weblogic安装目录（包括域目录）直接复制到目标服务器上，能正常启动和运行。但是发现在执行某些操作的时候，CPU占用很高（weblogic所在的java进程会100%占用4~5颗CPU），应用系统的表现就是反应很慢。

想登录Admin Console看看线程情况也登录不进去。

二、分析过程：

1、根据CPU占用极高的情况来判断，可能是线程很忙，而无法登录Admin Console可能是因为Admin在与出问题的Managed Server进行通讯时发生阻塞所致；如果应用系统中出现死循环，可能会导致这种现象出现，但迁移前后应用系统没有改动，所以死循环这个原因可能性不大；

2、为了查看线程使用情况，用kill -3 <pid>作ThreadDump，先后作了7次，然后对线程情况进行分析，发现有很多线程在读写数据库：

"ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d5f0 nid=21 lwp_id=428827 runnable [0x29f41000..0x29f40530]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at oracle.net.ns.Packet.receive(Unknown Source)
	at oracle.net.ns.DataPacket.receive(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteDescribe(TTC7Protocol.java:850)
	- locked <439f1b58> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2599)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a91e7bb8> (a oracle.jdbc.driver.OracleStatement)
	- locked <439f1bc0> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:295)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:266)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:449)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:359)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:80)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:87)
	at weblogic.jdbc.pool.Driver.connect(Driver.java:148)
	at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:507)
	at weblogic.jdbc.jts.Driver.connect(Driver.java:139)
	at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:305)
	at com.ursa.j2ee.J2eeDataSource.getConnection(J2eeDataSource.java:121)
	at com.ursa.acf.component.datasource.impl.StrutsDSManager.getConnection(StrutsDSManager.java:107)
	at com.ursa.acf.component.datasource.DatabaseHelper.getConnection(DatabaseHelper.java:40)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:217)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

"ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d1b8 nid=18 lwp_id=428824 runnable [0x2a181000..0x2a180530]
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at oracle.net.ns.DataPacket.send(Unknown Source)
	- locked <53bfdb28> (a java.net.SocketOutputStream)
	at oracle.net.ns.NetOutputStream.flush(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:1231)
	- locked <528ca5a8> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2616)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a9200028> (a oracle.jdbc.driver.OracleStatement)
	- locked <52850b28> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.releaseResource(ResourcePoolImpl.java:517)
	at weblogic.jdbc.common.internal.ConnectionPool.release(ConnectionPool.java:604)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.release(ConnectionPoolManager.java:105)
	at weblogic.jdbc.wrapper.PoolConnection.doClose(PoolConnection.java:164)
	at weblogic.jdbc.wrapper.PoolConnection.close(PoolConnection.java:124)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:229)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

还有一些是：

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=00074508 nid=100 lwp_id=428908 runnable [0x26400000..0x26400530]
	at weblogic.socket.PosixSocketMuxer.poll(Native Method)
	at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
	- locked <3ba3f728> (a java.lang.String)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

也有一些在输出日志：

"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e568 nid=32 lwp_id=428840 runnable [0x29701000..0x29700530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:110)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.write(PrintStream.java:258)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c12d840> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c12d700> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)
	at com.ursa.acf.util.SQLUtils.getContents(SQLUtils.java:398)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.getAttrs(DbProcessAdmin.java:40)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.setID(DbProcessAdmin.java:56)
	at com.ursa.acf.plugins.workflow.WorkflowHelper.getAllProcessAdmin(WorkflowHelper.java:148)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:220)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

剩下的多数就是在waitForRequest：

"ExecuteThread: '61' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00072078 nid=74 lwp_id=428882 in Object.wait() [0x27780000..0x27780530]
	at java.lang.Object.wait(Native Method)
	- waiting on <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
	- locked <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)

从以上情况，首先看到有比较多的数据库操作，考虑会不会是数据库出现问题或应用程序中数据库操作部分效率不高，但是本次迁移没有动数据库，应用程序也没改动，所以这种可能性也不是很大；

再综合这些线程来看，这些线程都在处理I/O相关的事情（除去那些“waitForRequest”的线程），会不会是I/O效率低导致的问题呢？我以前曾经遇到过因为weblogic上的“Native I/O”选择没有打开而导致性能低下的案例，这次会不会也是这个问题呢？登录到Admin Console（服务器已经重启过了），看到出问题服务器的“Native I/O”选择是选上的，看来也不是“Native I/O”的问题了。

在ThreadDump日志里，最严重的一个地方是一个线程在写日志：

"ExecuteThread: '39' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00070188 nid=52 lwp_id=428860 runnable [0x28802000..0x28800530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.flush(PrintStream.java:136)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c13bb50> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)

            ……

，另外有55个线程在等待：

"ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e9a0 nid=35 lwp_id=428843 waiting for monitor entry [0x294c2000..0x294c0530]
	at java.util.logging.StreamHandler.publish(StreamHandler.java:173)
	- waiting to lock <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:87)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java
                ……

从这些现象综合来看，应该还是I/O效率的问题。

我再次到网上搜索，找到Weblogic的老家关于I/O调优的部分：

http://download.oracle.com/docs/cd/E13222_01/wls/docs100/perform/WLSTuning.html#wp1152246

里面讲到了“Performance Packs”的问题，根据该文指示的方向：

http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html

，我核对了一下操作系统与weblogic版本的对应的Performance Packs，发现在“HP-UX 11i v2 on PA-RISC”上，并不支持“weblogic SP3”的“Performance Packs”（http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html）。

最后，我分析的结论是：

在迁移时weblogic是直接Copy到目标机上的，复制过去的Performance Packs(

lib/hpux11/libmuxer.sl

)不适合操作系统的版本“HP-UX 11i v2 on PA-RISC”，导致I/O效率低下，进而导致线程池繁忙，占用CPU高。

解决办法是：重新安装weblogic8.1 SP4以上版本，启用“Native I/O”。

=============================

另外，有同事发现weblogic运行在Development Mode，这也可能是影响性能的因素之一，已改成Produectio Mode，正在观察效果。

HP-UX上一次高CPU占用问题的分析

继续阅读

nginx location中斜线的位置的重要性

27 Best Free Eclipse Plug-ins for Java Developer to be ProductiveCode Quality PluginsText Editor PluginsDependency ManagementVersion Control Integration PluginsFramework Development Continuous Integration Related PluginsOther Utility Plugins

Java String.format方法的简单使用

neo4j之cypher使用文档

GitHub连夜封杀！这份阿里 10W 字内部 Java 字面试手册到底有多强？

spark/scala关于【资源文件】加载方法概述外部文件加载方案测试资源文件打包入jar包中小结

mybatis_入门程序Mybatis入门

AOP编程_Android优雅权限框架(1)概念基础，2021金三银四前言正文大纲正文

Effective Java 8:通用程序设计

OOM三种类型

工厂模式-三种类型

【递归】高效率求2的n次幂

win10本地scala和spark安装安装scala安装spark

JBoss,Geronimo和Glassfish初窥

Oracle 批量查询传入List 返回List

scala (3) Function 和 Method