Spring Cloud Gateway Direct Memory溢出, diagnosis与解决方案

一、错误描述

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 939524103, max: 954728448)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:802)
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:731)
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:648)
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:623)
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:202)
        at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:186)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:136)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
        at org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71)
        at org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
        at org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:91)
        at org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:75)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
                 

二、环境配置

• springboot 2.6.4

• springcloud 2021.0.1

• jdk 1.8

三、原因分析

3.1 Java内存区域

Java内存主要分为堆区和非堆区:

• 堆区:新生代和老年代,用于存储对象实例

• 非堆区:方法区、直接内存、本地方法栈等,方法区用于存储类信息,直接内存用于NIO

3.2 直接内存

直接内存不是JVM运行时数据区的一部分,用于NIO操作。它通过DirectByteBuffer对象进行管理,避免在Java堆和Native堆中复制数据,提高性能。

直接内存的大小不受Java堆限制,但受限于物理内存和处理器寻址空间。如果各个内存区域的总和大于物理内存限制,会导致OutOfMemoryError异常。

3.3错误原因

Spring WebFlux中大量使用了直接内存及Netty操作。在两个地方DirectByteBuffer未正确释放:

(1) 未捕获异常,MyCachedBodyOutputMessage中的DirectByteBuffer未释放。修复方式:

java
MyCachedBodyOutputMessage outputMessage = new MyCachedBodyOutputMessage(exchange, headers);  
return bodyInserter.insert(outputMessage, new BodyInserterContext())
   .then(Mono.defer(() -> {  
       //...  
   }))
   .onErrorResume((Function<Throwable, Mono<Void>>) throwable -> release(exchange, outputMessage, throwable));
           

(2) 使用DataBufferUtils.release()无法释放DefaultDataBuffer的直接内存。修复方式:

DataBufferFactory dataBufferFactory = new NettyDataBufferFactory(ByteBufAllocator.DEFAULT);  
DataBuffer join = dataBufferFactory.join(dataBuffers);  
//...  
DataBufferUtils.release(join);             

四、总结

通过对Java内存区域的理解,查找直接内存的使用场景,corrected其释放问题,解决了直接内存溢出的错误。我们在编程过程中,需要警惕各内存区域的使用和监控,特别是直接内存,以避免此类问题的发生。希望能起到梳理思路与知识点的作用。欢迎检阅,不吝赐教。