Featured: Why streaming not closing can cause a memory leak

introduction

You're often told to turn off your stream when it's run out, otherwise it will lead to a memory leak, but have you ever considered the following questions:

1. Why does not close the stream cause a memory leak?
2. Doesn't the JVM have a garbage collection mechanism? These references don't become garbage when I use them, so why aren't they recycled?
3. The stream is not closed except for causing a memory leak? Does it raise other questions?

In this paper, we will discuss the underlying working principle of IO flow again.

The problem is repeated

Code demo

Let's start with a piece of sample code, 1w file input streams will be created every time we request, and it will not be closed after the creation is completed, and then we will request this interface through the stress testing tool.

java           

Copy the code

@RequestMapping("noClose") public String noClose() throws FileNotFoundException { //Create 1w input file input streams every time a request comes in for (int i = 0; } i < 10000; i++) { openFileStream(); } return "success"; } private static void openFileStream() throws FileNotFoundException { InputStream is = new FileInputStream("data.txt"); }

To see the effect faster, let's adjust the heap memory to 50m:

java           

Copy the code

-Xmx50m

Problem location

Then we did the interface stress test with JMeter, and soon the problem appeared:

java           

Copy the code

Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded ..... Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause analysis

To do this, we use jps to locate the process number, and then export the memory information:

java           

Copy the code

jmap -dump:live,format=b,file=xxxx.hprof pid

If you open the exported xxxx.hprof through mat, you can see that the top 3 classes contain File-related, and the memory leak problem is obviously due to our operation on the file.

Let's start with the second-ranked FileDescriptor, each FileInputStrean maintains a FileDescriptor, which can be thought of as a file descriptor, which is a handle to open a file.

java           

Copy the code

public class FileInputStream extends InputStream { /* File Descriptor - handle to the open file */ private final FileDescriptor fd; //略 }

The corresponding call to our constructor above is as follows:

1. Generate a File object with the incoming file name and call another constructor.
2. Another construct method for security as well as file validity checks.
3. Create a file descriptor and associate it with the current flow to ensure that it can be closed later.
4. Open the file by calling the open function of the operating system to open the file and get the file handle, and then our flow is associated with the system resources.

java           

Copy the code

public FileInputStream(String name) throws FileNotFoundException { this(name != null ? new File(name) : null); } public FileInputStream(File file) throws FileNotFoundException { //Security check String name = (file != null ? file.getPath() : null); } SecurityManager security = System.getSecurityManager(); if (security != null) { security.checkRead(name); } //File validity check if (name == null) { throw new NullPointerException(); } } } if (file.isInvalid()) { throw new FileNotFoundException("Invalid file path"); } //Create a file descriptor, get a file handle and set it on fd fd = new FileDescriptor(); fd.attach(this); path = name; open(name); }

Therefore, if we do not close the stream after using it, the FileDescriptor will always hold the operating system resources, so when the JVM does garbage collection, these classes will not be GC in time because the file resources have not been released.

In order to verify whether the stream holds the resource, we can also try to delete the file on the computer after the above code is executed, and the final result will be as shown in the following figure, you can see that the file can never be deleted, and it is obvious that it is owned by FileDescriptor.

Therefore, when the GC performs garbage collection, the FileDescriptor object cannot be recycled in time, and the stream not closed will not only lead to memory leakage, but also lead to continuous occupation of system resources, affecting the use of system resources by other processes.

Let's take a look at the number one Finalizer, because it's related to garbage collection, we can directly use the Finalizer class to locate the problem, so we can click with outgoing references to see which classes it references:

You can see that the class internally references FileInputStream (the 3rd class that consumes the 3rd memory), which in turn refers to FileDescriptor (the 2nd class).

As we can see in FileInputStream, it overrides the finalize method, which can be seen from the code to release the stream and return the system resources to the FileDescriptor that is not collected in time.

java           

Copy the code

protected void finalize() throws IOException { if ((fd != null) && (fd != FileDescriptor.in)) { close(); } }

The author found that the classes that rewrite the finalize method will be referenced by the Finalizer, and because they are referenced by the Finalizer, even if we use the complete and exit function, these objects will not be recycled during GC.

Only when the GC is completed, the JVM will mark these classes that are only referenced by the Finalizer and store them in the ReferenceQueue queue, until the Finalizer thread discovers and calls finalize. The Finalizer thread will remove our FileInputStream and FileDescriptor from the Finalizer reference and will be recycled on the next GC.

Because the Finalizer thread has a very low priority, the garbage is collected very infrequently, which is why we see a large number of classes in memory snapshots that the Finalizer points to are not collected in time.

Because we manually close the stream, resulting in a large number of FileDescriptor classes holding file streams and system resources, so that the FileDescriptor can not be recycled by the GC, you need to use the Finalizer thread to call finalize to release system resources before it has the qualification to be GC, because the Finalizer thread priority is extremely low, the creation speed of the stream is much greater than the recycling speed, and eventually the heap memory cannot be released in time and memory leak.

solution

The solution is also very simple, just close the stream in time, and jdk7 also provides us with try-with-resource, and the syntax needs to be more concise.

java           

Copy the code

private static void openFileStream() { try(InputStream is = new FileInputStream("data.txt")) { } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } }

Further

In fact, some classes can not close the stream after we complete the operation, for example: ByteArrayOutputStream and ByteArrayInputStream, we look at its close method, we can see that it is an empty implementation, the reason is very simple, they operate the data stream in memory when they operate the bytes, and will not hold the operating system file resources, of course, in order to unify the development habits, we still recommend that the reader call close when operating the stream.

java           

Copy the code

public void close() throws IOException { }

brief summary

If the I/O flow is not closed, the following objects may not be reclaimed:

1. FileInputStream or other input stream objects: If you don't close the FileInputStream object, it will always hold the handle to the underlying file, which may result in file resources not being freed. Such objects will not be able to be collected by the garbage collector.
2. FileOutputStream or other output stream objects: Similarly, if you don't close the FileOutputStream object, it may hold a handle to the underlying file and may cause data in the write buffer to not be flushed to disk. This can lead to resource leakage and data loss.
3. Sockets or other network-related objects: If you don't close sockets or other network-related objects, they may remain connected to the remote host, which will result in network resources not being freed up, which will not be reclaimed by the garbage collector, and may also lead to a situation where the port number is hogged and other threads cannot use the port.
4. BufferedReader, BufferedWriter, or other buffered stream objects: If you don't turn them off, they may hold the underlying input or output stream objects and may cause the data to fail to refresh or the buffer data to fail to clear. This can lead to resource leakage and data loss.

It's important to note that even if these objects are not explicitly turned off, in some cases they may be automatically collected when the garbage collector executes. However, this depends on the specific garbage collection algorithm and implementation, so we can't rely on this behavior. The correct thing to do is to explicitly call their close() method to close the stream and release the associated resources after you've used them to prevent resource leakage and data loss.