Scaling Threads on Linux – Per Account or Per Application

Failures such as:

java.lang.OutOfMemoryError: unable to create new native thread

can drive you to do much googling and after that much frustration.  I know it did us. Here’s our current compendium of answers.

Before I get going here, one thing we found really handy was to test whether the change was making the expected difference.  You can write something yourself or see below at the end for a trivial example.

Linux Tunables:

First you’ve got to enable a large process count since each thread consumes one ProcessID on Linux.

    1.a. Use “sysctl” to manage “kernel.Threads-max” – must set to level that is sufficient for entire system, including all threads and processes.

    1.b. Use “sysctl” to manage “kernel.pid_max” – must be set to at least 4X times “kernel.threads-max” in order to provide reasonably fast allocation of ProcessIDs during process and thread creation.

    1.c. There are other system limits that may cause the thread creation to fail.  google around a bit and you’ll find “vm.max_map_count” mentioned but there are definitely others.  Beware the “max” tunables in the sysctl set (there are a lot of them).

Second, you need to make sure that you can use them (assuming that your executing account is not “root”).

    2.a. Edit “/etc/security/limits.conf” (or create files in “/etc/security/limits.d/”) and enable “nproc” to be big enough for all your heavy-weight and thread “processes”.

    2.b. Same for “nofile”.  This one can be a bit strange.  You might think that you’ll need 1 file for every process’s STDIN/STDOUT/STDERR but that’s not so: threads will share the same one.  Forked parent and child processes share them too (until changed). So this number can be much smaller than you might think. Base the number on the expected concurrent number of files to be opened by your Application – then multiply by a safety factor (2X, 4X, … depending on how accurate you think you are).  If you get this wrong you’ll need to make it bigger in the future.

Third, there’s “SystemD”.  I suspect that where the old rule was that “every application grows to include email”, the new rule is that “SystemD will grow to include all system tunables”.  There are now 2 places you need to be aware of.

    3.a. If your application is running via a SystemD unit file, then you will probably need to have them set the “TasksMax” tunable (depends on the version of SystemD).  This should match your setting in 2.a. above.

    3.b. If your application is running via a login process then you may need to edit the file “/etc/systemd/logind.conf” and set the “UserTasksMax” tunable (again, depends on the version of SystemD).  This should match your setting in 2.1. above.

Notes:

• The “sysctl” command makes changes immediately.  However those changes will not survive a reboot. To do so you must edit one of the configuration files that it reads on startup (e.g. “/etc/sysctl.conf”).

• Changes to “limits.conf”, et. al., require a logout/login before you’ll see them take effect.

• Changes to SystemD Unit files require “systemctl daemon-reload; systemctl restart serviceName” before they take effect.

• Changes to “logind.conf” require a “systemctl restart systemd-logind” and a logout/login before you’ll see them take effect.

• Also, depending on your application and just how many threads you need to create, you might just want to drop the stack size a bit.  In java this is the “-Xss” option.

• Trivial Example:  below is some simple java code you can compile/run on Linux.  When it’s “done” you’ll have to SIGSTOP (normally Control-Z) the java process and then kill it using “kill -KILL %” (the ‘%’ represents the newly stopped process).  Or wait until all the thread sleep()’s timeout (hard coded to 45 seconds – should be plenty to run the experiment).

import java.util.concurrent.TimeUnit;

class CreateThreads
{
    private static long count = 0;

    public static void main(String args[])
    {
        while(true) {
            new Thread(new Runnable() {
                public void run() {
                    try {
                        Thread.sleep(TimeUnit.SECONDS.toMillis(45));
                    } catch(Exception e) {
                        e.printStackTrace();
			Runtime.getRuntime().halt(1);
                    } 
                }    
            }).start();
            count++;
            System.out.printf("Thread count is %d%n" , count);
        }
    }
}

Hopefully the information above is all you need to know.  If not, let us know!

#linux #scaling #java #threads #suse #rhel #centos #systemd