Application server diagnosis: Tomcat workers, JVM threads and downstream waits
Diagnose Java application servers by separating accepted connections, request workers, downstream pools and JVM memory. Read thread evidence and distinguish JMX monitoring from JMeter test plans.
TL;DR: A request can wait before a worker, inside an occupied worker, or at a downstream dependency. Locate that wait with connector metrics, repeated thread evidence and request timing before increasing Tomcat threads or JVM heap.
A connection is not a busy request thread
A Tomcat connector accepts network connections and arranges request processing. With the usual NIO connector, idle keep-alive connections do not each require a permanently occupied request worker. For conventional synchronous handlers, a busy worker can remain occupied while waiting for a database connection or remote response.
Tomcat 10.1 distinguishes worker limits, connection limits and the operating system's incoming connection queue. maxThreads concerns the internal processing pool; maxConnections limits accepted connections, and acceptCount concerns the pending connection backlog once the connector reaches its connection limit. If the connector uses an external executor, tune that executor's worker settings instead of assuming the connector's maxThreads controls it. See the HTTP connector and executor reference.
The arrows show separate capacity constraints. Increasing the first pool does not enlarge the database behind the last one.
Turn the busy-thread count into a hypothesis
In a fictional service, throughput is 100 requests per second and a handler occupies a worker for an average of 200 milliseconds. Under stable conditions that implies about 100 × 0.2 = 20 busy workers. If downstream time grows to two seconds, maintaining the same rate would require about 200 workers. A cap of eighty means the old throughput cannot continue through that synchronous path without growing waits or rejecting demand.
CPU can remain low because the threads are waiting. Raising the cap might merely send more concurrent work to an already constrained database. Measure pool acquisition time separately from query time, as in connection pool exhaustion.
| Evidence | Candidate explanation | Discriminating observation |
|---|---|---|
| Many workers waiting in pool acquisition | Downstream connection scarcity | Checked-out leases and hold time |
| Repeated runnable stacks plus high CPU | Expensive computation | CPU samples and hot stack locations |
| Threads blocked on the same monitor | Lock contention | Lock owner and repeated stack snapshots |
| Latency spikes with long GC pauses | Memory/collector pressure | GC events, allocation and live-set behavior |
These observations narrow an investigation; a single thread-state label does not identify the root cause. Java RUNNABLE is not a measurement of CPU consumption. Compare stacks over time and correlate them with operating-system and request evidence.
Collect bounded JVM evidence
The following commands require a compatible JDK, the correct JVM process ID and attachment permission. Replace the illustrative PID with the intended process. Diagnostic commands can pause or burden a busy JVM, so read their impact in the runtime's own help before using them during an incident.
APP_PID=12345
jcmd "$APP_PID" help Thread.print
jcmd "$APP_PID" Thread.print -l
jcmd "$APP_PID" GC.heap_info
The JDK diagnostic-command reference describes available commands and their impact. Thread printing depends on thread count; a heap dump is a substantially different operation with storage and pause implications. Do not automatically escalate to a heap dump because a dashboard shows high heap occupancy.
Compare live memory after collection across representative workload cycles. A heap that grows while caches warm can stabilize; a retained live set that keeps increasing suggests a different problem. Native allocations, thread stacks and direct buffers can also make process memory exceed Java heap usage. Use the memory retention concept to structure that comparison.
Keep JMX, JMeter and execution models distinct
JMX is Java's management and monitoring mechanism, exposing managed attributes and operations through MBeans. Remote access needs deliberate authentication, transport and network controls. A JMeter .jmx file is an XML test-plan file; its extension does not make it a JVM management session. The two can cooperate during a test, but they provide different evidence.
Async request processing and virtual-thread executors change the relationship between requests and platform threads. State the actual connector and executor before applying a fixed-worker calculation. Database connections, CPU and downstream quotas remain finite even when creating another execution context becomes cheaper.
Self-check: doubling workers reduces front-door queueing briefly, then database waits grow and completed throughput stays flat. Has capacity doubled? No. Work moved farther into the same constrained path. Restore a bounded concurrency policy while investigating database service time and connection retention; verify completed requests and tail latency together.