Alfresco, JVM tuning and monitoring with JMX

Tuning and deeper inspection of the Java Virtual Machine is a very important activity for java-based applications. In an enterprise-class installation of Alfresco where often one or multiple clustered instances are running, a huge amounts of data are stored with a lot of user access that cause often a http overload. Like many other Java applications, Alfresco can be optimized by a careful and meticulous tuning of the hardware resources, specifically cpu and ram used by the JVM. I am not versed in the details regarding java thread management, you can found further details here. To make a succesfull JVM tuning, the Alfresco official wiki suggests the following approach.

Once the heap memory has the maximum allowed value, you can proceed with additional configuration parameters by editing the JAVA_OPTS variable which is located in the Alfresco startup script.Here the JAVA_OPTS settings in a default Alfresco installation.

JAVA_OPTS="-XX:MaxPermSize=512m -Xms128m -Xmx1024m -XX:-DisableExplicitGC"
JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true"
JAVA_OPTS="${JAVA_OPTS} -Dalfresco.home=/opt/alfresco-4.1.1.3"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"

In a recent documentation, Alfresco has released a table of possible values ​​for the heap memory based on the number of users accessing the system.

 

Note that for these metrics, N concurrent users is considered equivalent to 10xN casual users that the server could support. Follow a best practice for an Alfresco installation where about 5 milions of documents are stored and  frequently solr searches are performed.

– Guest Alfresco = 4.0.0 Enterprise (2 cluster nodes)
– Guest RAM = 10 GB
– Guest CPU = 4 virtual cores
– Hypervisor = Esxi 5.0
– Host CPU = Intel Xeon E5640 2.67GHz
– Storage = 1 TB SAN (via NFS)

JAVA_OPTS="-XX:MaxPermSize=512m -Xms8000m -Xmx8000m -XX:-DisableExplicitGC"
JAVA_OPTS="${JAVA_OPTS} -Djava.awt.headless=true"
JAVA_OPTS="${JAVA_OPTS} -Dalfresco.home=/opt/alfresco-4.1.1.3"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
JAVA_OPTS="${JAVA_OPTS} -Xss1024K -XX:NewSize=1G -XX:+UseConcMarkSweepGC -server"
JAVA_OPTS="${JAVA_OPTS} -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80"
JAVA_OPTS="${JAVA_OPTS} -XX:ParallelGCThreads=4 -XX:+UseParNewGC"

Now, let’s see how to perform a deep monitoring of the JVM.

Case 1. Using jstad and jvisualvm client.

– Server JVM

$ vi JAVA_HOME_SERVER/bin/jstatd.policy

grant codebase "file:${java.home}/../lib/tools.jar" {
        permission java.security.AllPermission;
};

$  ./jstatd -J-Djava.security.policy=./jstatd.policy -J-Djava.rmi.server.hostname=192.168.71.64

 

– Client JVM

$ JAVA_HOME/jvisualvm

 

Case 2. Using JMX and jvisualvm client (Alfresco Enterprise 3.2 and higher).

– Server JVM

By default the JMX interface allows you to access Alfresco Enterprise via a standard JMX console that supports JMX Remoting like jvisualvm. There are no more settings to do for Alfresco server. Only a important note for Redhat/Centos operative systems: Alfresco has an in process RMI registry that will be started on the IP specificed by the command line property -Djava.rmi.server.hostname and the port in alfresco.rmi.services.host. If you have multiple network interfaces, you must start Alfresco defining the “java.rmi.server.hostname” with the server IP address. A second solution is to add a new local dns entry in the /etc/hosts file.

$ vi ALFRESCO_HOME/tomcat/scripts/ctl.sh
JAVA_OPTS="-Djava.rmi.server.hostname=192.168.1.23 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false"
$ vi /etc/hosts
192.168.1.23	myalfresco

 

– Client JVM

Run jvisualvm and attach a JMX connection using the default Alfresco credentials.

service:jmx:rmi:///jndi/rmi://192.168.1.23:50500/alfresco/jmxrmi
-username (default): controlRole
-password (default): change_asap