java not enough memory error.

After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12) 12 posts by 7 authors

Patrick Heppler

10/28/13

 

Hi,

I'm running railo 4.1.1.009 on a centos box. Today I updated JRE (Oracle) from jre-7_21 to jre-7_45. And now when starting jetty I get:

Starting Jetty: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007e0000000, 536870912, 2097152, 0) failed; error='Cannot allocate memory' (errno=12)

 

With jre-7_21 I had no issue. Railo is running fine, anyway. 

 

My Java Args (from a Railo Blog ;) ):

-server -Djava.io.tmpdir=/dev/shm -javaagent:/opt/jetty/lib/ext/railo-inst.jar -Dsun.io.useCanonCaches=false -Xms1024m -Xmx2048m -XX:PermSize=512M -XX:MaxPermSize=512M -XX:NewSize=512M -XX:MaxNewSize=512M  -XX:GCTimeRatio=5 -XX:ThreadPriorityPolicy=42 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -XX:MaxGCPauseMillis=50 -XX:+DisableExplicitGC -XX:MaxHeapFreeRatio=70 -XX:MinHeapFreeRatio=40 -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseTLAB -XX:+ScavengeBeforeFullGC -XX:CompileThreshold=1500 -XX:+TieredCompilation -XX:+UseBiasedLocking -Xverify:none -XX:+UseThreadPriorities -XX:+UseLargePages  -XX:LargePageSizeInBytes=2m -XX:+UseFastAccessorMethods -XX:+UseCompressedOops -Djava.security.egd=/dev/urandom -Dfile.encoding=UTF-8 -Duser.language=de -Duser.region=DE -Duser.timezone=Europe/Berlin -Dcoldfusion.jsafe.defaultalgo=FIPS186Random

 

 

 

 

Phil Sweeney

11/2/13

 

To be honest, I think using a bunch of JVM flags from blogs/google searches is a bad idea, and is potentially why you have run into this problem.

When I went through this exercise late last year, I researched each flag I saw mentioned around the place.  Most of them were already set as default in modern JVMs and others were simply misunderstood.

For our high traffic (and now very fast) site, this is all I have for my JVMs:

export CATALINA_OPTS="-Xms768m \
                      -Xmx768m \
                      -XX:NewSize=448m \
                      -XX:MaxNewSize=448m \
                      -XX:MaxTenuringThreshold=32 \
                      -XX:MaxPermSize=128m \
                      -XX:+UseConcMarkSweepGC \
                      -XX:+CMSClassUnloadingEnabled \
                      -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses \

The New size relates to the nature of the site, in that it has much more temporary allocations than permanent.  But I did that based on analysis with visualvm and then testing.
Obviously you would choose your memory sizes based on your site profile.

But every single one is there for a demonstrable reason.

Phil

- show quoted text -

 

 

 

 

Igal

11/2/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

I think using a bunch of JVM flags from blogs/google searches is a bad idea, and is potentially why you have run into this problem.

I agree

but having said that I think that you also set some flags that you shouldn't mess with (like MaxNewSize and MaxTenuringThreshold) unless you have a very good reason to do so.

I recommend setting the following flags:

    -Xms={minimum-heap-memory-in-mb} m
    -Xmx={maximum-heap-memory-in-mb}m
    -XX:MaxPermSize ={maximum-perm-gen-in-mb} m
    -XX:+HeapDumpOnOutOfMemoryError

the memory values should be set according to physical RAM size.  MaxPermSize should be 256m if possible (though 128m can be used as minimum), and -Xmx should be between 1024m and 4096m, again, depends on available RAM.

- show quoted text -

- show quoted text -

-- 
Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at  https://github.com/getrailo/railo/wiki
--- 
You received this message because you are subscribed to the Google Groups "Railo" group.
To view this discussion on the web visit  https://groups.google.com/d/msgid/railo/88afc251-7daa-4ad4-8d1b-bcd7a651ea3b%40googlegroups.com.
For more options, visit  https://groups.google.com/groups/opt_out.

-- 

Igal Sapir

Railo Core Developer

http://getRailo.org/

 

 

 

 

Patrick Heppler

11/4/13

 

Just FYI  https://www.farbeyondcode.com/Optimizing-Tomcat-s-JVM-Options-for-Railo-4-1-5-3001.html

This is where I got this settings from. But I'll try the JVM Settings recommended

Am Montag, 28. Oktober 2013 12:53:35 UTC+1 schrieb Patrick Heppler:

- show quoted text -

 

 

 

 

Kai Koenig

11/5/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

That's fine, but be aware and warned that JVM/GC settings are always heavily depending on one's hardware, OS, configuration, application as well as _optimisation goals_. 

 

I strongly advise against copying any sort of JVM/GC settings from anyone's blog or tutorial. They might really well work for Bruce's setup but they might (and apparently do) cause issue for you.

 

Cheers

Kai

- show quoted text -

 

 

 

 

Igal

11/5/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

This is where I got this settings from

then I suggest that Bruce update his blog post ;)

- show quoted text -

- show quoted text -

-- 
Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at  https://github.com/getrailo/railo/wiki
--- 
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit  https://groups.google.com/d/msgid/railo/585C7698-F6A1-4422-9DA1-D26019A8AE9D%40gmail.com.

For more options, visit  https://groups.google.com/groups/opt_out.

-- 

Igal Sapir

Railo Core Developer

http://getRailo.org/

 

 

 

 

Patrick Heppler

11/11/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

I now switched back from custom jetty setup to railo/tomcat (installer bundle) because I had an issue with javaagent.

 

I now have this Java Opts:

JAVA_OPTS="-server -Djava.io.tmpdir=/dev/shm -javaagent:/usr/share/nginx/html/ example.com/WEB-INF/lib/railo-inst.jar -Dsun.io.useCanonCaches=false -Xms1024m -Xmx2048m -XX:MaxPermSize=256M -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=/dev/urandom -Dfile.encoding=UTF-8 -Duser.language=de -Duser.region=DE -Duser.timezone=Europe/Berlin -Dcoldfusion.jsafe.defaultalgo=FIPS186Random";

 

I still get the memory errors, not on railo start but on restart. Linux top command shows more then 1.4 GB free while railo is running.

Railo Server Admin shows 30% Heap and 20% Non-Heap used.

 

Am Montag, 4. November 2013 20:40:06 UTC+1 schrieb Igal:

- show quoted text -

 

 

 

 

Chris Blackwell

11/11/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

Patrick (and Bruce) 

 

Do not use JAVA_OPTS to pass arguments to tomcat, use CATALINA_OPTS

 

for an explanation of why, see ->  http://www.wellho.net/mouth/2163_CATALINA-OPTS-v-JAVA-OPTS-What-is-the-difference-.html

 

basically, if you set -Xms1024m in JAVA_OPTS the java process that is spawned to restart tomcat also uses that amount of memory, so only use global options in JAVA_OPTS, not per process stuff which for tomcat should be placed in CATALINA_OPTS.

 

Chris

- show quoted text -

- show quoted text -

To view this discussion on the web visit  https://groups.google.com/d/msgid/railo/39c735b9-e2ee-4486-8647-ee7f25621066%40googlegroups.com.

For more options, visit  https://groups.google.com/groups/opt_out.

 

 

 

 

Jordan Michaels

11/12/13

 

Re: [railo] Re: After Update from jre-7_21 to jre-7_45: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory() failed; error='Cannot allocate memory' (errno=12)

Good find Chris! 

https://github.com/utdream/CFML-Installers/issues/57 

Warm Regards, 
Jordan Michaels 

On 11/11/2013 07:28 AM, Chris Blackwell wrote: 
> Patrick (and Bruce) 
> 
> Do not use JAVA_OPTS to pass arguments to tomcat, use CATALINA_OPTS 
> 
> for an explanation of why, see -> 
>  http://www.wellho.net/mouth/2163_CATALINA-OPTS-v-JAVA-OPTS-What-is-the-difference-.html 
> 
> basically, if you set -Xms1024m in JAVA_OPTS the java process that is 
> spawned to restart tomcat also uses that amount of memory, so only use 
> global options in JAVA_OPTS, not per process stuff which for tomcat 
> should be placed in CATALINA_OPTS. 
> 
> Chris 
> 
> 
> On 11 November 2013 12:05, Patrick Heppler < [email protected] 

> <mailto: [email protected]>> wrote: 
> 
>     I now switched back from custom jetty setup to railo/tomcat 
>     (installer bundle) because I had an issue with javaagent. 
> 
>     I now have this Java Opts: 
>     JAVA_OPTS="-server -Djava.io.tmpdir=/dev/shm 
>     -javaagent:/usr/share/nginx/html/ example.com/WEB-INF/lib/railo-inst.jar 

>     < http://example.com/WEB-INF/lib/railo-inst.jar> 

>     -Dsun.io.useCanonCaches=false -Xms1024m -Xmx2048m 
>     -XX:MaxPermSize=256M -XX:+HeapDumpOnOutOfMemoryError 
>     -Djava.security.egd=/dev/urandom -Dfile.encoding=UTF-8 
>     -Duser.language=de -Duser.region=DE -Duser.timezone=Europe/Berlin 
>     -Dcoldfusion.jsafe.defaultalgo=FIPS186Random"; 
> 
>     I still get the memory errors, not on railo start but on restart. 
>     Linux top command shows more then 1.4 GB free while railo is running. 
>     Railo Server Admin shows 30% Heap and 20% Non-Heap used. 
> 
> 
>     Am Montag, 4. November 2013 20:40:06 UTC+1 schrieb Igal: 
> 
>>         This is where I got this settings from 
>         then I suggest that Bruce update his blog post ;) 
> 
> 
>         On 11/4/2013 11:38 AM, Kai Koenig wrote: 
>>         That's fine, but be aware and warned that JVM/GC settings are 
>>         always heavily depending on one's hardware, OS, configuration, 
>>         application as well as _optimisation goals_. 
>> 
>>         I strongly advise against copying any sort of JVM/GC settings 
>>         from anyone's blog or tutorial. They might really well work 
>>         for Bruce's setup but they might (and apparently do) cause 
>>         issue for you. 
>> 
>>         Cheers 
>>         Kai 
>> 
>>>         Just FYI 

>>>          https://www.farbeyondcode.__com/Optimizing-Tomcat-s-JVM-__Options-for-Railo-4-1-5-3001.__html 

>>>         < https://www.farbeyondcode.com/Optimizing-Tomcat-s-JVM-Options-for-Railo-4-1-5-3001.html> 
>>> 
>>>         This is where I got this settings from. But I'll try the JVM 
>>>         Settings recommended 
>>> 
>>>         Am Montag, 28. Oktober 2013 12:53:35 UTC+1 schrieb Patrick 
>>>         Heppler: 
>>> 
>>>             Hi, 
>>>             I'm running railo 4.1.1.009 on a centos box. Today I 
>>>             updated JRE (Oracle) from jre-7_21 to jre-7_45. And now 
>>>             when starting jetty I get: 
>>>             Starting Jetty: Java HotSpot(TM) 64-Bit Server VM 

>>>             warning: INFO: os::commit_memory(__0x00000007e0000000, 

>>>             536870912, 2097152, 0) failed; error='Cannot allocate 
>>>             memory' (errno=12) 
>>> 
>>>             With jre-7_21 I had no issue. Railo is running fine, anyway. 
>>> 
>>>             My Java Args (from a Railo Blog ;) ): 
>>>             -server -Djava.io.tmpdir=/dev/shm 

>>>             -javaagent:/opt/jetty/lib/ext/__railo-inst.jar 

>>>             -Dsun.io.useCanonCaches=false -Xms1024m -Xmx2048m 
>>>             -XX:PermSize=512M -XX:MaxPermSize=512M -XX:NewSize=512M 
>>>             -XX:MaxNewSize=512M  -XX:GCTimeRatio=5 
>>>             -XX:ThreadPriorityPolicy=42 -XX:ParallelGCThreads=4 
>>>             -XX:+UseParNewGC -XX:MaxGCPauseMillis=50 
>>>             -XX:+DisableExplicitGC -XX:MaxHeapFreeRatio=70 
>>>             -XX:MinHeapFreeRatio=40 -XX:+UseStringCache 
>>>             -XX:+OptimizeStringConcat -XX:+UseTLAB 
>>>             -XX:+ScavengeBeforeFullGC -XX:CompileThreshold=1500 
>>>             -XX:+TieredCompilation -XX:+UseBiasedLocking 
>>>             -Xverify:none -XX:+UseThreadPriorities -XX:+UseLargePages 
>>>              -XX:LargePageSizeInBytes=2m -XX:+UseFastAccessorMethods 

>>>             -XX:+UseCompressedOops -Djava.security.egd=/dev/__urandom 
>>>             -Dfile.encoding=UTF-8 -Duser.language=de -Duser.region=DE 
>>>             -Duser.timezone=Europe/Berlin 
>>>             -Dcoldfusion.jsafe.__defaultalgo=FIPS186Random 

>>> 
>> 
>> 
>>         -- 
>>         Did you find this reply useful? Help the Railo community and 
>>         add it to the Railo Server wiki at 

>>          https://github.com/getrailo/__railo/wiki 

>>         < https://github.com/getrailo/railo/wiki> 
>>         --- 
>>         You received this message because you are subscribed to the 
>>         Google Groups "Railo" group. 
>>         To view this discussion on the web visit 

>>          https://groups.google.com/d/__msgid/railo/585C7698-F6A1-__4422-9DA1-D26019A8AE9D%__40gmail.com 
>>         < https://groups.google.com/d/msgid/railo/585C7698-F6A1-4422-9DA1-D26019A8AE9D%40gmail.com>. 
>>         For more options, visit 
>>          https://groups.google.com/__groups/opt_out 
>>         < https://groups.google.com/groups/opt_out>. 

> 
>         -- 
>         Igal Sapir 
>         Railo Core Developer 
>          http://getRailo.org/ 
> 
>     -- 
>     Did you find this reply useful? Help the Railo community and add it 
>     to the Railo Server wiki at  https://github.com/getrailo/railo/wiki 
>     --- 
>     You received this message because you are subscribed to the Google 
>     Groups "Railo" group. 
>     To view this discussion on the web visit 
>      https://groups.google.com/d/msgid/railo/39c735b9-e2ee-4486-8647-ee7f25621066%40googlegroups.com. 
> 
>     For more options, visit  https://groups.google.com/groups/opt_out. 
> 
> 

> -- 
> Did you find this reply useful? Help the Railo community and add it to 
> the Railo Server wiki at  https://github.com/getrailo/railo/wiki 
> --- 
> You received this message because you are subscribed to the Google 
> Groups "Railo" group. 
> To view this discussion on the web visit 

>  https://groups.google.com/d/msgid/railo/CAB%3DtfToJGsJU%2BC5PiVU%3DJ6mq5Yz9voDkEAuuG7Tpy0_%2B6bSKyg%40mail.gmail.com. 

- show quoted text -

 

 

 

 

I made some tweaks to the blog article now, but I still prefer the non-default options I use.  Some of them probably have no effect, others have a real effect on garbage collection timings.

 

My app runs a bunch of CMS sites that have very little write activity.  OS / Hardware is pretty typical.  Works good still.

- show quoted text -

 

 

 

 

Patrick Heppler

11/12/13

 

Changed JAVA_OPTS to CATALINA_OPTS, works fine.

Am Montag, 28. Oktober 2013 12:53:35 UTC+1 schrieb Patrick Heppler:

Hi,

- show quoted text -

 

 

 

 

Bruce Kirkpatrick

11/12/13

 

If that was the problem, I'd guess the system has not much memory available to the rest of the system.  You might want to leave around 25% of the memory unused so linux can cache stuff and avoid swapping.  I usually give Railo less then 1/3 of the total memory so database and other services have a 1/3 and system has a third.  I'd give database a greater share in a larger system.

- show quoted text -