This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 9 - Pages: 1 - Last Post: 05-Apr-2018 18:08 Last Post By: karlmistelberger Threads: [ Previous | Next ]
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Java Heap Space
Posted: 10-Jan-2015 10:04
 
  Click to reply to this thread Reply
Available heap space depends on the java version installed:
karl@erlangen:~> java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.8) (suse-24.17.1-x86_64)
OpenJDK 64-Bit Server VM (build 24.55-b03, mixed mode)
karl@erlangen:~> java -Xmx4000m -DuseDesktop=true -jar /usr/share/jalbum/JAlbum.jar
The above invocation works fine with 64 bit Java. Memory is allocated as needed up to the maximum declared by -Xmx4000m. There is no penalty for small albums when declaring a huge heap size, but for large albums build times are lower.
Nordschleife

Posts: 76
Registered: 18-Dec-2012
Re: Java Heap Space
Posted: 11-Jan-2015 17:48   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
Thanks for the tip - this speeds up my large album a lot.
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Re: Java Heap Space
Posted: 25-Jun-2016 21:33   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
Wondering about out of memory errors encountered by some users and reading about changing heap size with effects unsupported by own experience I had a closer look at the java virtual machine and found some facts which may be helpful to other users:

  • You may query the maximum heap size supported by your own machine:
    erlangen:~ # /usr/lib/jalbum/jre64/bin/java -XshowSettings:vm
    VM settings:
        Max. Heap Size (Estimated): 3.41G
        Ergonomics Machine Class: server
        Using VM: Java HotSpot(TM) 64-Bit Server VM
     
    Usage: java [-options] class [args...]
               (to execute a class)
       or  java [-options] -jar jarfile [args...]
               (to execute a jar file)
    where options include:
        -d32          use a 32-bit data model if available
        -d64          use a 64-bit data model if available
        -server       to select the "server" VM
                      The default VM is server,
                      because you are running on a server-class machine.
    ... 
    See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details.
    erlangen:~ # 
    
    Obviously my machine has a Max. Heap Size of 3.41GB.

  • checking the parent process tree:
    erlangen:~ # pstree 20744
    java─┬─ffmpeg───7*[{ffmpeg}]
         └─69*[{java}]
    erlangen:~ 
    
    Java does not care about processor numbers. It spawns 69(!) children of its own and one instance of ffmpeg which spawns another 7 children of ffmpeg.

  • Making a large album (33,761 images and videos, 80 GB altogether) with jAlbum 13.2.4 consumes all of the memory available without impairing the responsiveness of the GUI:
    erlangen:~ # free -h
                 total       used       free     shared    buffers     cached
    Mem:           15G        15G        86M       568M       181M       9.5G
    -/+ buffers/cache:       5.6G       9.7G
    Swap:         7.5G       1.0M       7.5G
    erlangen:~ # 
    

  • omitting the (default) -Xmx1200m value from the command line causes the virtual machine to use 3.41GB and does decrease build times moderately: from 2:29h to 2:24h for a build from scratch and from 2:16m to 2:11m for a build of all skin files only.

Conclusions

While increasing heap size is necessary with the 32 bit version of Java, it does not matter much when using the 64 bit version. You may safely delete the parameter from the command line and will not experience any inconvenience.

More about jvm
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Re: Java Heap Space
Posted: 05-Jul-2016 07:45   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
Checked Virtual machine parameters on a very old 32 bit machine:
hofkirchen:~ # java -XshowSettings:vm
VM settings:
    Max. Heap Size (Estimated): 910.25M
    Ergonomics Machine Class: server
    Using VM: Java HotSpot(TM) Server VM
 
Usage: java [-options] class [args...]
           (to execute a class)
   or  java [-options] -jar jarfile [args...]
           (to execute a jar file)
where options include:
    -d32          use a 32-bit data model if available
    -d64          use a 64-bit data model if available
    -client       to select the "client" VM
    -server       to select the "server" VM
    -minimal      to select the "minimal" VM
                  The default VM is server,
                  because you are running on a server-class machine.
Both the 64 bit and the 32 bit of Oracle Java now use -server as a default, so specifying -Xmx on the command line has become obsolete in most cases as a very reasonable heap size is selected by default. Users building large albums should check whether -server is default on their machines and if not specify it on the command line:
java -server -jar /usr/share/jalbum/JAlbum.jar 
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Transcoding the camera originals
Posted: 12-Nov-2016 17:19   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
Attachment jalbum1.png (155.5 KB)
Viewings jAlbums Image directory with the Smart TV via a DLNA Media Server is an interesting option. But most Smart TV do not support older video formats, such as .avi and .mov. Transcoding these videos with ffmpeg is painless and can be done issuing a single command:
 for i in $(find Bilder/ -name *.mov) ; do ffmpeg -i $i -pix_fmt yuv420p ${i%*mov}mp4; done
Transcoding jAlbum's input movies requires rebuilding videos and skin files. jAlbum was invoked as /usr/lib/jalbum/jre64/bin/java -jar /usr/lib/jalbum/JAlbum.jar and claimed some 3.5 GB of heap space (see screenshot) and another 20 GB of OS cache. Processing 680 video clips (12.1 GB, equivalent to several DVDs) took some 29 minutes to complete which compares favorably with total playtime of several hours.
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Processing large images
Posted: 05-Apr-2018 08:41   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
No tuning of heap space is necessary when processing large images. A test was run using the following invocation of jAlbum:
 /usr/lib/jalbum/jre64/bin/java -jar /usr/lib/jalbum/JAlbum.jar
The above invocation assigns heap size as displayed by the following command:
karl@erlangen:/home/Albums/testjAlbum> /usr/lib/jalbum/jre64/bin/java -XshowSettings:vm
VM settings:
    Max. Heap Size (Estimated): 3.84G
    Using VM: Java HotSpot(TM) 64-Bit Server VM
...
karl@erlangen:/home/Albums/testjAlbum>
Upon the above invocation jAlbum successfully processed an image with 31033x5051 pixels (size 1.2 GB).

Users may consider to remove jAlbum's default limit -Xmx1200m from their settings.
jGromit

Posts: 34,221
Registered: 31-Jan-2006
Re: Processing large images
Posted: 05-Apr-2018 11:05   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
karlmistelberger wrote:
Users may consider to remove jAlbum's default limit -Xmx1200m from their settings.

The jAlbum 15.3.4 default on a 64-bit Windows system is -Xmx4000M.

Edit: On macOS, the default is -Xmx2000M.
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Re: Processing large images
Posted: 05-Apr-2018 13:37   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:
karlmistelberger wrote:
Users may consider to remove jAlbum's default limit -Xmx1200m from their settings.

The jAlbum 15.3.4 default on a 64-bit Windows system is -Xmx4000M.

Edit: On macOS, the default is -Xmx2000M.


Of course your results depend on OS type, phases of the moon and many more. Users may want to consult the Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide of the Java Platform.

Java comes with a nice feature:

java -Xloggc:log -XX:+UseParallelGC -jar /usr/lib/jalbum/JAlbum.jar


The above selects the Parallel Garbage Collector and logs to file log:

 [0.007s][info][gc] Using Parallel
[0,911s][info][gc] GC(0) Pause Young (Metadata GC Threshold) 40M->6M(236M) 5,258ms
[0,933s][info][gc] GC(1) Pause Full (Metadata GC Threshold) 6M->6M(236M) 22,283ms
[1,876s][info][gc] GC(2) Pause Young (Metadata GC Threshold) 44M->11M(236M) 8,136ms
[1,926s][info][gc] GC(3) Pause Full (Metadata GC Threshold) 11M->11M(236M) 49,808ms
[3,696s][info][gc] GC(4) Pause Young (Allocation Failure) 73M->19M(236M) 20,167ms
[5,051s][info][gc] GC(5) Pause Young (Allocation Failure) 81M->27M(236M) 18,415ms
[8,351s][info][gc] GC(6) Pause Young (Allocation Failure) 88M->32M(234M) 19,594ms
[8,634s][info][gc] GC(7) Pause Young (GCLocker Initiated GC) 182M->131M(240M) 18,879ms
[9,664s][info][gc] GC(8) Pause Young (GCLocker Initiated GC) 203M->151M(244M) 19,657ms
[10,455s][info][gc] GC(9) Pause Young (GCLocker Initiated GC) 182M->159M(248M) 20,602ms
[10,584s][info][gc] GC(10) Pause Young (GCLocker Initiated GC) 162M->159M(252M) 22,003ms
[10,756s][info][gc] GC(11) Pause Full (Ergonomics) 159M->39M(221M) 171,491ms
[12,502s][info][gc] GC(12) Pause Young (Allocation Failure) 100M->44M(221M) 8,360ms
[13,004s][info][gc] GC(13) Pause Young (Allocation Failure) 105M->47M(261M) 14,353ms
[13,032s][info][gc] GC(14) Pause Young (Allocation Failure) 149M->46M(263M) 3,291ms
[13,248s][info][gc] GC(15) Pause Young (Allocation Failure) 148M->123M(318M) 17,606ms
[13,398s][info][gc] GC(16) Pause Young (Allocation Failure) 280M->262M(421M) 45,155ms
[13,537s][info][gc] GC(17) Pause Full (Ergonomics) 262M->187M(721M) 138,172ms
[13,710s][info][gc] GC(18) Pause Young (Allocation Failure) 467M->529M(810M) 60,102ms
[13,951s][info][gc] GC(19) Pause Full (Ergonomics) 529M->466M(1081M) 241,189ms
[14,122s][info][gc] GC(20) Pause Young (Allocation Failure) 746M->796M(1200M) 76,316ms
[14,376s][info][gc] GC(21) Pause Full (Ergonomics) 796M->744M(1556M) 253,822ms
[14,559s][info][gc] GC(22) Pause Young (Allocation Failure) 1036M->1069M(1655M) 78,423ms
[15,221s][info][gc] GC(23) Pause Young (Allocation Failure) 1361M->1340M(1706M) 139,537ms
[15,337s][info][gc] GC(24) Pause Full (Ergonomics) 1340M->43M(1205M) 115,939ms
[16,407s][info][gc] GC(25) Pause Young (Allocation Failure) 329M->48M(1253M) 9,585ms
[20,249s][info][gc] GC(26) Pause Young (Allocation Failure) 391M->73M(1263M) 14,821ms
jGromit

Posts: 34,221
Registered: 31-Jan-2006
Re: Processing large images
Posted: 05-Apr-2018 15:50   in response to: karlmistelberger in response to: karlmistelberger
 
  Click to reply to this thread Reply
karlmistelberger wrote:
Of course your results depend on OS type, phases of the moon and many more.

No, these are simply the current jAlbum default values. For 99% of jAlbum users, these are the values they will get with the current release of the application. There are no other complications.

jAlbum is automatically invoking the PGC. It doesn't log the results, but there are almost no users who would know what the log files meant, anyway.
karlmistelberger

Posts: 832
Registered: 5-Dec-2013
Re: Processing large images
Posted: 05-Apr-2018 18:08   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:
jAlbum is automatically invoking the PGC.

Checked jAlbum's v15.3.4 defaults and found it uses G1.

Most users experiencing problems related to memory usage will want to follow Oracle's recommendations:

https://docs.oracle.com/javase/10/gctuning/garbage-first-garbage-collector-tuning.htm

General Recommendations for G1

The general recommendation is to use G1 with its default settings, eventually giving it a different pause-time goal and setting a maximum Java heap size by using -Xmx if desired.

G1 defaults have been balanced differently than either of the other collectors. G1's goals in the default configuration are neither maximum throughput nor lowest latency, but to provide relatively small, uniform pauses at high throughput. However, G1's mechanisms to incrementally reclaim space in the heap and the pause-time control incur some overhead in both the application threads and in the space-reclamation efficiency.

If you prefer high throughput, then relax the pause-time goal by using -XX:MaxGCPauseMillis or provide a larger heap. If latency is the main requirement, then modify the pause-time target. Avoid limiting the young generation size to particular values by using options like -Xmn, -XX:NewRatio and others because the young generation size is the main means for G1 to allow it to meet the pause-time. Setting the young generation size to a single value overrides and practically disables pause-time control.
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums