Compiling an Album reads meta data all ...

MarkusD

Posts: 759
Registered: 13-Apr-2006

Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 18:50

Hi,

95% of the time I'm compiling albums without new objects (pictures or videos). All that has changed are titles or comments. But I see in the status dialog of jAlbum during the whole compile process "reading meta data". And I don't understand why is jAlbum reading meta data every time, even if nothing has changed? That makes totally no sense to me.

I've a video which shows that process.
https://www.markus-drueck.de/public/jAlbum/Videos/jAlbum%20Core/index.html#img=Video_2025-04-22_183955.mp4

I'm using jAlbum "37.5b3" and "Tiger Version 6.9".

Any idea why jAlbum is reading all objects during compile time?

Thanks, Markus

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 19:58

in response to: MarkusD

Reply

jAlbum needs to gather all input data in order to compile the output data (images and pages). It isn't before jAlbum has read the input data that it can determine that it doesn't need to update a certain page. Scripts may refer to variables in other pages, so if one simply skips the processing of untouched objects (determined by last-modified-date for instance), then there may be broken scripts.

What I could potentially do to speed things up is to cache the read metadata in a faster-to-parse format, but I don't know how much we'd gain in reality. What I've noticed most is that relying on an SSD for album builds has a tremendous effect in metadata processing time, so much so that the net gain will be very small over an SSD. I have, for example, a 6000 image gallery that updates in 2.8s on my computer (Tiger skin, no new images).

There is plenty of metadata in PNG files as well, xmp data for instance. Right-click a PNG image, then select "extenstions/List metadata"

(When reading metadata, it's not that the whole file needs to be read into RAM. The metadata processor tries to use file IO operations like SEEK to locate the section to read)

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:05

in response to: MarkusD

Reply

Markus, is your D drive a mechanical hard drive or a networked drive, and how many CPU cores does your computer have?

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:12

in response to: davidekholm

Reply

davidekholm wrote:
Markus, is your D drive a mechanical hard drive or a networked drive

It is a HDD (mechanical hard).

No networked drives are involved in the process.

and how many CPU cores does your computer have?

8.

During the compiling, the involved hard drive makes noises which are scary. I don't know any other application which "tortures" that drive in that way. Even no heavy database applications.

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:29

in response to: davidekholm

Reply

davidekholm wrote:
jAlbum needs to gather all input data in order to compile the output data (images and pages). It isn't before jAlbum has read the input data that it can determine that it doesn't need to update a certain page. Scripts may refer to variables in other pages, so if one simply skips the processing of untouched objects (determined by last-modified-date for instance), then there may be broken scripts.

I'm aware of this.

What I could potentially do to speed things up is to cache the read metadata in a faster-to-parse format, but I don't know how much we'd gain in reality. What I've noticed most is that relying on an SSD for album builds has a tremendous effect in metadata processing time, so much so that the net gain will be very small over an SSD. I have, for example, a 6000 image gallery that updates in 2.8s on my computer (Tiger skin, no new images).

I totally believe that.

There is plenty of metadata in PNG files as well, xmp data for instance. Right-click a PNG image, then select "extenstions/List metadata"

Yes, you are right:

This is what the extractor spits out

Directories:
PNG-IHDR (class com.drew.metadata.png.PngDirectory)
XMP (class com.drew.metadata.xmp.XmpDirectory)
PNG-pHYs (class com.drew.metadata.png.PngDirectory)
File Type (class com.drew.metadata.file.FileTypeDirectory)
File (class com.drew.metadata.file.FileSystemDirectory)
 
File type.Detected File Type Long Name=Portable Network Graphics
File type.Detected File Type Name=PNG
File type.Detected MIME Type=image/png
File type.Expected File Name Extension=png
File.File Modified Date=Fr. Apr. 04 18:59:42 +02:00 2025
File.File Name=2025-04-04_185940.png
File.File Size=3780650 bytes
Png-ihdr.Bits Per Sample=8
Png-ihdr.Color Type=True Color
Png-ihdr.Compression Type=Deflate
Png-ihdr.Filter Method=Adaptive
Png-ihdr.Image Height=1600
Png-ihdr.Image Width=2560
Png-ihdr.Interlace Method=No Interlace
Png-phys.Pixels Per Unit X=3780
Png-phys.Pixels Per Unit Y=3780
Png-phys.Unit Specifier=Metres
Xmp.XMP Value Count=2
xmp.exif:GPSLatitude=60,36.11321999999987N
xmp.exif:GPSLongitude=7,30.23051999999998E

But, are the last two values are really in the PNG file? The GPS infos are stored in the "info" files:

<?xml version="1.0" encoding="UTF-8"?>
<java version="23.0.1" class="java.beans.XMLDecoder">
 <object class="java.util.HashMap">
  <void method="put">
   <string>keywords</string>
   <string>Bahnhof, Finse, WebCam</string>
  </void>
  <void method="put">
   <string>userVariables</string>
   <object class="java.util.LinkedHashMap">
    <void method="put">
     <string>xWeight</string>
     <string>0.5</string>
    </void>
    <void method="put">
     <string>yWeight</string>
     <string>0.4947</string>
    </void>
   </object>
  </void>
  <void method="put">
   <string>title</string>
   <string>Finse</string>
  </void>
 </object>
</java>

(When reading metadata, it's not that the whole file needs to be read into RAM. The metadata processor tries to use file IO operations like SEEK to locate the section to read)

Yes, I read that in an article. But nevertheless, nearly 11,000 files have to be opened every time to read meta data, which hasn't changed.

And, regarding info about last-modified-date and so on, isn't this what we find in files like ".info"

:
<?xml version="1.0" encoding="UTF-8"?>
<java version="23.0.1" class="java.beans.XMLDecoder">
 <object class="java.util.HashMap">
  <void method="put">
   <string>lastFolderThumbCrop</string>
   <boolean>true</boolean>
  </void>
  <void method="put">
   <string>lastFolderThumbPath</string>
   <string>Misc/Finse/2025 99 Finse/04/2025-04-02_101135.png</string>
  </void>
  <void method="put">
   <string>lastFolderImageSize</string>
   <string>1880x360</string>
  </void>
  <void method="put">
   <string>lastFolderThumbSize</string>
   <string>620x349</string>
  </void>
  <void method="put">
   <string>firstCameraDate</string>
   <long>1743581498103</long>
  </void>
  <void method="put">
   <string>cameraDatesUpdatedDate</string>
   <long>1744984434001</long>
  </void>
  <void method="put">
   <string>lastFolderImagePath</string>
   <string>Misc/Finse/2025 99 Finse/04/2025-04-02_101135.png</string>
  </void>
  <void method="put">
   <string>lastFolderThumbName</string>
   <string>04_2025-04-02_101135.jpg</string>
  </void>
  <void method="put">
   <string>lastCameraDate</string>
   <long>1744872433254</long>
  </void>
  <void method="put">
   <string>title</string>
   <string>April</string>
  </void>
  <void method="put">
   <string>lastFolderThumbDims</string>
   <string>620x349</string>
  </void>
 </object>
</java>

Wouldn't it make sense to store in the ".info" file when jAlbum has read meta data in this folder the last time. And only if the modified timestamp of a file is newer jAlbum has to read meta data again from only these newer files? I'm just thinking out loud.

And yes a SSD would speed up things. But the whole structure under "D:\Eigene Dateien\MDrück\Pictures" is > than 2 TB. Yes, SSDs with this size are affordable these days, still quite some money, just to please jAlbum.

Thanks.

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:43

in response to: davidekholm

Reply

davidekholm wrote:
What I could potentially do to speed things up is to cache the read metadata in a faster-to-parse format, but I don't know how much we'd gain in reality. What I've noticed most is that relying on an SSD for album builds has a tremendous effect in metadata processing time, so much so that the net gain will be very small over an SSD. I have, for example, a 6000 image gallery that updates in 2.8s on my computer (Tiger skin, no new images).

BTW. The folders with my PNG files sum up to 26,4 GB, 5,950 files.

I'll prepare a testing scenario...

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:45

in response to: MarkusD

Reply

You'll get a dramatical improvement if you use an SSD instead. The "torture" likely comes from the head of the drive going back and forth as metadata from different files is read. What could speed things up is if one could read these files in such an order that these movements are minimized. On an SSD, they are no movements at all - naturally.

jAlbum is actually using 24 parallel "IO threads" when reading metadata. I've found this to work well for both SSDs and slow NAS devices, but maybe it's better and faster to use fewer IO threads on your drive (it will certainly cause less forced read head travels). The new b5 version of jAlbum 37.5 allows you to configure the # of IO threads. On Mac you do this by editing jAlbum.cfg (see Tools->Open directories->Program directory) and adding the following under JavaOptions:

java-options=-DioThreads=4

On Windows, you edit jAlbum.ini and add -DioThreads=4 to the VirtualMachineParameters section.

Now when making albums, jAlbum will print the number of IO threads used to the system console. Please play with some different values and see what gives optimal results for you.

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 20:53

in response to: MarkusD

Reply

MarkusD wrote:
But, are the last two values are really in the PNG file? The GPS infos are stored in the "info" files:

GPS coordinates may be written to both the main file and the .info file. Writing to .info in this regard is a backup move.

And, regarding info about last-modified-date and so on, isn't this what we find in files like ".info"

You find some cached data there, also in albumfiles.txt, but not all metadata.

Wouldn't it make sense to store in the ".info" file when jAlbum has read meta data in this folder the last time. And only if the modified timestamp of a file is newer jAlbum has to read meta data again from only these newer files? I'm just thinking out loud.

jAlbum could rely on .info files to cache just the metadata it uses, thereby avoiding parsing it from the main files, but it has to populate the internal variable structure with correct values prior to processing scripts and generating pages. However, I don't know how much speed gain we'd get in reality.

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 21:13

in response to: davidekholm

Reply

davidekholm wrote:
java-options=-DioThreads=4

On Windows, you edit jAlbum.ini and add -DioThreads=4 to the VirtualMachineParameters section.

Did that, compiling the album took half a minute longer, the sound of the hard drive was pretty much the same. Log file her:
https://www.markus-drueck.de/JA/Compiling%20Norge%20at%202025%2004%2022%20-%2020%2056.7z

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 22:28

in response to: MarkusD

Reply

I'm missing the 1:st line from the log file. It should read like this:

Making project name (changes). IO threads: 4

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 22:30

in response to: davidekholm

Reply

davidekholm wrote:
I'm missing the 1:st line from the log file. It should read like this:

Making project name (changes). IO threads: 4

It is in line 24:
Erzeuge "Norge" (�nderungen). IO threads: 4

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 22 Apr 25, 23:41

in response to: MarkusD

Reply

David, I created an archive with most of my PNG files. Just include these folders in your project with the 6,000 other files and lets see what happens.

The archive (be warned, this is 26.4 GB):
The link comes via a private message.

My project file is:
https://www.markus-drueck.de/JA/jalbum-settings.jap

How long does it take if you compile your project the second time? When there is nearly nothing to do.

Thanks. Cheers, Markus

P.S. Please let me know if you have loaded the 7z file, so I can delete that "monster" again.

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 15:20

in response to: MarkusD

Reply

You can remove the monster archive now. Here are my results:

Plain skin:
1:st make: 15 min, 34s (lots of video processing, but I used hardware accelerated video processing)
2:nd make: 5.8s
3:rd make: 4.98s

Tiger skin:
1:st make: 1 min, 39s (all images reprocessed)
2:nd make: 6.53s
3:rd make: 5.8s
4:th make: 4.84s

(Album information says 2508 objects. I made the project with ONLY the files in the archive)

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 15:33

in response to: davidekholm

Reply

I'm impressed!

I did a test with moving these elements to a SSD too.

2:nd make:
"jAlbum-Finse-Test SSD" -> 36,393s

Time to get new equipment. Nevertheless I'm really curious which equipment other users have. More SSD than HDD? If SSD, which speed? 8 or more cores?

David, thanks for your time! I suspect that in the medium term it makes no sense to invest time in determining the metadata in a more resource-efficient way.

JeffTucker

Posts: 8,090
Registered: 31-Jan-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 15:46

in response to: MarkusD

Reply

My 7-year-old mid-priced desktop PC came with a 512GB SSD, and a 6-core, dual-threading Intel i7 chip. I'm thinking about retiring it.

These days, it's almost impossible to buy a new PC with a spinning hard drive. Even the low-end, pre-configured desktops from Dell arrive with a 512GB SSD as the primary drive. In a dual-drive configuration, the second drive is often another SSD, and only sometimes a regular hard drive. I'll wager that "moving parts" drives disappear entirely within a year or two.

The bargain PC's are also showing up routinely with 20-core processors.