Compiling an Album reads meta data all ...

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 16:36

in response to: JeffTucker

JeffTucker wrote:
These days, it's almost impossible to buy a new PC with a spinning hard drive. Even the low-end, pre-configured desktops from Dell arrive with a 512GB SSD as the primary drive. In a dual-drive configuration, the second drive is often another SSD, and only sometimes a regular hard drive.

The bargain PC's are also showing up routinely with 20-core processors.

I do have several "drives" in my machine:

C: SSD 0.5 GB (System-Partition)
D: HDD 10.0 TB
E: HDD 4.0 TB
F: SSD 1.0 TB (Virtual Machines)

The D: drive holds pictures and videos. Replacing the 10 TB and 4 TB HDD with SSDs is quite a challenge.

I'll wager that "moving parts" drives disappear entirely within a year or two.

This is true for 95% of regular home users. But those who really have a lot of data (videos for example) HDDs still make sense, I think.

JeffTucker

Posts: 8,090
Registered: 31-Jan-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 17:01

in response to: MarkusD

Reply

At this point, the 4TB SSD's still fall into the "consumer" category - not cheap, but the prices don't make all the blood drain from your head. When you get into the bigger ones, however, you're in the realm of "stuff that people who run data centers buy by the caseload." Kind of out of reach for the rest of us. But that will change, and probably quickly.

RobM

Posts: 3,940
Registered: 4-Aug-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 17:31

in response to: MarkusD

Reply

MarkusD wrote:
Time to get new equipment. Nevertheless I'm really curious which equipment other users have. More SSD than HDD? If SSD, which speed? 8 or more cores?

Mac laptop, 8 cores, built in SSD, three external 1TB SSD 1050MB/s, a time machine, an archive and one for storing media and caches for use with other applications.

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 18:57

in response to: RobM

Reply

By the way. The specs on my laptop are:
MacBook Pro M4 (10 core), 24GB RAM, 2TB SSD.

The disk performance is excellent as everything is basically integrated on the same chip (not possible to change/upgrade)

Apple charges crazy prices for such a large SSD, but I really value large storage, running several virtual machines on this computer in order to test jAlbum. I also run a virtual Win 11 + two Linux distros on this machine.

Luckily, SSDs are far more affordable if you're on non Macs or desktop computers.

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 19:03

in response to: davidekholm

Reply

I issued a Profiler.instance in the system console in order to see how much time jAlbum spends in various locations during the make of your huge project. Here's how it sums up:

       AlbumBean.registerVariables:  2838 calls      3,836s
     AlbumObjectImpl.getXmpManager:  3105 calls       3,72s
       AlbumObjectImpl.getMetadata:  2688 calls      3,545s
       Process compiled expression:   118 calls       1,29s
                 predir processing:   115 calls      1,272s
     AlbumObjectImpl.getProperties:  2669 calls      1,136s
        AlbumObjectProperties.load:  2671 calls       1,12s
          AlbumBean.makeIndexPages:   115 calls      0,455s
      Executing compiled scriptlet:   741 calls      0,389s
                   Create MediaRSS:   115 calls      0,241s
       RecoveryTool.createLifeboat:     1 calls      0,214s
     FileFilters.getBasicImageInfo:  2669 calls      0,213s
                 AlbumBean.popVars: 11084 calls      0,194s
   Executing interpreted scriptlet: 13859 calls      0,134s
                AlbumBean.pushVars: 11085 calls      0,116s
           Reading file attributes:  1347 calls      0,083s
           JSONMaker.makeDataPages:   115 calls      0,043s
             Making deep-data.json:     1 calls      0,035s
                     Writing pages:   347 calls      0,026s
                              init:     1 calls      0,025s
   JAlbumUtilities.countCategories:  1032 calls      0,024s
JAlbumUtilities.getDeepCameraDates:   115 calls      0,022s
     AlbumBean.getFolderProperties:   199 calls      0,011s
         AlbumBean.countTotalFiles:     1 calls      0,007s
              Compiling scriptlets:    57 calls      0,001s
                Process expression:     1 calls      0,001s
                    Copy res files:     1 calls      0,001s
                JSONMaker.makeTree:     1 calls          0s

The majority is metadata related, but all in all, it's so fast on a modern computer, so why worry?

It will be interesting to see your timings as well

MarkusD

Posts: 759
Registered: 13-Apr-2006

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 21:47

in response to: davidekholm

Reply

davidekholm wrote:
It will be interesting to see your timings as well

Here we go:

       AlbumBean.registerVariables:  2678 calls     27,659s
     AlbumObjectImpl.getXmpManager:  2438 calls     25,783s
       AlbumObjectImpl.getMetadata:  2581 calls     25,217s
       Process compiled expression:    80 calls      10,59s
                 predir processing:    77 calls      9,618s
     AlbumObjectImpl.getProperties:  2613 calls      5,795s
        AlbumObjectProperties.load:  2615 calls      5,715s
          AlbumBean.makeIndexPages:    77 calls      2,054s
                     Writing pages:   233 calls      1,921s
                              init:     1 calls      1,593s
                   Create MediaRSS:    77 calls      1,288s
     FileFilters.getBasicImageInfo:  2548 calls      1,256s
      Executing compiled scriptlet: 16865 calls      1,104s
   JAlbumUtilities.countCategories:   690 calls      0,838s
                 AlbumBean.popVars: 10401 calls      0,776s
   Executing interpreted scriptlet: 13391 calls      0,744s
           JSONMaker.makeDataPages:    77 calls       0,66s
                  Compiling script:     4 calls      0,556s
                AlbumBean.pushVars: 10402 calls      0,469s
              Compiling scriptlets: 16409 calls      0,323s
     AlbumBean.getFolderProperties:   201 calls      0,229s
           Reading file attributes:   714 calls      0,211s
         AlbumBean.countTotalFiles:     1 calls      0,202s
             Making deep-data.json:     1 calls      0,163s
JAlbumUtilities.getDeepCameraDates:    77 calls      0,097s
             FileFilters.loadImage:    36 calls      0,095s
                JSONMaker.makeTree:     1 calls      0,028s
                Process expression:     1 calls      0,022s
                    Copy res files:     1 calls      0,007s
                             Total: 85290 calls   2m 5,013s

davidekholm

Posts: 3,699
Registered: 18-Oct-2002

Re: Compiling an Album reads meta data all the time, why?
Posted: 23 Apr 25, 22:36

in response to: MarkusD

Reply

Thanks. Same pattern but a slower computer and slower IO. Comparing AlbumObjectProperties.load and AlbumObjectImpl.getMetadata reveals a 4-5x performance difference but AlbumObjectProperties.load is also faster because it's less metadata to load, so if we were to cache all metadata into the AlbumObjectProperties format (the .info files), then we'll likely "only" get a 2-3x speed gain at best, at metadata reading, that is, compared to the 20x speed gain in total a new computer + SSD gives.