Permlink Replies: 31 - Pages: 3 [ Previous | 1 2 3 | Next ] - Last Post: 13 Jan 25, 20:32 Last Post By: MarkusD Threads: [ Previous | Next ]
MarkusD

Posts: 685
Registered: 13-Apr-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 06:38   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
David, here we go:
1,014s ==> 
               AlbumBean.registerVariables: 17113 calls  7m 48,747s
               AlbumObjectImpl.getMetadata: 15838 calls  6m 27,289s
             AlbumObjectImpl.getXmpManager: 18489 calls  6m 23,727s
               Process compiled expression:   765 calls   3m 44,56s
                         predir processing:   762 calls  3m 35,128s
             AlbumObjectImpl.getProperties: 16362 calls  2m 12,943s
                AlbumObjectProperties.load: 16100 calls  1m 30,693s
                     FileFilters.loadImage:   278 calls     44,964s
              Executing compiled scriptlet: 100529 calls     29,299s
             FileFilters.getBasicImageInfo: 15653 calls     28,091s
             AlbumBean.getFolderProperties:  1419 calls      26,59s
                  AlbumBean.makeIndexPages:   762 calls      22,07s
                HardwareSmoothScaler.scale:   264 calls     20,119s
                             Writing pages:  2327 calls     20,111s
                   Reading file attributes:  8727 calls     19,498s
                 AlbumBean.countTotalFiles:     1 calls     18,126s
           JAlbumUtilities.countCategories:  6953 calls     11,224s
                                      init:     1 calls     10,569s
        JAlbumUtilities.getDeepCameraDates:   762 calls     10,467s
                           Create MediaRSS:   762 calls      7,725s
                     FileFilters.saveImage:   356 calls       6,56s
                         AlbumBean.popVars: 64413 calls       5,25s
                        AlbumBean.pushVars: 64414 calls      5,165s
           Executing interpreted scriptlet: 105169 calls      4,054s
                   JSONMaker.makeDataPages:   762 calls      3,237s
               FileFilters.unifyColorModel:   277 calls      2,853s
                     Making deep-data.json:     1 calls      1,955s
JAlbumUtilities.countWebLocationCategories:   997 calls      1,743s
                      Compiling scriptlets: 95769 calls      1,104s
                        JSONMaker.makeTree:     1 calls      0,879s
                          Compiling script:     4 calls      0,601s
                  AlbumBean.processFilters:   722 calls      0,058s
                        Process expression:     1 calls      0,018s
                            Copy res files:     1 calls      0,007s
                                     Total: 556754 calls 36m 45,424s




AlbumObjectImpl.getMetadata: 15838 calls 6m 27,289s

Everything from the console during compilation:
https://www.markus-drueck.de/JA/2025%2001%2008%20-%20Compiling%20Project%20Norge.7z

I'll delete that file in about 11 hours.

Thanks! Cheers, Markus

Edited by: MarkusD on 8 Jan 2025, 06:52
Laza

Posts: 1,964
Registered: 6-Sep-2005
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 10:14   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
Here are the measurements on my PC (Win11, AMD Ryzen 5 5600G @3.90 GHz, 16GB)

jAlbum 36.1b10
Making "Sample Project" (All)
"Sample Project" made in 9.187s
 
                             FileFilters.saveImage:   210 calls      7.188s
                             FileFilters.loadImage:    57 calls      1.133s
                        HardwareSmoothScaler.scale:   159 calls      0.834s
                       AlbumBean.registerVariables:    58 calls      0.104s
                       FileFilters.unifyColorModel:    57 calls      0.086s
                          AlbumBean.makeIndexPages:     4 calls      0.063s
                                              init:     1 calls       0.04s
                       Process compiled expression:     7 calls      0.035s
                       RecoveryTool.createLifeboat:     1 calls      0.034s
                                   Create MediaRSS:     4 calls      0.031s
                      Executing compiled scriptlet:    63 calls       0.03s
                     AlbumObjectImpl.getProperties:    57 calls       0.03s
                        AlbumObjectProperties.load:    13 calls      0.028s
                     AlbumObjectImpl.getXmpManager:    63 calls      0.023s
                       AlbumObjectImpl.getMetadata:    57 calls       0.02s
                   Executing interpreted scriptlet:   368 calls      0.019s
                     FileFilters.getBasicImageInfo:    57 calls      0.017s
                                     Writing pages:    12 calls      0.017s
                                Process expression:     1 calls      0.006s
                                 predir processing:     4 calls      0.006s
                   JAlbumUtilities.countCategories:     8 calls      0.005s
                         AlbumBean.countTotalFiles:     1 calls      0.005s
                                AlbumBean.pushVars:   158 calls      0.005s
                           Reading file attributes:    55 calls      0.005s
                                 AlbumBean.popVars:   157 calls      0.004s
                              Compiling scriptlets:    46 calls      0.001s
                                    Copy res files:     1 calls      0.001s
                          AlbumBean.processFilters:   264 calls      0.001s
                     AlbumBean.getFolderProperties:     5 calls          0s
                                             Total:  1948 calls      9.771s

jAlbum 36.1.2
Making "Sample Project" (All)
"Sample Project" made in 3.033s
0.001s ==> 
                             FileFilters.saveImage:   210 calls      0.953s
                        HardwareSmoothScaler.scale:   159 calls      0.816s
                             FileFilters.loadImage:    57 calls      0.451s
                       AlbumBean.registerVariables:    58 calls      0.117s
                          AlbumBean.makeIndexPages:     4 calls      0.083s
                                              init:     1 calls      0.062s
                       Process compiled expression:     7 calls      0.055s
                                     Writing pages:    12 calls      0.048s
                      Executing compiled scriptlet:    63 calls      0.042s
                                   Create MediaRSS:     4 calls      0.041s
                       RecoveryTool.createLifeboat:     1 calls      0.036s
                     AlbumObjectImpl.getProperties:    57 calls      0.035s
                        AlbumObjectProperties.load:    13 calls       0.03s
                     AlbumObjectImpl.getXmpManager:    63 calls      0.028s
                   Executing interpreted scriptlet:   368 calls      0.027s
                       AlbumObjectImpl.getMetadata:    57 calls      0.023s
                     FileFilters.getBasicImageInfo:    57 calls      0.022s
                                    Copy res files:     1 calls      0.019s
                                Process expression:     1 calls      0.009s
                                AlbumBean.pushVars:   158 calls      0.009s
                         AlbumBean.countTotalFiles:     1 calls      0.008s
                                 predir processing:     4 calls      0.008s
                           Reading file attributes:    55 calls      0.008s
                                 AlbumBean.popVars:   157 calls      0.006s
                   JAlbumUtilities.countCategories:     8 calls      0.004s
                              Compiling scriptlets:    46 calls      0.001s
                     AlbumBean.getFolderProperties:     5 calls      0.001s
                          AlbumBean.processFilters:   264 calls      0.001s
                                             Total:  1891 calls      2.943s

Both are second "Force Make" runs. It's 66% improvement.
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:01   in response to: JeffTucker in response to: JeffTucker
  Click to reply to this thread Reply
JeffTucker wrote:
That still leaves me head-scratching a bit about why I've never noticed this message. Could it be because I'm not using any JSON-based skins? Does generating those files account for the overhead?

Not the generation, but the gathering of the data stored within the json files takes time. I'd really need to brush up on using the profiler to get a better understanding on where exactly to optimise. EXIF/xmp reading is a good candidate for optimisation though.
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:12   in response to: MarkusD in response to: MarkusD
  Click to reply to this thread Reply
@Markus, I inspected the log file. You have zillions of "Camera date cache miss for ..." warnings. This indicates that jAlbum has to do far more file reading to compile the camera dates than it should have. Do you get the same warnings if you remake the album again?

jAlbum should cache the camera dates to .jalbum/albumfiles.txt. Please open a random .jalbum/albumfiles.txt and pass it to me for inspection.

What skin are you using, and are you using an ordinary hard drive or networked drive or local SSD?
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:15   in response to: Laza in response to: Laza
  Click to reply to this thread Reply
Awesome data Laza! It seems to go from 9.187s to 3.033s, doesn't it? Isn't that a 200% improvement?
MarkusD

Posts: 685
Registered: 13-Apr-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:30   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
davidekholm wrote:
@Markus, I inspected the log file. You have zillions of "Camera date cache miss for ..." warnings. This indicates that jAlbum has to do far more file reading to compile the camera dates than it should have. Do you get the same warnings if you remake the album again?

Unfortunately, this is not possible. I use so many and such long videos that a complete recompile would take several weeks.

jAlbum should cache the camera dates to .jalbum/albumfiles.txt. Please open a random .jalbum/albumfiles.txt and pass it to me for inspection.

Here we go:
https://www.markus-drueck.de/JA/albumfiles.txt-for-norge-2024.7z

What skin are you using,
Tiger.

and are you using an ordinary hard drive or networked drive or local SSD?


Cheers, Markus

Edited by: MarkusD on 8 Jan 2025, 13:39
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:34   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
davidekholm wrote:
Awesome data Laza! It seems to go from 9.187s to 3.033s, doesn't it? Isn't that a 200% improvement?

Back to math class! A 100% improvement would be from 9.187s to 0.000s. A time saving of more than 100% would involve some sort of quantum time travel.

(9.187 - 3.033) / 9.187 = 0.67, or a 67% saving of time.
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 13:43   in response to: MarkusD in response to: MarkusD
  Click to reply to this thread Reply
MarkusD wrote:
Unfortunately, this is not possible. I use so many and such long videos that a complete recompile would take several weeks.

David: This is why we need to have Force Remake split into two options, Force Remake Images and Force Remake Videos. Anyone who wants to remake everything can just delete the local album and then Make Album.
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 16:47   in response to: MarkusD in response to: MarkusD
  Click to reply to this thread Reply
Thanks Markus. Your albumfiles.txt files are missing camera dates. I don't know why though.
You don't need to do a "Force rebuild", only another "make album" to test, and you can also test on a smaller project.
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 16:48   in response to: JeffTucker in response to: JeffTucker
  Click to reply to this thread Reply
JeffTucker wrote:
davidekholm wrote:
Awesome data Laza! It seems to go from 9.187s to 3.033s, doesn't it? Isn't that a 200% improvement?

Back to math class! A 100% improvement would be from 9.187s to 0.000s. A time saving of more than 100% would involve some sort of quantum time travel.

(9.187 - 3.033) / 9.187 = 0.67, or a 67% saving of time.


Can't you say that 100% faster means twice as fast? The comparison is against the current speed.
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 17:11   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
100% more apples is twice as many. 100% fewer apples is zero.

ETA: And remember, "fastness" is a bit of a misnomer. You're not trying to get more "fastness." You're trying to use less time.
davidekholm

Posts: 3,626
Registered: 18-Oct-2002
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 22:04   in response to: JeffTucker in response to: JeffTucker
  Click to reply to this thread Reply
JeffTucker wrote:
100% more apples is twice as many. 100% fewer apples is zero.

I agree, and how is that different to saying 100% faster means twice as fast?
MarkusD

Posts: 685
Registered: 13-Apr-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 22:13   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
davidekholm wrote:
Thanks Markus. Your albumfiles.txt files are missing camera dates. I don't know why though.
You don't need to do a "Force rebuild", only another "make album" to test, and you can also test on a smaller project.
David, the only thing I do to "compile" an album for years is to press <F9> (which is, as far as I know, "Make album"). So, I don't know what to do to populate the missing camera dates.

Please keep in mind that I do have a lot of "pictures" which are actually only PNG files. They don't have any "camera" dates anyway.

Cheers, Markus
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 22:45   in response to: davidekholm in response to: davidekholm
  Click to reply to this thread Reply
davidekholm wrote:
JeffTucker wrote:
100% more apples is twice as many. 100% fewer apples is zero.

I agree, and how is that different to saying 100% faster means twice as fast?


"Twice as fast" means that it takes 50% less time, not 100% less time.

In some ways, the problem is a linguistic one. There is no measurable thing called a "fast." You can't have 12 fasts, or a kilo of fast, or a liter of fast. So, "100% more fast" does not compute.

Stick to phrasing like "twice as fast," or "three times as fast," and stay away from percentages, and you'll stay out of trouble. :)
RobM

Posts: 4,008
Registered: 4-Aug-2006
Re: Faster image reading and writing with jpeg-turbo
Posted: 8 Jan 25, 23:42   in response to: MarkusD in response to: MarkusD
  Click to reply to this thread Reply
MarkusD wrote:
davidekholm wrote:
Thanks Markus. Your albumfiles.txt files are missing camera dates. I don't know why though.
You don't need to do a "Force rebuild", only another "make album" to test, and you can also test on a smaller project.
David, the only thing I do to "compile" an album for years is to press <F9> (which is, as far as I know, "Make album"). So, I don't know what to do to populate the missing camera dates.

Please keep in mind that I do have a lot of "pictures" which are actually only PNG files. They don't have any "camera" dates anyway.

Cheers, Markus

You could use exiftool to add date information. For a single file setting the dates to 'now' you could use
exiftool pathToFile -allDates=now

exiftool can also batch process files in a directory using -P DIR switches instead of a filename.

Just a thought.
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums