This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 13 - Pages: 1 - Last Post: 18-Jul-2018 12:46 Last Post By: davidekholm Threads: [ Previous | Next ]
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Don't write out file if unchanged
Posted: 10-Jul-2018 00:11
 
  Click to reply to this thread Reply
I won't bore you with the details, but under some conditions, several of my skins generate image files apart from those cranked out by the jAlbum core. Without going through a lot of elaborate record-keeping, I can't really tell whether a given image file needs to be regenerated or not (there are all sorts of changes that could affect it), but it would be helpful if I could, at least, not write out the file if the existing output file is identical.

Condensed, it looks like this:
AlbumImage ai = new AlbumImage(rif, engine);
ai = ai.scaleToFit(new Dimension(fThumbW, fThumbH));
ai.saveImage(new File(outputDirectory, "me-folderthumb" + folderCounter + ".png"));
Before saving that image, I'd like to know if that file already exists (the easy part), and if it does, if it's different from the one I'm about to write out.
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 10-Jul-2018 00:40   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
Thinking about this a bit more, I'd be willing to settle for a "cheat." The conditions that would require the generation of a new image file would be things like a different image (the rif in the code above) or different bounds. The chances that the old and new image files would have exactly the same number of bytes would be very, very close to zero. Simply comparing the old and new sizes would be a "good enough" proxy for actually examining all of the bytes.
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 10-Jul-2018 12:25   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
We have a new API called EmbeddedProperties to assist in figuring out whether an image file needs reprocessing or not. Here's a usage example from ThemeImageProcessor:

    public void processThemeImage(AlbumObject folder, File outputDir) throws IOException {
        AlbumObject theme = JAlbumUtilities.getThemeObject(folder);
        Dimension themeDim = engine.getThemeImageDim();
        if (theme != null) {
            if (themeDim == null) {
                themeDim = new Dimension(theme.getImageInfo().width, theme.getImageInfo().height);
            }
            File themeFile = new File(outputDir, getThemePath(folder));
            EmbeddedProperties existing = new EmbeddedProperties(themeFile);
            EmbeddedProperties current = new EmbeddedProperties();
            current.put("themeSrc", theme.getPathFrom(folder));
            current.put("themeDim", themeDim.width + "x" + themeDim.height);
            if (!engine.isAppendImages() || JAlbumUtilities.isDirty(themeFile, theme) || !existing.equals(current)) {
                JAlbum.logger.log(Level.FINE, "Generating theme image for {0}", folder);
                AlbumImage ai = new AlbumImage(theme);
                if (engine.getThemeImageDim() != null) {
                    CropFilter cf = new CropFilter();
                    double ratio = (double) themeDim.width / themeDim.height;
                    int width = ai.getBufferedImage().getWidth();
                    int height = ai.getBufferedImage().getHeight();
                    Dimension bounds = new Dimension(width, (int) (width / ratio));
                    if (bounds.height > height) {
                        bounds = new Dimension((int) (height * ratio), height);
                    }
                    cf.setBounds(bounds);
                    ai = ai.applyFilter(cf, (Map) theme.getVars());
                    ai = ai.scaleToFit(themeDim);
                }
                ai.saveImage(themeFile);
                current.write(themeFile);
            }
        }
    }
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 10-Jul-2018 13:54   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Yes, I see.

I'll have to mull this one over. For obscure layout purposes, I need a lot of information about that cropped and scaled image, information that can be derived only by doing the cropping and scaling and then extracting info about the resulting image. So, I'd need to store a number of value pairs in the properties when the image is initially created, then retrieve them to decide whether to crop and scale again on a subsequent album build, and to fetch the image info I need.

It might actually be more efficient to do the cropping and scaling every time, and just use my "file size" cheat to decide whether to keep the existing file, or replace it with the new one.

(BTW, on a new machine with a fast chip and SSD hard drive, the time to process all of this is trivial, even if the skin reprocesses all of the folder images every time. I'm only fiddling with it to avoid unnecessary file uploads. But the files aren't very large, so even the extra uploads don't matter very much!)
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 10-Jul-2018 21:35   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
It's less development work to just reprocess these files, but naturally always faster for the end user to avoid reprocessing and re-uploading. With this API you have the ability to embed whatever properties in a generated file and later check those properties.
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 11-Jul-2018 13:36   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Before I wade into this, one quick question. If I'm reading the API correctly, the name/value pairs are both Strings. Is that right? So if I'm putting an integer, I need to do a toString() on it, and if I'm getting a value, but want to use it for doing arithmetic, I need to do an Integer.parseInt().

Yeah, I could try this and find out, but rather than endure a series of crashes.... ;)

ETA: A crash or two answered the question. Yup, we're talking Strings here, and nothing else.
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 00:06   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
BTW, if anyone else wants to play with EmbeddedProperties(), it pays to remember that the properties are stored in the file as xmp metadata. So you can take the image file you've generated, drag it into an empty album project, tell jAlbum to List metadata, and see exactly what your routine has planted in the file. Very handy.
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 00:37   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
I've got this working, despite the byzantine requirements. There are half a dozen settings a user could change, both jAlbum and skin variables, that would trigger the need to regenerate the folder images. And of course, they're an unholy mix of booleans, Strings, and ints, so it requires some gymnastics.

Despite all the logic processing overhead, the album generation speed difference is noticeable, even on this new machine. Not a big deal, but when the images don't need to be cropped and scaled again, you don't get that "Just a sec... I'm doing the folder thumbs" pause. And of course, no unchanged files to be uploaded again.

Coming soon to several skins near you. (Not until August, however - not doing any releases just before heading to the Alps.)
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 01:06   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
Another tidbit to pass along - you can't use EmbeddedProperties() on a PNG image file. So if the image file you're cranking out has to be a PNG (to preserve transparency, for example), you're out of luck. You'll end up regenerating it every time.
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 10:22   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:
Before I wade into this, one quick question. If I'm reading the API correctly, the name/value pairs are both Strings. Is that right? So if I'm putting an integer, I need to do a toString() on it, and if I'm getting a value, but want to use it for doing arithmetic, I need to do an Integer.parseInt().

Yeah, I could try this and find out, but rather than endure a series of crashes.... ;)

ETA: A crash or two answered the question. Yup, we're talking Strings here, and nothing else.


Yes, your question is answered by this piece of code:
public class EmbeddedProperties extends TreeMap<String, String>
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 10:25   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:
I've got this working, despite the byzantine requirements. There are half a dozen settings a user could change, both jAlbum and skin variables, that would trigger the need to regenerate the folder images. And of course, they're an unholy mix of booleans, Strings, and ints, so it requires some gymnastics.

Despite all the logic processing overhead, the album generation speed difference is noticeable, even on this new machine. Not a big deal, but when the images don't need to be cropped and scaled again, you don't get that "Just a sec... I'm doing the folder thumbs" pause. And of course, no unchanged files to be uploaded again.

Coming soon to several skins near you. (Not until August, however - not doing any releases just before heading to the Alps.)


Looking forward to it! Sorry for my late answers. On vacation here and today we have plenty of wind for me to tackle on the waters :-)
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 10:27   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:
Another tidbit to pass along - you can't use EmbeddedProperties() on a PNG image file. So if the image file you're cranking out has to be a PNG (to preserve transparency, for example), you're out of luck. You'll end up regenerating it every time.

Good point. Let's see if there is an update to the Apache Sanselan library capable of embedding xmp data into png images too.
jGromit

Posts: 8,621
Registered: 31-Jan-2006
Re: Don't write out file if unchanged
Posted: 12-Jul-2018 16:10   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
davidekholm wrote:
Looking forward to it!

I was doing this coding in an as-yet-unreleased new skin, but I just back-ported it to several others. My primary "travel" site has over 3000 images in 54 folders, so it's a good test. Doing an album build with no new material, but regenerating all of the folder thumbnails, build time was 37s. With the new "don't do it if you don't have to" code, 9s. I'd say that's convincing.

Sorry for my late answers. On vacation here and today we have plenty of wind for me to tackle on the waters :-)

Not a problem. Leave me to thrash about on my own for a while, and I usually figure things out. :)
davidekholm

Posts: 3,853
Registered: 18-Oct-2002
Re: Don't write out file if unchanged
Posted: 18-Jul-2018 12:46   in response to: jGromit in response to: jGromit
 
  Click to reply to this thread Reply
jGromit wrote:

I was doing this coding in an as-yet-unreleased new skin, but I just back-ported it to several others. My primary "travel" site has over 3000 images in 54 folders, so it's a good test. Doing an album build with no new material, but regenerating all of the folder thumbnails, build time was 37s. With the new "don't do it if you don't have to" code, 9s. I'd say that's convincing.

Great!

Sorry for my late answers. On vacation here and today we have plenty of wind for me to tackle on the waters :-)

Not a problem. Leave me to thrash about on my own for a while, and I usually figure things out. :)


I trust you on that. You're a clever guy :-)
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums