Hard links for originals ...

rothlisbergers

Posts: 13
Registered: 18-Jan-2009

Re: Hard links for originals
Posted: 15 Sep 17, 20:20

in response to: JeffTucker

jGromit wrote:
The effects of modifying the source image may be very different in each case. Even without hardlinking, that confuses users.

But hardlinking does not appear to make it more confusing, except in the "change and upload without rebuild" case.

RobM

Posts: 3,838
Registered: 4-Aug-2006

Re: Hard links for originals
Posted: 15 Sep 17, 20:32

in response to: JeffTucker

Reply

hardlink.jpg (70.3 KB)

On a duplicate of the Sample portfolio - copied to 'My albums' - with the Flower.jpg image selected I executed

import java.nio.file.*;
File f = selectedObjects[0].getFile();
Files.createLink(new File(f.getParentFile(), "/album/hardlink.jpg").toPath(), f.toPath());

Then I deleted the actual file and emptied the trash.
Finally I attached the hard link.jpg file from the 'album' folder here.
What do you see if you download this file?

rothlisbergers

Posts: 13
Registered: 18-Jan-2009

Re: Hard links for originals
Posted: 15 Sep 17, 20:48

in response to: RobM

Reply

RobM wrote:
On a duplicate of the Sample portfolio - copied to 'My albums' - with the Flower.jpg image selected I executed

import java.nio.file.*;
File f = selectedObjects[0].getFile();
Files.createLink(new File(f.getParentFile(), "/album/hardlink.jpg").toPath(), f.toPath());

Then I deleted the actual file and emptied the trash.
Finally I attached the hard link.jpg file from the 'album' folder here.
What do you see if you download this file?

It's color, but I don't have enough information to know whether that's a problem.
1) What is the 'When adding files' setting for the album?
2) What is the exact sequence of steps (copies, edits, hardlinks, etc...)?

RobM

Posts: 3,838
Registered: 4-Aug-2006

Re: Hard links for originals
Posted: 15 Sep 17, 21:00

in response to: rothlisbergers

Reply

rothlisbergers wrote:

RobM wrote:
On a duplicate of the Sample portfolio - copied to 'My albums' - with the Flower.jpg image selected I executed

import java.nio.file.*;
File f = selectedObjects[0].getFile();
Files.createLink(new File(f.getParentFile(), "/album/hardlink.jpg").toPath(), f.toPath());

Then I deleted the actual file and emptied the trash.
Finally I attached the hard link.jpg file from the 'album' folder here.
What do you see if you download this file?

It's color, but I don't have enough information to know whether that's a problem.
1) What is the 'When adding files' setting for the album?
2) What is the exact sequence of steps (copies, edits, hardlinks, etc...)?

I just copied the whole Sample portfolio from the jAlbum application to 'My albums'.
What is happening, I think, is that when you have a hard link (a pointer to the file) and you delete the 'file' - really just another pointer to the file, the OS is keeping the contents on disc and only really deleting the file when all pointers have been deleted.

That means if a user 'deletes' the original from the project the album pointer keeps the file active, so there is a loss of disc space and the album has an orphan file that would get uploaded - unless jAlbum could synchronise the hard link with the project and delete both together.

rothlisbergers

Posts: 13
Registered: 18-Jan-2009

Re: Hard links for originals
Posted: 15 Sep 17, 21:11

in response to: RobM

Reply

RobM wrote:
What is happening, I think, is that when you have a hard link (a pointer to the file) and you delete the 'file' - really just another pointer to the file, the OS is keeping the contents on disc and only really deleting the file when all pointers have been deleted.

This is correct.

RobM wrote:
That means if a user 'deletes' the original from the project the album pointer keeps the file active

Correct

RobM wrote:
the album has an orphan file that would get uploaded - unless jAlbum could synchronise the hard link with the project and delete both together.

This is true whether you use hardlinks or not. An album rebuild would see that the file was deleted and remove the hardlink from the output (which is now the only reference to the file), just as it would delete the copied file if you weren't using hardlinks.

Edited by: rothlisbergers on 15-Sep-2017 21:13

RobM

Posts: 3,838
Registered: 4-Aug-2006

Re: Hard links for originals
Posted: 15 Sep 17, 21:36

in response to: rothlisbergers

Reply

rothlisbergers wrote:

RobM wrote:
the album has an orphan file that would get uploaded - unless jAlbum could synchronise the hard link with the project and delete both together.

This is true whether you use hardlinks or not. An album rebuild would see that the file was deleted and remove the hardlink from the output (which is now the only reference to the file), just as it would delete the copied file if you weren't using hard links.

As long as the synchronise option is enabled under preferences.

By the way, we (jGromit & myself) are trying to think of potential problems (Murphey's law), to ensure they are considered so if/when implemented users won't affected, rather than being anti-implementing this.

rothlisbergers

Posts: 13
Registered: 18-Jan-2009

Re: Hard links for originals
Posted: 15 Sep 17, 21:55

in response to: RobM

Reply

RobM wrote:
By the way, we (jGromit & myself) are trying to think of potential problems (Murphey's law), to ensure they are considered so if/when implemented users won't affected, rather than being anti-implementing this.

Absolutely. I'm a developer in a separate domain where changes are tightly controlled - but we also don't want to invent or overstate problems to the deprivation of the benefits. I think your prior suggestion to make this a non-default option that is monitored for some period before making it the default behavior is a very reasonable first step.

RobM

Posts: 3,838
Registered: 4-Aug-2006

Re: Hard links for originals
Posted: 17 Sep 17, 00:39

in response to: rothlisbergers

Reply

MinimalHL.jaskin (78.3 KB)

I have modified the Minimal skin to work with hard links, so we can test it out more easily.
It does not make any changes to video files.

The attached MinimalHL has a poster.bsh file that deletes any original files.
It also has a finally.bsh file that places a hard link in the output folders wherever an original would have been.

Because it is a hack, it will regenerate the links on every make album - does anyone know how to detect hard links? The Java docs show detecting a symbolic link but not a hard one.

It all seems to work fine. I have tried it with various linking methods and including just specific items.

davidekholm

Posts: 3,478
Registered: 18-Oct-2002

Re: Hard links for originals
Posted: 7 Oct 17, 13:52

in response to: davidekholm

Reply

texts.properties (68.5 KB)

I've implemented hard links for jAlbum 14.2 beta now. Get it from http://jalbum.net/download/beta/jalbum-core.jar and put the attached "texts.properties" in jAlbum's "texts" folder to test it.

When switched on, hard links applies to the original and resource files written to the album folder. The default setting is off. You can activate it under Preferences->Album.

As hard links are implemented at a very low level in the file system, checking the album folder size doesn't reveal any change, but you should be able to notice that the free space reported for the disk volume used is affected by hard links. I saw a massive impact: For my sample album of 89 images, the disk usage went down from 400MB to 10MB - i.e. 2.5% of the earlier size. Here's the data when monitoring disk usage after making album with hard links off and on:

Before 229,35 GB free
After making ordinary album: 228,95 GB free. Album size: 400MB
After making hard linked album: 229,34 GB free. Album size: 10 MB
Effective compression: 2.5% of original size

You have to issue a "Make all" in order to have the files re-generated.

Warning: There is a potential for data loss: If you make an album with hard links to originals and then make the same album in an older version of jAlbum (a make all operation), then your originals will be truncated to 0 bytes length, and I'm talking about your real originals, not the hard-linked copies

. The reason for this is that, when instructed to copy an original file to the album folder, it effectively uses the same file for the source and destination without knowing it. This causes the source file to be truncated as soon as the destination is opened for writing. jAlbum 14.2 prevents this from happening by deleting the destination file before writing to it as it makes a copy, thereby breaking the hard link, but this delete instruction is naturally not present in earlier jAlbum versions.

To conclude: Hard links gives a massive improvement in lowered disk usage but comes with a potential for data loss if you move back and forth between jAlbum versions. The impact is luckily limited by jAlbum's default behavior to copy files to the album project and not link to them.

Sidenote: jAlbum 14.2 has further preparation for Java 9. This unfortunately causes it to consider album projects changed so the first time you move to jAlbum 14.2 it will want to reprocess your images, but after that, it should be as clever as before to determine if images needs to be reprocessed or not. Moving on from now, it's less likely that we will see this unnecessary reprocessing happening again.

RobM

Posts: 3,838
Registered: 4-Aug-2006

Re: Hard links for originals
Posted: 7 Oct 17, 17:08

in response to: davidekholm

Reply

davidekholm wrote:
I've implemented hard links for jAlbum 14.2 beta now. Get it from http://jalbum.net/download/beta/jalbum-core.jar and put the attached "texts.properties" in jAlbum's "texts" folder to test it.

When switched on, hard links applies to the original and resource files written to the album folder. The default setting is off. You can activate it under Preferences->Album.

Not sure if this is new or happened before this core update, but if you change from link to scaled images only with 'copy originals if needed unchecked' to link to originals via scaled images the 'copy originals if needed' checkbox is not automatically checked.

If the new preference is checked would it not be better now to have the 'copy originals if needed checked and disabled?

davidekholm

Posts: 3,478
Registered: 18-Oct-2002

Re: Hard links for originals
Posted: 7 Oct 17, 17:34

in response to: RobM

Reply

RobM wrote:

If the new preference is checked would it not be better now to have the 'copy originals if needed checked and disabled?

I can't say that for sure.

rothlisbergers

Posts: 13
Registered: 18-Jan-2009

Re: Hard links for originals
Posted: 14 Oct 17, 22:59

in response to: davidekholm

Reply

Thanks for putting this together David. I've been running this for the past week or so and haven't seen any issues yet. I've now built 160GB of album output while only consuming 4GB of disk space. Right in line with your 2.5% observation. That's awesome.

davidekholm

Posts: 3,478
Registered: 18-Oct-2002

Re: Hard links for originals
Posted: 16 Oct 17, 12:23

in response to: rothlisbergers

Reply

rothlisbergers wrote:
Thanks for putting this together David. I've been running this for the past week or so and haven't seen any issues yet. I've now built 160GB of album output while only consuming 4GB of disk space. Right in line with your 2.5% observation. That's awesome.

Thanks for bringing this idea and testing it. I'm truly happy you're so content with the result. Just remember to keep backups so you don't suffer from any data loss. As I warned, moving back to an earlier jAlbum version now can cause data loss if you remake the albums again.