Posts:
3,464
Registered:
18-Oct-2002
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 17:43
in response to: RobM
|
|
|
In the answers section the comment 'The reason for this behavior is rather straightforward, and it relates to how files are saved in most Mac OS X applications: Atomically. What happens is that a copy of the file is written to a temporary staging area, and then moved to replace the original file. *This, quite naturally, breaks hard links*'
With "break", they mean that the two file references will point to two separate files after an edit to the original file, not the kind of broken link you get with symbolic links.
|
|
|
Posts:
3,464
Registered:
18-Oct-2002
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 17:44
in response to: RobM
|
|
|
You can play with hard links already! Open jAlbum, select an image and execute this from the system console: import java.nio.file.*;
File f = selectedObjects[0].getFile();
Files.createLink(new File(f.getParentFile(), "hardlink.jpg").toPath(), f.toPath());
|
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:03
in response to: rothlisbergers
|
|
|
|
Wow, glad to see that this sparked such interest. I'm not a JAlbum developer, so anything I say should be taken with a grain of salt, but I thought I would offer some thoughts based on the discussions I've seen so far:
1) It seems obvious that this is valuable at the very least as an optional mode of operation, even if it isn't on by default.
2) No matter what, there needs to be a fall-back to a standard copy in the case where the createLink call fails (e.g., as it would on FAT32). So that seems like a non-issue.
3) Any decision to make this the standard mode of operation is a risk benefit analysis. So far, the risks seem to be outweighed by the benefit. We're very likely talking about people not having to be concerned with migrating to new, larger hard drives if they want to switch to including originals in their JAlbum outputs (that is my primary motivation for asking about this). There's also a lot of potential for a performance boost in building existing albums containing originals, and the freeing up of a lot of disk space for those folks. The "upload after changing dimensions without having remade album" scenario is an edge case that few would ever see and depends on the application used to modify the image (see below).
4) The issue raised with OSX appears to be related specifically to the use of the Cocoa platform (rather than the filesystem) and presumably wouldn't apply to JAlbum. Not confident about this, but that's the impression I get. And as David noted, this would just result in two separate files - not in any corruption (see more explanation below).
Lastly, it's important to understand how a hard link differs from what we traditionally conceive of as "links" and to understand what happens under the hood. It's helpful to know that a normal file is actually two parts: a block of data and a pointer to that data in the filesystem (like a house and a mailing address). A hard link is just adding another pointer to the same block of data (having two mailing addresses for the same house), whereas other links (e.g., symbolic links or shortcuts) are pointers to the pointer to the block of data (a mailing address that forwards to another mailing address). So when a file (pointer+data) is changed by an application, it could operate directly on the file, or it could copy the file to memory, change it there and then overwrite the existing file (similar to when you drag and drop through a file manager to overwrite an existing file).
In the first case (operating directly on the file), the block of data is changed and so all file pointers will see the change.
In the second case, the overwrite is effectively the same as a delete followed by the creation of a new file. When a file is deleted and there was the only one pointer to the block of data, then the data block is effectively gone (though it still physically resides on the disk until the space is used for something else). However, with hard links there is the possibility that there are multiple file pointers. So in this second case (overwrite), the "change" will just delete the pointer of interest (keeping the other pointers valid), create a new block of data with the changes and recreate the pointer to go to the new block of data.
Whether an application does the first type of change or the second type of change is dependent on the application design.
I don't know if this is helpful, but it looks like there was some confusion about exactly what hard links do, so I thought I would try to help clarify.
Edited by: rothlisbergers on 15-Sep-2017 19:05
|
|
|
Posts:
3,823
Registered:
4-Aug-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:06
in response to: davidekholm
|
|
|
You can play with hard links already! Open jAlbum, select an image and execute this from the system console: import java.nio.file.*;
File f = selectedObjects[0].getFile();
Files.createLink(new File(f.getParentFile(), "hardlink.jpg").toPath(), f.toPath());
I selected a file and then executed the code, but putting the hard link in the 'album' directory.
I then edited the original, to make it B&W and rebuilt the album.
The original is B&W but the hard link is not. The hardlink is showing as taking up the same space on the disk.
I then deleted the original file, the hard link is still there and looks just like the original colour image.
Edited by: RobM on 15-Sep-2017 18:08
Update, the hard link has changed to black & white, but it is still there after deleting the original and is taking up the same space on the disk |
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:12
in response to: RobM
|
|
|
|
Hardlinks aren't built into JAlbum (yet) so your rebuild of the album wouldn't have done anything to change the file you created in the output by running the code in the console.
Hardlinks will always show as taking up the same amount of space on disk, but if you look at the total that is actually consumed on disk, you'll see the difference. You could have 1,000 hardlinks to a 1MB block of data and your file manager would make it look like you have 1,000 1MB files (1GB) but there would actually only be a single 1MB block of data on disk.
|
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:32
in response to: rothlisbergers
|
|
|
|
To further clarify how this would interact with JAlbum, let's walk through a scenario where JAlbum is using hardlinks.
An image is in /pictures and included in an album that outputs to /web. After making the album, there is a single block of data and two pointers to it, one from /pictures and one from /web. Now let's say you use an editor to redimension the file from /pictures, there are two possibilities (as per my previous post):
1) The editor modifies the file directly. Now /pictures and /web both point to the modified block of data. The output album is out of sync with regard to image dimensions. Upon rebuilding the album, however, JAlbum sees the changed file, updates the metadata and deletes and recreates the hardlink (which effectively doesn't change anything with respect to the hardlink). The album is now in sync.
2) The editor modifies the file via an overwrite. Now /pictures and /web point to different blocks of data. The output album dimensions are in sync. Upon rebuilding the album, JAlbum sees the changed file and updates the metadata and deletes and recreates the hardlink. In this case, the old image data will be gone (because all hardlinks have been removed) and the album will point to the new image data. Everything is still in sync.
This exercise revealed for me what I think is probably the biggest risk associated with the change - that somebody would use an editor which operates directly on the file to modify the output file in the album, thinking that their source original wouldn't be changed in the process. That seems like another severe edge case and I don't know how common that mode of operation is with editors, but it is probably sufficient to warrant making this an optional feature. I'll leave it to the developers to decide whether it outweighs the benefits of making it the default behavior.
Edited by: rothlisbergers on 15-Sep-2017 19:33
Edited by: rothlisbergers on 15-Sep-2017 19:34
|
|
|
Posts:
8,081
Registered:
31-Jan-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:37
in response to: rothlisbergers
|
|
|
|
There's another potential twist buried in all of this, one that hadn't occurred to me until now.
The effects of using hard links in the output would also be entirely dependent upon whether you had chosen to Copy images into the project directory, or to Link to them.
Consider the first case. You've added Pictures/mydog.jpg to the Dogs album, chosen Link to originals, and made the album. The Dogs/album/mydog.jpg file would be a hard link, but not to Pictures/mydog.jpg. It would be a hard link to Dogs/mydog.jpg, the copy of the image that's sitting in the project directory.
Changes made to Pictures/mydog.jpg, no matter how they're carried out, would have no effect on the album. Even making the album again wouldn't pick up those changes. Only deleting and re-adding the image to the project would retrieve the modified image, and only making the album again would result in the changes appearing in the album (in fact, deleting it from the project would delete the output version entirely).
But if the project is linking to the source file, rather than copying it into the project directory, then Dogs/album/mydog.jpg is a hard link to Pictures/mydog.jpg. So changes to the source image might or might not be reflected in the output, depending upon the software used to alter the image, and depending upon whether or not you've remade the album.
It all makes me woozy.
|
|
|
Posts:
3,823
Registered:
4-Aug-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 19:51
in response to: rothlisbergers
|
|
|
Hardlinks aren't built into JAlbum (yet) so your rebuild of the album wouldn't have done anything to change the file you created in the output by running the code in the console.
I took that into account by putting the hard link outside of the project files and looking at the file itself as well as manually changing the slide.html code to point to the hard link instead of the original - copied to and deleted from the album.
Hardlinks will always show as taking up the same amount of space on disk, but if you look at the total that is actually consumed on disk, you'll see the difference. You could have 1,000 hardlinks to a 1MB block of data and your file manager would make it look like you have 1,000 1MB files (1GB) but there would actually only be a single 1MB block of data on disk.
With the hard link on the disc
Used bytes = 225,380,796,770 bytes
After deleting the hard link
Used bytes = 225,380,886,882 bytes
Difference = 90112 bytes.
90KB instead of 900 Kbytes |
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:02
in response to: JeffTucker
|
|
|
|
@jGromit,
It's possible that I've misunderstood or that I don't actually know what JAlbum is doing, but I think you're confusing the effects of the and the .
If 'When adding files' is set to copy, then the output from 'Link to originals' will hardlink to the copied file in the album directory. If 'When adding files' is set to link, then the output from 'Link to originals' will hardlink to the source file. The hardlink will always reference the same file that is being used by the album.
|
|
|
Posts:
8,081
Registered:
31-Jan-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:06
in response to: rothlisbergers
|
|
|
|
That's pretty much what I said. But in the former case, changes to the source file, no matter how they're done, won't change the image in the album - there's nothing pointing to that source file. In that latter case, changes to the source file might change the image in the album, depending upon other factors.
Edit: But of course, things change again if the user edits the copy of the image that's sitting in the project directory. Then we're back to having to concern ourselves with whether the image processing software alters the data block, or deletes and recreates it, and with whether the user has made the album again.
|
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:06
in response to: RobM
|
|
|
After deleting the hard link
Difference = 90112 bytes.
90KB instead of 900 Kbytes
I would be surprised if deleting the hardlink actually removed 90kB - there was probably some other stuff that changed in the process to reflect that difference. The actual hardlink consumption is probably closer to 1kB.
Regardless, this demonstrates the benefit. Full image data isn't duplicated. Oh! We haven't even talked about videos. Imagine the savings for people putting videos in their albums! |
|
|
Posts:
8,081
Registered:
31-Jan-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:07
in response to: rothlisbergers
|
|
|
Imagine the savings for people putting videos in their albums!
Oh, don't get us started on that subject. Link to originals doesn't apply to videos at all. Long story....
Edit: In fact, I can't think of any reliable way for jAlbum to make use of hard links for videos unless major changes are made to the way videos are handled. Even Do not re-encode isn't the same thing as linking to originals. |
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:14
in response to: JeffTucker
|
|
|
That's pretty much what I said. But in the former case, changes to the source file, no matter how they're done, won't change the image in the album - there's nothing pointing to that source file. In that latter case, changes to the source file might change the image in the album, depending upon other factors.
I see. I thought you were highlighting a problem rather than explaining the associations. My point was that the hardlink behavior is consistent with expectations regardless of whether somebody has chosen either 'Link' or 'Copy' for their album. |
|
|
Posts:
8,081
Registered:
31-Jan-2006
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:16
in response to: rothlisbergers
|
|
|
My point was that the hardlink behavior is consistent with expectations regardless of whether somebody has chosen either 'Link' or 'Copy' for their album.
Whose expectations? 
The effects of modifying the source image may be very different in each case. Even without hardlinking, that confuses users. |
|
|
Posts:
13
Registered:
18-Jan-2009
|
|
|
|
Re: Hard links for originals
Posted:
15 Sep 17, 20:16
in response to: JeffTucker
|
|
|
Imagine the savings for people putting videos in their albums!
Oh, don't get us started on that subject. Link to originals doesn't apply to videos at all. Long story....
Edit: In fact, I can't think of any reliable way for jAlbum to make use of hard links for videos unless major changes are made to the way videos are handled. Even Do not re-encode isn't the same thing as linking to originals.
Bummer. Maybe this would create incentive to fix that. |
|
|
|
Legend
|
|
Forum admins
|
|
Helpful Answer
|
|
Correct Answer
|
|
