This question is not answered. Helpful answers available: 1. Correct answers available: 1.


Permlink Replies: 15 - Pages: 2 [ 1 2 | Next ] - Last Post: 1 Nov 20, 14:26 Last Post By: ufalbum
ufalbum

Posts: 91
Registered: 2-Aug-2010
Make Album (Update)
Posted: 13 Oct 20, 14:33
 
  Click to reply to this thread Reply
Hi,
I wonder why all *.html files -even in subfolders- are re-created during 'Make album' if absolute nothing has changed in a subfolder. This re-creation causes updated timestamps and further an unnecessary upload to the server.

E.g. My main album is
https://webalben.rezkonv.de/albums/stuttgart21/album/
with a considerable amount of subfolders and images.

My theme is tiger and my albums reside on an own server.

davidekholm

Posts: 3,581
Registered: 18-Oct-2002
Re: Make Album (Update)
Posted: 13 Oct 20, 16:33   in response to: ufalbum in response to: ufalbum
Helpful
  Click to reply to this thread Reply
Do like this: Save a copy of one of the generated index.html pages, then make the album again. Now compare the copies using a "diff" tool. That will instantly reveal what change triggered a re-generation of that page. It could be something as simple as there being a date stamp on your page. I see for instance this:

GeƤndert 12.10.20, 22:05
ufalbum

Posts: 91
Registered: 2-Aug-2010
Re: Make Album (Update)
Posted: 22 Oct 20, 11:26   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Hi David,
thank you for your answer: The problem seems to be that even untouched folders/subfolders get the last date of modification of the whole album.
Even subfolders from 2017 with unchanged images all taken in 2017 get the new modification date.
Regards
Ulli
davidekholm

Posts: 3,581
Registered: 18-Oct-2002
Re: Make Album (Update)
Posted: 28 Oct 20, 14:41   in response to: ufalbum in response to: ufalbum
 
  Click to reply to this thread Reply
ufalbum wrote:
Hi David,
thank you for your answer: The problem seems to be that even untouched folders/subfolders get the last date of modification of the whole album.
Even subfolders from 2017 with unchanged images all taken in 2017 get the new modification date.
Regards
Ulli

Do you mean sub folders of the project of the generated album?
JeffTucker

Posts: 7,682
Registered: 31-Jan-2006
Re: Make Album (Update)
Posted: 28 Oct 20, 14:53   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
davidekholm wrote:
Do you mean sub folders of the project of the generated album?

Yes, and that's the expected behavior. If you just hit Make Album, each index.html file gets regenerated. If it's different from the existing one, it gets written to the output. Since most skins include a timestamp within the page code, the index.html will, indeed, be "new," and will get written out, even if the rest of the code on the page hasn't changed.

The only way to avoid that is to tell jAlbum to process only updated subdirectories. But for a host of reasons, we tell people not to do that.

So, why not get rid of the timestamps in the page code? Skin developers do that because it's a vital debugging tool. When users have problems, we don't have to play "20 Questions" with them about what they did. We can look at the page code to determine whether or not they've got pages that haven't all been made with the current skin settings. We can also tell whether the uploaded album is, in fact, the one they just generated, rather than an older version.

The extra uploading is a small price to pay - the HTML files upload in milliseconds.
ufalbum

Posts: 91
Registered: 2-Aug-2010
Re: Make Album (Update)
Posted: 29 Oct 20, 20:41   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
Okay, I understand. In principle I don't complain about the really small index.html files, but about the duration of the ftp transfers, which takes a considerable time with my nearly 30,000 files, even if only a small part of the files has changed since the last build. My ftp client (FileZilla) has to check for each file if the content or the date has changed. If unmodified folders could be skipped immediately, this would be an advantage. But this is prevented by the newly generated index files.

Regards
Ulli
JeffTucker

Posts: 7,682
Registered: 31-Jan-2006
Re: Make Album (Update)
Posted: 29 Oct 20, 21:22   in response to: ufalbum in response to: ufalbum
 
  Click to reply to this thread Reply
ufalbum wrote:
My ftp client (FileZilla) has to check for each file if the content or the date has changed.

This is why I switched to using the jAlbum uploader. Because it makes use of a manifest file to keep track of what needs to be changed, it is much faster than FileZilla. The first time you use it, it will be no faster than FileZilla, because it has to create the initial manifest file. But subsequent uploads happen in a small fraction of the time.

My family photos/videos album clocks in with over 11,000 files in 132 folders. When I add a few things to it, the upload is done in under fifteen seconds.
JeffTucker

Posts: 7,682
Registered: 31-Jan-2006
Re: Make Album (Update)
Posted: 30 Oct 20, 00:02   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
Attachment ss007978.png (20.7 KB)
Attachment ss007979.png (12.1 KB)
I tell a lie. It's even faster than that. See the screenshots.

Adding an image to the family album, then making and uploading the album without even bothering to restrict it to processing only updated subdirectories. The album build (over 5500 objects in 39 folders) takes 25 seconds, and the upload (11,600 files in 132 folders) is done in less than 6 seconds.
ufalbum

Posts: 91
Registered: 2-Aug-2010
Re: Make Album (Update)
Posted: 31 Oct 20, 08:50   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
Hi Jeff,
Good prospects!
I tried it with an smaller album with a total size of 1,3GB in almost 6000 files and 139 folders. It worked fine and fast on the second try.
Then I tried it with my large album https://webalben.rezkonv.de/albums/stuttgart21/ with nearly 10 GB in 43,991 files, 1,811 folders. As it was the first time I used the jAlbum uploader instead of FileZilla for it, and all files and folders except the already existing folder 'album' (4,3GB in 28.691 files, and 988 folders) which head been already uploaded by FileZilla had to be uploaded. jAlbum's uploader reported a forecast of about 26 min which I found acceptably for the first time. But the remaining time increased and increased. After hours this remaining time was still about 1 hr and didn't decrease considerably. The uploader reported the upload of folders slides and thumbs within the subfolders of folder album with very low progress. Window's task manager reported an almost constant CPU load of 98%, RAM consumption >3 GB from (32 GB) in my system. What was jAlbum doing all the time? After 6 hours I tried to abort the upload, but this was impossible. I had to shut down jAlbum from the task manager. After a restart of my PC I tried it again, same behaviour. My VDSL-Router reported 32 Mbit/s as upload speed.
Any idea?
Regards Ulli
davidekholm

Posts: 3,581
Registered: 18-Oct-2002
Re: Make Album (Update)
Posted: 31 Oct 20, 10:23   in response to: ufalbum in response to: ufalbum
 
  Click to reply to this thread Reply
Before attempting that huge upload again, open jAlbum's system console (F7). It has a "Dump threads" button in the top-right corner that you can press if jAlbum seems to eat >90% CPU during upload. Once the "Dump threads" button has been pressed, a file is generated. Please attach that file here, but pay attention that you only press it when you clearly see that jAlbum is taking >90% CPU time.

Also, IF it's possible to cancel the upload, then select Tools->External tools->Perform garbage collection. This will produce information of jAlbum's memory usage that is helpful for us.
ufalbum

Posts: 91
Registered: 2-Aug-2010
Re: Make Album (Update)
Posted: 31 Oct 20, 14:05   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Attachment thread-dump-1.txt (262.7 KB)
Hi David,
I did as you told me. I open windows' task manager, jAlbum, and its system console. Then I invoked an album's upload. At the beginning It says remaining time 26', the CPU load was with ca. 3% in the low single-digit percentage range. 5 min. later the load was 98% again.The reported RAM consumption was low. I pressed the 'Dump threads' button (see attached file). Afterwards I was able to abort the upload and to close the upload window. However the CPU load was still 98% and I was unable to select anything on jAlbums' user interface. Therefore I had to terminate jAlbum from windows' task manager.
Any idea?
Regards
davidekholm

Posts: 3,581
Registered: 18-Oct-2002
Re: Make Album (Update)
Posted: 31 Oct 20, 23:06   in response to: ufalbum in response to: ufalbum
 
  Click to reply to this thread Reply
Thanks for this thread dump. It's really odd, there is not much work going on there but you have hundreds of worker threads for the upload. Around 20 should be enough for an ordinarily working upload process. I guess all these threads are guilty of both CPU and memory usage. For some reason, the upload worker threads aren't being returned properly to the pool, hence new threads are created again and again.

Please try this: Switch ftp connection type from ftp (ftp4j) to ordinary ftp (edtftpj) and see if the behavior changes.

I stay tuned.
ufalbum

Posts: 91
Registered: 2-Aug-2010
Re: Make Album (Update)
Posted: 1 Nov 20, 12:54   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Hi David,
-))
Thank you, I am delighted!
The problem seemed to be that I selected FTPES as connection type.

With ftp (edtfpj) the CPU load was between 3 and 4% and the memory load about 500 MB during the whole transmission..
What I also noticed was that the GPU was mostly running with about 30% load. With FTPES the GPU load was negligible and only for short times more than 0%.

After the completed upload I tried it again, and - as I didn't make modifications to the album - the upload was completed within a few seconds. FileZilla would have needed a considerable amount of time for this.

A further question: Now the whole album - not only the folder 'album' - will be uploaded, so the server unnecessarily uses about twice the disk space. Can this be prevented?

Best regards
your jAlbum fan
Ulli
JeffTucker

Posts: 7,682
Registered: 31-Jan-2006
Re: Make Album (Update)
Posted: 1 Nov 20, 13:50   in response to: ufalbum in response to: ufalbum
 
  Click to reply to this thread Reply
Why would the server be using twice as much disk space?

When you upload the finished album with FileZilla, you should be uploading the contents of the output album folder. You don't upload the folder - you upload only what's in the folder.

When you upload the finished album with the jAlbum uploader, it does exactly the same thing - it uploads the contents of the album folder.

In either case, your dogpictures album on the server should end up looking like this:

public_html/dogpictures
public_html/dogpictures/res
public_html/dogpictures/slides
public_html/dogpictures/thumbs
public_html/dogpictures/index.html
public_html/dogpictures/other files (depending on which skin you used)
public_html/dogpictures/manifest.jmf (the small file that jAlbum uses to coordinate the upload)
JeffTucker

Posts: 7,682
Registered: 31-Jan-2006
Re: Make Album (Update)
Posted: 1 Nov 20, 13:55   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
BTW, I always had uploading problems with my web host when using FTPS or FTPES, both in FileZilla and jAlbum. I discovered that SFTP was much more robust. It's secure, and I've never seen any odd problems. See if your web host supports it - they might be using an odd port number - mine uses 2233 instead of 22!
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums