This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 43 - Pages: 3 [ 1 2 3 | Next ] - Last Post: 20 Sep 22, 13:57 Last Post By: davidekholm Threads: [ Previous | Next ]
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Some DiskCache questions
Posted: 14 Sep 22, 00:48
 
  Click to reply to this thread Reply
I've been tinkering with the new DiskCache class, and I've got some questions.

1. My quick test tells me that the load(URL) method can't be used with a local URL, like
 file:///C:\Users\jefft\Downloads\Bearpaw.woff
It appears to require something with an http or https protocol. Is that correct?

2. The API says, " Files are only loaded if... in need of update." How does it determine that a file needs to be updated?

3. Is there no constructor like this?
public DiskCache(File primaryCacheDir)
In other words, a constructor that effectively wildcards the extension? If it can't be used for local files, it doesn't really matter, but in my skins, I'm copying all sorts of local files to a hidden cache, and the extensions might be woff, woff2, jpg, jpeg, png, gif, svg, mp3, etc.
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 14 Sep 22, 17:20   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
JeffTucker wrote:
I've been tinkering with the new DiskCache class, and I've got some questions.

1. My quick test tells me that the load(URL) method can't be used with a local URL, like

 file:///C:\Users\jefft\Downloads\Bearpaw.woff
It appears to require something with an http or https protocol. Is that correct?

That's right, but if it helps debugging I can add support for the file: protocol too. It will appear in the next beta

2. The API says, " Files are only loaded if... in need of update." How does it determine that a file needs to be updated?

DiskCache checks the last modification date. (Over http this is easily done using the "If-Modified-Since" HTTP request parameter)

3. Is there no constructor like this?
public DiskCache(File primaryCacheDir)
In other words, a constructor that effectively wildcards the extension? If it can't be used for local files, it doesn't really matter, but in my skins, I'm copying all sorts of local files to a hidden cache, and the extensions might be woff, woff2, jpg, jpeg, png, gif, svg, mp3, etc.

I see. Well at this point I require a file extension in order to avoid accidental file deletion. Say that you first load file "a.woff", "b.woff" and "c.woff", but later only load "a.woff and "b.woff", now DiskCache will delete "c.woff" for you. If it didn't have an extension filter, it would delete all other files from the target folder. Another way to avoid this whole thing with file extensions would be to store a local text file listing downloaded files. Perhaps better?
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 14 Sep 22, 17:36   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
davidekholm wrote:
JeffTucker wrote:
1. My quick test tells me that the load(URL) method can't be used with a local URL....
That's right, but if it helps debugging I can add support for the file: protocol too. It will appear in the next beta

I think that would be a useful addition. Of course, these can be simple file copies, without going through all the folderol of opening a stream, etc.

3. Is there no constructor like this?
public DiskCache(File primaryCacheDir)
I see. Well at this point I require a file extension in order to avoid accidental file deletion. Say that you first load file "a.woff", "b.woff" and "c.woff", but later only load "a.woff and "b.woff", now DiskCache will delete "c.woff" for you. If it didn't have an extension filter, it would delete all other files from the target folder. Another way to avoid this whole thing with file extensions would be to store a local text file listing downloaded files. Perhaps better?

I'm stashing files in .jalbum/.neptune during the album build. The user is never putting files there manually. So, if he's included mydog.jpg as a logo file, init.bsh grabs it from its source and stashes it there. During processing, I keep track of the files that the skin is stashing there, and in finally.bsh, it gets rid of the leftover ones that aren't used in this album. In short, accidental deletion isn't possible.

If I were using DiskCache, instead, it would be much the same. In init.bsh, I would assemble a string array of all the URL's (http and file) that are needed for this album, and then do a load(urls). Anything else that's sitting in .jalbum/.neptune would, by definition, not be needed for this album, and could safely be deleted.

But if I have to create a new DiskCache for each file type I encounter, it kind of stops being a "helper," if you know what I mean. ;)
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 14 Sep 22, 17:55   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
BTW, when it comes to grabbing local files, I'd rather not have to prepend file:/// to everything. When I use file selectors, I'm getting the full path, like C:\User\jefft\Pictures\mydog.jpg. It would be nice if I could feed that to DiskCache as a URL, just as it is. If the class would treat anything that doesn't start with http or https as a local file, that would do nicely.
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 14 Sep 22, 18:43   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
It seems, for your use case, that i could do more aggressive deletion of files from both the primary and secondary folders, but if other skin developers put files not managed by DiskCache under either of these foldes, they would be deleted. I guess the only safe way around this is to keep track of what files have been downloaded last time. I could use a little control file for this, for instance .diskcache-files.json
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 14 Sep 22, 18:53   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
Well, other skin developers could, for safety, stick to the current constructor:

DiskCache(File primaryCacheDir, String extension)

But provide the wildcarded version for the more daring among us:

DiskCache(File primaryCacheDir)

Sort of a "use at your own risk" option.

If the cache directory is imageDirectory/res, I agree that a lot of caution would be called for, since users would sometimes be putting their own stuff there. But in the case of .jalbum/.neptune, it's totally under skin control.

And the skin wouldn't even have to keep track of which files had been added. Let's say the project is using two Google fonts, a logo file, and a soundtrack. In init.bsh, a routine would add those four files to the list of URL's to be fed to load(urls). On every subsequent build, those same four URL's would be included. Only if the user stopped using one of the Google fonts, or stopped using the logo file or soundtrack, would that URL go away, and the file get deleted.
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 14 Sep 22, 19:02   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
BTW, I wouldn't delay jAlbum 29 for any of this. It's mostly "inside baseball," stuff that a user might never notice. The GDPR problem with Google fonts can easily be addressed with what's currently available. Laza's included it already in a couple of skins, and I could add just that aspect of it to my skins fairly easily.

And in my skins, there's already another workaround, since they let the user add WOFF files directly to the project. Manually getting a WOFF version of a Google font takes only a couple minutes. Hey, presto, no more "phoning home" to Google.
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 15 Sep 22, 00:41   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
Picking up on something from one of our email exchanges, yes, it would be nice to have the option to rename the stashed file. Easy enough to do with some quick BeanShell Java, but then the automatic DiskCache cleanup of unused files would fail, because it wouldn't know about the renaming.

Some of you may be asking why this is desirable. It's because when you fetch, for example, the Roboto font from Google, you don't end up with a file named:

Roboto.woff2

Instead, you get fed this file:

KFOmCnqEu92Fr1Mu72xKOzY.woff2

Makes debugging just a little messy. ;)
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 15 Sep 22, 17:19   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
I've now rewritten DiskCache in https://jalbum.net/download/beta/jalbum-core.jar

Changes:
  • A local text database now keeps track of managed files
  • You can now use an alternative local name by using the format "localName<space>url" in the load call
  • The "ext" parameter is gone. There is instead an optional second parameter containing the desired database name. If not specified, "diskcache.json" is used. The local database is stored within the primaryDirectory
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 15 Sep 22, 17:47   in response to: davidekholm in response to: davidekholm
 
  Click to reply to this thread Reply
And one more goodie that you didn't even mention: it's perfectly happy grabbing, e.g.,
file:///C:/Users/jefft/Pictures/Icons/airplane.jpg
Not sure if you intended that, or if it's just a lucky accident. :)

The "local name" option also seems to be working nicely.

I'm still at the "hard coded test cases" stage, so I'm not ready to give it a good workout.

Better warn Laza about the removal of the "ext" parameter, since he's got that coded into Story and Tiger already, and if you try to use that, you end up in a very strange situation. In short, you end up with a database file named woff2 instead of diskcache.json. It works, of course, but certainly isn't the intended behavior. ;)
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 15 Sep 22, 17:53   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
I guessed this was going to be a problem, and it is. If I try to grab:
file:///C:/Users/jefft/Pictures/Icons/airplane 80.jpg
It sees that space between "airplane" and "80" as the delimiter for a local name, and things collapse. Spaces in local paths, of course, don't get encoded - they're not URL's.

I'm thinking you may want to choose something else as the delimiter. Not sure what to suggest.

ETA: Perhaps a tab, as \t, since that can never occur in any URL or path? Or a two-element array?
JeffTucker

Posts: 8,438
Registered: 31-Jan-2006
Re: Some DiskCache questions
Posted: 15 Sep 22, 18:32   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
Also, local files get copied successfully to the primary cache, but their names are not added to diskcache.json, so they never get cleared out.
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 16 Sep 22, 08:49   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
JeffTucker wrote:
And one more goodie that you didn't even mention: it's perfectly happy grabbing, e.g.,
file:///C:/Users/jefft/Pictures/Icons/airplane.jpg
Not sure if you intended that, or if it's just a lucky accident. :)

I added support for file: protocol, but forgot to mention it.

The "local name" option also seems to be working nicely.

I'm still at the "hard coded test cases" stage, so I'm not ready to give it a good workout.

Better warn Laza about the removal of the "ext" parameter, since he's got that coded into Story and Tiger already, and if you try to use that, you end up in a very strange situation. In short, you end up with a database file named woff2 instead of diskcache.json. It works, of course, but certainly isn't the intended behavior. ;)


I'll do that, but he's currently using his own copy of this class (under another package name)
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 16 Sep 22, 08:54   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
JeffTucker wrote:
I guessed this was going to be a problem, and it is. If I try to grab:
file:///C:/Users/jefft/Pictures/Icons/airplane 80.jpg
It sees that space between "airplane" and "80" as the delimiter for a local name, and things collapse. Spaces in local paths, of course, don't get encoded - they're not URL's.

I'm thinking you may want to choose something else as the delimiter. Not sure what to suggest.

ETA: Perhaps a tab, as \t, since that can never occur in any URL or path? Or a two-element array?


I think you should simply URL encode spaces to %20, otherwise they're not correct URLs. Now if you also want a space in the local name, then we had a problem, but I'm now handing it: The last space (if exists) is guaranteed to be a delimiter. Two-element arrays looks so clunky, same goes for pair objects, just a lot of syntax.
davidekholm

Posts: 4,398
Registered: 18-Oct-2002
Re: Some DiskCache questions
Posted: 16 Sep 22, 08:57   in response to: JeffTucker in response to: JeffTucker
 
  Click to reply to this thread Reply
JeffTucker wrote:
Also, local files get copied successfully to the primary cache, but their names are not added to diskcache.json, so they never get cleared out.

Good that you spotted that one. Now corrected. I've also ensured that if you pass a name for the cache that doesn't contain a file extension, then ".json" is appended. This means that if anyone adds "woff" to that 2:nd parameter then the cache file will be called "woff.json" and that's better than just "woff" :-)
Legend
Forum admins
Helpful Answer
Correct Answer

Point your RSS reader here for a feed of the latest messages in all forums