refresh performance of a large font folder

Get help with MainType here. Please do not post feature requests or bug reports here.
Post Reply
thibaud
Posts: 68
Joined: Fri Jan 27, 2006 8:20 pm
Contact:

refresh performance of a large font folder

Post by thibaud » Mon Jan 18, 2016 1:21 pm

My main font library is quite large (around 100k fonts)
it's added via a single "keep in sync" folder set to manual refresh mode.
whenever I refresh the whole folder via CTRL-R or individual subfolders via either the button "synchronize the selected font folder" in the group pane or the contextual menu of the subfolder.
the library refresh takes a very long time: 7 minutes :shock:

Considering that the library is stored on a quite fast local raid5 array (8 disks) are these times considered normal ?

I assumed refreshing a single subfolder holding a few fonts would take considerably less time, but it isn't the case (it seems like the whole library is updated each time) is this normal ?

What can I do to improve this refresh performance ?

Erwin Denissen
Moderator
Moderator
Posts: 6899
Joined: Fri Oct 04, 2002 12:41 am
Location: De Bilt, The Netherlands
Contact:

Re: refresh performance of a large font folder

Post by Erwin Denissen » Mon Jan 18, 2016 2:36 pm

The sync feature will be applied to the whole folder and during that process it has to read all font resources to ensure they aren't changed. 7 minutes sounds pretty fast to me.
Erwin Denissen
High-Logic
Proven Font Technology

thibaud
Posts: 68
Joined: Fri Jan 27, 2006 8:20 pm
Contact:

Re: refresh performance of a large font folder

Post by thibaud » Mon Jan 18, 2016 3:32 pm

ok, but isn't there room for improvement here ?
most library manager (be it music or pictures) even when dealing with large libraries don't impose a 7 minutes delay every time you import new items.

Bhikkhu Pesala
Top Typographer
Top Typographer
Posts: 7077
Joined: Tue Oct 29, 2002 5:28 am
Location: Seven Kings, London UK
Contact:

Re: refresh performance of a large font folder

Post by Bhikkhu Pesala » Mon Jan 18, 2016 10:20 pm

thibaud wrote:My main font library is quite large (around 100k fonts)
the library refresh takes a very long time: 7 minutes :shock:

I assumed refreshing a single subfolder holding a few fonts would take considerably less time, but it isn't the case (it seems like the whole library is updated each time) is this normal ?

What can I do to improve this refresh performance ?
My maths tells me that 420 seconds to examine 100,000 font files isn't too bad. My library is only about 4K of fonts, and refreshing the fonts folders takes less than 10 seconds on a standard PC.

I have my fonts organised into subfolders on my hard drive, and MainType creates groups from these folders. Most folders never change. The folders that do change regularly, like my OpenType folder for fonts generated by FontCreator are refreshed, the other are not. The Fonts Catalogue folder doesn't change often, but it makes it easier to update my online catalogue if it's included in my Font Folders.

I think you should organise your font collection so that MainType only has to refresh those folders that are likely to change.
Font Folders.png
Font Folders.png (21.3 KiB) Viewed 1009 times
My FontsReviews: MainTypeFont CreatorHelpFC11.0 Pro + MT7.0 @ Win10 1703

thibaud
Posts: 68
Joined: Fri Jan 27, 2006 8:20 pm
Contact:

Re: refresh performance of a large font folder

Post by thibaud » Tue Jan 19, 2016 10:43 am

Bhikkhu Pesala wrote: I think you should organise your font collection so that MainType only has to refresh those folders that are likely to change.
My font library is tightly organized, it consist of OTF files only, structured like so:

Code: Select all

+ FONT LIBRARY
|   + FoundryName
|   |   + FamilyName
|   |   |    FontName-bold.otf
|   |   |    FontName-light.otf
|   |   |    FontName-regular.otf
|   |   |    ...
|   |   - FamilyName
|   |   - ...
|   - FoundryName
|   - ...
Whenever I add a new font family it naturally goes into this structure.
Changing the way I store and manage incoming fonts to work around how the refresh works, (like having a separate incoming structure that gets flushed and integrated into the main structure periodically) doesn't sound like a practical solution (extra work, delayed duplicate management, etc..)

Just to clarify (I noticed the subject line is missleading) I'm not suggesting there are performance issues in the refresh routine itself (I suppose it's indeed quite fast for what it does- though I still have to make some test because I suspect something fishy here as the scaling isn't linear 4k/10sec & 100k/420sec :? ) but rather questioning the way the refresh works by forcing an entire re-scan of the library.
Erwin Denissen wrote:The sync feature will be applied to the whole folder and during that process it has to read all font resources to ensure they aren't changed. 7 minutes sounds pretty fast to me.
Why is this so ?
Are there any technical reason/limitation why the refresh is not operated on the selected folder only ?
Having a action that says "synchronize the selected font folder" update the whole library instead of the selected folder looks like a bug from a user perspective.

Would you consider a more optimized solution that would rely on some sort of directory change notification from the filesystem for a future Maintype version perhaps ?

Erwin Denissen
Moderator
Moderator
Posts: 6899
Joined: Fri Oct 04, 2002 12:41 am
Location: De Bilt, The Netherlands
Contact:

Re: refresh performance of a large font folder

Post by Erwin Denissen » Tue Jan 19, 2016 11:19 am

thibaud wrote:
Erwin Denissen wrote:The sync feature will be applied to the whole folder and during that process it has to read all font resources to ensure they aren't changed. 7 minutes sounds pretty fast to me.
Why is this so ?
Are there any technical reason/limitation why the refresh is not operated on the selected folder only ?
Having a action that says "synchronize the selected font folder" update the whole library instead of the selected folder looks like a bug from a user perspective.
I understand, so I'll add this to our to-do list.
thibaud wrote:Would you consider a more optimized solution that would rely on some sort of directory change notification from the filesystem for a future Maintype version perhaps ?
We initially spend months working on this approach, but came to the sad conclusion the file system change notifications are not always guaranteed to get fired, so we decided take another approach.
Erwin Denissen
High-Logic
Proven Font Technology

thibaud
Posts: 68
Joined: Fri Jan 27, 2006 8:20 pm
Contact:

Re: refresh performance of a large font folder

Post by thibaud » Tue Jan 19, 2016 12:46 pm

Thanks a lot for the prompt answers.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests