Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

On Tue, Nov 19, 2019, at 9:38 AM, Deborah Pickett wrote:
> > Food for thought.  Maybe instead of having one "%SHARED" backup, having one "%SHARED.foo" backup per top-level shared folder would be a better implementation?  I haven't seen shared folders used much in practice, so it's interesting to hear about it.
> >
> > Looking at your own data, if you had one "%SHARED.foo" backup per top level shared folder, would they be roughly user-sized pieces, or still too big?  If too big, how deep would you need to go down the tree until the worst offenders are a manageable size?  (If I make it split shared folders like this, maybe "how-deep-to-split-shared-folders" needs to be a configuration parameter, because I guess it'll vary from installation to installation.)
> >
> For my data, %SHARED.foo would be the perfect granularity level. Each 
> foo is a shared email address like "sales" or "accounts" and it gets 
> about as much traffic as a user account does.  (Two months ago when we 
> were on Exchange, they _were_ user accounts.)

Ah yep, that makes sense!
 
> foo also includes "#calendars" and "#addressbooks" on my server so there 
> are weird characters to deal with.

Now that's an interesting detail to consider.  I think, with a hypothetical depth setting, I would treat any level that contains '#directories' as being "too deep" for splitting, regardless of the depth setting, because at that point we're looking at things that I guess we expect to belong together.  Like, if hypothetical_depth is 3, but foo.#calendars exists, then I think we'd want to treat the entirety of foo as a single backup (as if hypothetical_depth were 1), regardless of what else is deeper in there.  I need to think about this more.

I'm gonna have a go at implementing this (I've opened https://github.com/cyrusimap/cyrus-imapd/issues/2915) but I'll step through it one level of complication at a time.