<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div>On Tue, Nov 12, 2019, at 14:50, Anatoli wrote:<br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Bron,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">The proposed algo is a barrier before any single-lock. In itself it's a<br></div><div style="font-family:Arial;">single lock, but the same code (the pseudocode for the *worker thread*<br></div><div style="font-family:Arial;">in my previous mail) should be inserted at *every* single-lock/write<br></div><div style="font-family:Arial;">operation location. If there's no need to pause, the overhead is<br></div><div style="font-family:Arial;">non-existent. If a pause is requested, all worker threads would pause at<br></div><div style="font-family:Arial;">the entrance to any single-lock/write code.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">It would make the entire Cyrus daemon to complete all pending write<br></div><div style="font-family:Arial;">operations and pause new ones. At this stage, if I understand it<br></div><div style="font-family:Arial;">correctly, the data on disk would be in a consistent state, ready to<br></div><div style="font-family:Arial;">take a snapshot or to perform some other operation.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">"complete all pending write operations and pause new ones"<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">How do you know when the current write operations are finished?<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Without that, if we just take a snapshot of the fs, it could happen that<br></div><div style="font-family:Arial;">a) some files are not written entirely (i.e. caught in the middle of a<br></div><div style="font-family:Arial;">write operation) or b) the contents of some files are newer than the<br></div><div style="font-family:Arial;">other, i.e. the logical write operation was not atomic (e.g. mail data<br></div><div style="font-family:Arial;">is written but indexes are not updated yet or something similar).<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Maybe I didn't understand you correctly. Do you mean that finishing all<br></div><div style="font-family:Arial;">writes and pausing new ones is not enough to guarantee an integral state<br></div><div style="font-family:Arial;">of files on disk? If it's the case, what would have to be done to<br></div><div style="font-family:Arial;">guarantee it (i.e. to make it like Cyrus was shutdown normally)?<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I mean that to finish all writes and pause new ones, you need to know that the writes are finished. And not just writes, but sets of writes that are held under a lock together. The way I know to do this is a single global lock with the following properties:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">1) every action which locks any file within Cyrus for writing takes a SHARED global lock before it takes the write lock on the file.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">2) the SHARED lock is held for the duration of the writes, and released once the writes are finished.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">3) the "backup utility" takes an EXCLUSIVE lock on the global lock, which will only be granted once each write is finished. It then takes a snapshot, and releases the EXCLUSIVE lock.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">This guarantees full consistency.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">The question that always exists for locks is "what granularity" - too wide, and you hold the lock for a long time. Too narrow, and you take and release it very frequently, adding overhead.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">My first and most dumbest theory is to go quite wide - add the lock in every runloop and command line utility such that it's held for the entire running of the loop or the utility! Mostly these are done within a fraction of a second. The one place that might be interesting is FETCH 1:* RFC822.PEEK or similar in imapd, where we already have some locking magic that holds a shared namelock on the mailbox to stop repacking while it releases the index lock to allow other actions on the mailbox in the meanwhile.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">So we could go down a layer and only lock when we lock mailboxes or cyrusdbs, and refcount the global lock. This seems more likely to be a good layer, and not too horrible.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">The other thing is that we'll need to assert that the lock isn't being held during each daemon's command loop, so that bugs don't leak out to deadlock entire servers.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">And I think that's nearly it :)<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, Fastmail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>