sync_client failing with "Fatal error: failed to finish reading file!"

[Nic Bernstein]

On 06/17/2013 01:00 AM, Bron Gondwana wrote:
> On Wed, Jun 12, 2013, at 06:07 AM, Nic Bernstein wrote:
>>  Checking the source for dlist.c, here is the context for this error
>>  [printfile()]:
> <SNIP>
> No, if size < 0 then we've read PAST the end of the file - which is also
> totally bogus.  I can see a case for while (size > 0) on the loop though,
> for this exact case, so we drop out earlier than EOF.
>
> There are only two ways to exit that loop: either size gets down to zero,
> or fread returns a non-positive value.  From the man page:
>
> RETURN VALUE
>         On success, fread() and fwrite() return the number of
>         items read or written.
>         This number equals the number of bytes transferred only
>         when size is 1.  If an error occurs, or the end of the
>         file is reached, the return value is a short item count
>         (or zero).
>
>         fread() does not distinguish between end-of-file and
>         error, and callers must use feof(3) and ferror(3) to
>         determine which occurred.
>
> So we could certainly make the code better about reporting the
> cause of the error.  I suspect 99% probability file permissions
> problems (your 'cyrus' user can't read it, but can stat it) and
> 1% probability a corrupt filesystem with that file contents
> being unreadable.  It's probably the very first fread which is
> failing.

The directories and files are all user=cyrus, group=mail and all perms
seem to be correct: 700 for directories and 600 for files.  Given that
we only see this on the middle system in a replication chain, we suspect
that perhaps the sync_server is still writing the file while the
sync_client starts to slurp it in.  Any chance this is the case?

What sucks for us about this is that sync_client dies, so the host at
the end of the chain falls out of sync, requiring manual intervention.

Cheers,
    -nic

-- 
Nic Bernstein                             nic at onlight.com
Onlight, Inc.                             www.onlight.com
219 N. Milwaukee St., Suite 2a            v. 414.272.4477
Milwaukee, Wisconsin  53202