sync_client errors out after 2.3.16 -> 2.5.9 upgrade

Mon Aug 1 22:34:13 EDT 2016

2.3.2 wasn't version 10, it was version 7!  It would have upgraded through 8, 9, 10 - and maybe you needed to reconstruct to get GUIDs for those versions - that sounds familiar.

Bron.

On Tue, Aug 2, 2016, at 12:22, Kenneth Marshall wrote:
> Hi Bron and Ellie,
> 
> I think that is what happened. Replication was working and the 'reconstruct -G'
> was optional in terms of working, so some of the oldest mailboxes could be in
> that state. I will check the versions and see if there are any older than 10,
> but we started with version 2.3.2 so unless the version changed within the
> patch series, they will all be 10.
> 
> Regards,
> Ken
> 
> On Tue, Aug 02, 2016 at 11:39:01AM +1000, Bron Gondwana via Info-cyrus wrote:
> > Yeah, could be.  In that case then the mailbox needs a reconstruct -G first :(
> > Version 10 mailboxes have a GUID space available, but I guess they could
> > wind up zero depending on how they got upgraded in the past.
> > 
> > Bron.
> > 
> > On Tue, Aug 2, 2016, at 11:26, ellie timoney via Info-cyrus wrote:
> > > I've been under the impression that Ken's mailboxes were version 10,
> > > which seems like they *should* have guids in them.   If this is the
> > > case, then it's interesting that the replica is coming up with zeroed
> > > ones.
> > > 
> > > If his mailboxes are older than version 10, then it all makes sense,
> > > nothing to see here... though it would be good if the sync code detected
> > > mailboxes being too old to replicate and reported that, instead of
> > > naively trying to sync them and crashing out on the guid check.
> > > 
> > > I guess it's plausible that version 10 mailboxes had a field in the
> > > index for the guid, but that it wasn't necessarily populated?  That
> > > might complicate things?  I don't know the history here.
> > > 
> > > On Tue, Aug 2, 2016, at 10:01 AM, Bron Gondwana via Info-cyrus wrote:
> > > > You can't sync a mailbox without GUID for messages.  You need to upgrade
> > > > the mailboxes before you can use them for replication.  The GUID is used
> > > > for replication - if we allowed zero GUIDs, then every message would
> > > > deduplicate to the same message!
> > > > 
> > > > On Tue, Aug 2, 2016, at 07:56, Kenneth Marshall via Info-cyrus wrote:
> > > > > Hi Cyrus Developers,
> > > > > 
> > > > > Thank you for your patch to address the folder move problem between
> > > > > un-reconstructed mailboxes after the 2.3.16 -> 2.5.9 upgrade. I am
> > > > > not sure, but it looks like there may be another overly aggressive
> > > > > check. I keep getting these fatal errors from sync_client:
> > > > > 
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch user.robot 1696 (56412de8678bfb53f6cdb5fe4502031af5484056 0000000000000000000000000000000000000000)
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch user.robot 1697 (1b0024218a4419973b83ae3e84ac7133a4ab7d40 0000000000000000000000000000000000000000)
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch user.robot 1698 (f17084425d83bccb28a4dfa195846c7ef88c7567 0000000000000000000000000000000000000000)
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch user.robot 1699 (7a751e41e1d3a58e541298ab724be4c29d96e49d 0000000000000000000000000000000000000000)
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: SYNCERROR: guid mismatch user.robot 1700 (724a013d0ae97d27a1da33832487df1719681659 0000000000000000000000000000000000000000)
> > > > > Aug  1 16:24:16 cyrus1a imap/sync_client[14886]: Fatal error: Internal error: assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(&record->guid)
> > > > > 
> > > > > And then the sync_client has to be run manually and if lucky, it will
> > > > > process the full log successfully. I was looking in imap/mailbox.c
> > > > > and it looks like the assert at line 2850 may need a similar override
> > > > > for non-upgraded folders:
> > > > > 
> > > > > -----------imap/mailbox.c---------------
> > > > > /* append a single message to a mailbox - also updates everything
> > > > >  * automatically.  These two functions are the ONLY way to modify
> > > > >  * the contents or tracking fields of a message */
> > > > > EXPORTED int mailbox_append_index_record(struct mailbox *mailbox,
> > > > >                                 struct index_record *record)
> > > > > {
> > > > >     indexbuffer_t ibuf;
> > > > >     unsigned char *buf = ibuf.buf;
> > > > >     size_t offset;
> > > > >     int r;
> > > > >     int n;
> > > > >     struct utimbuf settime;
> > > > >     uint32_t recno;
> > > > > 
> > > > >     assert(mailbox_index_islocked(mailbox, 1));
> > > > > 
> > > > >     /* Append MUST be a higher UID than any we've yet seen */
> > > > >     assert(record->uid > mailbox->i.last_uid)
> > > > > 
> > > > >     /* Append MUST have a message with data */
> > > > >     assert(record->size);
> > > > > 
> > > > > =====>    /* GUID must not be null */
> > > > > =====>    assert(!message_guid_isnull(&record->guid));
> > > > > 
> > > > >     /* belt AND suspenders - check the previous record too */
> > > > >     if (mailbox->i.num_records) {
> > > > >         struct index_record prev;
> > > > > 
> > > > > -----------imap/mailbox.c---------------
> > > > > 
> > > > > What do you think?
> > > > > 
> > > > > Regards,
> > > > > Ken
> > > > > ----
> > > > > Cyrus Home Page: http://www.cyrusimap.org/
> > > > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > > > To Unsubscribe:
> > > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > > > 
> > > > 
> > > > -- 
> > > >   Bron Gondwana
> > > >   brong at fastmail.fm
> > > > ----
> > > > Cyrus Home Page: http://www.cyrusimap.org/
> > > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > > To Unsubscribe:
> > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > > ----
> > > Cyrus Home Page: http://www.cyrusimap.org/
> > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > > To Unsubscribe:
> > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 
> > 
> > -- 
> >   Bron Gondwana
> >   brong at fastmail.fm
> > ----
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 

-- 
  Bron Gondwana
  brong at fastmail.fm