sync_client bails when encountering a deleted message

John Capo jc at irbs.com
Sun May 13 12:19:55 EDT 2007


These patches handle the message deleted case and the folder rename
case properly.  Should handle all cases of messages and folders
disappearing while working on a folder.  May handle deleting a user
while working on that user.

John Capo



Quoting John Capo (jc at irbs.com):
> Pages in the middle of the night the last few days prompted me to
> look into this long standing problem.
> 
> upload_message_work() returns IMAP_IOERROR when it can't map the
> message being uploaded due to a race between reading the index and
> actually doing the upload work.  upload_message_work() returns with
> these items in the stream to the server.
> 
>  PARSED msgid uid internaldate sent-date last-updated modseq flags
> 
> The server is expecting the header_size to arrive next but most
> anything can arrive depending on what client work is left to do.
> 
> Partially sanitized and shortened log entries for a case where a
> folder was renamed with other operations on the same mailbox and
> other mailboxes in the work queue.
> 
> 03:25:01 sync_client: DELSUB gersham_domain1^com user.gersham_domain1^com.mova
> 03:25:01 sync_client: MAILBOX user.gersham_domain1^com
> 03:25:01 sync_client: MAILBOX user.gersham_domain1^com.mova
> 03:25:01 sync_client: MAILBOX user.gersham_domain1^com.something.mova
> 03:25:01 sync_client: MAILBOX user.1040001_1051001-domain2^com
> 03:25:01 sync_client: MAILBOX user.gersham_domain1^com.something
> 03:25:01 sync_client: MAILBOX user.gersham_domain1^com.Trash
> 03:25:01 sync_client: MAILBOX user.rismi_domain3^co^id
> 03:25:01 imap: Rename: gersham_domain1^com something/mova => mova
> 03:25:01 sync_client: IOERROR: opening message file 594 of user.gersham_domain1^com.something.mova: No such file or directory
> 03:25:01 sync_client:   Promoting: MAILBOX user.gersham_domain1^com.mova -> USER gersham_domain1^com
> 03:25:01 imap: Deleted mailbox user.gersham_domain1^com.something.mova
> 03:25:03 sync_client: MAILBOX user.gersham_domain1^com.something.mova
> 03:25:03 sync_client: MAILBOX user.1040001_1051001-domain2^com
> 03:25:03 sync_client: MAILBOX user.gersham_domain1^com.something
> 03:25:03 sync_client: MAILBOX user.gersham_domain1^com.Trash
> 03:25:03 sync_client: MAILBOX user.rismi_domain3^co^id
> 03:25:03 sync_client: MAILBOXES received BAD response: Syntax error in Append at item 3: Invalid flags
> 03:25:03 sync_client:   Promoting: MAILBOX user.gersham_domain1^com.something.mova -> USER gersham_domain1^com
> 03:25:07 sync_client: MAILBOX user.1040001_1051001-domain2^com
> 03:25:07 sync_client: MAILBOX user.gersham_domain1^com.something
> 03:25:07 sync_client: MAILBOX user.gersham_domain1^com.Trash
> 03:25:07 sync_client: MAILBOX user.rismi_domain3^co^id
> 03:25:07 sync_client: USER gersham_domain1^com
> 03:25:07 sync_client: UPLOAD received BAD response: Syntax error in Append at item 3: Unknown Reserved message
> 03:25:09 sync_client: USER gersham_domain1^com
> 03:25:09 sync_client: UPLOAD received BAD response: Syntax error in Append at item 3: Unknown Reserved message
> 03:25:13 sync_client: USER gersham_domain1^com
> 03:25:13 sync_client: UPLOAD received BAD response: Syntax error in Append at item 1: Unknown Reserved message
> 03:25:13 sync_client: Error in do_sync(): bailing out!
> 
> The failure message depends on how much work is in the queue.
> 
> sync_client: MAILBOXES: Invalid unsolicited response type 1 from server: (null)
> sync_client: Discarding: 104056 010400463ca07b23db000000 ()
> 
> A few to many thousands of those depending on the mailbox size.
> 
> sync_client: MAILBOXES received BAD response: Syntax error in Append at item 3: Invalid flags
> 
> Locking the mailbox would fix it the race but it may stay locked
> for longer that one would want.
> 
> I tinkered with sending a token to the server signaling it to cleanup
> the message_list and the upload_list and wait for a new command but
> doing that right is rather involved.
> 
> A really ugly hack is to and fake up a message and send that to
> keep the stream in sync.  The fake message is deleted from the
> replica on the next MAILBOX command.
> 
>     if (r) {
> 	char *message = "Message vanished from primary, probably deleted\r\n";
> 
> 	syslog(LOG_ERR, "IOERROR: opening message file %lu of %s: %m",
> 	       record->uid, mailbox->name);
>  
> 	prot_printf(toserver, " 0 0 0 {0+}\r\n");
> 	prot_printf(toserver, "{%lu+}\r\n", strlen(message));
> 	prot_write(toserver, message, strlen(message));
> 	mailbox_unmap_message(mailbox, record->uid, &msg_base, &msg_size);
> 	sequence++;    /* This variable is not used */
> 	return (0);
>     }
> 
> Monitoring and restarting the sync_client always works but that's
> ugly too.
> 
> John Capo
> 
> 
> 
-------------- next part --------------
diff -ru cyrus-imapd-2.3.7/imap/sync_client.c cyrus-imapd-2.3.7-abort-upload/imap/sync_client.c
--- cyrus-imapd-2.3.7/imap/sync_client.c	Sun May 13 11:56:58 2007
+++ cyrus-imapd-2.3.7-abort-upload/imap/sync_client.c	Sun May 13 12:06:27 2007
@@ -1179,6 +1179,9 @@
         if (r) {
             syslog(LOG_ERR, "IOERROR: opening message file %lu of %s: %m",
                    record->uid, mailbox->name);
+	    prot_printf(toserver, " %lu ", 0xdeadbeef);
+	    prot_flush(toserver);
+	    sync_msgid_remove(msgid_onserver, &record->uuid);
             return(IMAP_IOERROR);
         }
 
diff -ru cyrus-imapd-2.3.7/imap/sync_server.c cyrus-imapd-2.3.7-abort-upload/imap/sync_server.c
--- cyrus-imapd-2.3.7/imap/sync_server.c	Sun May 13 11:56:58 2007
+++ cyrus-imapd-2.3.7-abort-upload/imap/sync_server.c	Sun May 13 12:05:41 2007
@@ -1780,6 +1780,8 @@
     upload_list = sync_upload_list_create(new_last_uid, mailbox->flagname);
 
     do {
+	unsigned long hdr_size;
+
         err  = NULL;
         item = sync_upload_list_add(upload_list);
 
@@ -1870,19 +1872,28 @@
                 goto parse_err;
             }
 
-            message = sync_message_add(message_list, &item->uuid);
-
             /* Parse Message (header size, content lines, cache, message body */
             if ((c = getastring(sync_in, sync_out, &arg)) != ' ') {
                 err = "Invalid Header Size";
                 goto parse_err;
             }
-            message->hdr_size = sync_atoul(arg.s);
             
             if ((c = getastring(sync_in, sync_out, &arg)) != ' ') {
                 err = "Invalid Content Lines";
                 goto parse_err;
             }
+
+	    if ((hdr_size = sync_atoul(arg.s)) == 0xdeadbeef)
+	    {
+		/* Make sure cache data flushed to disk before we commit */
+		sync_message_fsync(message_list);
+		sync_message_list_cache_flush(message_list);
+		sync_upload_list_free(&upload_list);
+		syslog(LOG_ERR, "IOERROR: UPLOAD ABORT");
+		return;
+	    }
+            message = sync_message_add(message_list, &item->uuid);
+            message->hdr_size = sync_atoul(arg.s);
             message->content_lines = sync_atoul(arg.s);
             
             if ((r=sync_getcache(sync_in, sync_out, message_list, message)) != 0) {
diff -ru cyrus-imapd-2.3.7/imap/sync_support.c cyrus-imapd-2.3.7-abort-upload/imap/sync_support.c
--- cyrus-imapd-2.3.7/imap/sync_support.c	Sun May 13 11:56:59 2007
+++ cyrus-imapd-2.3.7-abort-upload/imap/sync_support.c	Sun May 13 12:03:25 2007
@@ -544,6 +544,25 @@
     return(NULL);
 }
 
+struct sync_msgid *sync_msgid_remove(struct sync_msgid_list *l,
+				     struct message_uuid *uuid)
+{
+    int offset = message_uuid_hash(uuid, l->hash_size);
+    struct sync_msgid *msgid;
+
+    if (message_uuid_isnull(uuid))
+        return(NULL);
+
+    for (msgid = l->hash[offset] ; msgid ; msgid = msgid->hash_next) {
+        if (message_uuid_compare(&msgid->uuid, uuid))
+	{
+	    msgid->uuid.value[0] = 0;
+	    return;
+	}
+    }
+    return(NULL);
+}
+
 /* ====================================================================== */
 
 struct sync_folder_list *sync_folder_list_create(void)
diff -ru cyrus-imapd-2.3.7/imap/sync_support.h cyrus-imapd-2.3.7-abort-upload/imap/sync_support.h
--- cyrus-imapd-2.3.7/imap/sync_support.h	Sun May 13 11:56:59 2007
+++ cyrus-imapd-2.3.7-abort-upload/imap/sync_support.h	Sun May 13 12:11:21 2007
@@ -143,6 +143,9 @@
 struct sync_msgid *sync_msgid_add(struct sync_msgid_list *list,
 				  struct message_uuid *uuid);
 
+struct sync_msgid *sync_msgid_remove(struct sync_msgid_list *list,
+				  struct message_uuid *uuid);
+
 struct sync_msgid *sync_msgid_lookup(struct sync_msgid_list *list,
 				     struct message_uuid *uuid);
 


More information about the Cyrus-devel mailing list