Murder problems, tracked down do updatelist, followup.

[Sergio Devojno Bruder]

Sergio Devojno Bruder wrote:
> There are some problems with one cyrus murder deployment,
> where our frontends arent getting updates, we tracked it
> down to updatelist.

More info: we are in initial test phase, linux environment, CentOS 3.3 (a recompiled free version of RHEL 3):
2 backends (2xP4 Xeon, 2GB RAM, local SCSI now);
mupdate master (P4 Xeon with HT, 2GB RAM);
2 frontends (2xP4 Xeon, 2GB RAM, local SCSI);

This time mupdate master was using a -UP kernel, we tested :

	- all boxes already started (backends, master and frontends).
	- stop mupdate and frontends.
	- mupdate master start, frontend2 start, frontend1 start;

In the case of the second frontend starting updatelist was
NULL, with no reference of the first frontend. How that can be?
[12921] and [12931] arent threads of the same process?
why updatelist mod in 12921 wasnt visible in 12931?

(12921 and 12931, pids of mupdate master processes that handled
the connections of the 2 frontends).


Patch and syslog follow:

patch for cyrus 2.2.10:
--- trunk/imap/mupdate.c
+++ trunk/imap/mupdate.c
@@ -54,6 +54,7 @@
  #include <assert.h>
  #include <syslog.h>
  #include <errno.h>
+#include <stdarg.h>

  #include <netdb.h>
  #include <sys/socket.h>
@@ -347,6 +348,27 @@
      return C;
  }

+
+static void syslog_updatelist(const char *msg, ...)
+{
+    va_list argp;
+    struct conn *upc;
+    int i;
+    char buffer[1024];
+
+    va_start(argp, msg);
+    vsprintf(buffer, msg, argp);
+    va_end(argp);
+
+    syslog(LOG_DEBUG, "<updatelist printout msg=\"%s\">", buffer);
+    i = 0;
+    for (upc = updatelist; upc != NULL; upc = upc->updatelist_next) {
+       syslog(LOG_DEBUG, "\tupdatelist element #%d\t0x%x\tfd:%d\t%s;", i++, (int) upc, upc->fd, upc->clienthost);
+    }
+    syslog(LOG_DEBUG, "</updatelist printout>");
+}
+
+
  static void conn_free(struct conn *C)
  {
      assert(!C->idle); /* Not allowed to free idle connections */
@@ -1734,8 +1758,10 @@
      /* indicate interest in updates */
      pthread_mutex_lock(&mailboxes_mutex); /* LOCK */

+    syslog_updatelist("before cmd_startupdate(%s)", C->clienthost);
      C->updatelist_next = updatelist;
      updatelist = C;
+    syslog_updatelist("after cmd_startupdate(C: 0x%x, tag:%s, partial:0x%x)", (int)C, tag, (int) partial);
      C->streaming = xstrdup(tag);
      C->streaming_hosts = partial;


syslog (edited for readability):
19:15:43 mupdate[12921]: login: frontend2 [192.168.115.19] cyrus PLAIN User logged in
19:15:43 mupdate[12921]: <updatelist printout msg="before cmd_startupdate(frontend2 [192.168.115.19])">
19:15:43 mupdate[12921]: </updatelist printout>
19:15:43 mupdate[12921]: <updatelist printout msg="after cmd_startupdate(C: 0x8793940, tag:U01, partial:0x0)">
19:15:43 mupdate[12921]:         updatelist element #0   0x8793940       fd:10   frontend2 [192.168.115.19];
19:15:43 mupdate[12921]: </updatelist printout>
19:16:02 mupdate[12931]: accepted connection
19:16:02 mupdate[12931]: login: frontend1 [192.168.115.18] cyrus PLAIN User logged in
19:16:02 mupdate[12931]: <updatelist printout msg="before cmd_startupdate(frontend1 [192.168.115.18])">
19:16:02 mupdate[12931]: </updatelist printout>
19:16:02 mupdate[12931]: <updatelist printout msg="after cmd_startupdate(C: 0x844f940, tag:U01, partial:0x0)">
19:16:02 mupdate[12931]:         updatelist element #0   0x844f940       fd:10   frontend1 [192.168.115.18];
19:16:02 mupdate[12931]: </updatelist printout>

pstree of relevant processes:
   |-cyrus-master,12915 -d
   |   |-mupdate,12921 -m
   |   |-mupdate,12929 -m
   |   |-mupdate,12930 -m
   |   |-mupdate,12931 -m
   |   |-mupdate,12935 -m
   |   |-mupdate,12938 -m

--
Sergio Devojno Bruder


---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html