From augie.schwer at gmail.com Mon Oct 6 19:42:59 2008 From: augie.schwer at gmail.com (Augie Schwer) Date: Mon, 6 Oct 2008 16:42:59 -0700 Subject: [imapproxy-info] Error connecting to IMAP server: localhost. Message-ID: <192e943c0810061642pbfe33e6wf14a1e7c513c93c6@mail.gmail.com> Anyone seen any problems with a long running imapproxy (v1.2.4) process gives intermittent timeouts to clients? I had a up-imapproxy v1.2.4 instance that had been running since January, then a few weeks ago rumors of Webmail users behind the proxy receiving timeouts started. There were no obvious errors in the logs, and restarting the long running imapproxy process seems to have solved the problem; I plan on updating the imapproxy to the latest release, but I was hoping to find a more definitive answer on why this occurred in the first place. I did not see anything in the Changelog to indicate that this specific bug was fixed (if it even is a bug in imapproxy). I have an strace with what looks like an example of a reported timeout, and I see what looks to be futex lock contention between two threads, but I'll save that until there is some interest in that. Note: this is Linux CentOS 4.6 kernel 2.6.9-67.0.1.ELsmp. Any advice greatly appreciated. -- Augie Schwer - Augie at Schwer.us - http://schwer.us Key fingerprint = 9815 AE19 AFD1 1FE7 5DEE 2AC3 CB99 2784 27B0 C072 From slusarz at curecanti.org Fri Oct 17 00:51:27 2008 From: slusarz at curecanti.org (Michael M. Slusarz) Date: Thu, 16 Oct 2008 22:51:27 -0600 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response Message-ID: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> Hi all, I'm currently in the process of rewriting the IMAP library used in Horde/IMP - we are converting to a fully PHP socket based system. Now that we have much more control over the details of the IMAP transaction, rather than the abstracted interface of PHP's imap extension, it gives me an opportunity to play around with some webmail/imapproxy limitations that have bugged me for awhile :) Attached find a patch that does the following: 1. Changes the names of the internal imapproxy commands, from P_[CMDNAME] to XPROXY_[CMDNAME], and clean up the output a bit Rationale: imapproxy is a RFC 3501 IMAP server (essentially the only requirements needed to be an RFC 3501 IMAP server is to be able to connect to it and have the server return a 'IMAP4rev1' capability, which imapproxy will do if connected to an IMAP4rev1 server). As such, it should follow the RFCs. To conform, these imapproxy commands should follow the requirements of RFC 3501 [6.5.1] which requires an 'X' to prefix any command that does not appear in an RFC. Additionally, clean up the response from the commands (specifically the stats and the dumpicc commands) to conform to an IMAP untagged response - previously, these commands were not returning the command name in the responses. Also, tweak the OK tagged response for a couple of the commands a bit to make it a bit easier to follow what happened, especially if a human is reading/debugging the output. 2. Adds an XIMAPPROXY capability string to the CAPABILITY list returned from the remote server before we login. Rationale: Although proxies generally are meant to transparent, that does not need to be the case. imapproxy is already not fully transparent since it is broadcasting, and serving, those internal IMAP commands. Adding an IMAPPROXY capability alerts a client that these imapproxy functions MAY be available. It is also useful for debugging (to make sure one is connecting through the proxy) and, since the imapproxy commands are not part of the RFC, is required by 3501 to indicate that additional commands are available. Unknown capabilities are ignored by clients so this doesn't raise any additional issues. 3. Adds a status response [XPROXYREUSE] if the connection is restored from the cache as opposed to newly created. Rationale: Right now, imapproxy is useful to prevent the overhead of creating/tearing down an IMAP connection on each page access. However, in the present version, it does not leverage another benefit of keeping the connection open - initialization of the remote IMAP server. Although there is little/no initialization needed with a straight RFC 3501 server, various extensions should only need to be run once and will just add extra overhead if they need to be sent to the server on every connection. These are just a few examples: RFC 2971 (ID) No need to send an ID/get the server ID on every page access - once per webmail login is more than sufficient. RFC 5161 (ENABLE) Certain newer IMAP extensions are required to be enabled before using. CONDSTORE (RFC 4551) is optionally enabled using the ENABLE command and QRESYNC is REQUIRED to be enabled via the command. Again, the enable command only needs to be sent once per login. RFC 5255 (LANGUAGE/COMPARATOR) Setting the LANGUAGE and/or COMPARATOR on the server should only be done once per webmail login also. Minor note: it *is* useful to set the language before logging in on servers that support it because this may result in failed logins to return a translated error message. However, an imapproxy connection should wait until after login to do this to prevent sending this request every login. Additionally, with the current code, I'm not even sure the proxy would send this request on to the remote server. It might be smart to remove LANGUAGE from the CAPABILITY string before login... Anyway, enough talk - here's the code (hopefully this list accepts text attachments). As of tonight, the Horde_Imap_Client lib will support all three features so even if the patch is not incorporated into imapproxy, I will eventually make it available on the Horde website once the patch matures. Comments/criticisms are welcome. Especially regarding the quality/security of the code - I honestly haven't written C code since college 10 years ago. Hopefully I have not exposed huge security holes. :) Thanks, michael -------------- next part -------------- --- README.old 2008-10-16 22:21:12.742888208 -0600 +++ README 2008-10-16 21:40:51.137261255 -0600 @@ -185,6 +185,31 @@ ############################################################################## +NEW STATUS RESPONSE: +############################################################################## + +Upon a successful LOGIN or AUTHENTICATION, if imapproxy reused an existing +connection an untagged IMAP response with the status response XPROXYREUSE +will be issued. This status response indicates to a client that any previous +initialization does not need to be performed again. + +Examples: +--------- +1. Login without an active cached connection. + +C: a001 LOGIN user password +S: [... untagged response(s) from remote IMAP server ...] +S: a001 OK User logged in + +2. Login with an active cached connection. + +C: a001 LOGIN user password +S: [... untagged response(s) from remote IMAP server ...] +S: * OK [XPROXYREUSE] IMAP connection reused by imapproxy +S: a001 OK User logged in + + +############################################################################## ADDITIONAL COMMANDS: ############################################################################## @@ -199,37 +224,105 @@ If you secure the listen_port, you can then set the enable_admin_commands option in imapproxy.conf to enable the use of these commands. -P_NEWLOG --------- +These commands are available only until you log into the remote IMAP server. + +If these commands are available, the string 'XIMAPPROXY' will be listed in +the CAPABILITY string before login/authentication (see RFC 3501 [6.1.1]). + +These commands are prefixed with 'X' to comply with RFC 3501 [6.5.1]. + + +XPROXY_NEWLOG +------------- Since the protocol log file is held open as long as the server is running, it's difficult to clear the logfile. This command was added to take care of -that for you. It doesn't accept any arguments. +that for you. -P_DUMPICC ---------- +Usage: + XPROXY_NEWLOG + +Example: +C: a001 XPROXY_NEWLOG +S: a001 OK Logfile cleared + + +XPROXY_DUMPICC +-------------- The Dump ICC command allows you to display the internal data structures of the proxy server. It can tell you how many connections you currently have open, what users they're for, and the status of the connections (active or cached). Use this sparingly if the proxy server is extremely busy. -P_TRACE -------- +Usage: + XPROXY_DUMPICC + +Untagged response(s) (1 for each cached connection): + * XPROXY_DUMPICC <#> + + <#> = Hash lookup # of internal IMAP connection + = Username of connection + = Either 'Active' or 'Cached' + +Example: +C: a001 XPROXY_DUMPICC +S: * XPROXY_DUMPICC 1 foo Cached +S: * XPROXY_DUMPICC 2 foo2 Active +S: a001 OK Completed + + +XPROXY_TRACE +------------ This is used to turn on or off protocol logging. If issued without any arguments, it will disable protocol logging. If issued with a username argument, it will turn on logging for that particular user. Protocol log output will show up in the file configured as "protocol_log_filename" in the configuration file. -P_RESETCOUNTERS ---------------- +Usage: + XPROXY_TRACE [user] + + [user] = (OPTIONAL) Username to log + +Examples: +C: a001 XPROXY_TRACE +S: a001 OK Tracing disabled + +C: a001 XPROXY_TRACE foo +S: a001 OK Tracing enabled for user foo + + +XPROXY_RESETCOUNTERS +-------------------- This allows you to reset the internal counters that pimpstat reports on without having to restart the server. These are the counters that are mmap()ed to the file configured as "stat_filename" in the configuration file. -P_VERSION -------- +Usage: + XPROXY_RESETCOUNTERS + +Example: +C: a001 XPROXY_RESETCOUNTERS +S: a001 OK Counters reset + + +XPROXY_VERSION +-------------- This is used to show the version number of the IMAP proxy. +Usage: + XPROXY_VERSION + +Untagged response(s) (1 for each cached connection): + * XPROXY_VERSION + + = Current version number of imapproxy. + +Example: +C: a001 XPROXY_VERSION +S: * XPROXY_VERSION 1.2.6 +S: a001 OK Completed + + Happy proxying, --- src/imapcommon.c.old 2008-10-16 12:53:11.472952773 -0600 +++ src/imapcommon.c 2008-10-16 12:33:07.340974680 -0600 @@ -505,6 +505,10 @@ "LOGIN: '%s' (%s:%s) on existing sd [%d]", Username, ClientAddr, portstr, ICC_Active->server_conn->sd ); + + /* Set the ICD as 'reused' */ + ICC_Active->server_conn->reused = 1; + return( ICC_Active->server_conn ); } } @@ -520,6 +524,10 @@ */ Server.conn = ( ICD_Struct * ) malloc( sizeof ( ICD_Struct ) ); memset( Server.conn, 0, sizeof ( ICD_Struct ) ); + + /* As a new connection, the ICD is not 'reused' */ + Server.conn->reused = 0; + Server.conn->sd = socket( ISD.srv->ai_family, ISD.srv->ai_socktype, ISD.srv->ai_protocol ); if ( Server.conn->sd == -1 ) --- src/main.c.old 2008-10-16 14:46:04.698942614 -0600 +++ src/main.c 2008-10-16 22:20:08.930810243 -0600 @@ -264,7 +264,7 @@ */ static void SetBannerAndCapability( void ); static int ParseBannerAndCapability( char *, unsigned int, - char *, unsigned int ); + char *, unsigned int, unsigned int ); static void ServerInit( void ); static void Daemonize( const char* ); static void Usage( void ); @@ -943,10 +943,14 @@ static int ParseBannerAndCapability( char *DestBuf, unsigned int DestBufSize, char *SourceBuf, - unsigned int SourceBufSize ) + unsigned int SourceBufSize, + unsigned int capability ) { char *fn = "ParseBannerAndCapability"; char *CP; + char *bracket; + unsigned int CPlen; + unsigned int inCap = 0; if ( SourceBufSize >= DestBufSize ) { @@ -994,57 +998,108 @@ if ( !CP ) break; - if ( !strncasecmp( CP, "UNSELECT", strlen( "UNSELECT" ) ) ) - { - PC_Struct.support_unselect = UNSELECT_SUPPORTED; - } - /* - * If this token happens to be an auth mechanism, we want to - * discard it unless it's an auth mechanism we can support. + * Don't parse CAPABILITY extensions until we know we are either in + * a CAPABILITY untagged response or a CAPABILTY status response. */ - if ( ! strncasecmp( CP, "AUTH=", strlen( "AUTH=" ) ) && - ( strncasecmp( CP, "AUTH=LOGIN", strlen( "AUTH=LOGIN" ) ) ) ) - { - continue; - } + if ( ( capability && + !strncasecmp( CP, "CAPABILITY", strlen( "CAPABILITY" ) ) ) || + ( !capability && + !strncasecmp( CP, "[CAPABILITY", strlen( "[CAPABILITY" ) ) ) ) + { + /* + * Indicate that we are now in CAPABILITY data + */ + inCap = 1; + } + else if ( inCap ) + { + /* + * If we are in a CAPABILITY status response, we keep processing + * until we see a ']' character. + */ + if ( capability == 0 ) + { + CPlen = strlen( CP ); + bracket = memchr( CP, ']', CPlen ); + + if ( bracket ) + { + inCap = 0; + + /* + * Add the XIMAPPROXY capability string at the end of the + * CAPABILITY listing. + */ + strcat( DestBuf, " " ); + CP[CPlen - 1] = '\0'; + strcat( DestBuf, CP ); + strcat( DestBuf, " XIMAPPROXY]" ); + + continue; + } + } + + if ( !strncasecmp( CP, "UNSELECT", strlen( "UNSELECT" ) ) ) + { + PC_Struct.support_unselect = UNSELECT_SUPPORTED; + } - /* - * If this token happens to be SASL-IR, we want to discard it - * since we don't support any auth mechs that can use it. - */ - if ( !strncasecmp( CP, "SASL-IR", strlen( "SASL-IR" ) ) ) - { - continue; - } + /* + * If this token happens to be an auth mechanism, we want to + * discard it unless it's an auth mechanism we can support. + */ + if ( ! strncasecmp( CP, "AUTH=", strlen( "AUTH=" ) ) && + ( strncasecmp( CP, "AUTH=LOGIN", strlen( "AUTH=LOGIN" ) ) ) ) + { + continue; + } - /* - * If this token happens to be STARTTLS, we want to discard it - * since we don't support it on the client-side. - */ - if ( ! strncasecmp( CP, "STARTTLS", strlen( "STARTTLS" ) ) ) - { - PC_Struct.support_starttls = STARTTLS_SUPPORTED; - continue; - } + /* + * If this token happens to be SASL-IR, we want to discard it + * since we don't support any auth mechs that can use it. + */ + if ( ! strncasecmp( CP, "SASL-IR", strlen( "SASL-IR" ) ) ) + { + continue; + } + + /* + * If this token happens to be STARTTLS, we want to discard it + * since we don't support it on the client-side. + */ + if ( ! strncasecmp( CP, "STARTTLS", strlen( "STARTTLS" ) ) ) + { + PC_Struct.support_starttls = STARTTLS_SUPPORTED; + continue; + } - /* - * If this token happens to be LOGINDISABLED, we want to discard it - * since we don't support it on the client-side. - */ - if ( ! strncasecmp( CP, "LOGINDISABLED", strlen( "LOGINDISABLED" ) ) ) - { - PC_Struct.login_disabled = LOGIN_DISABLED; - continue; + /* + * If this token happens to be LOGINDISABLED, we want to discard it + * since we don't support it on the client-side. + */ + if ( ! strncasecmp( CP, "LOGINDISABLED", strlen( "LOGINDISABLED" ) ) ) + { + PC_Struct.login_disabled = LOGIN_DISABLED; + continue; + } } strcat( DestBuf, " "); strcat( DestBuf, CP ); } - + /* + * Add a 'XIMAPPROXY' CAPABILITY response to indicate that the + * current connection is handled by imapproxy. + */ + if ( capability == 1 ) + { + strcat( DestBuf, " XIMAPPROXY" ); + } + strcat( DestBuf, "\r\n" ); - + return( strlen( DestBuf ) ); } @@ -1129,7 +1184,7 @@ BannerLen = ParseBannerAndCapability( Banner, sizeof Banner - 1, - itd.ReadBuf, BytesRead ); + itd.ReadBuf, BytesRead, 0 ); /* * See if the string we got back starts with "* OK" by comparing the @@ -1179,8 +1234,7 @@ } CapabilityLen = ParseBannerAndCapability( Capability, sizeof Capability - 1, - itd.ReadBuf, BytesRead ); - + itd.ReadBuf, BytesRead, 1 ); /* Now read the tagged response and make sure it's OK */ BytesRead = IMAP_Line_Read( &itd ); --- src/request.c.old 2008-10-16 12:53:32.856511372 -0600 +++ src/request.c 2008-10-16 21:37:58.956491193 -0600 @@ -269,7 +269,7 @@ return( -1 ); } - snprintf( SendBuf, BufLen, "%s OK Completed\r\n", Tag ); + snprintf( SendBuf, BufLen, "%s OK Logfile cleared\r\n", Tag ); if ( IMAP_Write( itd->conn, SendBuf, strlen(SendBuf) ) == -1 ) { @@ -333,7 +333,7 @@ IMAPCount->TotalServerConnectionsReused = 0; IMAPCount->TotalClientLogins = 0; - snprintf( SendBuf, BufLen, "%s OK Completed\r\n", Tag ); + snprintf( SendBuf, BufLen, "%s OK Counters reset\r\n", Tag ); if ( IMAP_Write( itd->conn, SendBuf, strlen(SendBuf) ) == -1 ) { @@ -389,7 +389,7 @@ while ( HashEntry ) { - snprintf( SendBuf, BufLen, "* %d %s %s\r\n", HashEntry->server_conn->sd, + snprintf( SendBuf, BufLen, "* XPROXY_DUMPICC %d %s %s\r\n", HashEntry->server_conn->sd, HashEntry->username, ( ( HashEntry->logouttime ) ? "Cached" : "Active" ) ); if ( IMAP_Write( itd->conn, SendBuf, strlen(SendBuf) ) == -1 ) @@ -449,7 +449,7 @@ return( 0 ); } - snprintf( SendBuf, BufLen, "* %s\r\n", IMAP_PROXY_VERSION ); + snprintf( SendBuf, BufLen, "* XPROXY_VERSION %s\r\n", IMAP_PROXY_VERSION ); if ( IMAP_Write( itd->conn, SendBuf, strlen(SendBuf) ) == -1 ) { syslog(LOG_WARNING, "%s: IMAP_Write() failed: %s", fn, strerror(errno) ); @@ -554,7 +554,8 @@ strncpy( TraceUser, Username, sizeof TraceUser - 1 ); TraceUser[ sizeof TraceUser - 1 ] = '\0'; - snprintf( SendBuf, BufLen, "%s OK Tracing enabled\r\n", Tag ); + snprintf( SendBuf, BufLen, "%s OK Tracing enabled for user %s.\r\n", + Tag, TraceUser ); if ( IMAP_Write( itd->conn, SendBuf, strlen(SendBuf) ) == -1 ) { syslog(LOG_WARNING, "%s: IMAP_Write() failed: %s", fn, strerror(errno) ); @@ -854,6 +855,20 @@ } Server.conn = conn; + + /* + * If the connection has been reused, send a status response indicating + * this. + */ + if (Server.conn->reused == 1) + { + sprintf( SendBuf, "* OK [XPROXYREUSE] IMAP connection reused by imapproxy\r\n" ); + if ( IMAP_Write( Client->conn, SendBuf, strlen(SendBuf) ) == -1 ) + { + syslog(LOG_ERR, "%s: IMAP_Write() failed: %s", fn, strerror(errno) ); + return( -1 ); + } + } snprintf( SendBuf, BufLen, "%s OK User authenticated\r\n", Tag ); if ( IMAP_Write( Client->conn, SendBuf, strlen( SendBuf ) ) == -1 ) @@ -986,6 +1001,20 @@ Server.conn = conn; /* + * If the connection has been reused, send a status response indicating + * this. + */ + if (Server.conn->reused == 1) + { + sprintf( SendBuf, "* OK [XPROXYREUSE] IMAP connection reused by imapproxy\r\n" ); + if ( IMAP_Write( Client->conn, SendBuf, strlen(SendBuf) ) == -1 ) + { + syslog(LOG_ERR, "%s: IMAP_Write() failed: %s", fn, strerror(errno) ); + return( -1 ); + } + } + + /* * Send a success message back to the client * and go into raw proxy mode. */ @@ -1491,7 +1520,8 @@ * of the following IMAP commands (rfc 2060): NOOP, CAPABILITY, * AUTHENTICATE, LOGIN, and LOGOUT. Also, it handles the * commands that are internal to the proxy server such as - * P_TRACE, P_NEWLOG, P_DUMPICC, P_RESETCOUNTERS and P_VERSION. + * XPROXY_TRACE, XPROXY_NEWLOG, XPROXY_DUMPICC, + * XPROXY_RESETCOUNTERS and XPROXY_VERSION. * * None of these commands should ever have the need to send * a boatload of data, so we avoid some error checking and @@ -1767,7 +1797,7 @@ close( Client.conn->sd ); return; } - else if ( ! strcasecmp( (const char *)Command, "P_TRACE" ) ) + else if ( ! strcasecmp( (const char *)Command, "XPROXY_TRACE" ) ) { if ( Client.LiteralBytesRemaining ) { @@ -1780,7 +1810,7 @@ cmd_trace( &Client, S_Tag, Username ); continue; } - else if ( ! strcasecmp( (const char *)Command, "P_DUMPICC" ) ) + else if ( ! strcasecmp( (const char *)Command, "XPROXY_DUMPICC" ) ) { if ( Client.LiteralBytesRemaining ) { @@ -1792,7 +1822,7 @@ cmd_dumpicc( &Client, S_Tag ); continue; } - else if ( ! strcasecmp( (const char *)Command, "P_RESETCOUNTERS" ) ) + else if ( ! strcasecmp( (const char *)Command, "XPROXY_RESETCOUNTERS" ) ) { if ( Client.LiteralBytesRemaining ) { @@ -1804,7 +1834,7 @@ cmd_resetcounters( &Client, S_Tag ); continue; } - else if ( ! strcasecmp( (const char *)Command, "P_NEWLOG" ) ) + else if ( ! strcasecmp( (const char *)Command, "XPROXY_NEWLOG" ) ) { if ( Client.LiteralBytesRemaining ) { @@ -1816,7 +1846,7 @@ cmd_newlog( &Client, S_Tag ); continue; } - else if ( ! strcasecmp( (const char *)Command, "P_VERSION" ) ) + else if ( ! strcasecmp( (const char *)Command, "XPROXY_VERSION" ) ) { if ( Client.LiteralBytesRemaining ) { --- include/imapproxy.h.old 2008-10-16 12:52:49.452946282 -0600 +++ include/imapproxy.h 2008-10-16 12:19:47.796530898 -0600 @@ -229,6 +229,7 @@ SSL *tls; /* TLS connection context */ #endif struct IMAPSelectCache ISC; /* Cached SELECT data */ + unsigned int reused; /* Was the connection reused? */ }; From slusarz at curecanti.org Fri Oct 17 01:01:28 2008 From: slusarz at curecanti.org (Michael M. Slusarz) Date: Thu, 16 Oct 2008 23:01:28 -0600 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response In-Reply-To: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> References: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> Message-ID: <20081016230128.5g4x5a96um80wsg04cgg84@bigworm.curecanti.org> Quoting "Michael M. Slusarz" : > 3. Adds a status response [XPROXYREUSE] if the connection is > restored from the cache as opposed to newly created. > Rationale: Right now, imapproxy is useful to prevent the overhead of > creating/tearing down an IMAP connection on each page access. > However, in the present version, it does not leverage another > benefit of keeping the connection open - initialization of the > remote IMAP server. Although there is little/no initialization > needed with a straight RFC 3501 server, various extensions should > only need to be run once and will just add extra overhead if they > need to be sent to the server on every connection. Got a bit ahead of myself, and never explained what this is useful for. On a normal login, or a login through imapproxy where there is no cached connection, the following will occur: C: a001 LOGIN user password S: [... untagged response(s) from remote IMAP server ...] S: a001 OK User logged in To a client that is aware of the XIMAPPROXY extension, this tells the client that this is a brand new connection and all initialization should be done. Conversely, if a login occurs when imapproxy does have a cached connection, the following exchange occurs: C: a001 LOGIN user password S: [... untagged response(s) from remote IMAP server ...] S: * OK [XPROXYREUSE] IMAP connection reused by imapproxy S: a001 OK User logged in A client that is not aware of the [XPROXYREUSE] response code will ignore it (MANDATORY under RFC 3501 [7.1]). For an XIMAPPROXY aware client, this tells the client that the IMAP connection is in the same state as when the client disconnected from the proxy server last (with the exception that the connection will be in an unselected state). Thus, there is no reason to issue any additional initialization commands and the client can proceed to issuing the "meat" of his IMAP commands. Hope that explains it better. michael From dave64 at andrew.cmu.edu Fri Oct 17 07:12:50 2008 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Fri, 17 Oct 2008 07:12:50 -0400 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response In-Reply-To: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> References: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> Message-ID: <48F87332.9000302@andrew.cmu.edu> Hi Michael, Thanks much for the patch. It contains some excellent new features. I'll try to get it incorporated this weekend along with a few other small patches and get a release candidate out next week. Dave From japc at co.sapo.pt Fri Oct 17 13:23:40 2008 From: japc at co.sapo.pt (Jose Celestino) Date: Fri, 17 Oct 2008 18:23:40 +0100 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response In-Reply-To: <20081016230128.5g4x5a96um80wsg04cgg84@bigworm.curecanti.org> References: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> <20081016230128.5g4x5a96um80wsg04cgg84@bigworm.curecanti.org> Message-ID: <20081017172340.GC26919@co.sapo.pt> Words by Michael M. Slusarz [Thu, Oct 16, 2008 at 11:01:28PM -0600]: > Quoting "Michael M. Slusarz" : > > > 3. Adds a status response [XPROXYREUSE] if the connection is > > restored from the cache as opposed to newly created. > > Rationale: Right now, imapproxy is useful to prevent the overhead of > > creating/tearing down an IMAP connection on each page access. > > However, in the present version, it does not leverage another > > benefit of keeping the connection open - initialization of the > > remote IMAP server. Although there is little/no initialization > > needed with a straight RFC 3501 server, various extensions should > > only need to be run once and will just add extra overhead if they > > need to be sent to the server on every connection. > > Got a bit ahead of myself, and never explained what this is useful for. > > On a normal login, or a login through imapproxy where there is no > cached connection, the following will occur: > > C: a001 LOGIN user password > S: [... untagged response(s) from remote IMAP server ...] > S: a001 OK User logged in > > To a client that is aware of the XIMAPPROXY extension, this tells the > client that this is a brand new connection and all initialization > should be done. > > Conversely, if a login occurs when imapproxy does have a cached > connection, the following exchange occurs: > > C: a001 LOGIN user password > S: [... untagged response(s) from remote IMAP server ...] > S: * OK [XPROXYREUSE] IMAP connection reused by imapproxy > S: a001 OK User logged in > > A client that is not aware of the [XPROXYREUSE] response code will > ignore it (MANDATORY under RFC 3501 [7.1]). For an XIMAPPROXY aware > client, this tells the client that the IMAP connection is in the same > state as when the client disconnected from the proxy server last (with > the exception that the connection will be in an unselected state). > Thus, there is no reason to issue any additional initialization > commands and the client can proceed to issuing the "meat" of his IMAP > commands. > > Hope that explains it better. > Yes. That's definitely usefull. We have very uncommon problems, but problems nevertheless with imapproxy reusing connections left in IDLE: [japc at morgoth:~]$ telnet imap.sapo.pt 143 Trying 212.55.154.47... Connected to imap.sapo.pt. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] PTMail-IMAP v.1.1.1 ready. 2003/2008 - PTC/Saber e Lazer. imapproxy02 ... x idle + idling ^]q [japc at morgoth:~]$ telnet imap.sapo.pt 143 Trying 212.55.154.47... Connected to imap.sapo.pt. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] PTMail-IMAP v.1.1.1 ready. 2003/2008 - PTC/Saber e Lazer. imapproxy02 ... x examine inbox x BAD Expected DONE. Being so uncommon I left this on the lowest priority for resolution :) -- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc ---------------------------------------------------------------- "One man?s theology is another man?s belly laugh." -- Robert A. Heinlein From slusarz at curecanti.org Fri Oct 17 13:47:07 2008 From: slusarz at curecanti.org (Michael M. Slusarz) Date: Fri, 17 Oct 2008 11:47:07 -0600 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response In-Reply-To: <20081017172340.GC26919@co.sapo.pt> References: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> <20081016230128.5g4x5a96um80wsg04cgg84@bigworm.curecanti.org> <20081017172340.GC26919@co.sapo.pt> Message-ID: <20081017114707.2fbo6pr4r1gk00gksc88g0@bigworm.curecanti.org> Quoting Jose Celestino : > Yes. That's definitely usefull. We have very uncommon problems, but > problems nevertheless with imapproxy reusing connections left in IDLE: > > [japc at morgoth:~]$ telnet imap.sapo.pt 143 > Trying 212.55.154.47... > Connected to imap.sapo.pt. > Escape character is '^]'. > * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES > MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE > LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] > PTMail-IMAP v.1.1.1 ready. 2003/2008 - PTC/Saber e Lazer. imapproxy02 > ... > x idle > + idling > ^]q > > [japc at morgoth:~]$ telnet imap.sapo.pt 143 > Trying 212.55.154.47... > Connected to imap.sapo.pt. > Escape character is '^]'. > * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES > MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE > LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] > PTMail-IMAP v.1.1.1 ready. 2003/2008 - PTC/Saber e Lazer. imapproxy02 > ... > x examine inbox > x BAD Expected DONE. > > Being so uncommon I left this on the lowest priority for resolution :) I'm guessing that code needs to be added to catch/flag IDLE commands when in proxy mode, and if the connection is still flagged IDLE when the client->imapproxy connection terminates, imapproxy should send the 'DONE' before marking the connection as Cached. This won't happen when using the new Horde IMAP lib because we don't send IDLE commands - IDLE isn't very useful when your connection to the IMAP server lasts all of a second :) I'm guessing those IDLE commands are coming from other, non-web clients. michael From japc at co.sapo.pt Fri Oct 17 13:49:03 2008 From: japc at co.sapo.pt (Jose Celestino) Date: Fri, 17 Oct 2008 18:49:03 +0100 Subject: [imapproxy-info] PATCH: CAPABILITY/command cleanups, new XIMAPREUSE response In-Reply-To: <20081017114707.2fbo6pr4r1gk00gksc88g0@bigworm.curecanti.org> References: <20081016225127.v30wk855o3kw008cwosc8@bigworm.curecanti.org> <20081016230128.5g4x5a96um80wsg04cgg84@bigworm.curecanti.org> <20081017172340.GC26919@co.sapo.pt> <20081017114707.2fbo6pr4r1gk00gksc88g0@bigworm.curecanti.org> Message-ID: <20081017174903.GE26919@co.sapo.pt> Words by Michael M. Slusarz [Fri, Oct 17, 2008 at 11:47:07AM -0600]: > Quoting Jose Celestino : > >> Yes. That's definitely usefull. We have very uncommon problems, but >> problems nevertheless with imapproxy reusing connections left in IDLE: >> >> [japc at morgoth:~]$ telnet imap.sapo.pt 143 >> Trying 212.55.154.47... >> Connected to imap.sapo.pt. >> Escape character is '^]'. >> * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES MULTIAPPEND >> UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS >> LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] PTMail-IMAP v.1.1.1 ready. >> 2003/2008 - PTC/Saber e Lazer. imapproxy02 >> ... >> x idle >> + idling >> ^]q >> >> [japc at morgoth:~]$ telnet imap.sapo.pt 143 >> Trying 212.55.154.47... >> Connected to imap.sapo.pt. >> Escape character is '^]'. >> * OK [CAPABILITY IMAP4rev1 STARTTLS SORT THREAD=REFERENCES MULTIAPPEND >> UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS >> LIST-EXTENDED I18NLEVEL=1 QUOTA AUTH=LOGIN] PTMail-IMAP v.1.1.1 ready. >> 2003/2008 - PTC/Saber e Lazer. imapproxy02 >> ... >> x examine inbox >> x BAD Expected DONE. >> >> Being so uncommon I left this on the lowest priority for resolution :) > > I'm guessing that code needs to be added to catch/flag IDLE commands when > in proxy mode, and if the connection is still flagged IDLE when the > client->imapproxy connection terminates, imapproxy should send the 'DONE' > before marking the connection as Cached. This won't happen when using the > new Horde IMAP lib because we don't send IDLE commands - IDLE isn't very > useful when your connection to the IMAP server lasts all of a second :) > I'm guessing those IDLE commands are coming from other, non-web clients. > You are right. And yes, the solution is in imapproxy itself. You just reminded me of this issue :) -- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc ---------------------------------------------------------------- "One man?s theology is another man?s belly laugh." -- Robert A. Heinlein From dave64 at andrew.cmu.edu Mon Oct 20 14:48:57 2008 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 20 Oct 2008 14:48:57 -0400 Subject: [imapproxy-info] New release candidate, imapproxy-1.2.7rc1 available for download Message-ID: <48FCD299.3000401@andrew.cmu.edu> Hi, A new release candidate is now available for download from http://www.imapproxy.org/ Fixes and new features include: - Advertises XIMAPPROXY in the capability string so clients written to know they're talking to an imapproxy will know. - Adds XPROXYREUSE status response, again for clients written to know they're talking to an imapproxy. - Fixes (hopefully) the buffer overflow condition when doing AUTH LOGIN. - disables IDLE capability if the server advertises it, since imapproxy currently does not support IDLE. - The internal command set has been renamed to be RFC-compliant. Instead of the internal commands being P_, they're now prefixed with an X. The full changelog is available at http://www.imapproxy.org/downloads/ChangeLog Essentially, all the new features will make imapproxy integrate much better with IMP since I'm not aware of any other clients specifically written to take advantage of XIMAPPROXY and XPROXYREUSE. Thanks to Michael M. Slusarz and Noel B for their patch contributions. Thanks also to the ~50 people who provided various patches to fix the buffer overflow problem. Please post to the list if you discover any issues with this new release candidate. Thank you, Dave From adrieder at sbox.tugraz.at Tue Oct 21 03:26:57 2008 From: adrieder at sbox.tugraz.at (Didi Rieder) Date: Tue, 21 Oct 2008 09:26:57 +0200 Subject: [imapproxy-info] New release candidate, imapproxy-1.2.7rc1 available for download In-Reply-To: <48FCD299.3000401@andrew.cmu.edu> References: <48FCD299.3000401@andrew.cmu.edu> Message-ID: <20081021092657.17qwm4xcsbuo0o4cw@webmail.tugraz.at> Quoting Dave McMurtrie : > Please post to the list if you discover any issues with this new release > candidate. The new release candidate advertises LOGINDISABLED in the capability string, this should not be passed from the imap server through the proxy to the client though. imap server: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED] mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready imap proxy: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED XIMAPPROXY] mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready Didi -- ------------------------- Didi Rieder adrieder at sbox.tugraz.at PGPKey ID: 3431D0B0 ------------------------- From slusarz at curecanti.org Sat Oct 25 00:55:47 2008 From: slusarz at curecanti.org (Michael M. Slusarz) Date: Fri, 24 Oct 2008 22:55:47 -0600 Subject: [imapproxy-info] New release candidate, imapproxy-1.2.7rc1 available for download In-Reply-To: <20081021092657.17qwm4xcsbuo0o4cw@webmail.tugraz.at> References: <48FCD299.3000401@andrew.cmu.edu> <20081021092657.17qwm4xcsbuo0o4cw@webmail.tugraz.at> Message-ID: <20081024225547.4fro4ag9xbwg80c4sc80wg@bigworm.curecanti.org> Quoting Didi Rieder : > Quoting Dave McMurtrie : > >> Please post to the list if you discover any issues with this new release >> candidate. > > The new release candidate advertises LOGINDISABLED in the capability > string, this > should not be passed from the imap server through the proxy to the > client though. > > imap server: > * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED] > mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready > > imap proxy: > * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED XIMAPPROXY] > mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready If the last capability in a banner header is one of the non-allowed IMAPPROXY headers, the code in 1.2.7rc1 will allow it to slip through. We need to add the IMAPPROXY capability after we check to see if that last capability is valid. This patch should do the trick: --- main.c.old 2008-10-24 22:45:15.857985053 -0600 +++ main.c 2008-10-24 22:52:33.247986911 -0600 @@ -958,6 +958,7 @@ char *bracket; unsigned int CPlen; unsigned int inCap = 0; + unsigned int endbracket = 0; if ( SourceBufSize >= DestBufSize ) { @@ -1000,6 +1001,20 @@ for( ; ; ) { + if (endbracket) + { + /* + * Add the XIMAPPROXY capability string at the end of the + * CAPABILITY listing. + */ + endbracket = 0; + inCap = 0; + strcat( DestBuf, " " ); + strcat( DestBuf, " XIMAPPROXY]" ); + + continue; + } + CP = strtok( NULL, " " ); if ( !CP ) @@ -1032,18 +1047,8 @@ if ( bracket ) { - inCap = 0; - - /* - * Add the XIMAPPROXY capability string at the end of the - * CAPABILITY listing. - */ - strcat( DestBuf, " " ); + endbracket = 1; CP[CPlen - 1] = '\0'; - strcat( DestBuf, CP ); - strcat( DestBuf, " XIMAPPROXY]" ); - - continue; } } @@ -1077,7 +1082,7 @@ */ if ( ! strncasecmp( CP, "IDLE", strlen( "IDLE" ) ) ) { - continue; + continue; } /* michael From adrieder at sbox.tugraz.at Sat Oct 25 05:30:34 2008 From: adrieder at sbox.tugraz.at (Didi Rieder) Date: Sat, 25 Oct 2008 11:30:34 +0200 Subject: [imapproxy-info] New release candidate, imapproxy-1.2.7rc1 available for download In-Reply-To: <20081024225547.4fro4ag9xbwg80c4sc80wg@bigworm.curecanti.org> References: <48FCD299.3000401@andrew.cmu.edu> <20081021092657.17qwm4xcsbuo0o4cw@webmail.tugraz.at> <20081024225547.4fro4ag9xbwg80c4sc80wg@bigworm.curecanti.org> Message-ID: <20081025113034.17qwm4xcsglc0ogw8@webmail.tugraz.at> Quoting "Michael M. Slusarz" : > Quoting Didi Rieder : > >> Quoting Dave McMurtrie : >> >>> Please post to the list if you discover any issues with this new release >>> candidate. >> >> The new release candidate advertises LOGINDISABLED in the capability >> string, this >> should not be passed from the imap server through the proxy to the >> client though. >> >> imap server: >> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED] >> mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready >> >> imap proxy: >> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED XIMAPPROXY] >> mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready > > If the last capability in a banner header is one of the non-allowed > IMAPPROXY headers, the code in 1.2.7rc1 will allow it to slip through. > We need to add the IMAPPROXY capability after we check to see if > that last capability is valid. This patch should do the trick: This seems to fix the problem, thanks. Didi -- ------------------------- Didi Rieder adrieder at sbox.tugraz.at PGPKey ID: 3431D0B0 ------------------------- From dave64 at andrew.cmu.edu Sat Oct 25 11:41:41 2008 From: dave64 at andrew.cmu.edu (dave64) Date: Sat, 25 Oct 2008 11:41:41 -0400 Subject: [imapproxy-info] New release candidate, imapproxy-1.2.7rc1 available for download In-Reply-To: <20081025113034.17qwm4xcsglc0ogw8@webmail.tugraz.at> References: <48FCD299.3000401@andrew.cmu.edu> <20081021092657.17qwm4xcsbuo0o4cw@webmail.tugraz.at> <20081024225547.4fro4ag9xbwg80c4sc80wg@bigworm.curecanti.org> <20081025113034.17qwm4xcsglc0ogw8@webmail.tugraz.at> Message-ID: <49033E35.8030100@andrew.cmu.edu> Didi Rieder wrote: > Quoting "Michael M. Slusarz" : > >> Quoting Didi Rieder : >> >>> Quoting Dave McMurtrie : >>> >>>> Please post to the list if you discover any issues with this new release >>>> candidate. >>> The new release candidate advertises LOGINDISABLED in the capability >>> string, this >>> should not be passed from the imap server through the proxy to the >>> client though. >>> >>> imap server: >>> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED] >>> mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready >>> >>> imap proxy: >>> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED XIMAPPROXY] >>> mail.tugraz.at Cyrus IMAP v2.3.12p2-TUGraz server ready >> If the last capability in a banner header is one of the non-allowed >> IMAPPROXY headers, the code in 1.2.7rc1 will allow it to slip through. >> We need to add the IMAPPROXY capability after we check to see if >> that last capability is valid. This patch should do the trick: > > This seems to fix the problem, thanks. I'll apply this tomorrow and try to get rc2 out the door. Thanks, Dave From home158 at gmail.com Mon Oct 27 02:02:40 2008 From: home158 at gmail.com (chu) Date: Mon, 27 Oct 2008 14:02:40 +0800 Subject: [imapproxy-info] imaps-proxy (imap+ssl)-proxy Message-ID: <7c4771060810262302k6144601ew47e306d24bf672a7@mail.gmail.com> Hi all, How to setting /etc/imapproxy.conf allow my server to connect imaps? I try using README.ssl to configure CA, alse modify imapproxy below. listen_port 993 server_port 993 force_tls yes tls_ca_path /etc/CA/ tls_ca_file /etc/CA/cacert.pem tls_cert_file /etc/CA/newcert.pem tls_key_file /etc/CA/newreq.pem Than starting imap-proxy, it will wait a long time on my screen. #in.imapproxyd -f /etc/imapproxy/imapproxy.conf. imaps [nothing on my screen ..............] Who know I forgot something? or someone can share you experience? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/imapproxy-info/attachments/20081027/c460f5b6/attachment.html From dave64 at andrew.cmu.edu Mon Oct 27 06:46:30 2008 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 27 Oct 2008 06:46:30 -0400 Subject: [imapproxy-info] imaps-proxy (imap+ssl)-proxy In-Reply-To: <7c4771060810262302k6144601ew47e306d24bf672a7@mail.gmail.com> References: <7c4771060810262302k6144601ew47e306d24bf672a7@mail.gmail.com> Message-ID: <49059C06.2070008@andrew.cmu.edu> chu wrote: > Hi all, > How to setting /etc/imapproxy.conf allow my server to connect imaps? > I try using README.ssl to configure CA, alse modify imapproxy below. From README.ssl: "imapproxy does not support the deprecated notion of imaps using port 993. It only supports the use of the STARTTLS command to initiate SSL/TLS from within a regular imap connection." Thanks, Dave From home158 at gmail.com Thu Oct 30 22:47:57 2008 From: home158 at gmail.com (chu) Date: Fri, 31 Oct 2008 10:47:57 +0800 Subject: [imapproxy-info] build a gmail imaps proxy Message-ID: <7c4771060810301947g7ae2d36g4874e0b9656d9e8f@mail.gmail.com> Hi all, The gmail was support imaps now. Could I use the imapproxy package to build a proxy server to receive my messages from imap.gmail.com:993. I get a response from README.ssl, and it seem NOT support imaps(imap + SSL). If probably, someone know which GNU project can deal with it. Thanks for you answer. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/imapproxy-info/attachments/20081031/8ce49532/attachment.html From pont_imapproxy_info at soua.net Fri Oct 31 11:16:18 2008 From: pont_imapproxy_info at soua.net (Pontus Freyhult) Date: Fri, 31 Oct 2008 16:16:18 +0100 Subject: [imapproxy-info] build a gmail imaps proxy In-Reply-To: <7c4771060810301947g7ae2d36g4874e0b9656d9e8f@mail.gmail.com> References: <7c4771060810301947g7ae2d36g4874e0b9656d9e8f@mail.gmail.com> Message-ID: <490B2142.5010203@soua.net> Hi, > The gmail was support imaps now. Could I use the imapproxy package to > build a proxy server to receive my messages from imap.gmail.com:993 > . I get a response from README.ssl, and it > seem NOT support imaps(imap + SSL). If probably, someone know which GNU > project can deal with it. It's not at all clear to me what you want to do, but I'm pretty sure you should be able to do it by combining imapproxy and stunnel (). /Pontus -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.andrew.cmu.edu/pipermail/imapproxy-info/attachments/20081031/665e4d77/attachment.bin