Running a script with cyradm throwing ReadLine errors

[Binarus]

Dear ellie,

On 19.12.2018 01:38, ellie timoney wrote:

> I did a bit of reading, and apparently Term::ReadLine is a stub module that just loads "an implementation", which in your case wants to be Term::ReadLine::Gnu.  My guess is that, when you uninstall Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because it's missing an implementation, and consequently the fallback code I pointed out previously is used instead.  So, from this I'm concluding that the "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real ReadLine implementation.  Sounds like we've found our bug!

Some additional findings:

1) Cyrus::IMAP::DummyReadLine
-----------------------------

Looking again at that code

# ugh.  ugh.  suck.  aieee.
my $use_rl = 'Cyrus::IMAP::DummyReadline';
{
  if (eval { require Term::ReadLine; }) {
    $use_rl = 'Term::ReadLine';
  }
}

I believe that $use_rl *always* equals 'Term::ReadLine' after having
executed it. This is for the following reason: In newer Perl versions,
Term::ReadLine is a core module. Everybody has it installed. This means
that the require Term::ReadLine will always be successful.

I did a test to prove that. I uninstalled Term::ReadLine::Gnu again and
changed the code above to the following (note the last line):

# ugh.  ugh.  suck.  aieee.
my $use_rl = 'Cyrus::IMAP::DummyReadline';
{
  if (eval { require Term::ReadLine; }) {
    $use_rl = 'Term::ReadLine';
  }
}
print $use_rl."\n";

As expected, perl -MCyrus::IMAP::Shell -e 'run("./000")' now prints

Term::ReadLine

as first line on the terminal. This was still the case (as expected
again) after reinstalling Term::ReadLine::Gnu.

*That means:*

Cyrus::IMAP::DummyReadLine is not related to the problem or its solution
in any way. It never gets pulled in, at least with recent Perl
distributions which have Term::ReadLine included [as a core module].

2) *__DATA__ variable / file handle
-----------------------------------

After having read the Perl docs about that mysterious __DATA__ variable
(see below), grep'ing the whole Perl module trees for the string
__DATA__, and analyzing the results, I came to the conclusion that the
*__DATA__ variable *never* is assigned any value during normal program
execution, meaning that _run() always is called with undef as its last
parameter.

As a proof, I have replaced the following code

# trivial; wrapper for _run with correct setup
sub run {
  my $cyradm;
  _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
}

by

# trivial; wrapper for _run with correct setup
sub run {
  my $cyradm;
  print Dumper(${*Cyrus::IMAP::Shell::__DATA__})."\n";
  _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
}

and have added use Data::Dumper at the beginning of the file.

Now, when executing perl -MCyrus::IMAP::Shell -e 'run("./000")', it printed

$VAR1 = undef;

as the first line on the terminal. This was the case whether
Term::ReadLine::Gnu was installed or not.

To further back that finding, I reverted my changes and then changed the
code again as follows (note the last parameter to _run()):

# trivial; wrapper for _run with correct setup
sub run {
  my $cyradm;
  _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], undef);
}

This did not change the module's behavior compared to the original code.
While it now threw the errors described in my first post again (as
expected) when Term::ReadLine::Gnu was installed, it threw no errors
when it was not installed.

*That means:*

*__DATA__ (the third parameter to _run) is always undef, and this does
not lead to errors being thrown or the compilation / execution being
aborted as long as Term::ReadLine::Gnu is not installed, but makes
Term::ReadLine::Gnu (if it is installed) throw errors and abort the
compilation / execution of the script.

(Too) short explanation of the __DATA__ variable:

This is a predefined filehandle in Perl which could be used as follows.
Suppose you have a script:

package ...

[code here]

__DATA__
data value 1
data value 2
...

Then you can access the data values (i.e. all values which come behind
the __DATA__ statement) using the special filehandle [PACKAGE
NAME]::DATA (or __DATA__ as well?) from within the package code.

For details, see https://perldoc.perl.org/perldata.html#Special-Literals

Since there is no __DATA__ statement in any of Cyrus' Perl modules or in
modules they use, it is clear that the *__DATA__ filehandle is always
undef. To be honest, I can't understand why it is used. I originally
thought that it would be initialized by some other module (directly or
indirectly) which is used by Cyrus::IMAP::Shell, but my analysis showed
that it isn't (unless I have missed something, which might well be the
case).

3) No script execution at all
-----------------------------

I have to apologize that I didn't mention this in my first post; the
reason was that I did my first tests with an *empty* script.

However, now that my script is meaningful, I noticed that it did not get
executed at all even if Term::ReadLine::Gnu was not installed. In other
words, when I uninstalled Term::ReadLine::Gnu again and ran

perl -MCyrus::IMAP::Shell -e 'run("./000")'

the script "000" was *not* executed when the original version of
Cyrus::IMAP::Shell was in place. I didn't notice this from the beginning
on because I did my first tests with an empty script, and no errors were
thrown.

When I changed the module's code and assigned *__DATA__ a handle to the
file desired (as shown in my previous post), the script was executed.

To put it all together:
-----------------------

- Cyrus::IMAP::DummyReadLine is never used if the Perl distribution is
recent (i.e. already includes Term::ReadLine). This applies whether
Term::ReadLine::Gnu is installed or not.

- _run() is always called with undef as third parameter. If
Term::ReadLine::Gnu is installed, errors are thrown if we execute perl
-MCyrus::IMAP::Shell -e 'run("./000")' in a terminal. No errors are
thrown if Term::ReadLine::Gnu is not installed. The errors
Term::ReadLine::Gnu throws are clearly due to the fact that the third
argument to _run() is undef (the error thrown reads "Bad filehandle:
__DATA__ at ...").

- With the original version of Cyrus::IMAP::Shell, even when
Term::ReadLine::Gnu is not installed, the script <name> is not executed
when we issue perl -MCyrus::IMAP::Shell -e 'run("<name>")' in a terminal.

- If we assign *__DATA__ the filehandle to the desired script in the
run() function before calling _run(), the desired script does get
executed when we issue perl -MCyrus::IMAP::Shell -e 'run("./000")'. This
works whether Term::ReadLine::Gnu is installed or not.

Hence, the problem is clearly related to Cyrus::IMAP::Shell. It does
either just not execute the script given (if Term::ReadLine::Gnu is not
installed), or it makes Term::ReadLine::Gnu throw errors (if it is
installed) due to the uninitialized file handle.

It would be very nice if somebody could fix that bug or correct
Cyrus::IMAP::Shell's man page accordingly.

Fortunately, there is a workaround. We could make cyradm execute a
script directly with no problem:

cat 000 | cyradm

Shame on me that I didn't see this earlier - it would have solved my
problem and would have saved me a whole day or two. I guess
Cyrus::IMAP::Shell's man page had distracted me too much...

However, I'd still be strongly interested in a bug fix for the module.
In fact, I already have a clean solution in mind, but I'd like to test
it before posting it here. I'll report back in a few hours.

Thank you very much again!

Regards,

Binarus