Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data

[Alex Masidlover]

-------------8<---------- Start Bug Report ------------8<----------
1. Problem Description:


I'm running Perl 5.20.2 on a Gentoo virtual machine. We've had some
code in place that uses a mod_perl handler to send complex hashref
based structures between two servers using nfreeze / thaw; the code
needed to be high performance therefore we used nfreeze/thaw rather
than XML or similar. This has been working correctly on a variety of
architectures since July 2014.

The mod_perl server does:

print nfreeze( $message_hash )

The client connects to the server use LWP::UserAgent and does:

my $message = thaw( $response->decoded_content() );

The message from client to server is similarly encoded with nfreeze /
thaw in a cgi parameter in multi-part-form-data.

This has all worked perfectly up until I upgraded to Apache 2.4 /
mod_perl 2.0.10 - I've even downgraded just those two packages to
mod_perl 2.0.8 / Apache 2.2.31 and it starts working again. The message
from the client gets to the server correctly and the server logs and
processes the message correctly, but it then fails with an 'Out of
memory!' error from the client when the response is received (due to
this bug in Storable.pm) - however, regardless of that bug, something
is corrupting the data in transit...

I've picked apart what's going on and found there are an additional two
bytes being inserted somehow. I've recorded the data in a /tmp/ file
before the print statement in the mod_perl handler and I've used perl
-d to inspect what's received:

  DB<3> say "0x$_" for unpack "(H2)*", $response->content()
0x05
0x0a
0x03
0x00
0x00
0x00
0x01
0x08
0xc2
0x81
0x00
0x00
0x00
0x02
0x6f
0x6b

However, the server sent:

0x05
0x0a
0x03
0x00
0x00
0x00
0x01
0x08
0x00
0x00
0x00
0x02
0x6f
0x6b

So somehow 2 bytes have appeared in the middle of the message 0x08 and
0xc2.

I've also tried adding Content-Type => 'x-application/binary' and
Content-Length headers to see if that fixes the issue.

Thanks!

2. Used Components and their Configuration:

*** mod_perl version 2.000010

*** using /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-
multi/Apache2/BuildConfig.pm

*** Makefile.PL options:
  MP_APR_CONFIG  => /usr/bin/apr-1-config
  MP_APR_LIB     => aprext
  MP_APXS        => /usr/sbin/apxs2
  MP_COMPAT_1X   => 1
  MP_DEBUG       => 0
  MP_GENERATE_XS => 1
  MP_LIBNAME     => mod_perl
  MP_TRACE       => 0
  MP_USE_DSO     => 1


*** The httpd binary was not found


*** (apr|apu)-config linking info

 -laprutil-1 -lldap -llber -ldb-4.8 -lgdbm  -lexpat
-L/var/tmp/portage/dev-libs/apr-util-1.5.4/temp
 -lapr-1 -luuid -lrt -lcrypt  -lpthread -ldl 



*** /usr/bin/perl -V
Summary of my perl5 (revision 5 version 20 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=2.6.32-openvz-feoktistov.1, archname=x86_64-
linux-thread-multi
    uname='linux alex.zednax.com 2.6.32-openvz-feoktistov.1 #2 smp wed
apr 6 14:38:37 bst 2011 x86_64 intel(r) xeon(r) cpu l5630 @ 2.13ghz
genuineintel gnulinux '
    config_args='-des -Duseshrplib -Darchname=x86_64-linux-thread
-Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O -pipe -Dldflags=-Wl,-O1
-Wl,--as-needed -Dprefix=/usr -Dinstallprefix=/usr
-Dsiteprefix=/usr/local -Dvendorprefix=/usr -Dscriptdir=/usr/bin
-Dprivlib=/usr/lib64/perl5/5.20.2
-Darchlib=/usr/lib64/perl5/5.20.2/x86_64-linux-thread-multi
-Dsitelib=/usr/local/lib64/perl5/5.20.2
-Dsitearch=/usr/local/lib64/perl5/5.20.2/x86_64-linux-thread-multi
-Dvendorlib=/usr/lib64/perl5/vendor_perl/5.20.2
-Dvendorarch=/usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-
multi -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
-Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3
-Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.20.2
-Dlocincpth=/usr/include  -Dglibpth=/lib64 /usr/lib64  -Duselargefiles
-Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost -Dperladmin=root
@localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm -Di_ndbm -Di_gdbm
-Di_db -Dusethreads -DDEBUGGING=none -Dinc_version_list=5.20.0/x86_64-
linux-thread-multi 5.20.0 5.20.1/x86_64-linux-thread-multi 5.20.1  -
Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dnoextensions=ODBM_File'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='x86_64-pc-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE
-fwrapv -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
    optimize='-O -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing
-pipe'
    ccversion='', gccversion='4.7.3', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='x86_64-pc-linux-gnu-gcc', ldflags ='-Wl,-O1 -Wl,--as-needed'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/lib/gcc/x86_64-pc-
linux-gnu/4.7.3/include-fixed /usr/lib /lib/../lib64 /usr/lib/../lib64
/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
-lgdbm_compat
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=libc-2.21.so, so=so, useshrplib=true,
libperl=libperl.so.5.20.2
    gnulibc_version='2.21'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O -pipe -Wl,-O1 -Wl,--as-
needed'


Characteristics of this binary (from libperl): 
  Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
                        USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
                        USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
                        USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO
                        USE_PERL_ATOF USE_REENTRANT_API
  Locally applied patches:
gentoo/hints_hpux - Fix hpux hints
gentoo/aix_soname - aix gcc detection and shared library soname
support
gentoo/EUMM-RUNPATH - https://bugs.gentoo.org/105054
cpan/ExtUtils-MakeMaker: drop $PORTAGE_TMPDIR from LD_RUN_PATH
gentoo/config_over - Remove -rpath and append LDFLAGS to
lddlflags
gentoo/opensolaris_headers - [PATCH] Add headers for
opensolaris
gentoo/patchlevel - List packaged patches for perl-5.20.2(#1)
in patchlevel.h
gentoo/cpanplus_definstalldirs - Configure CPANPLUS to use the
site directories by default.
gentoo/cleanup-paths - [PATCH] Cleanup PATH and shrpenv
gentoo/enc2xs - Tweak enc2xs to follow symlinks and ignore
missing @INC directories.
gentoo/enc2xs_checksums -
gentoo/darwin-cc-ld - https://bugs.gentoo.org/297751 [PATCH]
darwin: Use $CC to link
gentoo/cpan_definstalldirs - Provide a sensible INSTALLDIRS
default for modules installed from CPAN.
gentoo/interix - [PATCH] Fix interix hints
gentoo/create_libperl_soname - https://bugs.gentoo.org/286840
[PATCH] Set libperl soname
gentoo/mod_paths - Add /etc/perl to @INC
gentoo/EUMM_delete_packlist -
gentoo/drop_fstack_protector - https://bugs.gentoo.org/348557
[PATCH] Don't force -fstack-protector on everyone
gentoo/usr_local - [PATCH] Remove /usr/local paths
gentoo/D-SHA-CFLAGS - https://bugs.gentoo.org/506818 [PATCH] Do
not set custom CFLAGS in cpan/Digest-SHA
gentoo/io_socket_ip_tests -
debian/cpan-missing-site-dirs - Fix CPAN::FirstTime defaults
with nonexisting site dirs if a parent is writable
debian/regcomp-mips-optim - Downgrade the optimization of
regcomp.c on mips and mipsel due to a gcc-4.9 bug
debian/perldoc-less-R - Tell the 'less' pager to allow terminal
escape sequences
debian/makemaker-pasthru - Pass LD settings through to
subdirectories
fixes/net_smtp_docs - [rt.cpan.org #36038] Document the
Net::SMTP 'Port' option
fixes/memoize_storable_nstore - [rt.cpan.org #77790]
Memoize::Storable: respect 'nstore' option not respected
fixes/document_makemaker_ccflags - [rt.cpan.org #68613]
Document that CCFLAGS should include $Config{ccflags}
  Built under linux
  Compiled at Mar 27 2016 10:44:32
  %ENV:
    PERL_LWP_USE_HTTP_10="1"
  @INC:
    /etc/perl
    /usr/local/lib64/perl5/5.20.2/x86_64-linux-thread-multi
    /usr/local/lib64/perl5/5.20.2
    /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread-multi
    /usr/lib64/perl5/vendor_perl/5.20.2
    /usr/local/lib64/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/lib64/perl5/5.20.2/x86_64-linux-thread-multi
    /usr/lib64/perl5/5.20.2
    .

*** Packages of interest status:

Apache2            : -
Apache2::Request   : 2.13
CGI                : 3.65, 4.21, 4.22
ExtUtils::MakeMaker: 6.98
LWP                : 6.15
mod_perl           : -
mod_perl2          : 2.000010


3. This is the core dump trace: (if you get a core dump):

  [CORE TRACE COMES HERE]

This report was generated by /usr/bin/mp2bug on Sun Apr  3 13:00:59
2016 GMT.

-------------8<---------- End Bug Report --------------8<----------

Note: Complete the rest of the details and post this bug report to
modperl <at> perl.apache.org. To subscribe to the list send an empty
email to modperl-***@perl.apache.org.

[Vincent Veyron]

On Sun, 03 Apr 2016 14:11:23 +0100

Post by Alex Masidlover
This has all worked perfectly up until I upgraded to Apache 2.4 /
mod_perl 2.0.10 -

After upgrading to Apache 2.4 and mod_perl 2.0.9, I had to make those two changes to my application :

In a PerlOutputFilterHandler, change '$content .= $buffer' to '$content .= decode_utf8($buffer)'

And in response handlers, change '$args{$_} = $req->param($_)' to '$args{$_} = decode_utf8($req->param($_))'

Not sure it applies to your case, but something changed in Apache 2.4 concerning UTF-8 data.

If I understood correctly, anything that goes through APR::Table is considered UTF-8, however the SvUTF8 flag is not set, so you get double encoding when processing your data.