Sporadic End Of Script Output Before Headers In Admin Panel

[Alex]

Hey guys,

When I'm navigating the admin panel sometimes a page will come back with "Internal Server Error". If I just refresh the page, it usually works fine, which is quite odd and annoying.

I've been tailing logs for a few days and I have narrowed it down in the Apache error logs. Every time I get one of these errors, the following appears in the Apache log for the relevant page:

[Mon Sep 16 16:01:28.748494 2013] [:warn] [pid 459211] (104)Connection reset by peer: [client 24.129.40.105:52451] mod_fcgid: error reading data from FastCGI server, referer: https://fullambit.net/portal/admin/packages/index/inactive/
[Mon Sep 16 16:01:28.748584 2013] [core:error] [pid 459211] [client 24.129.40.105:52451] End of script output before headers: index.php, referer: https://fullambit.net/portal/admin/packages/index/inactive/
[Mon Sep 16 16:01:30.001015 2013] [:error] [pid 459210] mod_fcgid: process /usr/local/cpanel/cgi-sys/php5(465614) exit(communication error), get unexpected signal 11

Can any of the developers provide insight as to what is going on here?

We have a lot of other PHP/MySQL applications running on this server without any issues. Additionally, this never seems to happen on the client-end of Blesta, only sporadically on the admin-end.

Thanks,

Alex

[Cody]

Enable error reporting in php.ini:

 

error_reporting = -1
log_errors = On
error_log = /path/to/php/log/error.log

[Alex]

Enable error reporting in php.ini:

 

error_reporting = -1

log_errors = On

error_log = /path/to/php/log/error.log

PHP error logging is enabled. It has nothing related to Blesta in it, only an entirely unrelated script in a different directory has any logging in there. That script's errors are simply warning of a PEAR script using deprecated code.

Nagged the error happening in the admin panel again while tailing the Apache log, on a different page this time:

 

[Mon Sep 16 18:16:41.815076 2013] [:warn] [pid 482817] (104)Connection reset by peer: [client 24.129.40.105:59536] mod_fcgid: error reading data from FastCGI server, referer: https://fullambit.net/portal/admin/settings/company/
[Mon Sep 16 18:16:41.815170 2013] [core:error] [pid 482817] [client 24.129.40.105:59536] End of script output before headers: index.php, referer: https://fullambit.net/portal/admin/settings/company/
[Mon Sep 16 18:16:46.012278 2013] [:error] [pid 482633] mod_fcgid: process /usr/local/cpanel/cgi-sys/php5(483392) exit(communication error), get unexpected signal 11

Really no rhyme or reason to when it is happening. Which page it happens on seems variable, and it only happens to maybe 1 out of 10 requests. If I refresh, the page works.

[Cody]

PHP error logging is enabled. It has nothing related to Blesta in it, only an entirely unrelated script in different directory.

 

Hm... that doesn't seem to make sense.

The apahce log error

End of script output before headers: index.php, referer: https://fullambit.net/portal/admin/packages/index/inactive/

indicates an error with the application (PHP) processing that URL. I would think PHP would log such an error. Unless I'm mistaken, they even log 500 internal errors thrown by PHP itself, as opposed to the script.

 

Not sure what else to try, except maybe searching the PHP changelog or bug tracker for similar issues. It's still entirely possible it's an issue with Blesta, but without an PHP error being captured it's impossible to know for sure.

[Alex]

Hm... that doesn't seem to make sense.

The apahce log error indicates an error with the application (PHP) processing that URL. I would think PHP would log such an error. Unless I'm mistaken, they even log 500 internal errors thrown by PHP itself, as opposed to the script.

 

Not sure what else to try, except maybe searching the PHP changelog or bug tracker for similar issues. It's still entirely possible it's an issue with Blesta, but without an PHP error being captured it's impossible to know for sure.

I agree. My first thought is that it was a server issue when I saw "Internal Server Errors" and no PHP error logs which are relevant. As things unfolded, it became clear that the issue is inconsistent, limited to the Blesta admin-end, and quite difficult to diagnose. I'll let you know if I figure anything more out.

[Alex]

Hm... that doesn't seem to make sense.

The apahce log error indicates an error with the application (PHP) processing that URL. I would think PHP would log such an error. Unless I'm mistaken, they even log 500 internal errors thrown by PHP itself, as opposed to the script.

 

Not sure what else to try, except maybe searching the PHP changelog or bug tracker for similar issues. It's still entirely possible it's an issue with Blesta, but without an PHP error being captured it's impossible to know for sure.

 

Just found this: http://serverfault.com/questions/251418/premature-end-of-script-headers-occurring-seemingly-randomly

 

It seems similar.

 

Also found these:

 

http://galleryproject.org/node/8955

http://forum.joomla.org/viewtopic.php?p=2511696

https://bugzilla.redhat.com/show_bug.cgi?id=868893

http://htmlfixit.com/cgi-tutes/tutorial_Common_Web_dev_error_messages_and_what_they_mean.php#premature

 

Maybe these will help to generate ideas.

[Alex]

We just made a few tweaks to our fcgi settings in Apache and with any luck it will help. I'll wait a couple of days to make a call, as it can be quite intermittent.

[Alex]

I think we solved the mystery. CloudLinux has per-cPanel-account restrictions on resources. After increasing the default allowed concurrent requests and CPU usage per account as a last ditch effort, these issues seem to have disappeared. However, this means that the pages which we had this problem with require quite a bit of resources.

Editing packages, or modifying modules, seems to trigger this most often. Pages which need to talk to an external module seem to be the affected pages. No other pages in any of our applications or our client's sites had this issue.

[Paul]

What are the default settings, and what did you change them to?

 

Interesting that it tossed an internal server error, and not something more descriptive.

[Alex]

What are the default settings, and what did you change them to?

 

Interesting that it tossed an internal server error, and not something more descriptive.

I jumped the gun, the issue quickly returned even with infinite resources. Running coredumps now to try to get to the bottom of it...

#0 0x00007f72338a96a7 in ?? () from /usr/local/IonCube/ioncube_loader_lin_5.4.so

No symbol table info available.

#1 0x00007f72338a9eac in _poisson_process () from /usr/local/IonCube/ioncube_loader_lin_5.4.so

No symbol table info available.

Looks like it is actually an ionCube related issue?

[Paul]

I wouldn't be surprised if it's an ioncube issue.. make sure you have the latest loaders (And if you do have the latest and it's still an issue, maybe see if you can get a slightly older loader).

[Alex]

I wouldn't be surprised if it's an ioncube issue.. make sure you have the latest loaders (And if you do have the latest and it's still an issue, maybe see if you can get a slightly older loader).

Well, hard to find version info on the loaders. But, I updated to the latest and the problem persists (same coredump) so I assume we already had the latest loader. Looking at http://www.ioncube.com/loaders.php I don't see a "older versions" section or anything... might be overlooking it though, I'm sick as a dog today.

[Paul]

I don't see anything either. I wonder if this is a bug with Ioncube for Cloudlinux. You might want to submit your findings to Ioncube and see what they say. You might also be able to ask them for an older version of the loader, I can't find anything either.

[Michael]

I don't see anything either. I wonder if this is a bug with Ioncube for Cloudlinux. You might want to submit your findings to Ioncube and see what they say. You might also be able to ask them for an older version of the loader, I can't find anything either.

Not a bug that I'm aware of mate in Cloudlinux as I tested Blesta on the magicserver.pw which is my customer server.

[Alex]

I don't see anything either. I wonder if this is a bug with Ioncube for Cloudlinux. You might want to submit your findings to Ioncube and see what they say. You might also be able to ask them for an older version of the loader, I can't find anything either.

Yeah, I'll start bugging cPanel and ionCube about this and see what I can come up with.

 

Not a bug that I'm aware of mate in Cloudlinux as I tested Blesta on the magicserver.pw which is my customer server.

I too doubt that it is CloudLinux related, since for the software it operates identically to CentOS. CloudLinux should be transparent at this layer of the stack. I think it's more likely an issue with the combination of fcgi and ionCube.