503 Service Unavailable after provisioning new package on Virtualmin

[adamjedgar]

Hi guys,

 

Thank you for a great product, and your great support.

 

I have experienced a problem this morning from the administrator dashboard in Blesta.

 

I have manually provisioned a new webhosting package for an existing client who has other services with me.

 

The provisioning is complete using the Virtualmin module (which i have been using for the other 3 services with this client.

When i complete the shopping cart procedure, i get the following error:

 

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

 

 

Strangely enough, when i log into the Virtualmin server, the new service (virtualmin>virtual server) appears to be setup correctly?

 

However, the service is not showing up as having been provisioned in Blesta?

 

1. Any ideas on what may be causing this?

2. How do i now reflect in Blesta, that this service is actually installed and active on my virtualmin server? (so a bill/invoice is created, and blesta controls the billing cycle to ensure payments are made when due in the future)

 

UPDATE

I have tried deleting the virtualmin virtual server and reprovisioning this package, however the same error pops up again (even though once again the service is active in virtualmin)

[Tyson]

The error sounds like a web server issue. You may want to debug the web server to see what is causing the issue (e.g. low resources or bad configuration?)

Blesta didn't complete provisioning the service because the error occurred in the middle of the process. If you check your module logs for this action, you may see a request was sent to VirtualMin to create the service, but no response was received back from VirtualMin to tell Blesta the service was provisioned. In that case, you should be able to Manage the service as an admin and set the domain, username, and password for the service in order to match the service credentials in Blesta with the credentials in VirtualMin. Be sure to uncheck 'Use module' when you do this.

[adamjedgar]

The module logs are empty....there are no errors on the virtualmin module?

 

From blesta side of things, it's as if nothing happened...but in fact virtualmin has received the command and has processed it correctly because the correct package has been setup.

 

Does blesta have any protection in its code against the module accidently "double provisioning" ? (This can cause 503 errors)

 

Also, even though i am apache and not lightspeed, could the cause be similar to this....https://www.litespeedtech.com/support/forum/threads/solved-503-service-unavailable-in-whmcs.7151/

 

I might change my php script execution mode and see if that makes any difference? (Perhaps to fpm, or if it is fpm, back to fcgi?)

[Paul]

Is Blesta installed on the same Virtualmin server?

When do you get the error exactly, as a client placing an order through the order form or as an admin provisioning it manually? Normally when a client places an order, the cron will provision the service. If the error happens at the same time that the API call is made to create the service with Virtualmin and that's when you get the error, and Blesta is on the same server, Virtualmin may be restarting/reloading Apache in the middle of the request which would explain everything.

[adamjedgar]

Yes it is on same server and yes it is when I am manually provisioning service through blesta admin.

 

Having said that, I have done it this way for this client account 3 times previously in the last few weeks...just prior to swapping from trial to paid blesta version.

This has only happened since that change in license. 

[adamjedgar]

So here are some details about my system

debian9 webmin/virtualmin

apache php 7.0.33

 

script execution mode php70-fpm

 

ah, i have managed to find something in my apache logs....(i am not sure if this is at the same time as i attempted to provision this service manually through blesta admin, however this error is repeated 3 times  on 2 separate occasions, and it appears to be consistent with me having tried this more than once on that day)

Mar 31 13:14:21.946639 2019] [proxy_fcgi:error] [pid 24919] (70007)The timeout specified has expired: [clientxxx.xxx.145.157:53996] AH01075: Error dispatching request to : (polling), referer: https://billing.<mdomain>.com.au/blesta/admin/settings/company/modules/addrow/

 

also, i am not sure if this is related, but the microsoft bingbot is causing the following error (Which is filling up my logs)

[Mon Apr 01 11:35:34.061210 2019] [autoindex:error] [pid 17333] [client 157.55.39.159:4276] AH01276: Cannot serve directory /home/user/domains/billing.<mydomain>.com.au/public_html/: No matching DirectoryIndex (index.html,index.htm,index.php,index.php4,index.php5) found, and server-generated directory index forbidden by Options directive
[Mon Apr 01 12:55:17.875152 2019] [proxy_fcgi:error] [pid 17114] [client 40.77.167.30:12708] AH01071: Got error 'Primary script unknown\n'

 

[adamjedgar]

my current timeout settings in /etc/apache2/apache2.conf 

Timeout 100
KeepAlive on
MaxKeepAliveRequests 100
KeepAliveTimeout 5

 

are the above settings ok for blesta?

 

i have changed the following in /etc/php/7.0/fpm/pool.d/xxxx.conf

php_admin_value[max_execution_time] = 180
php_admin_value[max_input_time] = 180

[Paul]

1 hour ago, adamjedgar said:

Having said that, I have done it this way for this client account 3 times previously in the last few weeks...just prior to swapping from trial to paid blesta version.

This has only happened since that change in license. 

A license change has no impact on functionality, so I 'm not sure how that could be related.

47 minutes ago, adamjedgar said:

my current timeout settings in /etc/apache2/apache2.conf 

Timeout 100
KeepAlive on
MaxKeepAliveRequests 100
KeepAliveTimeout 5

Does it take more than 100 seconds for Virtualmin to create the account? I'd expect only 5-10 seconds or so.

I think you should test with a Virtualmin server that is external to your Blesta install. I suspect that Virtualmin is dropping the connection after Blesta makes the request because it is restarting/reloading Apache. Alternatively you can add the service as a PENDING service, and let your CRON job run and try to provision the service. The cron may work because the connection to you as an admin would not be dropped if this is in fact the cause.

So, give that a try - either try to provision on another Virtualmin server OR let the cron provision the service by setting it to PENDING when you add it, and letting the cron run.

[adamjedgar]

49 minutes ago, Paul said:

A license change has no impact on functionality, so I 'm not sure how that could be related.

Does it take more than 100 seconds for Virtualmin to create the account? I'd expect only 5-10 seconds or so.

I think you should test with a Virtualmin server that is external to your Blesta install. I suspect that Virtualmin is dropping the connection after Blesta makes the request because it is restarting/reloading Apache. Alternatively you can add the service as a PENDING service, and let your CRON job run and try to provision the service. The cron may work because the connection to you as an admin would not be dropped if this is in fact the cause.

So, give that a try - either try to provision on another Virtualmin server OR let the cron provision the service by setting it to PENDING when you add it, and letting the cron run.

Yes i agree about the timeout settings...i would have thought that my default of 100 seconds should have been more than enough as i dont recall it taking more than a dozen seconds or so to provision the service.

hmmm this  is interesting. you are right...apache is restarting all the time.

 

I just attempted to manually provision another account and this time i went into webmin>system>system logs immediately after i got the browser error...

See below (this  PID is repeating itself quite often...every few minutes)

[Thu Apr 11 08:37:26.559931 2019] [mpm_prefork:notice] [pid 6145] AH00171: Graceful restart requested, doing restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using server1.<mydomain>.com.au. Set the 'ServerName' directive globally to suppress this message
[Thu Apr 11 08:37:28.641733 2019] [mpm_prefork:notice] [pid 6145] AH00163: Apache/2.4.25 (Debian) mod_fcgid/2.3.9 OpenSSL/1.0.2r configured -- resuming normal operations
 

 

[adamjedgar]

Also, 

I have manually run a cron job and, due to non payment...the cron is successfully disabling websites if users have not paid.

If i then go into blesta and manually unsuspend services for this clients virtualmin websites, they immediately come back up online again.

 

Having said that, i created a new account, then set it to pending, went into blesta admin and selected approve, manually ran the cron, and the service has not appeared as a new virtual server in virtualmin?

 

I am a little confused as to what is going on? Suspending and unsuspending services appears to be working normally on existing accounts, however,

1. manually creating new services on the virtualmin module are showing up in virtualmin but are not being reflected in blesta and

2. the cron for new accounts is not working at all (at least its been 10 minutes now and nothing is showing up

 

the cron setup in settings>system<automation is as follows

*/5 * * * * /usr/bin/php -q /home/user/domains/billing.domain.com.au/public_html/blesta/index.php cron > /dev/null 2>&1

 

is it possible to delete the virtualmin module and start again (and not lose the services i have already provisioned on the virtualmin server?)

 

How do i  add existing virtual servers on the virtualmin system to blesta so it can automate their billing/services management?

 

would reinstalling the blesta virtualmin module fix this problem? (it never used to do this before)

[Paul]

18 hours ago, adamjedgar said:

Having said that, i created a new account, then set it to pending, went into blesta admin and selected approve, manually ran the cron, and the service has not appeared as a new virtual server in virtualmin?

That's not what I mean. I mean your cron job on your server should run to provision it. You are doing it through a web browser, which is making an HTTP request to the same server that is restarting Apache and dropping the connection. That's the issue here. The problem is that Blesta and Virtualmin are on the same server, and Virtualmin drops the connection when you do it in a browser. Let the cron job run on the server automatically, and it should work. You won't have this problem if the Virtualmin server is external to Blesta.

[adamjedgar]

3 hours ago, Paul said:

That's not what I mean. I mean your cron job on your server should run to provision it. You are doing it through a web browser, which is making an HTTP request to the same server that is restarting Apache and dropping the connection. That's the issue here. The problem is that Blesta and Virtualmin are on the same server, and Virtualmin drops the connection when you do it in a browser. Let the cron job run on the server automatically, and it should work. You won't have this problem if the Virtualmin server is external to Blesta.

ok...i think i understand that.

May i ask the question...in terms of setting the service to active when manually provisioning services on Virtualmin, why is this happening now?

 

It has been working without issue since i first installed Blesta almost 1 month ago...its only now (the last week) all of a sudden it has stopped working (its always been on the same server and the three websites shown were all setup manually using active option)

[adamjedgar]

Also, I added a from to the virtualmin server...however I notice one is already there (now I have 2 cron for blesta)

Why do I need to manually add a cron job if one is already there?