Cron tasks stuck

[mobbdeep]

Hello,

Before anyone links me to all the threads regarding this issue back to 2013, I've already dug through a bunch and none of them resolved my issue. However, since I switch web hosting companies for my main site, I am getting the System Status error of "There are one or more cron tasks that have been executing for more than 60 minutes. View Automated Tasks." When I view the tasks, it just has the loading icon and that's it. 

Tasks it's referencing: Cancel Scheduled Services and Suspend Services

Troubleshooting steps I've done:

• Ran cron job manually
• Disabled the tasks > Ran cron manually > Enabled tasks again
• Removed cron job from cPanel and added it back
• Spread out the times the cron tasks are ran at every 20 minutes
• Upped the max_execution_time in PHP to 300
• Allowed them to run on their own over a full 24 hour period
• When running the cron jobs manually, there is no errors returned

Again, I'm not sure if I am missing something or doing something wrong but this only happened after I switched hosts.

Thanks,
Tyler

[Paul]

When you switched hosting companies, how did you move Blesta? Did you follow the guide? If you don't copy all your files + database, and instead do a fresh install and overwrite your database you will permanently break Blesta.

Now that that is out of the way..

It sounds like the issue is with 2 tasks, Cancel Scheduled Services and Suspend Services? Is that right?

Did you check your logs? By default, the logs are at ../logs_blesta/ but the path is defined under Settings > System > General.

When a cron task is hung, it can take 6 hours for it to be cleared out automatically and re-attempted. If you ran the cron manually I would highly suggest doing the following:

1. Disable the cron job in cPanel
2. Wait at least 6 hours
3. Enable error reporting in /config/blesta.php (Change ErrorReporting from "0" to "-1")
4. Run the cron manually under Settings > System > Automation. Observe the output, and see if there are any errors.
5. Disable error reporting in /config/blesta.php

The same errors you might get by following this method should also exist in your logs, so you could check there instead.

Since the issue occurred after moving Blesta to another provider, it's possible there is some dependency that is not met. A simple check for that would be to upload a fresh copy of Blesta in a sub-directory and access in your browser so that Blesta will run a system requirement check. Don't install, just delete the files when done checking the results of the system requirement check.

[mobbdeep]

9 hours ago, Paul said:

When you switched hosting companies, how did you move Blesta? Did you follow the guide? If you don't copy all your files + database, and instead do a fresh install and overwrite your database you will permanently break Blesta.

Now that that is out of the way..

It sounds like the issue is with 2 tasks, Cancel Scheduled Services and Suspend Services? Is that right?

Did you check your logs? By default, the logs are at ../logs_blesta/ but the path is defined under Settings > System > General.

 When a cron task is hung, it can take 6 hours for it to be cleared out automatically and re-attempted. If you ran the cron manually I would highly suggest doing the following:

1. Disable the cron job in cPanel
2. Wait at least 6 hours
3. Enable error reporting in /config/blesta.php (Change ErrorReporting from "0" to "-1")
4. Run the cron manually under Settings > System > Automation. Observe the output, and see if there are any errors.
5. Disable error reporting in /config/blesta.php

The same errors you might get by following this method should also exist in your logs, so you could check there instead.

Since the issue occurred after moving Blesta to another provider, it's possible there is some dependency that is not met. A simple check for that would be to upload a fresh copy of Blesta in a sub-directory and access in your browser so that Blesta will run a system requirement check. Don't install, just delete the files when done checking the results of the system requirement check.

When I moved hosting companies, I used JetBackup to take a full website backup and then the cPanel Backup tool to take a backup of my database. I then uploaded the full website backup and imported the database.

When checking the logs at ../logs_blesta/ that directory appears to be empty due to not defining the new path when switching web hosting companies. I defined the new path and made it writable. I will give it 24 hours to see if it logs anything. I also noticed the uploads directory should be defined as there's a text field for it. Is this accurate?

I will go ahead and perform the 1-2 steps now and give it 6-7 hours before running steps 3-5.

Checking the dependencies, they all checked out good to go.

 

[mobbdeep]

16 hours ago, Paul said:

When you switched hosting companies, how did you move Blesta? Did you follow the guide? If you don't copy all your files + database, and instead do a fresh install and overwrite your database you will permanently break Blesta.

Now that that is out of the way..

It sounds like the issue is with 2 tasks, Cancel Scheduled Services and Suspend Services? Is that right?

Did you check your logs? By default, the logs are at ../logs_blesta/ but the path is defined under Settings > System > General.

When a cron task is hung, it can take 6 hours for it to be cleared out automatically and re-attempted. If you ran the cron manually I would highly suggest doing the following:

1. Disable the cron job in cPanel
2. Wait at least 6 hours
3. Enable error reporting in /config/blesta.php (Change ErrorReporting from "0" to "-1")
4. Run the cron manually under Settings > System > Automation. Observe the output, and see if there are any errors.
5. Disable error reporting in /config/blesta.php

The same errors you might get by following this method should also exist in your logs, so you could check there instead.

Since the issue occurred after moving Blesta to another provider, it's possible there is some dependency that is not met. A simple check for that would be to upload a fresh copy of Blesta in a sub-directory and access in your browser so that Blesta will run a system requirement check. Don't install, just delete the files when done checking the results of the system requirement check.

Just updating..

I enabled the tasks I disabled, enabled error reporting, then ran the cron manually.

Attempting to run all tasks for SpdyHost.
Attempting to process service changes.
The process service changes task has completed.
Attempting to process renewing services.
The process renewing services task has completed.
Attempting plugin cron for order accept_paid_orders.
Finished plugin cron for order accept_paid_orders.
Attempting plugin cron for support_manager poll_tickets.
Finished plugin cron for support_manager poll_tickets.
Attempting plugin cron for mass_mailer export.
Finished plugin cron for mass_mailer export.
Attempting plugin cron for mass_mailer mass_mail.
Finished plugin cron for mass_mailer mass_mail.
All tasks have been completed.
Attempting to run all system tasks.
All system tasks have been completed.

Reviewing ../logs_blesta/ there are general-error, general-info, general-notice, and general-warning.

General-Error:

[2019-01-08 08:37:13] general.ERROR: Uncaught Exception Error: "Wrong parameters for BadMethodCallException([string $message [, long $code [, Throwable $previous = NULL]]])" at /home/domain/public_html/billing/core/Automation/TaskFactory.php line 59 {"exception":"[object] (Error(code: 0): Wrong parameters for BadMethodCallException([string $message [, long $code [, Throwable $previous = NULL]]]) at /home/domain/public_html/billing/core/Automation/TaskFactory.php:59)"} 
[2019-01-08 11:22:16] general.ERROR: Uncaught Exception Error: "Wrong parameters for BadMethodCallException([string $message [, long $code [, Throwable $previous = NULL]]])" at /home/domain/public_html/billing/core/Automation/TaskFactory.php line 59 {"exception":"[object] (Error(code: 0): Wrong parameters for BadMethodCallException([string $message [, long $code [, Throwable $previous = NULL]]]) at /home/domain/public_html/billing/core/Automation/TaskFactory.php:59)"} 

General-Notice:

[2019-01-08 08:04:30] general.NOTICE: E_USER_DEPRECATED: The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead. {"code":16384,"message":"The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead.","file":"/home/domain/public_html/billing/vendors/swiftmailer/swiftmailer/lib/classes/Swift/Transport/MailTransport.php","line":45} 
[2019-01-08 08:05:05] general.NOTICE: E_USER_DEPRECATED: The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead. {"code":16384,"message":"The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead.","file":"/home/domain/public_html/billing/vendors/swiftmailer/swiftmailer/lib/classes/Swift/Transport/MailTransport.php","line":45} 
[2019-01-08 08:10:44] general.NOTICE: E_USER_DEPRECATED: The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead. {"code":16384,"message":"The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead.","file":"/home/domain/public_html/billing/vendors/swiftmailer/swiftmailer/lib/classes/Swift/Transport/MailTransport.php","line":45} 
[2019-01-08 08:14:06] general.NOTICE: E_USER_DEPRECATED: The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead. {"code":16384,"message":"The Swift_Transport_MailTransport class is deprecated since version 5.4.5 and will be removed in 6.0. Use the Sendmail or SMTP transport instead.","file":"/home/domain/public_html/billing/vendors/swiftmailer/swiftmailer/lib/classes/Swift/Transport/MailTransport.php","line":45} 
[2019-01-08 08:37:13] general.NOTICE: E_NOTICE: PDOStatement::execute(): send of 107 bytes failed with errno=32 Broken pipe {"code":8,"message":"PDOStatement::execute(): send of 107 bytes failed with errno=32 Broken pipe","file":"/home/domain/public_html/billing/vendors/minphp/db/src/PdoConnection.php","line":196} 
[2019-01-08 11:22:16] general.NOTICE: E_NOTICE: PDOStatement::execute(): send of 107 bytes failed with errno=32 Broken pipe {"code":8,"message":"PDOStatement::execute(): send of 107 bytes failed with errno=32 Broken pipe","file":"/home/domain/public_html/billing/vendors/minphp/db/src/PdoConnection.php","line":196} 

General-Warning:

[2019-01-08 08:08:45] general.WARNING: E_WARNING: sprintf(): Too few arguments {"code":2,"message":"sprintf(): Too few arguments","file":"/home/domain/public_html/billing/vendors/minphp/language/src/Language.php","line":125} 
[2019-01-08 08:08:52] general.WARNING: E_WARNING: sprintf(): Too few arguments {"code":2,"message":"sprintf(): Too few arguments","file":"/home/domain/public_html/billing/vendors/minphp/language/src/Language.php","line":125} 

I did notice earlier there was an issue with a clients account where the domain was actually terminated from my cPanel server back in June but the service was still appearing as suspended in his Blesta account area. I went ahead and set the domain as cancelled without using the cPanel module so I'm not sure if maybe that was causing a problem. But above is all of the information I have gathered earlier.

Canceled Scheduled Services, Suspend Services, Unsuspend Services are the tasks I am looking into.

[Nelsa]

When you run cron manually not all tasks will be executed,there are many tasks sheduled to run at certain time or time period,in your case 2 of 3 tasks run once per day...

[Paul]

The tasks that did not appear to complete in your original post did not run in your manual run, they must have already run for the day. Looking at your errors, this one stands out:

Quote

PDOStatement::execute(): send of 107 bytes failed with errno=32 Broken pipe

It's possible that MySQL is timing out. You could try increasing your wait_timeout value, typically in my.cnf to something like:

wait_timeout=3600

To check the current value, and to see if the change took affect you can run the following query:

SHOW VARIABLES LIKE '%wait_timeout%';

Are any tasks listed as not having completed under Settings > Company > Automation any longer? If so, which ones?

[mobbdeep]

15 hours ago, Paul said:

The tasks that did not appear to complete in your original post did not run in your manual run, they must have already run for the day. Looking at your errors, this one stands out:

It's possible that MySQL is timing out. You could try increasing your wait_timeout value, typically in my.cnf to something like:

wait_timeout=3600

To check the current value, and to see if the change took affect you can run the following query:

SHOW VARIABLES LIKE '%wait_timeout%';

Are any tasks listed as not having completed under Settings > Company > Automation any longer? If so, which ones?

Unfortunately, my main website is on a cPanel account temporarily until I get things situated therefore I cannot change the wait_timeout in MySQL.

I did do additional logging and attached a .txt file containing logs for the cron job when it gets ran by cPanel.

In the logs, you will see "server.mydomain.com" which is referencing where my actual VPS is. Again, I am hosting just my website with Blesta on a separate cPanel service than where my clients are hosted. My clients are on a VPS I manage.

logs.txt

[Nelsa]

Well Error: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away  is common issue with many shared hosting providers,mostly problem is low limits for one or more  variables in mysql configuration file my.cnf  , in most cases if you  incrase wait_timeout and/or max_allowed_packet it will resolve issue and in case when problem is timeout even incresing php settings can solve issue but again only if it is timeout issue   but it is very rare..since you can't really get much with php tweaking...ask provider to incrase these values if server is not under your control

[mobbdeep]

I switched away from a shared hosting provide onto my own Linux VPS for the sake of my own freedom and ability to 100% manage my entire web server. Anyway, I increased the max_allowed_packet and wait_timeout, and I am still getting the same result with the Suspend Services task.

Below are my php.ini settings:

[Paul]

If the issue occurs when suspended services, the next question is: What services are trying to be suspended, and what modules are they using? Since suspended services invoke the module, and the module can make an API call to remotely suspend a service, the module has a lot of control. If you are running any 3rd party or custom modules, then they could be suspect.

[Blesta Addons]

wish module installed ?

[mobbdeep]

3 hours ago, Paul said:

If the issue occurs when suspended services, the next question is: What services are trying to be suspended, and what modules are they using? Since suspended services invoke the module, and the module can make an API call to remotely suspend a service, the module has a lot of control. If you are running any 3rd party or custom modules, then they could be suspect.

 

2 hours ago, Blesta Addons said:

wish module installed ?

I am running no 3rd party modules aside from the cPanel module.

3rd party plugins I am running:

• Auto Cancel Unpaid Orders
• Blesta Addons Widget
• Client Notice
• Delete Invoices
• Resend Welcome Email

How can I see what services are trying to be suspended? If I can see that information, I may be able to figure out the issue. I believe it may be an account that no longer exists on my cPanel server that was manually terminated and Blesta is now trying to suspend that non-existing account via the module which is causing an error.

[Blesta Addons]

13 hours ago, mobbdeep said:

 

I am running no 3rd party modules aside from the cPanel module.

3rd party plugins I am running:

• Auto Cancel Unpaid Orders 
• Blesta Addons Widget
• Client Notice 
• Delete Invoices
• Resend Welcome Email

How can I see what services are trying to be suspended? If I can see that information, I may be able to figure out the issue. I believe it may be an account that no longer exists on my cPanel server that was manually terminated and Blesta is now trying to suspend that non-existing account via the module which is causing an error.

how you can get more info ?

1 - disable the suspend cron task .
2 - after about 24 hours, change the suspend cron task time to the next 5 min from now time then disable the auto cron entirly from cPanel or plesk panel
3 - enable error reporting from bleta config
4 - run the cron from the CLI command the you will discouver the error output .

[MDHMatt]

ive had this when moving hosts and forgetting to suspend blesta beforehand. You need to go into you database and look for the cron jobs entries. you will find that one or two are missing an end date. add one that is 5 mins later than the start date then manually run the cron again this will fix it.

[Paul]

12 minutes ago, MDHMatt said:

ive had this when moving hosts and forgetting to suspend blesta beforehand. You need to go into you database and look for the cron jobs entries. you will find that one or two are missing an end date. add one that is 5 mins later than the start date then manually run the cron again this will fix it.

Tasks missing an end date should re-try again after at least 6 hours has passed since the failed task started, assuming that it crashed. What's fun is when people move Blesta but keep it on the old server and don't turn the cron off. Then we get reports about magic invoices and charges that don't appear in Blesta.

[MDHMatt]

15 hours ago, Paul said:

Tasks missing an end date should re-try again after at least 6 hours has passed since the failed task started, assuming that it crashed. What's fun is when people move Blesta but keep it on the old server and don't turn the cron off. Then we get reports about magic invoices and charges that don't appear in Blesta.

Oh really mine don't seem to... Well my method has fixed it before.. might not be the safest way if you have lots of invoices etc but if it's a small amount of clients it works fine.

I host mine in a vps so when i move hosts i make sure to delete the other.