Boilerplate Extension - publishing process times out

[rickyT]

Hello,

This is related to an existing ticket (submitted 4 months ago) which hasn’t been acknowledged.

[Feature Request] Boilerplate Extension - speed up publishing process

Currently, the process for publishing many Boilerplate posts to many child sites at the same time is very painful. Users need to go into each individual Boilerplate post and repeat the publish process. For 100+ posts, this takes hours. Proposed feature request for Boilerplate posts management: ability to select multiple boilerplate posts in Boilerplate Post Extension > Boilerplate Posts tab (similar UI as Posts > Manage Posts tab) ability to select multiple child sites / groups in Boiler…

The Boilerplate Extension publishing process is extremely slow when there are many posts and many child sites. The process is starting to time out as the list of child sites grows, which is creating a huge problem with duplicate posts because the Dashboard falls out of sync.

For example, we have 80 boilerplate posts and 100 child sites. Each time a single boilerplate post is updated, that creates a queue of 100 post updates which is likely to time out.

If we add a new child site, we need to update ALL boilerplate posts manually to ensure the new child site is included in the publishing queue. That’s over 8000 post updates in total, just to add one new child site.

So, in addition to the suggested improvements in the support ticket linked above, could the publishing process be made more efficient?

[bogdan (Bogdan Rapaić)]

Hi Ricky,

Can you please post the community system report from your MainWP Dashboard for review? The report is located in your Dashboard under Status → Server, on the top right of the page.

Be sure to use the button like the one below; this button hides all your private information:

Pressing the button auto-copies the report to your clipboard then just paste it in a reply here.

[rickyT]

### Server Info                    Required                                               Detected                                     Status   ###

MainWP Dashboard                   
MainWP Dashboard Version           4.1.4.1                                                4.1.4.1                                      Pass        
MainWP Upload Directory            Writable                                               Writable                                     Pass        
MainWP Extensions                  
Boilerplate Extension              4.0.2.1                                                API License Active                           Pass        
MainWP Branding Extension          4.0.2.1                                                API License Active                           Pass        
MainWP Bulk Settings Manager Extension4.0.2.1                                                API License Active                           Pass        
MainWP Custom Dashboard Extension  4.0.1                                                  API License Active                           Pass        
MainWP Favorites Extension         4.0.5                                                  API License Active                           Pass        
MainWP File Uploader Extension     4.0.1                                                  API License Active                           Pass        
MainWP Team Control                4.0.1.1                                                API License Active                           Pass        
WordPress                          
WordPress Version                  >=3.6                                                  5.6                                          Pass        
WordPress Memory Limit             >=64M                                                  40M                                          Warning     
MultiSite Disabled                 =true                                                  true                                         Pass        
FileSystem Method                  = direct                                               direct                                       Pass        
PHP                                
PHP Version                        >=5.6                                                  7.4.14                                       Pass        
PHP Safe Mode Disabled             =true                                                  true                                         Pass        
PHP Max Execution Time             >=30 seconds                                           3600                                         Pass        
PHP Max Input Time                 >=30 seconds                                           3600                                         Pass        
PHP Memory Limit                   >=128M                                                 512M                                         Pass        
PCRE Backtracking Limit            >=10000                                                1000000                                      Pass        
PHP Upload Max Filesize            >=2M                                                   50M                                          Pass        
PHP Post Max Size                  >=2M                                                   100M                                         Pass        
SSL Extension Enabled              =true                                                  true                                         Pass        
SSL Warnings                       = empty                                                                                             Pass        
cURL Extension Enabled             =true                                                  true                                         Pass        
cURL Timeout                       >=300 seconds                                          5                                            Warning     
cURL Version                       >=7.18.1                                               7.58.0                                       Pass        
cURL SSL Version                   >=OpenSSL/0.9.8l                                       OpenSSL/1.1.1                                Pass        
PHP Allow URL fopen                YES                                                    
PHP Exif Support                   YES ( V7.4.)                                           
PHP IPTC Support                   YES                                                    
PHP XML Support                    YES                                                    
PHP Disabled Functions             apache_child_terminate, apache_get_modules, apache_get_version, apache_getenv, apache_note, apache_setenv, disk_free_space, disk_total_space, diskfreespace, dl, exec, fastcgi_finish_request, link, opcache_compile_file, opcache_get_configuration, opcache_invalidate, opcache_is_script_cached, opcache_reset, passthru, pclose, pcntl_exec, popen, posix_getpid, posix_getppid, posix_getpwuid, posix_kill, posix_mkfifo, posix_setegid, posix_seteuid, posix_setgid, posix_setpgid, posix_setsid, posix_setuid, posix_uname, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, shell_exec, show_source, symlink, system,
PHP Loaded Extensions              Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, apache2handler, bcmath, bz2, calendar, ctype, curl, date, dba, dom, exif, fileinfo, filter, ftp, gd, gettext, gmp, gnupg, hash, iconv, igbinary, imagick, imap, intl, json, ldap, libxml, mbstring, memcached, msgpack, mysqli, mysqlnd, newrelic, openssl, pcre, pdo_mysql, posix, readline, redis, session, shmop, soap, sockets, sodium, ssh2, standard, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlreader, xmlrpc, xmlwriter, xsl, zip, zlib
MySQL                              
MySQL Version                      >=5.0                                                  5.7.32-35-log                                Pass        
MySQL Mode                         ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
MySQL Client Encoding              utf8                                                   
Server Info                        
Server Software                    Apache                                                 
Operating System                   Linux                                                  
Architecture                       64		 bit                                               
Server Protocol                    HTTP/1.0                                               
HTTPS                              ON                                                     
Server self connect                Not expected HTTP response body: 401 Authorization Required

401 Authorization Required
nginx
User Agent                         Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 Safari/537.36
Gateway Interface                  CGI/1.1                                                
Memory Usage                       32.56 MB                                               
Request Time                       1614889881                                             
Accept Content                     text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Charset Content             N/A                                                    
MainWP Settings                    
Number Of Child Sites              67                                                     
Use WP-Cron                        Yes                                                    
Optimize for Shared Hosting or Big NetworksNo                                                     
Automatic Daily Update             Disabled                                               
Abandoned Plugins/Themes Tolerance 365                                                    
Maximum number of posts to return  50                                                     
Maximum number of pages to return                                                         
Primary Backup System              Default MainWP Backups                                 
Maximum simultaneous requests      5                                                      
Minimum delay between requests     200                                                    
Maximum simultaneous requests per ip4                                                      
Minimum delay between requests to the same ip200                                                    
Maximum simultaneous sync requests 5                                                      
Minimum simultaneous install/update requests3                                                      
Active Plugins                     
MainWP Dashboard                   4.1.4.1                                                Active                                       
Redirection                        5.0.1                                                  Active                                       

[bogdan (Bogdan Rapaić)]

Hi Ricky,

I see that the WP Memory Limit is set low on your dashboard (40M) and cURL Timeout very low (5s). Can you try to increase these settings to at least 256M for WP Memory Limit and 300s for cURL timeout and see if that helps?

[rickyT]

Let me look into that…

[rickyT]

The WP Memory Limit will be increased, but WPEngine doesn’t allow the CURL timeout to be changed.

How does the Dashboard handle CURL timeouts - does it retry? Or give up?

[bogdan (Bogdan Rapaić)]

Hi Ricky,

thanks for getting back to me.
I am sorry to hear that WPEngine won’t allow you to increase cURL Timeout.

Regarding the cURL requests, it depends on actions. Some process retry, some are handled in smaller batches, some fail and can’t be re-tried.

Anyways, can you check and see if WP Memory Limit update made any difference?

[rickyT]

Yes - we’ll keep an eye on this.

In terms of the root cause of this issue, what can be done to make the Boilerplate extension more efficient? As the number of posts and child sites increases, this is only going to get worse.

[bogdan (Bogdan Rapaić)]

Hi Ricky,

thanks for getting back to me… Let me know how it goes with further testing. Also, my strong recommendation is to host MainWP Dashboard on server that meets at least minimum requirements (cURL Timeout >= 60).

In any case, I will check with our dev team and see if they can check this and see if there is anything that can be optimized on our side.

[rickyT]

Hi Bogdan - has there been any update on this?

[bogdan (Bogdan Rapaić)]

Hi rickyT,

I am still waiting for updates from the dev team, as soon as they get a chance to work on this, I will let you know how it goes.

[kwcjr (Keith Crain)]

@rickyT Just out of curiosity can you add this line to your wp-config.php file and see if the Server Reports a change on your cURL Timeout ?

ini_set(“default_socket_timeout”, 300);

[rickyT]

Hi @kwcjr - thanks for looking into this.

I tried forcing the default_socket_timeout and MainWP Dashboard successfully reported 300 seconds.

However, WPEngine ignores it.

So I ran some additional tests and discovered that WPEngine has a max execution time on curl_exec requests of 90 seconds.

This means that MainWP’s analysis of the server specs may not be accurate on some hosts.

Therefore, this issue is more likely related to the huge number of curl requests that need to be processed.

[rickyT]

Has there been any update on this?

Thanks

[bogdan (Bogdan Rapaić)]

Hi rickyT,

thanks for getting back to us.

We don’t have any update on this matter yet.

As I suggested earlier, moving your MainWP Dashboard to more suitable hosting may be the best option for you. cURL Timeout set 5s just can’t be enough to handle longer processes and bigger number of child sites.

I would recommend reading this topic on the hosting matter:

Why are you still using shared hosting in 2023?

Updated for 2023 Why am I posting this? This post first started brewing when users posted screenshots such as the two below showing hundreds of updates being done after WordPress 5.7 was released. These posts were followed by others wondering how they could process so many updates at once since their updates timeout after a certain amount. [400TotalUpdates] [694totalupdates] While I can’t guarantee it, I have a VERY strong suspicion their Dashboards are not on shared hosting. This does …

[rickyT]

Please read above… cURL timeout is 90 seconds.

The server is a VPS at WPEngine which has plenty of resource available.

@kwcjr what are your thoughts on this?

[rickyT]

@dennis perhaps you’re able to prioritize this issue?

[bogdan (Bogdan Rapaić)]

Hi Ricky,

As soon as we can, we will check this, but at the moment, this is not priority.

If you are in a hurry, you can process posting in smaller batches or move your MainWP Dashboard to server that meets recommended requirements.