Pro Reports Struggling

Hi guys,

The server we’re hosting the platform on is high on spec and with basically no specific limitations (60+ CPU 250G RAM). All PHP values are also much higher than the recommended ones. Here’s the server info report from the dashboard:


### WordPress Check                Required                                               Detected                                     Status   ###

FileSystem Method                  = direct                                               direct                                       Pass        
MultiSite Disabled                 =true                                                  true                                         Pass        
WordPress Memory Limit             >=64M                                                  1024M                                        Pass        
WordPress Version                  >=6.2                                                  6.5.4                                        Pass        

### PHP                            Required                                               Detected                                     Status   ###

cURL Extension Enabled             =true                                                  true                                         Pass        
cURL Timeout                       >=300 seconds                                          1000                                         Pass        
cURL Version                       >=7.29.0                                               7.81.0                                       Pass        
OpenSSL Version                    >=OpenSSL/1.1.0                                        OpenSSL/3.0.2                                Pass        
OpenSSL Working Status             Yes                                                    Yes                                          Pass        
PCRE Backtracking Limit            >=10000                                                1000000                                      Pass        
PHP Allow URL fopen                N/A                                                    YES                                                      
PHP Disabled Functions             N/A                                                    No functions disabled.                                   
PHP Exif Support                   N/A                                                    YES                                                      
PHP IPTC Support                   N/A                                                    YES                                                      
PHP Loaded Extensions              N/A                                                    Core, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, bcmath, brotli, calendar, cgi-fcgi, ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext, gmp, hash, iconv, imagick, imap, intl, json, ldap, libxml, mailparse, mbstring, mysqli, mysqlnd, openssl, pcntl, pcre, pdo_mysql, pdo_sqlite, posix, random, readline, redis, session, soap, sockets, sodium, sqlite3, standard, tidy, tokenizer, xml, xmlreader, xmlwriter, xsl, zip, zlib            
PHP Max Execution Time             >=30 seconds                                           3000                                         Pass        
PHP Max Input Time                 >=30 seconds                                           -1                                           Pass        
PHP Memory Limit                   >=256M                                                 4096M                                        Pass        
PHP Post Max Size                  >=2M                                                   1024M                                        Pass        
PHP Safe Mode Disabled             =true                                                  true                                         Pass        
PHP Upload Max Filesize            >=2M                                                   1024M                                        Pass        
PHP Version                        >=7.4                                                  8.2.16                                       Pass        
PHP XML Support                    N/A                                                    YES                                                      
SSL Extension Enabled              =true                                                  true                                         Pass        
SSL Warnings                       = empty                                                                                             Pass        

### MySQL                          Required                                               Detected                                     Status   ###

MySQL Client Encoding              N/A                                                    utf8                                                     
MySQL Mode                         N/A                                                    IGNORE_SPACE                                             
MySQL Version                      >=5.0                                                  8.0.35                                       Pass        

### Server Configuration           Detected Value                                      ###

Accept Content                     text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Charset Content             N/A                                                    
Architecture                       64		 bit                                               
Gateway Interface                  CGI/1.1                                                
HTTPS                              ON                                                     
Memory Usage                       9.97 MB                                                
Operating System                   Linux                                                  
Request Time                       1718095865                                             
Server Protocol                    HTTP/2.0                                               
Server self connect                Not expected HTTP response body:                       
Server Software                    Apache                                                 
User Agent                         Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36

### MainWP Dashboard Settings      Detected Value                                      ###

Abandoned plugins/themes tolerance 365                                                    
Basic uptime monitoring enabled    Yes                                                    
MainWP Dashboard Version           Latest: 5.0.3.2 | Detected: 5.0.3.2 Pass               
MainWP legacy backups enabled      No                                                     
Maximum number of pages to return                                                         
Maximum number of posts to return                                                         
Maximum simultaneous install and update requests6                                                      
Maximum simultaneous requests      4                                                      
Maximum simultaneous requests per ip2                                                      
Maximum simultaneous sync requests 13                                                     
Minimum delay between requests     200                                                    
Minimum delay between requests to the same ip1000                                                   
Number of connected sites          OVER 1k (edited)                                                  
Optimize data loading              Yes                                                    
Plugin advanced automatic updates enabledYes                                                    
Primary backup system              MainWP Legacy Backups                                  
REST API enabled                   Yes                                                    
Site health monitoring enabled     No                                                     
Theme advanced automatic updates enabledYes                                                    
Use WP Cron                        Yes                                                    
WP Core advanced automatic updates enabledYes                                                    

### Extensions                     Version                                                License                                      Status   ###

Advanced Uptime Monitor Extension  5.3.1                                                  Active                                       Pass        
Dashboard Lock                     5.0.1                                                  Active                                       Pass        
MainWP Custom Dashboard Extension  5.0.1                                                  Active                                       Pass        
MainWP Database Updater Extension  5.0.1                                                  Active                                       Pass        
MainWP Domain Monitor Extension    5.0.2                                                  Active                                       Pass        
MainWP Pro Reports Extension       5.0.2                                                  Active                                       Pass        
MainWP Vulnerability Checker Extension5.0.1                                                  Active                                       Pass        
MainWP White Label Extension       5.0                                                    Active                                       Pass        

### Plugin                         Version                                                Status                                    ###

Activity Log for MainWP            2.1.3                                                  Inactive                                     
Admin CSS MU                       2.10                                                   Active                                       
MainWP Dashboard                   5.0.3.2                                                Active                                       
Redis Object Cache                 2.5.2                                                  Active                                       
Wordfence Security                 7.11.6                                                 Active                                       
WP Activity Log                    5.0.0                                                  Inactive                                     
WP Mail SMTP                       4.0.1                                                  Active                                       

We’re managing quite a lot of sites and have been having performance issues with the “Pro Reports” extension and generating individual reports ever since.
The initial report sites list page loads fine and with no delays (page=Extensions-Mainwp-Pro-Reports-Extension), however everything else like “Create report” or “Edit” site, takes forever and in most cases doesn’t even complete (it just spins). Stracing the process in most cases loops through this:

sendto(13, "\246\10\0\0\3SELECT wp.*,wp_sync.*,wp_optionview.*\n                    FROM wp_mainwp_wp wp\n                    JOIN wp_mainwp_wp_sync wp_sync ON wp.id = wp_sync.wpid\n                    JOIN (SELECT intwp.id AS wpid ,(SELECT recent_comments.value FROM wp_mainwp_wp_options recent_comments WHERE  recent_comments.wpid = intwp.id AND recent_comments.name = \"recent_comments\" LIMIT 1) AS recent_comments,\n\t\t\t\t\t(SELECT recent_posts.value FROM wp_mainwp_wp_options recent_posts WHERE  recent_posts.wpid = intwp.id AND recent_posts.name = \"recent_posts\" LIMIT 1) AS recent_posts,\n\t\t\t\t\t(SELECT recent_pages.value FROM wp_mainwp_wp_options recent_pages WHERE  recent_pages.wpid = intwp.id AND recent_pages.name = \"recent_pages\" LIMIT 1) AS recent_pages,\n\t\t\t\t\t(SELECT phpversion.value FROM wp_mainwp_wp_options phpversion WHERE  phpversion.wpid = intwp.id AND phpversion.name = \"phpversion\" LIMIT 1) AS phpversion,\n\t\t\t\t\t(SELECT added_timestamp.value FROM wp_mainwp_wp_options added_timestamp WHERE  added_timestamp.w"..., 2218, MSG_DONTWAIT, NULL, 0) = 2218
poll([{fd=13, events=POLLIN|POLLERR|POLLHUP}], 1, 86400000

For the sake of troubleshooting, we tried enabling this on the child site’s report plugin settings:
" Comment Flood Tracking" , but no luck.

We’ve also tried adding Redis but it didn’t make a difference either (there’s limited info online if it would actually help or might cause issues with the overall performance and functionalities)

There are no browser console errors or processes running in the Network tab that eventually timeout (which is a bit odd).

My questions are:

Is there a setting somewhere in the reports plugin that could help with optimizing the SQL queries and perhaps exclude certain data?

How many sites can it handle realistically? It does say “Unlimited” but there is a point where it would eventually start to struggle?

Are there API end-points that can be used for generating monthly reports? This will be very, very handy since the functionality can be still used even if the UI struggles.

Any insight would be appreciated!
Thanks.

Hey @Fixed

Welcome to the MainWP community!

There aren’t Pro Reports specific settings, but there is Optimize data loading, which is a Dashboard-wide setting that, according to your System Report, you already have enabled.

Can you temporarily deactivate it and see if you get the same behavior on the Create and Edit report pages?

Generally we don’t advise using any caching/performance plugins on the Dashboard site. Since most of the content is dynamic, the performance improvements are going to be neglible, but they can sometimes cause issues.

We’ve had a number of users with over a thousands child sites and we aren’t aware of a actual limit at which the functionality breaks. It has always been a matter of adequate server resources.

Our API has Pro Reports calls but not for generating reports. They are mostly for fetching individual pieces of data that would otherwise be included in the report.

Hi, thanks for the reply!

We’ve tested by setting up a second MainWP installation and splitting the sites.
Everything works OK up until about 400-500 sites, once you start adding more, the Pro Reports extension starts to struggle.

Overall listing all the sites in the reports page loads fine, however this takes long to execute:

admin.php?page=Extensions-Mainwp-Pro-Reports-Extension**&tab=report**

It might be worth trying to optimize fetching the data:

  • class/class-mainwp-pro-reports-overview.php
  • class/class-mainwp-pro-reports-hooks.php
  • class/class-mainwp-pro-reports-schedule.php
  • js/mainwp-pro-reports-extension.js

This as well:

admin.php?page=Extensions-Mainwp-Pro-Reports-Extension**&tab=report&action=download_pdf&id=8**

You should be able to replicate the issue if you have 800-1000 sites in one dashboard and then going to the menu in question -

domain/wp-admin/admin.php?page=Extensions-Mainwp-Pro-Reports-Extension&tab=report

After some more in-depth troubleshooting I believe this is causing it:
(I’ve edited the sensitive data as “TEST”)

recvfrom(13, "\"nicename\":\"TEST-NICENAME\",\"email\":\"[email protected]\",\"registered\":\"2023-10-11 16:36:37\",\"status\":\"0\",\"display_name\":\"TEST-NAME fiddes\",\"role\":\"customer\",\"post_count\":\"0\"},{\"id\":7,\"login\":\"TEST-LOGIN\",\"nicename\":\"TEST_NICENAME\",\"email\":\"[email protected]\",\"registered\":\"2022-08-08 08:36:43\",\"status\":\"0\",\"display_name\":\"TEST NAME\",\"role\":\"administrator\",\"post_count\":\"58\"},{\"id\":194,\"login\":\"TEST.LOGIN2\",\"nicename\":\"TEST-NICNAME2\",\"email\":\"[email protected]\",\"registered\":\"2023-07-15 17:41:57\",\"status\":\"0\",\"display_name\":\"TEST2

etc. etc.

This seems to be a query/function that goes through all the existing USERS of ALL sites, which you can imagine could be thousands or even millions if it’s eCommerce shops.

You can confirm the above by running an “strace” of the the PHP process while the page is loading.

If this is a needed functionality, then it would be handy to have a global issue to disable ‘users data’ from the reports, but it needs to be “toggle-able” before going to the “Create report” page.

Thanks!

Thank you for the detailed report.

I will forward this to our development and have them take a closer look.

Can you please try generating a report with this Users updates section set to Hide, and let us know if that fixes the issue with generating reports?

If it does, the team will look into improving the process for collecting Users data while generating reports.

Hi Bojan,

Thanks for getting back to us.

We tried disabling it and it didn’t make a difference, I’m afraid.

Also, the issue is that this option is only available in the “Create Report” menu, which is the one struggling to load initially. I believe there should be a global option to disable this before accessing the menu, or as you mentioned - optimize the retrieving data function.

It looks like the underlying problem is that it’s retrieving the data for all sites once the “Create Report” button is clicked, not fetched individually for each one after it’s selected from the list and before generating/downloading the actual report. Basically, the “Users updates” data is always being retrieved for all sites, and then individually selected if it should be included in the report.

Thanks again.

Thanks for the update, @Fixed .

The development team has prepared a pre-release version with some performance improvements.

I’ve sent it to you via a direct message.

Let us know if it helps.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.