-
AuthorPosts
-
September 25, 2020 at 5:48 pm #27562Alex BillingtonGuest
I’m currently running WordPress version 5.4.2, PHP version 7.3.22. I’ve been using the W3TC for years, no issues at all until today. After updating the plugin from 0.14.4 to the latest 0.15.0 version, we experienced a MAJOR server issue today when Memcached ended up overloaded. My sysadmin discovered that “the connection count was increasing along with the newly created webserver processes on the frontend servers and as these were not terminated, they are staying connected to the memcached server.” We have never experienced this before. It occurred after publishing a new post, and suddenly the entire site crashed and memcached was overloaded.
I am also experiencing another weird caching and draft/revision issue with the WordPress backend. I can’t tell if it’s related to W3TC, but it only started acting this way after upgrading the plugin yesterday. I notice that if I write in a post, save the draft, the post sometimes reverts to an early draft that is from the previous save. However, it is saved in the database. But then if I refresh the page and try to load the “previous auto-draft” nothing new comes up. It shows up in the “revisions” section, but nothing happens when I try to restore this draft. Eventually it’s completely gone and there’s no way I can recover the previous text until I write and save the draft all over again. I have no idea why this is happening or how to fix it.
September 25, 2020 at 6:00 pm #27567Jesse OwensKeymasterHello Alex-
Thank you very much for the report. This is the first report we’ve had relating to Memcached overloading, so I’ve informed the developers about the issue.
For now, I’d recommend reverting to 0.14.4, using a plugin like WP Rollback.
After you revert, check to see if the issues with post revisions are fixed as well.
I’ll update you here as soon as I have news from the development team for you.
September 28, 2020 at 11:41 am #27569Alex BillingtonGuestHi. I did roll back the plugin to 0.14.4 for now. I think the Memcached issues are resolved, for the moment. Something about out configuration must cause issues with the updated plugin (since I noticed there was a change to the Memached configuration in this update). But that’s okay for now, and eventually I’ll update our WordPress version, PHP version, and try updating this plugin again in a few months once you have another new patched version.
As for the issues with the auto-draft in posts, it is still happening even after changing the plugin back. I have no idea why. I don’t even know where to look or where to start. Does it have to do with browser cache? Or does it have to do with the databases and the way it’s saving drafts/revisions while editing? I don’t know…
One other important cache question – I am already using the W3TC plugin with browser cache enabled through the plugin. My sysadmin also recently turned on the browser cache settings directly ON the server so it does this through its own configuration. Will this double setup cause problems? Will it override one cache setup? Is this causing issues or can we leave both browser cache settings enabled and it will still run okay?
September 28, 2020 at 12:21 pm #27595Jesse OwensKeymasterHi Alex-
Autosaves are stored in your browser, but I can’t think of a way that browser cache would affect the creation of autosaves. One thing you can try is to define the autosave interval in your wp-config.php file, like so:
define('AUTOSAVE_INTERVAL', 120);
Revisions aren’t saved to the database until you either save a draft or update the post. You mentioned that you are saving them, and it’s being included in the database, but the post isn’t updating to the newest revision. I have to say I’ve never heard of an issue like that before.
If the issue is related to W3 Total Cache in any way, you may be able to resolve it by checking the options to Don’t cache pages for logged in users in Performance > Page Cache and Disable minify for logged in users in Performance > Minify. That being said, W3 Total Cache doesn’t cache administration screens like the editor.
For your second question about browser cache, W3 Total Cache handles this by making modifications to your site’s main .htaccess file. Generally speaking, changes in .htaccess will override your server’s settings, so you can consult your sysadmin on which settings they’re setting up and make sure that your W3 Total Cache settings in Performance > Browser Cache match.
September 29, 2020 at 11:09 am #27664Alex BillingtonGuestThank you for the answers. I’ve never heard of this problem either! Very confusing, no idea why it was acting this way. So far since reverting back to 0.14.4 it has been running okay again. I also cleaned up and optimized the databases, incase there might’ve been some corruption.
We’re running a nginx server so there’s no htaccess file. But we have already activated the browser cache through W3TC (I’m not exactly sure how it does this on nginx?) and also directly on the server. I’ll ask my sysadmin to doublecheck the values. I noticed that the browser cache settings from W3TC are saved in the “nginx.conf” file, which I assume is fine, but I don’t know if this conflicts with the actual server-side browser caching systems. Perhaps it all works together and makes one setting redundant, but I’m not entirely sure if there are some sort of browser errors because of duplicate cache requests from the server.
September 29, 2020 at 11:33 am #27666Jesse OwensKeymasterHi Alex-
Similar to the way that Apache treats local .htaccess files, NGINX will use a local nginx.conf file to override the server-level settings on a per-directory basis.That means it won’t create any conflicts with your server-level settings, but it might change them if the plugin settings differ from what your System Administrator is setting up.
October 2, 2020 at 3:53 pm #27833Jesse OwensKeymasterHi Alex-
I wanted to let you know that W3 Total Cache 0.15.1 has been released with a bugfix for the memcached issue. Please let us know if you have any more issues!
October 5, 2020 at 12:14 pm #27841Alex BillingtonGuestThanks for the note! I’ll run the update weekend and do some testing to see if there’s any other issues. Hopefully this patched the problem – at least with memcache.
October 6, 2020 at 1:47 pm #27929Alex BillingtonGuestNope. Even more problems with memcache in this latest version. Not sure what happened, but today all of the memcache servers crashed. The plugin is giving me this error now:
The following memcached servers are not responding or not running:
Database Cache: 127.0.0.1:11211.
Object Cache: 127.0.0.1:11211.
Page Cache: 127.0.0.1:11211.This message will automatically disappear once the issue is resolved.
No idea why but something isn’t right with the way memcache works with this latest version of the plugin. Both updates keep causing problems.
October 6, 2020 at 3:21 pm #27955Jesse OwensKeymasterHi Alex-
You mentioned that the servers had crashed, this error would indicate that the communication to the server is failing altogether. Double-check your addresses and credentials. You mentioned that there were external Memcached servers (multiple) but these addresses all refer to a Memcached server running locally on your webserver.
Going back to your first message,
“the connection count was increasing along with the newly created webserver processes on the frontend servers and as these were not terminated, they are staying connected to the memcached server.”
Have you tried disabling the option for Persistent connection? Your sysadmin’s diagnosis would seem to indicate that the persistent connection might be causing issues.
October 6, 2020 at 6:26 pm #27958Alex BillingtonGuestThis time no the server itself was able to run. But the memcache servers are crashing entirely once they get overloaded. Everything worked fine for YEARS without any issue with memcache and the plugin. We only ever started experiencing issues like this with this server after upgrading to 0.15.0. I don’t know if there is some compatibility problem or something else misconfigured, but again, all of this began only after the upgrade to 0.15.0 (and 0.15.1). Considering the most recent version updates have made some significant changes to memcache and the way things are handled, it leads me to believe it’s all connected to the plugin’s latest updates and how things are working. But we can find any other errors or anything else in the logs to help clarify and explain the issues. All that we see are crashing memcache servers which (sometimes) leads to taking down the entire server (due to overloads or some other issue like that).
October 6, 2020 at 6:51 pm #27961Jesse OwensKeymasterHi Alex-
Thank you very much for the additional info, I’m consulting with the developers and the Support Team to see what else we can check for you. We’ll update you here when we have some more information for you.October 7, 2020 at 12:22 pm #27974Jesse OwensKeymasterHi Alex-
So far as we know, the bugfixes in 0.15.1 resolved the memcached problems for all the other users who reported issues in 0.15.0, so we’d like to get a little more information to help troubleshoot.
First, can you run the following commands, and reply back with the output?
memcached-tool 127.0.0.1:11211 display memcached-tool 127.0.0.1:11211 stats
If you’re not seeing any output from those, check to see if the service is running, and check if you can connect to it:
ps afux | grep memc telnet 127.0.0.1 11211 > stats
If you see the service running but not responding to status commands, try restarting the server:
service memcached restart
October 15, 2020 at 1:56 pm #28321Alex BillingtonGuestHere you go. I already reverted the plugin back to 0.14.4 again – just to be safe. Especially since it’s entirely stable on my WordPress and doesn’t cause any problems. I’m running on WP 5.4.2 and with this plugin. Here’s the memcache results. Everything runs fine and everything is smooth with this 0.14.4 version. But updating to anything 0.15+ just causes problems – both with the backend/saving drafts, and with the memcache crashing again. Still no idea why…
# Item_Size Max_age Pages Count Full? Evicted Evict_Time OOM 2 120B 113s 1 7 yes 0 0 0 4 192B 86392s 53 108591 yes 1 475439 0 5 240B 45599s 1 2791 yes 0 0 0 6 304B 127127s 1 2130 yes 0 0 0 7 384B 86379s 11 25579 yes 0 0 0 8 480B 274392s 24 43914 yes 0 0 0 9 600B 86392s 17 27959 yes 0 0 0 10 752B 86281s 34 44029 yes 0 0 0 11 944B 86373s 13 11875 yes 0 0 0 12 1.2K 86401s 13 9353 yes 0 0 0 13 1.4K 86401s 48 30682 yes 0 0 0 14 1.8K 86373s 54 28527 yes 0 0 0 15 2.3K 86373s 87 36385 yes 0 0 0 16 2.8K 58527s 66 18009 yes 0 0 0 17 3.5K 81817s 116 30340 yes 0 0 0 18 4.4K 7740s 52 1646 yes 0 0 0 19 5.5K 35222s 24 2032 yes 0 0 0 20 6.9K 3555s 4 213 yes 0 0 0 21 8.7K 85982s 156 17848 yes 7 86382 0 22 10.8K 43802s 104 5186 yes 0 0 0 23 13.6K 67093s 630 18672 yes 1 465122 0 24 16.9K 43802s 131 3656 yes 0 0 0 25 21.2K 3621s 45 405 yes 1 27265 0 26 26.5K 3645s 49 262 yes 0 0 0 27 33.1K 3316s 16 82 yes 0 0 0 28 41.4K 3651s 14 43 yes 0 0 0 29 51.7K 38820s 28 113 yes 0 0 0 30 64.7K 60655s 42 163 yes 0 0 0 31 80.9K 3319s 41 15 yes 0 0 0 32 101.1K 3401s 70 8 yes 0 0 0 33 126.3K 61109s 42 269 yes 0 0 0 34 157.9K 61206s 41 232 yes 0 0 0 35 197.4K 61235s 26 121 yes 0 0 0 36 246.8K 2825s 4 1 yes 0 0 0 37 308.5K 13659s 2 1 yes 0 0 0 38 385.6K 12056s 2 1 yes 0 0 0 39 482.0K 13659s 3 2 yes 0 0 0 40 602.5K 12056s 3 1 yes 0 0 0 41 753.1K 12057s 3 1 yes 0 0 0 ---- #127.0.0.1:11211 Field Value accepting_conns 1 auth_cmds 0 auth_errors 0 bytes 1041799960 bytes_read 44152075210 bytes_written 331445781756 cas_badval 0 cas_hits 0 cas_misses 0 cmd_flush 0 cmd_get 72482221 cmd_set 11693559 cmd_touch 0 conn_yields 0 connection_structures 32 curr_connections 28 curr_items 471212 decr_hits 0 decr_misses 0 delete_hits 53369 delete_misses 112172 evicted_unfetched 9 evictions 10 expired_unfetched 2272958 get_hits 54505775 get_misses 17976446 hash_bytes 4194304 hash_is_expanding 0 hash_power_level 19 incr_hits 0 incr_misses 0 libevent 2.0.21-stable limit_maxbytes 2147483648 listen_disabled_num 0 pid 23411 pointer_size 64 reclaimed 2435690 reserved_fds 20 rusage_system 1120.630942 rusage_user 484.521307 threads 4 time 1602723062 total_connections 265185 total_items 11693559 touch_hits 0 touch_misses 0 uptime 723666 version 1.4.15
Let me know if any of this helps clear up issues?
October 16, 2020 at 12:43 pm #28359MikhailGuestHi,
I am experience absolutely same issue. Memcacheed is overloaded. Using 0.15.1 version. Restarting memcache helps but only temporary.
Rolled back now to 0.14.4October 16, 2020 at 12:51 pm #28365Jesse OwensKeymasterHi Alex and Mikhail-
Thanks for the additional info and report. I’ve sent this information to the developers for more investigation. I’ll update you here as soon as I’ve heard back.
-
AuthorPosts
- The topic ‘Memcached crashes after W3TC 0.15.0 update’ is closed to new replies.