As the on-going chore of SEO is progressing for my site. I have noticed a number of 404 Errors are being generated all the time. They are really random URLS being generated. An example of a few is below:
As they are consistently generated, I have a feeling something is crawling the site. Does anyone know how to track down these errors and even more importantly, stop them.
I’d never seen “uenc” in a URL before, I just had to Google it. I found this article if you’re interested: WTF is UENC? Essentially, I believe the URL is used to Add and item to cart. So, in the example you gave above the first URL would/should add Product (with ID) 265 to the cart.
So, I don’t have a specific answer for you here… But I spotted some issues that might be related. At the very least, some leads for you to chase:
Default Cookie Policy: There is a setting in Configuration > General Web > Default Cookie Settings > Cookie Restriction Mode. Setting this to “Yes” is Advised for GDPR, however it kinda breaks the site until the customer hits “Accept Cookies”.
I tried adding MB19090253 to the cart via the Product Grid page using the Quick View function. However, nothing happened. This was before I clicked “Accept Cookies”.
So, I’m thinking:
404s might be generated from the Quick View function (in some way) when customers try to interact with the site before Cookies are enabled.
The Quick View function is broke in some form
Albeit, I could be leading you in the wrong direction… But at least it’s something for you to look into.
Bonus observations
Your sitemap.xml only contains categories (not products)
/pub/media/captcha/base/b8ee33bbc19360d621cee2e043ab439c.png is trying to load on the homepage but does not exist, which is a source of your 404. In fact, I ran a 404 report and the only automatically crawled 404s were related to this recpatcha URL. Perhaps this is a module you’re using that’s broken?
Because my 404 crawler didn’t find any URLs containing “uenc”, it makes me think the issues are caused by human interaction.
I am trying to recreate the error you mention above, but not having much luck any the minute. I will keep trying, as if I can record the error, I can look at resolving If you are right, and it is the Quick View function, then should be easily resolved, once I can prove that to be case.
Bonus observations
Your sitemap.xml only contains categories (not products) - Yes I have set that on purpose, I am not sure if that is correct. My thoughts was, you generally see category pages on SERP, so I don’t what to overload it with possible duplicate content from site and just focus on the category pages. (Could be the wrong approach, I don’t know. It is something I am researching)
You need to add trustpilot and cloudfront to your CSP list to stop generating Console reports. (Ref: Removing Content Security Policy Warning (Magento 2.3.5 and above) 1) - Good spot, only added trustpilot this weekend, but cloudfront was not setup correctly. All resolved!
/pub/media/captcha/base/b8ee33bbc19360d621cee2e043ab439c.png is trying to load on the homepage but does not exist, which is a source of your 404. In fact, I ran a 404 report and the only automatically crawled 404s were related to this recpatcha URL. Perhaps this is a module you’re using that’s broken? - I have had reported issues with the recpatcha module in the post. I thought it was disabled, I will see if I can track it down and disable it.
Because my 404 crawler didn’t find any URLs containing “uenc”, it makes me think the issues are caused by human interaction. - I would love to think its human interaction, that would be a lot of traffic, but I am only seeing around 50 hits per day (If i am lucky). On the other hand, 1000’s of 404’s are appearing! They are odd.
I have set the enforced cookie mode to NO for now. I want to see if the errors stop appearing.
Now that you’ve removed some of the “noise” (e.g. legitimate issues), you’ve got realistic list to investigate:
wp-login.php is a Wordpress login URL
phpMyAdmin-4.7.4-all-languages is a temporary phpMyAdmin directory
phpMyAdmin-4.7.3-all-languages is a temporary phpMyAdmin directory
downloader/index.php is a Magento 1.x directory
Looks like you have 1 or 2 malicious bots crawling your site, having a poke around. Not a lot you can do about that other than follow Security Best Practices. No matter what steps you take, you’ll always have bots trawl your website. It’s the nature of the internet.
Having said that, using Cloudflare as a DNS will filter out all “known” bots from accessing your website - Therefore, they do most of the heavy lifting and have a database for all their customers to compare against.
Yep I agree with the above and I can also correlate the requests to the firewall. Basically, there are matching requests to the website from a country “Tor” well, we know its not a country, but the firewall believes it to be.
I have blocked requests from TOR at the firewall level now, so hopefully them few requests won’t reappear.
Time to enable the other modules 1 by 1 and see when the errors return!
If you are right, and it is the Quick View function, then should be easily resolved, once I can prove that to be case.