Prevent development server getting indexed by google

Magento 2
1

Hi Craig,

Thanks again for all you great content, has really helped me out.

Do you know of a robust way of disabling the development server from being indexed by google?

Also would you know of a way of making it locked down to only allow certain ip addresses?

Thanks

2

Hi @william.davies,

Thank you for the positive comment.

To answer your question, there are a few ways you could go about this…

1 Put this in the robots.txt file:

User-agent: *
Disallow: /

It’s a bit wishy-washy and not completely foolproof. Infact, it’s more of a suggestion to Search Engines than an actual rule. I would not recommend it - Plus, competitors could still look if they wanted.

2 Password protect your /var/www/html/ (AKA Web) directory using Apache

Check this well-explained tutorial

It can be a hindrance when you require 3rd party service to ping your installation. Such as testing 3rd party payment gateways or verifying product keys for extensions you’ve purchased. I stopped using this method years ago.

3 Use a firewall - RECOMMENDED
This my go-to method and I highly recommend it. You can update firewall rules on most Server Providers using their easy-to-use interface. Alternatively, you could update these from the Command Line via SSH. You’d want to deny all connections via SSH (Port 22), HTTP (Port 80) & HTTPS (Port 443). Then add exceptions on a case-by-case basis. Now, obviously this works best when you work from an office uses has a Static IP - Otherwise you’ll forever be updating the rules whenever the router decides to reset.

Your Server Provider should have documentation to help you with firewalls. If you use Digital Ocean, this is their documentation.

4 Maintenance Mode - RECOMMENDED
Kind of an extension from the last point, but it’s one I also use. Maintenance Mode is a Magento 2 feature, controlled from the Command Line Interface (SSH). When in Maintenance Mode, anyone not on the Whitelist is presented with a Maintenance Mode message when trying to access Magento Frontend/Backend. Read more…

  • Enable Maintenance Mode: bin/magento maintenance:enable
  • Disable Maintenance Mode bin/magento maintenance:disable
  • Check Maintenance Mode Status: bin/magento maintenance:status
  • Add Whitelisted IPs: bin/magento maintenance:allow-ips 123.123.123.123 456.456.456.456
  • You can also add/remove Whitelisted IPs by editing the file var/.maintenance.ip. You just separate your IP addresses with a comma e.g. 123.123.123.123,456.456.456.456,789.789.789.789. Note: If you’ve never used whitelist IPs for maintenance mode, then you will have to create the file first.

Summary
The securest way is option 2, 3 & 4. Only you can decide which of those 3 options best suits your needs.

I hope this helps.emphasized text

EDIT: I added a fourth method.

1 Like
3

Could not have asked for a better answer. Thank you so much.

1 Like