New in Symfony 4.3: Automatic Search Engine Protection

March 13, 2019 • Published by Javier Eguiluz

Warning: This post is about an unsupported Symfony version. Some of this information may be out of date. Read the most recent Symfony Docs.

Contributed by Gary Pegeot in #30325

Search engines like Google, DuckDuckGo, Baidu and Yandex do a great job crawling and indexing your web sites and applications. However, this is not so great when you accidentally publish a development version of your app.

In Symfony 4.3 we improved this situation by disallowing the search engine indexing for development applications. How does it work? If the app kernel runs in debug mode (by default this happens when the Symfony environment is not prod) Symfony adds a X-Robots-Tag: noindex HTTP header to all the responses.

The X-Robots-Tag header is one of the HTTP headers that legit search engines take into account when crawling a web site. If your own app already added that header, this new feature doesn't change its value. Also, if you don't like this feature, you can disable it with this config option:

        1
2
3
4
        # config/packages/framework.yaml
framework:
    # ...
    disallow_search_engine_index: false
    

Published in #Living on the edge

Help the Symfony project!

As with any Open-Source project, contributing code or documentation is the most common way to help, but we also have a wide range of sponsoring opportunities.

Comments

Sébastien Lévêque said on Mar 13, 2019 at 10:22

Nice feature !

Ghost said on Mar 13, 2019 at 11:09

It's good protection, I usually add this manually to Nginx for demo environment.

BELHOMME Florian Certified said on Mar 13, 2019 at 12:41

Actually it's smart, we had the case in the past :x

Ulumuddin Cahyadi Yunus said on Mar 14, 2019 at 07:37

I think the value should be "true"

Indra Gunawan said on Mar 14, 2019 at 07:38

value of disallow_search_engine_index is false. then it should be allow search engine index, WDYT?

Javier Eguiluz Certified said on Mar 14, 2019 at 09:03

About the value of this option:

1) By default it has the same value as "debug". So if your app is in debug mode (dev environment) then it's true. If your app is not in debug mode (prod environment) then it's false.

2) The option name is negative ... so "true" means: "protect my site and don't index it".

So, if you don't like this option, you must use "false" to say: "don't protect my site".

Michael COULLERET said on Mar 19, 2019 at 14:13

Nice

Comments are closed.

To ensure that comments stay relevant, they are closed for old posts.