Here’s a simple way to create a dynamic Sitecore MVC robots.txt.
controller = "Robots",
action = "RobotsText"
2. Create a RobotsController
public class RobotsController : Controller
public FileContentResult RobotsText()
var contentBuilder = new StringBuilder();
return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");
private bool IsProduction
return (ConfigurationManager.AppSettings["IsProduction"] ?? "false") == "true";
3. Add /robots.txt to IgnoreUrlPrefixes setting in Sitecore.config
4. Ensure <modules runAllManagedModulesForAllRequests=”true”> is set to true within web.config.
If you are planning on using DI on your Sitecore 7.2 and later projects, you should be aware that Sitecore has itself started using MVC in some of it’s features. For example, if on an image field you click “Browse”, then “Upload”, this diaglog window will call the URL /api/sitecore/Media/Upload?sc_database=master which is an MVC route.
The issue is, if you have set the Controller Factory of your application to your own (this was done when using a Castle Windsor implementation) then you need to ensure you have registered the instances of IController within the Sitecore DLLs as well as your own. The DLLs you need to register are Sitecore.Speak.Client, Sitecore.Mvc and Sitecore.Mvc.DeviceSimulator. For example, in Castle Windsor you would need to add something like the following to your Installer which registers your controllers:
For further reading, look at Sitecore support ticket #429145
In a Sitecore instance that contains multiple sites, you may have found that links between different sites can result in incorrect URLs.
For example, suppose we have the following Sitecore structure:
- Site A (www.site-a.com)
- Site B (www.site-b.com)
Typically, a link from anywhere on Site A to Page B might result in a URL like “www.site-a.com/sitecore/site-b/page-b”, rather than the expected “www.site-b.com/page-b”.
There are a number of things you need to get right in order for this to work properly:
- The SiteResolving property needs to be set to true on the UrlOptions when calling LinkProvider.GetItemUrl().
There is a bug in that the Sitecore config setting (Rendering.SiteResolving) is never used to set this property automatically. This can be easily fixed by extending the LinkProvider with a few lines of code.
- Site definitions must have both hostName and targetHostName attributes set in order for site resolving to work (even if the sites in question all use the same hostnames)
If you want language to be preserved correctly across cross-site links then also:
- The Rendering.SiteResolvingMatchCurrentLanguage setting must be set to true
- The casing of the language names must be the same between the definitions in the Sitecore tree and the language attributes on the site site definitions, as a case-sensitive match is performed.
Some relevant articles:
When you are on the site in question:
- Left nav -> Site Contents
- Settings (on the right)
- People and groups