I've follow this post which explain one way of handling robots.txt in ASP MVC but as usual not everything worked as expected.
I knew the problem must be in the routing definitions but out-of-the-box ASP MVC does not provide any means of see which routing handle which request.
That's when I found this post which explain how to use one DLL to debug your ASP MVC routing.
In short, all you have to do is download DLL called RouteDebug, reference it and add one line of code to your global.asax.cs after you setup your routing:
RouteDebug.RouteDebugger.RewriteRoutesForTesting(RouteTable.Routes);From now on you will not see your actual views, instead you will see all the information about routing that was captured and created by ASP MVC.
It help me figure out right away that the order of my routing was wrong and another routing is capturing my request.
Many thanks to Karthik. A for publishing the CodeProject article.