Still doesn't work - Another observation: /robots.txt - shows Adobe's default file. but //robots.txt - shows the custom one that we want.

its AEM as a cloud service, so not possible to hit the publisher directly.

thanks @konstantyn_diachenko , but it didn't work...

thanks, yes this seems like cached at CDN level, this is AEM as a Cloud service and we are using Adobe bundled fastly CDN. so when i try publish+dispatcher url, its always with CDN. how to bypass CDN? 

Hello, We are implementing custom robots.txt which will be maintainable by content authors. the file is located in: "/content/dam/mycompany/config/robots.txt" We implemented following Dispatcher changes and deployed to AEMaaCS env: 1. filters.any# Allow robots.txt /0104 { /type "allow" /url "/conten...