spaCy/website/robots.txt.jade

10 lines
142 B
Plaintext
Raw Normal View History

2016-10-31 21:04:15 +03:00
//- 💫 ROBOTS.TXT
2016-10-03 21:19:13 +03:00
2017-06-05 21:07:52 +03:00
if environment != "deploy"
2016-10-03 21:19:13 +03:00
| User-agent: *
| Disallow: /
2017-06-05 21:07:52 +03:00
if ALPHA
| User-agent: Googlebot
| Disallow: /