[teiid-issues] [JBoss JIRA] (TEIID-5993) Add robots.txt to better crawl content

Add robots.txt to better crawl content -------------------------------------- Key: TEIID-5993 URL: https://issues.redhat.com/browse/TEIID-5993 Project: Teiid Issue Type: Quality Risk Components: Documentation Reporter: Steven Hawkins Assignee: Steven Hawkins Priority: Major Fix For: Backlog Original Estimate: 1 hour Remaining Estimate: 1 hour We should do a better job of directing what pages are crawled. I added a robots.txt to prevent the crawling of legacy. Something similar should be added for teiid.github.com to allow only the crawling of master.