robots.txt 질문입니다.
본문
요즘 트래픽이 급증하여, robots을 작성해봤는데요.
구글링해서 하긴 했습니다만...혹시나 잘못된 점이나 추가되면 좋을 것이 있을까요?
답변주시면 감사하겠습니다.
User-agent: AhrefsBot
Disallow: /
User-agent: archive.org_bot
Disallow: /
User-agent: BDCbot
Disallow: /
User-agent: BusinessBot
Disallow: /
User-agent: Dataprovider.com
Disallow: /
User-agent: DialogSearch.com Bot
Disallow: /
User-agent: Digincore crawler bot
Disallow: /
User-agent: DomainSigmaCrawler
Disallow: /
User-agent: kisaBot
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: MediavBot
Disallow: /
User-agent: memoryBot
Disallow: /
User-agent: mfibot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: msnbot-media
Disallow: /
User-agent: NeumobBot
Disallow: /
User-agent: PageAnalyzer
Disallow: /
User-agent: PagesInventory
Disallow: /
User-agent: Plukkie
Disallow: /
User-agent: roboto
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: RSNBOT
Disallow: /
User-agent: SafeDNSBot
Disallow: /
User-agent: semanticbot
Disallow: /
User-agent: SMTBot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: UXCrawlerBot
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: 360Spider-Image
Disallow: /
User-agent: 360Spider-Video
Disallow: /
User-agent: HaosouSpider
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Googlebot-Image
Disallow: /
User-agent: Yahoo-MMCrawler
Disallow: /
User-agent: wisebot
Disallow: /
User-agent: AdsBot-Naver
allow:
User-agent: bingbot
allow:
User-agent: cowbot
allow:
User-agent: Daum
allow:
User-agent: daumoa
allow:
User-agent: Googlebot
allow:
User-agent: Googlebot-Mobile
allow:
User-agent: Yeti
allow:
User-agent: ZumBot
allow:
User-agent: empas
allow:
Disallow: /adm/
Disallow: /extend/
Disallow: /install/
Disallow: /js/
Disallow: /lib/
Disallow: /plugin/
Disallow: /skin/
Disallow: /theme/
Disallow: /thema/
답변 1
Disallow: /adm/
Disallow: /extend/
Disallow: /install/
Disallow: /js/
Disallow: /lib/
Disallow: /plugin/
Disallow: /skin/
Disallow: /theme/
Disallow: /thema/
allow: / 로 하시고
이런식으로 엮어 버리면 사이트를 불러오는 부분이 서로 연결되어 있는데 이건 빼주셔도 될듯합니다.