관리하는 사이트를 살펴보니, bot 의 엄청난 방문을 확인했습니다.

그로 인해서 디비 용량만 마구 갈아먹는 그런 안타까운 상황이었습니다.

그래서 정리해봤습니다.

robots.txt  를 이용한 검색엔진 선택하기 ..... ( 업데이트는  https://www.happyjung.com/lecture/258 에서 확인 )

참고로 각 검색봇을 속성을 정리문서는 https://www.happyjung.com/lecture/2619  에서 확인할 수 있습니다

User-agent: AhrefsBot
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: BDCbot
Disallow: /

User-agent: BusinessBot
Disallow: /

User-agent: Dataprovider.com
Disallow: /

User-agent: DialogSearch.com Bot
Disallow: /

User-agent: Digincore crawler bot
Disallow: /

User-agent: DomainSigmaCrawler
Disallow: /

User-agent: kisaBot
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: MediavBot
Disallow: /

User-agent: memoryBot
Disallow: /

User-agent: mfibot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: msnbot-media
Disallow: /

User-agent: NeumobBot
Disallow: /

User-agent: PageAnalyzer
Disallow: /

User-agent: PagesInventory
Disallow: /

User-agent: Plukkie
Disallow: /

User-agent: roboto
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: RSNBOT
Disallow: /

User-agent: SafeDNSBot
Disallow: /

User-agent: semanticbot
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: UXCrawlerBot
Disallow: /

User-agent: Wotbox
Disallow: /

User-agent: YandexBot
Disallow: /


User-agent: AdsBot-Naver
allow:

User-agent: bingbot
allow:

User-agent: Daum
allow:

User-agent: Googlebot
allow:

User-agent: Googlebot-Mobile
allow:

User-agent: Yeti
allow:

User-agent: ZumBot
allow:

Disallow: /adm/ 
Disallow: /data/
Disallow: /extend/
Disallow: /install/
Disallow: /js/
Disallow: /lib/
Disallow: /plugin/
Disallow: /skin/
Disallow: /theme/

Sitemap: http://경로/sitemap.xml