이 ip 차단해도 괘찮을까요??
본문
이 ip들이 요 며칠 계속 사이트 트래픽 초과로 서버 폭파시키네요;;
그래서 차단해 놨는데, 찾아보니까 구글 관련된 거더라고요.
차단해도 괜찮은 건가요?
답변 1
차단하면 나중에 구글에 검색이 잘 안 될 수도.....
robots.txt 이용해 크롤링 주기를 조정해 보세요.
https://www.google.com/search?q=robots.txt+%EC%84%A4%EC%A0%95%EB%B2%95
헛. 이게 구글봇이었나요?ㅜ
바로 풀어야겠네요..ㄷㄷ
감사합니다!
채택~~
신비님 혹시, 아래처럼 설정하면 될까요?
User-agent: googlebot
Disallow:
Crawl-delay: 20
User-agent: googlebot-image
Disallow:
Crawl-delay: 20
User-agent: googlebot-mobile
Disallow:
Crawl-delay: 20
Disallow: 부분은 빼셔야 할 것 같아요.
옛날 구글링해서 저는 아래처럼 설정해 사용 중입니다.
# 허용 X
User-agent: AI-Bot-Name
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: archive.org_bot
User-agent: Bagoodex
User-agent: Bing AI
User-agent: BDCbot
User-agent: Brave
User-agent: BusinessBot
User-agent: CCBot
User-agent: Character.AI
User-agent: ChatGPT-User
User-agent: Claude
User-agent: CLOVA
User-agent: Codium
User-agent: Copilot
User-agent: CopyAI
User-agent: Dataprovider.com
User-agent: DeepL
User-agent: DialoGPT
User-agent: DialogSearch.com Bot
User-agent: Digincore crawler bot
User-agent: DomainSigmaCrawler
User-agent: ELSA
User-agent: Gemini
User-agent: Google-Extended
User-agent: Googlebot-Image
User-agent: GPTBot
User-agent: Hix
User-agent: iAsk
User-agent: Jasper
User-agent: KakaoBot
User-agent: kisaBot
User-agent: Komo
User-agent: Lilys
User-agent: Mail.RU_Bot
User-agent: MediavBot
User-agent: memoryBot
User-agent: mfibot
User-agent: msnbot-media
User-agent: NeumobBot
User-agent: Otter
User-agent: PageAnalyzer
User-agent: PagesInventory
User-agent: Perplexity
User-agent: Plukkie
User-agent: Quora
User-agent: roboto
User-agent: rogerbot
User-agent: RSNBOT
User-agent: SafeDNSBot
User-agent: semanticbot
User-agent: Sidetrade Indexer Bot
User-agent: SMTBot
User-agent: Socratic
User-agent: SurveyBot
User-agent: TabNine
User-agent: ThinkAny
User-agent: TurnitinBot
User-agent: UXCrawlerBot
User-agent: Vaiv
User-agent: Wotbox
User-agent: Writesonic
User-agent: YouChat
Disallow: /
# 허용 △
User-agent: AdsBot-Naver
User-agent: msnbot
User-agent: bingbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: DomainCrawler
User-agent: MegaIndex.ru
User-agent: AlphaBot
User-agent: ltx71
User-agent: CCBot
User-agent: Sogou
User-agent: DotBot
User-agent: PiplBot
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: MauiBot
User-agent: AhrefsBot
User-agent: ezooms
User-agent: sistrix
User-agent: Yandex
User-agent: ZumBot
Crawl-delay: 30
# 허용 O
User-agent: Daum
User-agent: Daumoa
User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Mediapartners-Google
User-agent: Yeti
Allow: /
와 감사합니다. 저도 똑같이 해야겠네요 ㅎ
