# Naughty list - all of these are blocked completely User-agent: Mail.Ru Disallow: / User-agent: MJ12Bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: PetalBot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SEMrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: MauiBot Disallow: / User-agent: YandexBot Disallow: / # everything else is permitted limited access User-agent: * # temp - block all robots to reduce site load. 11th March 2021 # Disallow: /ski-forum Disallow: /ousekjarr/ Disallow: /snowheads/ Disallow: /test-forum/ Disallow: /advertising/ Disallow: /media/ Disallow: /ski-forum/js Disallow: /ski-forum/templates Disallow: /ski-forum/search.php Disallow: /ski-forum/profile.php Disallow: /ski-forum/images/ Disallow: /ski-forum/resort-list.php # 22/11/21 - adjust because of Googlebot storm... # SEO - don't allow alternative ways of accessing the site, including MX (mobile experience) # as the content is the same but we get scanned 3-4 times as a result Disallow: /mx/ Disallow: /mx/* Disallow: /index.php?lmsg_start=* Disallow: /index.php?nstart=* Disallow: /?* Disallow: /mx/index.php?lmsg_start=* # 01/03/2022 - tinkering to reduce load # Don't permit direct linking to a post, as this causes a load of a whole page per post # when viewtopic.php?t=X has already done this Disallow: /ski-forum/viewtopic.php?p=* # 20/04/2022 - bandwidth and load amplification via image proxy not allowed, but thumbnail is OK Disallow: /ski-forum/includes/img_proxy.php?ref # site map applies to all bots Sitemap: https://snowheads.com/ski-forum/sitemap.php