The robots.txt War — Fighting Bots to Protect My Affiliate Revenue

There's a file on my server that's six lines long, and for two weeks, it was the most important document in my business. It's called robots.txt, and the fight I had with it taught me more about how the internet actually works than any course ever did.

The Problem

My affiliate revenue depends on organic search traffic. People search Google for a product, find my review, click my affiliate link, and I earn a commission. Simple. But I started noticing something strange in my analytics: my affiliate links were being crawled by bots. SEO bots, scraper bots, competitor research tools — they were all hitting my affiliate URLs, following redirects, and consuming my server resources.

This wasn't just a nuisance. It was a revenue problem. Some affiliate networks track clicks and penalize accounts with high bot traffic. If MaxBounty's system detected that 60% of my clicks were from bots, they could reduce my payouts or revoke my account. And the bot traffic was masking my real performance data — I couldn't tell how many actual humans were clicking my links because the bots were inflating the numbers.

I needed to block the bots. And the first line of defense was robots.txt.

What I Thought robots.txt Did

I thought robots.txt was simple: you list the user agents you want to block, and they respect it. I was wrong on two counts. First, not all bots respect robots.txt. Malicious scrapers ignore it entirely. Second, the bots I wanted to block — AhrefsBot, SemrushBot, MJ12bot, DotBot — are owned by legitimate companies who do respect robots.txt, but blocking them has side effects.

See, Ahrefs and Semrush are the same tools I use to track my own SEO performance. If I block their bots from crawling my site, I also block my own ability to analyze my backlinks and rankings. I'd be blinding myself to protect myself.

I spent three days researching which bots to block and which to allow. I made a list of pure scraper bots with no SEO value — things like MJ12bot, DotBot, PetalBot, YandexBot (I don't target Russian traffic). I blocked those. I kept the Google, Bing, and Facebook crawlers. I agonized over whether to block Ahrefs and Semrush, and ultimately decided to allow them because the SEO data was worth more than the bot traffic cost.

The .htaccess Layer

robots.txt alone wasn't enough. Good bots respect it; bad bots don't. For the bad bots, I needed .htaccess rules — server-level blocks that return a 403 Forbidden response before the bot even reaches my content.

I wrote rules blocking known scraper user agents. I blocked requests from specific IP ranges associated with scraping services. I added rate limiting so no single IP could hit more than ten pages per second. Each rule I added felt like building another wall around my site, and each wall potentially blocked a real visitor along with the bots.

The fear of false positives kept me up at night. What if I accidentally blocked a legitimate user? What if a real customer in a blocked IP range tried to click an affiliate link and got a 403 error? I'd never know — they'd just see an error page and leave. The affiliate network wouldn't register the click. The commission would be lost silently.

The Affiliate Link Protection

The most important .htaccess rules I wrote were the ones protecting my affiliate links directly. I set up redirects that strip tracking parameters from bots while preserving them for human visitors. I created rules that detect common scraper patterns — sequential page requests, identical user agents, rapid-fire requests — and block them at the server level.

I also disabled WordPress XML-RPC and pingbacks permanently. Those were the features that had been exploited in the pingback attack, and I wasn't going to leave the door open again. The Broken Link Checker plugin was next — it was a resource hog that added no value. Gone.

The Result

After two weeks of tweaking, my bot traffic dropped by 70%. Real human traffic stayed the same. The server CPU usage dropped from an average of 40% to 15%. My affiliate click data became clean for the first time since launch — every click in my analytics was a real person, not a bot.

And my MaxBounty quality scores improved. The network's fraud detection system had apparently been flagging my account because of the bot traffic. Once the bots were blocked, my click-to-conversion ratio improved, which meant I looked like a higher-quality affiliate. Better quality scores sometimes mean access to higher-paying campaigns.

The Bigger Picture

The robots.txt war taught me that running a website is not just about creating content. It's about protecting that content from the constant barrage of automated traffic that has no interest in your business but can undermine it anyway. The internet is not just humans visiting pages. It's mostly machines talking to machines, and some of those machines are working against you.

I now check my server logs weekly. I identify new bots and decide whether to block them. It's an ongoing process, not a one-time fix. The bots evolve, and so must the defenses. It's a quiet, unglamorous part of running an online business that nobody talks about in their success stories.

But it matters. A six-line file and a well-configured .htaccess might be the difference between a profitable affiliate site and one that gets devalued by bot traffic. Sometimes the most important work in business is the work nobody sees.