Search Engine Bans and Penalties, or Just Your Imagination?

After the recent Search Engine Strategies (SES) conference in San Jose, I was bombarded with requests to help get Web sites unbanned by Google and other major search engines. At first, I thought I was getting requests from former black-hat SEO (define) clients whose sites were getting their just desserts: exploit the search engines, get penalized.

After a few consultations, I realized a number of sites didn't even have search engine spam penalties. What I discovered were people with unrealistic expectations about search engine visibility; limited knowledge about the technical aspects of Web sites; and unclear concepts about how search engines work.

Search Engine Spider vs. Search Engine Index

I always have great fun at SES, not only as a speaker but also as an attendee. I often sit in the back of the room and listen to attendees' questions. They reveal how little or how much people know about the entire optimization process. If I hear the same question in multiple sessions, I know I should explain the concepts more clearly in forthcoming presentations.

The biggest misconception at this year's conference was the difference between the spidering and indexing processes. The index is a subset of the spider. Search engines access Web content through the spidering process. Then, they filter out duplicates and other bad content and create the index.

Maybe the reason SEO professionals, Web site owners, marketing professionals, and usability experts don't understand this concept is because there's so much emphasis on keywords. Before you build a Web site, you should perform keyword research. Keyword research is necessary to understand how users categorize and describe your products and services before you place site navigation and other sense-of-place labels on an advanced prototype. You can think keyword, keyword, keyword until you're blue in the face. Search engines won't see your Web pages' keywords unless you provide access to those keywords.

A Web site can be spidered and not included in a search engine index.

Instead of keywords, keywords, keywords, think access, access, access. Many Web site owners think their sites have been banned in the search engines when in reality they suddenly provided search engines with limited or no access to their sites' content.

Spam Penalty or Not?

Before assuming a site has been banned from or penalized by a search engine, look for technical reasons the site may not be included in the index.
Look at the robots exclusion protocol, both the text file (if used) and the robots exclusion meta tags. Was your site's content management system recently upgraded? Were the 301 redirects (define) properly implemented? Is your server delivering the appropriate HTTP status codes?

The 301 redirect is like a change-of-address card for a URL. It means a Web page address has permanently moved to a new address. If search engines have limited or no access to the new URLs, Web pages won't appear in search results.

Additionally, if 301 redirects aren't properly implemented, they can really mess up a site's link development. I recently had to go through a painstaking explanation (and some troubleshooting) of improperly formatted 301s for a client who assumed his site was banned by Google.

Ouch! Bad flashback. Excuse me for a moment while I take an aspirin.

Another common technical issue is a change in a page's linkage properties. Link farms are a form of search engine spam. A Web site owner has no control over how other sites link to his site, but he has complete control over how he links to other sites. Therefore, search engines rarely penalize a site for how other sites link to yours, with the exception of link farms. Additionally, by modifying all outbound links on a page to have a nofollow attribute, site owners change the hub or authority score of that page. That can have a negative impact on search engine visibility.

After all technical reasons for not being spidered or indexed are addressed, there's the genuine possibility a site is penalized or banned. Helping sites get unbanned is no fun task. It often takes months to undue all the damage created by black-hat SEO professionals.

I actually encountered a Web site banned for its affiliates' actions. The corporate site wasn't actually part of the link farm, but many of its affiliate sites participated in link farms to achieve instant link popularity. As a result, Google banned both the affiliate sites and the corporate site. Get a good handle on your affiliate SEO plan before embarking on an SEO campaign.

Ouch! Another bad flashback. Excuse me again while I take another aspirin.

Conclusion

First, look for technical reasons why Web pages aren't being spidered and indexed. Then, if the site has no technical issues, try to find items on a site that may violate the terms and conditions set forth by the commercial Web search engines. Sometimes, the simplest solution is turning to the black-hat SEO firm you hired and saying, "You're fired!"

All too often, the main reason people believe their sites have been banned or penalized is unrealistic expectations. All search engine positioning fluctuates. It doesn't matter whether pages of your site had top-10 positions for the past five years. No site is entitled to top search engine positions all the time.
New sites are added to the Web every day. Sometimes, these sites have better content and better site usability than your site. Search engine spam pages slip through the cracks, too. And search engines certainly don't always have effective algorithms. Keyword research, Web site development, usability, and search optimization are ongoing processes. By staying on top of all these processes, your site should receive consistent, qualified traffic over time from search engines and other reputable resources.