GoDaddy Surpressing Search Engine Indexing
I haven’t been impressed with GoDaddy from the day I signed up for their shard Linux hosting. I had heard lots of people rave about how good they are. They must have been referring to the price as that is about the only good thing about it. I would trade their custom control panel for CPanel anyday. However, on with my point.
Dispite the slow servers and poor quality of customer service I have managed to keep my sites running. My primary domain is my personal blog which I haven’t updated in a long time and haven’t paid much attention to. Basically it has a few posts plus I setup Google Analytics and Webmaster Tools and then forgot about it.
Today I was looking through Webmaster Tools, as I occasionally do, and decided to see how it has been indexed and if there were any backlinks. I was a bit surprised to find Google telling me that they could not index my site because they were being blocked by a robots.txt file. That’s odd, I didn’t setup a robots.txt file. I jumped into Filezilla and took a look at my site but couldn’t find the file. I enabled hidden files but still no robots.txt. At this point I am a bit confused so I browse to were the file should be (in Firefox) and sure enough I an greeted by a robots.txt file that is blocking all bots from indexing my site.
At this point I am getting miffed. GoDaddy not only setup a default robots.txt file that blocks all bots they also hide the file in hopes I, and most hobbiest and unaware users, wouldn’t find it. If I hadn’t setup Webmaster Tools I just might not have, at least not until after countless months wondering why I wasn’t getting indexed.
Uploading a blank robots.txt file easily overrides the hidden one and now I am once again free to be indexed. Are GoDaddy shared servers hurting and overloaded that much that they would need to sink that low to try and eliminate bots and search engine traffic from it’s users sites? Needless to say this is the last straw for me. I am tired of hosting companies messing with my sites and I certainly won’t fork out money for a dedicated server just for some personal sites. I plan to use my time off work over Christmas to get my space PC in shape to run a small web server and host my files myself.
With a little effort and a decent backup routine in place I will be much happier knowing I am in control of my sites.
RSS ?
12 comments on this post
I wonder, is it a wordpress blog? I know it’s common practice to rewrite non-existing files to the blogs index.php script when using wordpress. I also know that the default settings for a wordpress blog is to blanket deny access throught robots.txt when robots.txt is requested through index.php. Maybe that’s what’s going on? I’m a GoDaddy customer too, and I know I don’t have a default robots.txt that specifies spiders aren’t allowed…in fact, when it’s not present I get a 404.
Yes it is a WordPress blog. So maybe GoDaddy isn’t to blame but I have installed dozens of WordPress blogs and it has created a robots.txt file but not one that denies all bots let alone a hidden one.
The only thing different about this blog compared to the others is I used GoDaddy’s Hosting Connection to auto install WordPress for me so maybe that process installs WordPress differently. All other WordPress installs I have done have been uploaded and configured manually.
If someone can prove it is not GoDaddy that is to blame then I will gladly retract my statement and apologize. However, I am not convinced that it WordPress’s fault.
I just registered a domain a couple of days ago and I’ve run into this problem as well. I’ve uploaded a blank robots.txt file but Google is still referencing the old one in webmaster tools. So I email godaddy and they said it is a script issue and they cannot resolve for me.
It sucks right now…..
Try uploading a robots.txt that allows all bots not just a blank one. Google has still not indexed any pages form the site I was referencing in this article.
I just bought some different hosting today because I am fed up with their crap.
thanks for the reply justin..
Yeah I did that.. at least I think I did..
if you go to my robots.txt file
http://www.cutiesandhotties.com/robots.txt
You can see that it should allow all bots to access the site.. but google is still saying the robots.txt file is blocking google. =/
hmmm…you are right that should allow all bots. Google says that my robots.txt was correct but for some reason never crawled the site. I really think it is something that GoDaddy is doing behind the scenes.
i know i might need to switch..
well i think ur site is being indexed now though.. since i got here form google.. thx for the help.. and good luck with
I have two sites hosted with Godaddy , one quite old, and I see now using Google’s webmaster tools that neither have been indexed. I had added them both (one several times over 2 years !) to Google’s “add a url “ page - to no avail.
I am furious , and have written their support asking if it is their policy to block Google. This amounts to business fraud – I make some sites for commercial reasons and need to have them found; obviously this is the assumption made by all website designers – what the hell else is the point ?
Money back or small claims court for me if this turns out to be the case.
Thanks for posting your remarks - it got me going.
@Art, I hope you can get your issues resolved. I gave up and moved to another host. You can read about it here.
3 Trackback(s)