Who Hides Data From Search Engines And Why?
Posted by Bob Warfield on April 11, 2008
Fred Wilson just did an interesting post about delicious and its traffic after being acquired by Yahoo. On the face of it, looking at Compete or other stats, it appeared delicious had gone into decline after being acquired. Reality is a bit different as Joshu wrote back to Fred:
We continue to grow normally.
Unique users is not a good measure of our growth, though.
Much of our traffic is through the firefox and other browser extensions, which is not measured by these systems.
Additionally, we cut off search indexing several months ago, which also hurts the UU numbers.
It is fascinating to consider just how much of the web is not measurable even today due to such things as running traffic through browser extensions. What I really found interesting was the last line, though.
Why would delicious cut of search indexing?
As one commenter on the thread pointed out:
I would think Joshua would be delighted if the “funny-video” tag was the first search result at Google for the search team “funny video”… it would mean both greater distribution and influence.
Evidently not. There is no good cost reason to turn away web crawlers. For a property like delicious, crawling have to represent a tiny fraction of their traffic. It seems to me the reasons would have to be strategic.
For the conspiracy theorists out there, consider this. Perhaps Yahoo is doing this for a lot of their valuable properties and only letting Yahoo search engines index the data. This means a Yahoo search can find delicious posts but Google can’t.
It seemed a great theory, but alas I could not verify it at all. I tried half a dozen of the most popular posts on delicious and could not find them listed on searches for the title on either Google or Yahoo search.
So I’m stumped. It would be fascinating to see a list of sites that exclude search engines sorted by popularity. Even more fascinating would be understanding why they exclude the searchers.
Update: It seems that Yahoo does do something special with delicious, but maybe it isn’t all in production yet based on my limited testing. See TechCrunch for more.
Don’t you think this is a tacky way to compete? By limiting availability of search results? Search result integrity is essential, and here we have companies outright trying to sabotage the search results of their competition. This will get worse if MSFT has anything to say: they don’t believe in a fair fight!
engtech said
My take: Hiding Delicious results from Google is a competitive advantage for Yahoo Search.
twgonzalez said
I would have to agree with Engtech. Do I think it is “tacky”? Well, I think my life is a lot easier when I have the confidence that google is the be-all-end-all of search as I only have to go to one place. But, I am not sure I think it is fair that Google is basically re-purposing everyone’s content (albeit through their powerful indexing/retrieval) and making significant revenue from it? If MSFT and Yahoo started to do this (hide their data) for all of their properties I imagine it would seriously start to undermine the ubiquity in search that Google enjoys right now.
When you say the word “sabotage” I think there is an implicit assumption that by denying Google the ability to index your data you are somehow doing the community as a whole a disservice (which you might be at least in the short term), but if I were manning the sinking ship at Yahoo, I would be looking for every angle to create some value for my properties and drive eyeballs back to myself. I suspect there are other models for universal search that do not require one company to hold a monopoly on all the revenue generated from it. Very interesting nonetheless, I would have never thought of that tactic, I wonder if others might start to follow suit.
smoothspan said
Let’s suppose others do follow suit. In fact, let’s suppose all the big guys do, including Google. Search is no longer a reliable service for the Internet.
Instead, it has been co-opted from major corporate interests in a misleading way. Yes, Google does profit from search, but they give you an opportunity to profit as well by driving traffic to you site. That doesn’t seem evil to me. One of the reasons Google has so much share is because their stuff works. They don’t charge unless someone clicks through.
What’s next for these other companies? Will they charge you to even appear on their search engine?
This is bad voodoo.
Sincerely,
BW
twgonzalez said
I don’t disagree that Google works, and it works amazingly well. I like it both from a consumer (searching) standpoint and a content provider standpoint (ad-sense.) I just wonder if there is a different model that is more distributed but doesn’t put all the power in the hands of one company. I don’t have the answer to that solution, and I agree I would hate to see things go the way you describe with all the big-boys hoarding their data/content and destabilizing both search and the way I can drive eyeballs to my content.
Interestingly, after this post I was doing a search on Google for the Yahoo groups “Flexcoders” I did not have it in my bookmarks or delicious, and because I am lazy I just put “yahoo flexcoders” into Google search, expecting a hit. But alas, the result I wanted was not on the first page returned to google. I remembered that this had happened before and I found it frustrating that I couldn’t locate a popular yahoo group with 10k members and a very active mail group. I then looked forward in Google’s results for the next 50 results – still no luck. I then tried “yahoo groups flexcoders”, same result – nothing! So I did a test and went to Yahoo to search (which is something I never do) guess what the first result was in Yahoo? Thats right, the exact Yahoo group I was looking for. So I wonder if the problem is more pervasive than just delicious?
TG