The Burden of Truth
Discussion, opinion, information, and more opinion on search technology, the net, and the world.

Burden of Truth

Friday, January 09, 2004


Ethos and attitude are everything. If you wish to generically forecast the strategic actions of major company, scratch the surface and take a look at these values.

Those webmasters who suffered badly last year with the Florida update, and those who will suffer in the near future (I don't believe they have finished yet for one second) would do well to consider this.

The problem is that somewhere along the path, from a handful of techies working at the frontier of search technology, to Google as a multi-national, the culture of the company changed. Perhaps it was inevitable.

Take a look at any big corp. Who are they answerable to? Obviously stockholders and shareholders. What generally drives these people? Well.... profit... the best return possible on their investment. Basic stuff, but I would argue fundamental to what we are seeing at Google.

Clearly on the list of priorities therefore, a large post IPO organization will have return for stockholders pretty much at the top of their list of priorities. And yes, it has to be faced - well above maintaining the quality of the product set, which after all is only part of the means to the actual end.

There's the rub, and in a search environment the relationship between product quality and returns is even inversed in certain scenarios: poor search returns = more advert clicks = higher short term profit. This again is hardly a factor that will help push quality to the top of the company priority list.

I rather suspect we have seen a drift in how Google (at the top) sees itself... from a TECHNOLOGY company to an ADVERTISING company. There's a big difference between the two:

- A technology company behaves how Google behaved 3 years ago.
- An advertising company behaves how Google does now.

Think about it, and then read the tea leaves again.

Friday, November 21, 2003


I really wish I could believe it was as simple as that....

At time of publishing this entry, the Google returns are in a mess. For niche topic after niche topic the most appropriate focused niche sites are buried deep below the large vanilla portals: Amazon, ZDnet, etc.

The quality of the SERPS is diminished accordingly. So much so, that in the opinion of many, Inktomi, and a number of the other search engines, now produce better quality results.


A week ago all was fine. Google was producing normal results, and was almost the undisputed king of quality. Then.... it happened... Saturday the 15th November... a new algorithm was introduced. Perhaps not an algorithm, more of a filter.

The filter seems to work on the basis of a combination of anchor text of the incoming links, page content, and the authority status of the site (or otherwise).

The latter is particularly important, because it helps drive the bottom line phenomena: big deep-pocket sites rising up the SERPS to displace the niche focused sites.

Sadly for those countless niche site webmasters now facing ruin, this smacks of CALCULATED change, not error. A recent adjustment (Thursday) increased the rich-v-poor differential even further.


Yes, people are asking the obvious questions: Why? Can't Google see that the SERPS are now dodgy, to say the least? I would imagine they can see exactly that. But there is a bigger picture. Consider the following proposition:

If they change the balance of returns dramatically in favor of big money players, who own Amazon/etc, there are two potential fiscal effects:

1) A massive boost in income for those companies. A visible demonstration of what Google can do for them. And of course, guess what may around the corner? IPO.

The argument is that those companies will thus be more inclined to invest their mega profits to sustain the new status quo, which in turn will push the IPO value up.

2) A percentage of the fallen webmasters will rush straight into Adwords advertising in a desperate attempt to keep afloat. A short term but direct increase in revenue for Google.

Some people are thus suggesting that there are many ways to sell out to advertising. They argue that MSN/etc took the blunt route, filling many of their returns with paid ads. Maybe Google is being somewhat more subtle, playing a longer game. In the end these amount to the same thing.


Well, if quality is king, it shouldn't. So on this basis perhaps what happens next will be an indicator of whether there is any foundation at all to the above story.

In the meantime, spare a thought for all those niche webmasters, who spent all those hours producing high content quality web sites on the understanding (partially fostered by Google itself!) that this was the route to better ranking.

The general message from these guys? And a happy Christmas to you too Google!

Wednesday, November 12, 2003


With Yahoo now settled into ownership of AltaVista, Inktomi and FAST, speculation is mounting regarding the future of Yahoo's prime search facility. Which of it's acquisitions will be promoted to replace Google? And of course, when?

Most bets seem to be on Inktomi. But is it the foregone conclusion that the pundits seem to think?

One the face of it, yes. It has the most mature PFI model, and it has been in ownership for longer than the other two. In addition, there have been reported glimpses of returns which do seem to be aligned to the current Inktomi SERPS.

However, I would suggest that this is no betting certainty. There are simply too many variables and unresolved factors. For example:

a) The quality of Inktomi simply does not match that of Google yet. Yahoo will surely be aware of that

b) What about the prospects of a merger of results from two or even three of the above?

c) What of the Yahoo directory? This is still of fairly high quality and is a good earner. It is not difficult to imagine this playing a bigger role in the final search returns

d) And even Google. Yahoo owns a part of Google - the timing and nature of the change must surely have some regard for this.

These, in my opinion, must cloud the decision to some degree. In addition, I see no pressing need for a quick decision. The change could well take place only when Google's contract expires.

The race is far from over, and all the chips are still on the table.

Wednesday, August 13, 2003


1. A History Lesson

Consider the following quotes:

"But the company lost its way through repeated restructuring exercises and diversification." - The Times

"But there are some ominous signs out there, AV had been my primary choice for search since Digital rolled it out -I gave up on it a few weeks ago and switched to a collage of engines, with Google being first, AV second, etc., etc... Today, Brett posts that he had come to the same decision -to move on. How many more of us have just finally said "Enough!" and reluctantly chosen to leave an old workhorse that's come up lame? " - RC Jordan April 2000

"I think that the problem for AV stems from the fact that it seems to have taken it's eye off the ball.... some time back it seems to have forgotten that it was a search engine, first and last." - Napoleon, Feb 25 2001

"I received some feedback from correspondents on this issue and soon discovered that Altavista was no longer the mechanically objective tool that I once faithfully relied upon. As I surfed the new Altavista site, it was a little like experiencing the death of a friend. Companies can now purchase "relevant keyword result placements" that are sold via online auction." - Spies.Com

These all pertain to the decline of AltaVista of course. The question is, how many of these are starting to look applicable to Google at present?

2. Google Today

It's actually quite alarming. A search engine lives and dies by its quality, its focus and its reputation for objectivity.

All three of these have recently been challenged:

1) Quality
With the recent strangling of returns in some sectors by Amazon, PDFs, and so on, countless people, even on this board, are bemoaning a drastic reduction in quality. Other problems are also contributing to this, such as the 'missing index' phenomena

2) Focus
Google has its fingers in a growing number of pies... importantly, direct earning pies: Adwords, Adsense, etc. The diversification is stark and clear, giving the impression to many that, as quoted above, the company may be "taking its eye of the ball".

3) Objectivity
The allegations that somehow Google has a financial arrangement with Amazon are actually likely continue whilst the current status quo is maintained. Worrying for Google, this accusation may not remain on the fringes.

All this of course comes soon after a period of substantial instability due to the implementation of new technology.

3. Google Tomorrow

The question therefore becomes: are we at a turning point in search engine history? Has the decline of Google, which some view as inevitable, already started?

Of course, it's a very difficult question to answer... but there ARE signs

a) Consolidation

The consolidation in the industry certainly indicates that challengers are labouring up and preparing. Yahoo certainly, via acquisition. Microsoft via statement of intent. These and others are clearly developing a strategy to compete more effectively.

b) Google's Self Inflicted Damage

It never ceases to amaze how organizations fail to study the history of their own industry and learn from it. Search engine after search engine have trodden the same path to oblivion.

That path is loosely described above, and the signs over several months are that Google may be taking those first fateful steps. The current quality issue is a matter of fact. The index instability is a matter of fact. The diversification is a matter of fact.

Now look at AltaVista for example... we've seen much of this before.

4. Public Perception

Public perception is ABSOLUTELY CRITICAL to a search engine company. Unlike most other products and services, consumers can switch supplier freely and easily. If the public believe that results are somehow flawed, 'broken' or fixed in some way, they will switch. If they believe that a search engine is going down hill, and better returns can be found elsewhere, they will also switch.

Unfortunately, Google has been dicing with public perception for some time, perhaps riding the temporary situation of no obvious major challenger to the limits. This clearly cannot continue indefinitely.

The importance of webmasters in this scenario should not be underestimated. These are movers and shakers in the industry... major influencers. Some of the recent threads on this board demonstrate beyond doubt how webmaster perception has already started to shift.

Twelve months ago, Google was almost universally seen as infallible. Criticism was rare, heavy criticism almost unheard of. No more. Thread after thread is full of criticisms of Google quality. It is also generally believed that Google has had a series of bugs, some of which remain unfixed.

Despite GoogleGuy's best efforts (he's a Google Rep on the board), this is a seed change, and one which will have wider fallout in time.

To some degree Google is lucky that the main competitors are not yet ready for the challenge. That could change very quickly indeed though, and in my opinion at least, Google should be looking to manage perception a little more pro-actively.

Someone much wiser than me once said something like "It takes a lifetime to build a reputation, and minutes to lose one". If I was Google, I would take that to heart.

5. Neo: "I Don't Believe in that Fate Crap"

Neither do I. The ball is in play, and Google still has possession. Although significant medium term damage has already occurred, which will take a lot of effort to recover, Google can still shape the future if it reacts quickly enough to the warnings.

This latter point is the crux. Time is a commodity that will run down for the company sooner than it realizes if it fails to respond. Short terms measures for me would be: Beef up the QA department, sort out the current index (eg: Amazon), repair all the bugs (eg: index) and important... BE SEEN TO DO THIS.

Medium term (weeks not months) would be to create a strategy to grasp the initiative in terms of PR and perception. Explain a little more about the technological improvements (duel edged sword I know), herald the QA improvements (as just referenced), etc etc.

If they do nothing, but simply continue as they have been, by the time they IPO they will already be seen as an ailing company.

It's in their own hands.... and I for one hope they read it correctly. However, ultimately, they are likely have the future they deserve.


The item above was originally posted on a well known webmaster message board. It was immediately criticised (without rationale) by a 'moderator', and within 24 hours, was removed.

Interestingly, it had already received a substantial amount of positive feedback in its thread, and the author had received a steady flow of congratulatory emails.

Why was it removed? I offer no explanation... but it does make you think about what drives these boards.

Saturday, August 02, 2003


For me, the pending purchase of Overture by Yahoo is most interesting not because of the Pay Per Click (PPC) aspect, but for what else comes with it.

Yes, Yahoo will have its own PPC delivery system at a stroke. That fact alone though doesn't necessarily change the Yahoo onsite search landscape (although it may do so at MSN of course). The most interesting aspect is actually that Overture own Alta Vista and FAST, two substantial and highly respected search engines in their own right.

This gives Yahoo a set of three, because they already own Inktomi.

Three into one doesn't go, so the first big question is which search engine, or combination of search engines, Yahoo will use when it eventually gets to grips with all this technology, and uses it to replace Google (surely inevitable, given that Google is certainly seen as a competitor nowadays).

That's too difficult to call, but equally, what about the future of these engines, either as stand alone entities, or as providers for other portals? Will Yahoo merge them in some way, or simply replace one or even two of them? Of course there could be a degree of media management downstream, as they close one down, but with slight technical change to give the impression of merger.

All very murky and difficult to read. However, I would suggest that the likelihood of all three being maintained is quite low. I hope I'm wrong.


Well I'm back to the blog. I wish I could say that I had been on holiday, but I haven't. Instead I've been caught up in the stormy waters of Google, Summer 2003.

Anyone reading this log will understand that Google strayed from the norm a few months ago. The predictable monthly update never arrived. Instead, we were treated to a sequence of events that had many webmasters tearing their hair out. First there was the inactivity phase: new sites were simply not being collected, recorded, and displayed as expected.

Bad enough? It got worse!

Phase two included the 'missing index file' phenomena. For their main search terms (eg: company name) countless sites were AWOL - they didn't appear. Yes, there were common features to identify these sites, the most consistent being that they were under 2 years old. However, the impact was serious for most of those effected. Changes to content and link structure made little short term difference.

Serious webmasters of course analyzed this very closely. Even so, no consensus materialized. Camps were divided even at the high level plateau of whether this was actually an error, or a filter (algorithmic change). Evidence was substantial on both sides, which made the situation even more difficult to assess.

Then suddenly, Google changed again. The high quality of the search returns was re-established. Things seemed to resemble normality: companies appeared on their company names again; index pages appeared for searches as opposed to privacy statement pages; etc. Google was top of the SE pile again, almost at a stroke.

Unfortunately this didn't resolve all the ongoing disputes: Was it an error that had been fixed? Or was it a deliberate filter that had been withdrawn? If it was the latter of course.... it may re-appear again in the future!

I still believe though, on balance, that it was indeed caused by new technology related glitches. Somehow, I doubt we will ever know for certain.

Wednesday, June 25, 2003


The stress levels amongst webmasters and web promotion professionals are obvious at present. The message boards and newsgroups are humming with speculation, with a hint of trepidation.

Sometimes a Google employee steps into the fray and offers some calming insights and advice. Apparently not this time though - at least not thus far.

However, at times like this it is sometimes wise to look at the overall alternatives from a slightly higher perspective. I would suggest that there are THREE possibilities to explain the current search result flux (see footnote):

1) The 'Twilight Zone' proposition is correct, or on the correct lines. For anyone not familiar, this is the suggestion that Google is currently unable to manage all of the most recent link data, and shuffles it in and out of the index, perhaps using FreshBot. There are plenty of supporting arguments for this.

2) This is part of the expected and planned technology changeover cycle, in which case, the data centers will eventually align, and as the Google employee suggests, all the missing data will then be added. Normal looking search returns will eventually resume.

3) This is planned, and calculated, and permanent. No... I don't really want to think that either, because of the implications... these being that maybe the Google employee himself was mislead when he was told that there was no algorithmic tweak or filter affecting index pages (which he subsequently passed on to webmasters).

This would mean that in the bigger context the so-called Google 'social contract' has been unilaterally ripped up: Following the Google 'philosophy' of content-content-content and seeking relevant links will ultimately have been a pathway to oblivion for a lot of people (certainly on the most commercial terms). The rather sinister logic is that the main target terms have been purposefully selected by Google for this attention because they are of most commercial value. If the search results on these terms are relatively poor, this will drive up Adword clicks and ultimately revenue for the company. This conspiracy theory is far from new.

I'm personally sticking with the first two for now. Why? It's in the dates.

If this was a global tweak or filter of some sort, those sites whose link structures have been there for years would also be equally affected by the flux. They aren't. This fact actually re-enforces the 'trusted base' idea, with an outer twilight region (the TZ theory).

Also, for the second possibility, the Google employee did present a sort of roadmap to support it. His postings didn't preclude the first proposition by the way, but the roadmap did suggest that missing data would be added close to the end. But when is the end? Is it really in July as some people now suggest? Obviously I don't know the answer.

For the third, that type of theory has frequently raised its head before, but frankly I doubt it. The problem is, the longer things go unchecked, the more likely it will seem to many people, and the odds will ultimately change.

It's you're choice really... take your pick.

[FOOTNOTE: The main fluctuation is essentially that for many web sites the main index page doesn't feature on the most popular industry term. Sub-pages are available, but tend to attract much lower ranking.]

Friday, June 20, 2003


Life is always hectic when Google is re-indexing, but never previously as traumatic as with this cycle. Frankly, the index is all over the place.

Two issues have emerged:

a) Google has forgotten the importance of a site's index page. Hundreds of thousands of sites have had their main index page ditched or relegated on their main search terms. Sub-pages from the same sites still appear, but obviously with much lower ranking.

Webmasters are faced with their contact page or links pages (for example) being presented as the entry page for visitors. Ooops!

b) The index is still not up to date. It seems to have moved along slightly, but recent links are prone to come and go, almost within the hour.

The distinction between 'fresh' (news) and core index has become blurred. Core index fodder is being treated like temporary fresh fodder. Many new sites are still nowhere to be seen, or appear only fleetingly.

And no, it isn't just the little guy who is currently suffering. A major news group aggregation firm, EasyNews, for example, was recently forced to offer a reward to anyone who could help them out of the mire. Incredibly, they have been ditched on the search term 'Easynews'. They are far from being alone, even though other factors seem to apply in their case.

Trying to establish some pattern is proving to be extremely difficult, although some have tried. Indeed, there may not even be a pattern. Some of these features may indeed be unforeseen by Google itself, as it continues to wrestle with it's new software.

With the ins and outs constantly occurring, many webmasters are clinging to the hope that all the missing links/etc will be added as the final ingredient to the pot when the new index is distributed across all 9 data centers.

Only time will tell whether that will happen, but certainly, the waters are very rough at present. One would imagine that Google is aware that the currently situation isn't exactly healthy for them either - or maybe not.

Sunday, June 15, 2003


One of the worst occurrences for any webmaster or search engine optimizer is of course web site unavailability. Yes, it happens. Unfortunately, sometimes all too often, and sometimes for far too long. Not all web hosts are equal - FAR from it.

For my pains I have a significant number of hosts. Some are excellent, some are fine and some are... let's just say there is plenty of room for improvement.

The first group of hosts provide almost 100% availability, decent speeds, and are there when I need them. These are probably the big 3 needs in web hosting terms, and a host that fails to make the grade for any one of them is likely to be a real headache.

But how do you go about ensuring that you are not stuck with bad host? How do you know they are bad unless you actually try them? Well fortunately, there are certain tell tale signs.

Before I list a few though - a golden rule: NEVER let your host acquire and administer your domain name for you. That's a definite no no. If you hit problems downstream the last thing you will need is trouble getting out of there and pointing your DNS at another host's servers. Allowing your host to control your domain name is like blocking your own escape route. Don't even think about it.

OK.... what about that list? Consider the following:

1) Does the host have proper contact details on their web site? This MUST include a telephone number. Apart from the fact that dodgy hosts often don't give their number, you won't want to sit on your hands if your site is down and you get no response via email.

2) Check their reputation. Take a look at their own forum, if they have one, and somewhere like Do the homework.

3) Check out their infrastructure. If they are decent they are likely to have this information on their site. For an important site you are unlikely to want a reseller of a reseller of a reseller. Know something about where and how your site will physically be hosted.

4) Send them an email asking a question. How quickly did they respond? How professional was their response?

These four are an absolute minimum, and go side by side with cost and the package on offer. Ducking the responsibility of checking these things could well come back to haunt... your site could be down when Googlebot is crawling, or when a customer is buying. It's not a great feeling to be sat upon a lengthy outage with matters outside your control.


No sign of a traditional deepcrawl, but hints are emerging from the Plex that we may see a re-index based upon deeper than usual freshbot crawls in the next week or two. This will certainly help matters, at least to a degree, even though it may still be lacking the usual thoroughness in mapping the current web.

With all 9 data centers also largely aligned at present, the signs are good for at least some update activity in the very near future. We can only hope that this resolves most of the outstanding issues.

Thursday, June 12, 2003


I am sometimes asked why I rate Google so highly relative to most others. The answer is DNA (I'm not the first to use the term in this context by the way).

What I mean by that is simply that Google is a search technology company. It created the Google engine. It has fostered it, improved it, and expanded it over the years. It is obviously committed to it and is actually interested in search technology itself. As a result we see a product which is focused upon relevancy of returns - which is actually exactly what searchers want.

But what of the other main players?

Firstly, if you look at the main portals, they are not actually owned by search technology firms. MSN and AOL are also simply portals for example, using technology provided by others to deliver search functionality and other services. Yahoo owns its own directory, and has recently (relatively) bought the Inktomi search engine. But you still get the feeling that they see search as a side product rather than the core offering of the company (look at the way they have relegated the directory over the last couple of years).

All these firms offer search, but none have a core interest in driving search technology and search quality forward.

What about Alta Vista and AllTheWeb (FAST)? Certainly, these WERE search technology driven, but they have recently been bought out by Overture. Of course as a PPC (Pay Per Click) firm, Overture is very much an ADVERTISING company. That is its roots, and that is in its DNA. I don't really think it is going to suddenly acquire a cultural or academic interest in search technology.

Those two established engines will most likely have been procured to provide platforms for Overture PPC, or a platform to make PPC/Search bids to major portals. Sadly, I foresee a bleak future for both in terms of quality.

Of course there are other smaller search engines out there, some utilizing leading edge search technology and producing excellent results. The problems they face of course are visibility, scalability and market share. Some of them do have the right DNA though, and hopefully will eventually emerge to challenge Google in the quality stakes. Time, as ever, will tell.

However, at present, none of the leading players really has the focus or DNA to seriously challenge Google in terms of product quality. That isn't a healthy situation at all.


I've just been reading the Q&A by GoogleGuy at WMW. What always strikes me about it is not the information provided, which of course has to be limited, but that Google allows to happen. This open approach to stakeholder communication is actually very refreshing. It's a pity other major players aren't equally enlightened. It clearly benefits all parties.

This page is powered by Blogger. Isn't yours?
Link To Us

If so, please
do link to us
from your
blog or site

Thank You.


Google News

OK, the links are boring! We are new, so more coming soon

06/01/2003 - 06/30/2003
08/01/2003 - 08/31/2003
11/01/2003 - 11/30/2003
01/01/2004 - 01/31/2004





Please do
link to us
from your
blog or site

Thank You