Google, Yahoo, MSN to Standardize Site Maps

[Kelvyn]

It seems that Yahoo & MSN have now decided that Google had a good idea when they recommended xml sitemaps and introduced the tools to implement them. See this news item.

For more information on Google sitemaps see here.

[iamback]

Originally Posted by Kelvyn

It seems that Yahoo & MSN have now decided that Google had a good idea when they recommended xml sitemaps and introduced the tools to implement them. See this news item.

Good news - of a sort. But a pity that these sitemaps don't incorporate the ROR formats, created specifically to make it easy to create feeds (including site map feeds) that are accessible to all search engines - not just Google, as they say, but now they should say "not just Google, Yahoo and MSN"...

Why can't we all agree on a single format? Yet another proprietary format (even if it's shared by three - only three! - search engines instead of used by a single one) is not really the way forward.

Of course, given that it's XML, you could merge them all... but what a bother!

[dthomsen8]

Next week I intend go generate a Google sitemap for a website that I am upgrading to use the PayPal shopping cart. Does this announcement mean that I can generate the Google sitemap and then submit it to Yahoo! and MSN also?

[iamback]

Originally Posted by dthomsen8

Next week I intend go generate a Google sitemap for a website that I am upgrading to use the PayPal shopping cart. Does this announcement mean that I can generate the Google sitemap and then submit it to Yahoo! and MSN also?

You should not need to submit anything anywhere but have the site map available at a defined place for the search engines - all search engines! - to find by themselves!

[Kelvyn]

Originally Posted by dthomsen8

Next week I intend go generate a Google sitemap for a website that I am upgrading to use the PayPal shopping cart. Does this announcement mean that I can generate the Google sitemap and then submit it to Yahoo! and MSN also?

You can now submit the .xml sitemap to Google using the webmaster tools and to Yahoo as a feed (in each case you need accounts with them and validate site ownership). MSN does not yet seem to have a system in place for this, but their new MSN live search is in a mess, anyway. Yahoo used to require a .txt sitemap, but now also accept .xml as a feed. Note that the sitemap should be used to inform the SEs of the update periods for pages, so that frequently updated pages are spidered more often, and static pages less often to reduce bandwidth. This usually means editing the file produced by most generators before SE notification.

For the minor SEs I still use a simple XHTML sitemap, and many generators will produce this at the same time as the .xml map.

[iamback]

Originally Posted by Kelvyn

You can now submit the .xml sitemap to Google using the webmaster tools and to Yahoo as a feed (in each case you need accounts with them and validate site ownership).

This is where it's completely backward. There should be no need to submit anything at all! Just put the sitemap in an agreed place (like we do with robots.txt files), or link to it, so both humans and bots can find and use it. No need to prove site ownership either, that way.

I have a small (incomplete) sitemap at google, as an experiment, but I am seriously considering removing it. And putting in a ROR sitemap (maybe integrated with this new format); ROR can do it all and then some.

[dthomsen8]

Originally Posted by iamback

...
I have a small (incomplete) sitemap at google, as an experiment, but I am seriously considering removing it. And putting in a ROR sitemap (maybe integrated with this new format); ROR can do it all and then some.

How did this sitemap experiment work out, Marjolein? There is a new thread by a newbie about SEO, and I want to mention sitemaps in that thread.

[iamback]

Originally Posted by dthomsen8

How did this sitemap experiment work out, Marjolein? There is a new thread by a newbie about SEO, and I want to mention sitemaps in that thread.

First, I need to say that due to the way my completely dynamic site is structured, it's not realistically possible to construct a "complete" site map. I tried a generator - but again because of the way teh site is structured, it needs to work by spidering (no way to do any director & file-based site map), and that wasn't very successful. So I started with a "necessary basic" portion in a manually-constructed flat-file. (The disadvantage, of course, is that any extension to the site would require manual updating of the sitemap file as well.)

At first, google bots got busy. It was like they had renewed interest, and indexed many more pages than were actually in the sitemap file: they retrieved the resources in the map but followed links as well. Great.

Then for whatever reason, google quietly stared dropping pages again. Right now it has only 107 pages in its index, which is actually fewer than there are lines in my sitemap file!

Meanwhile, Yahoo! and MSN have much more "complete" content indexed, based purely on spidering (and incoming links) - thousands as opposed to a measly 107, which seems to remain stable even after I added new entries for my Yemen trip. It looks like it just maintains those 107 pages, and no longer follows any links.

So, the sitemap got me exactly nowhere - Google had more pages indexed before that. Most of my incoming hits from searches are now from Yahoo! search. AdSense results have gone down, too - but of course that's also directly related to the fact that Google doesn't really know what top show for pages that it doesn't have indexed. And because Yahoo! has many more pages indexed, I also just signed up for Yahoo! site search (which I've yet to integrate) - it doesn't look pretty but it gets relevant results, based on a few trials when configuring it.

(And I see I'm not the only one grumbling about decreasing relevance of Google search results for my own searches. I've - also recently - now switched to Yahoo! as my default search tool...)

I have no idea what will happen when I pull the sitemap file, but it can't really get much worse. But I'm planning to only do that when I launch a new release with some different page-building algorithms.

[Steve Rindsberg]

Ah, hang on. The article Kelvyn linked to sends you on to sitemaps.org whose protocol specifies several methods of informing the search engine of the whereabouts of the site map file. One of those seems to be exactly what you're suggesting (quoted from sitemap.org):

Specifying the Sitemap location in your robots.txt file

You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line:
Sitemap: <sitemap_location>

The <sitemap_location> should be the complete URL to the Sitemap, such as: http://www.example.com/sitemap.xml

This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don't need to list each individual Sitemap listed in the index file

==========================================

Whether Google, Yahoo et al actually abide by this provision is another matter. Anybody have any idea?

[back again after poking at google]

Google says that you can add

Sitemap: http://yourdomain.com/full_url.xml

to your robots.txt file and they'll use it. They still recommend registering the sitemap with them as well so you can use their webmaster tools.

[Steve Rindsberg]

Originally Posted by dthomsen8

Next week I intend go generate a Google sitemap for a website that I am upgrading to use the PayPal shopping cart. Does this announcement mean that I can generate the Google sitemap and then submit it to Yahoo! and MSN also?

That makes sense to me, at least most of the time. But how would you deal with a situation where you actually have several "subsites" under the same directory tree and don't want the site maps of the two mixed up?

Then again, I suppose it'd work anyhow, since

http://www.mysubsite.com/

would point to

/myserver/mydirectory/mysubsite/index.html

and the spider would look in the same directory as index.html for the site map.

OK. Never mind. You have my vote.
Marlene for Empress of the Internet.
Where do I get my t-shirt and bumpersticker?