Web crawler scheduler that utilizes sitemaps from websites

Details Web crawler scheduler that utilizes sitemaps from websites Web crawler scheduler that utilizes sitemaps from websites

2010-06-25

2011-10-11

Lu, Kuen (Department: 2167)

Data processing: database and file management or data structures

Database and file access

Search engines

Reexamination Certificate

active

08037054

ABSTRACT:
Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

REFERENCES:
patent: 5935210 (1999-08-01), Stark
patent: 5958008 (1999-09-01), Pogrebisky et al.
patent: 6124966 (2000-09-01), Yokoyama
patent: 6144959 (2000-11-01), Anderson et al.
patent: 6269370 (2001-07-01), Kirsch
patent: 6271840 (2001-08-01), Finseth et al.
patent: 6285999 (2001-09-01), Page
patent: 6321265 (2001-11-01), Najork et al.
patent: 6418433 (2002-07-01), Chakrabarti et al.
patent: 6421724 (2002-07-01), Nickerson et al.
patent: 6424966 (2002-07-01), Meyerzon et al.
patent: 6516337 (2003-02-01), Tripp et al.
patent: 6525748 (2003-02-01), Belfiore et al.
patent: 6636854 (2003-10-01), Dutta et al.
patent: 6732105 (2004-05-01), Watson, Jr. et al.
patent: 6957383 (2005-10-01), Smith
patent: 6976053 (2005-12-01), Tripp et al.
patent: 6983282 (2006-01-01), Stern et al.
patent: 7133870 (2006-11-01), Tripp et al.
patent: 7139747 (2006-11-01), Najork
patent: 7191210 (2007-03-01), Grossman
patent: 7774782 (2010-08-01), Popescu et al.
patent: 7844610 (2010-11-01), Hillis et al.
patent: 2002/0032772 (2002-03-01), Olstad et al.
patent: 2002/0052928 (2002-05-01), Stern et al.
patent: 2002/0061029 (2002-05-01), Dillon
patent: 2002/0087515 (2002-07-01), Swannack et al.
patent: 2002/0138582 (2002-09-01), Chandra et al.
patent: 2003/0028896 (2003-02-01), Swart et al.
patent: 2004/0030683 (2004-02-01), Evans et al.
patent: 2004/0093327 (2004-05-01), Anderson et al.
patent: 2004/0122686 (2004-06-01), Hill et al.
patent: 2004/0158617 (2004-08-01), Shanny et al.
patent: 2004/0168066 (2004-08-01), Alden
patent: 2004/0221289 (2004-11-01), D'Souza et al.
patent: 2005/0060286 (2005-03-01), Hansen et al.
patent: 2005/0256865 (2005-11-01), Ma et al.
patent: 2006/0004691 (2006-01-01), Sifry
patent: 2006/0070022 (2006-03-01), Ng et al.
patent: 2006/0080405 (2006-04-01), Gibson
patent: 2006/0106866 (2006-05-01), Green et al.
patent: 2006/0212451 (2006-09-01), Serdy et al.
patent: 2007/0011168 (2007-01-01), Keohane et al.
patent: 2008/0021904 (2008-01-01), Garg et al.
“Technorati: Ping Configurations,” http://web.archive.org/web/20040829035832/www.technorati.com/de..., Aug. 2004.
“SOAP Meets RSS,” http://blogs.law.harvard.edu/tech/soapMeetsRss, Jul. 17, 2003.
“The Open Archives Initiative Protocol for Metedata Harvesting,” Ver. 2.0, http://www.openarchives.org/OAI/openarchivesprotocol.html, Jun. 14, 2002.
“Hermetic Sitemap Builder,” http://www.hermetic.ch/smb.htm, pp. 1-3 [cited by examiner, no date supplied or on document].
“Yahoo! Free Sitemaps,” http://www.seroundtable.com/archives/002421.html, Aug. 2005, p. 1-2.
“Yahoo! Sitemap Feed Submission . . . Worth the Effort?” http://www.antezeta.com/yahoo/site-map-feed.html, 2005, pp. 1-7.
“What are Sitemaps?” www.sitemaps.org, Aug. 2005, 1 page.
“Archive for the ‘Site Explorer’ Category,” http://www.ysearchblog.com/category/site-explorer/page/3/, 2005, pp. 1-3.
“The Optimizer-Weekly SEO News,” http://www.increased-online-traffic.com/2005/8/yahoo-adopts-site-maps-urllisttxt.asp, Aug. 23, 2005, pp. 1-5.
“Archive for the ‘Site Explorer’ Category,” http://www.ysearchblog.com/category/site-explorer/page/2/, Jun. 2, 2006, p. 1-11.
“Build your Site Map Online,” http://www.xml-sitemap.com, 2005, pp. 1-2.
Microsoft Compute Dictionary, Fifth Edition, © 2002, 3 pages.

Affiliated with

Also associated with

Rating

Web crawler scheduler that utilizes sitemaps from websites does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Web crawler scheduler that utilizes sitemaps from websites, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Web crawler scheduler that utilizes sitemaps from websites will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-4292693