Data processing: database and file management or data structures – Database and file access – Record – file – and data search and comparisons
Reexamination Certificate
2008-04-24
2011-11-29
Vital, Pierre (Department: 2156)
Data processing: database and file management or data structures
Database and file access
Record, file, and data search and comparisons
C707S723000, C707S748000
Reexamination Certificate
active
08069179
ABSTRACT:
The claimed subject matter provides a system that trains or evaluates ranking techniques by employing or obtaining relative preference judgments. The system can include mechanisms that retrieve a set of documents from a storage device, combine the set of documents with a query or judgment task received via an interface to form a comparative selection panel, and present the comparative selection panel for evaluation by an assessor. The system further requests the assessor to make a selection as to which document included in the set of documents and presented in the comparative selection panel most satisfies the query or judgment task, and thereafter produces a comparative assessment of the set of documents based on the selections elicited from the assessor and associated with the set of documents.
REFERENCES:
patent: 5535382 (1996-07-01), Ogawa
patent: 5808615 (1998-09-01), Hill et al.
patent: 6598058 (2003-07-01), Bird et al.
patent: 6766316 (2004-07-01), Caudill et al.
patent: 6829599 (2004-12-01), Chidlovskii
patent: 6886010 (2005-04-01), Kostoff
patent: 7111000 (2006-09-01), Wen et al.
patent: 7295967 (2007-11-01), Corman et al.
patent: 7647314 (2010-01-01), Sun et al.
patent: 7657507 (2010-02-01), Shi et al.
patent: 7685078 (2010-03-01), Chen et al.
patent: 2007/0073748 (2007-03-01), Barney
patent: 2007/0088695 (2007-04-01), Bleyendaal et al.
patent: 2007/0208730 (2007-09-01), Agichtein et al.
Chris Buckley, et al. Retrieval Evaluation with Incomplete Information. SIGIR'04, Jul. 25-29, 2004, Sheffield, South Yorkshire, UK. ACM 1-58113-881-04/04/0007. http:lldoi.acm.org110.114511008992.1009000. Last accessed 1125108, 8 pages.
J. Diez, et al. A Clustering Algorithm to Find Groups with Homogeneous Preferences h t tp://re sea rch. m ic ros oft. co m/-s d u m a is/SIGIR 2003/ExtendedAbstracts/BAHAMONDE—clustering—preferences.pdf.Last accessed Jan. 25, 2008, 4 pages. Last accessed Jan. 25, 2008, 4 pages.
Filip Radlinski, et al. Query Chains: Learn to Rank from Implicit Feedback, Research Track Paper, KDD'05, Aug. 21-24, 2005, Chicago, Illinois, USA. ACM 1-59593-135-X/05/0008. http://delivery.acm.org/10.1145/1090000/1081899/p239-radlinski.pdf?key1=1081899&key2=1504511021&coll=GUIDE&dl=GUIDE&CFID=51337974&CFTOKEN=47070968. Last accessed Jan. 25, 2008, 10 pages.
Filip Radlinski, et al. Evaluating the Robustness of Learning from Implicit Feedback. W4: Learning in Web Search, at the 22nd International Conference on Machine Learning, Bonn, Germany, 2005. http://www.cs.cornell.edu/People/tj/publications/radlinski—joachims—05b.pdf. Last accessed Jan. 25, 2008, 8 pages.
J. Diez, et al. A Clustering Algorithm to Find Groups with Homogeneous Preferences http://research.microsoft.com/˜sdumais/SIGIR2003/ExtendedAbstracts/BAHAMONDE—clustering—preferences.pdf. Last accessed Jan. 25, 2008, 4 pages.
T. Joachims, et al. Accurately Interpreting Clickthrough Data as Implicit Feedback, SIGIR'05, Aug. 15-19, 2005, Salvador, Brazil. ACM 1-59593-034-5/05/0008. http://www.cs.cornell.edu/People/tj/publications/joachims—etal—05a.pdf. Last accessed Jan. 25, 2008, 8 pages.
Ellen M. Voorhees. Evaluation by Highly Relevant Documents. SIGIR'01, Sep. 9-12, 2001, New Orleans, Louisiana, USA. ACM 1-58113-331-6/01/0009. http://www-ufrima.imag.fr/FORMATION/FILIERE/MASTER/SI/SiteMasterSI/Documents/MORI/p74-voorhees.pdf. Last accessed Jan. 25, 2008, 9 pages.
K. Jarvelin, et al. IR evaluation methods for retrieving highly relevant documents. Published in: Belkin, N.J., Ingwersen, P. and Leong, M.-K. (eds.) Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY: ACM, pp. 41-48. SIGIR 2000 7•00 Athens, Greece. ACM 1o58113-226-3/0010007. http://www.info.uta.fi/tutkimus/fire/archive/KJJKSIGIR00.pdf., Last accessed Jan. 25, 2008, 9 pages.
Ben Carterette, et al. Learning a Ranking from Pairwise Preferences, SIGIR'06, Aug. 6-10, 2006, Seattle, Washington, USA, ACM 1595933697/06/0008. http://ciir.cs.umass.edu/˜carteret/sigir06-poster.pdf. Last accessed Jan. 25, 2008, 2 pages.
Chris Burges, et al. Learning to Rank using Gradient Descent, Proceedings of the 22nd International Confer-ence on Machine Learning, Bonn, Germany, 2005. http://research.microsoft.com/˜cburges/papers/icml—ranking.pdf. Last accessed Jan. 25, 2008, 8 pages.
Brian Bartell, et al. Learning to Retrieve Information. In Current trends in connectionism: Proceedings of the Swedish Conference on Connectionism (LEA: Hillsdale, 1995), L. Niklasson and M. Boden, Eds. http://citeseer.ist.psu.edu/cache/papers/cs/44/http:zSzzSzwww-cse.ucsd.eduzSzuserszSzgaryzSzpubszSzsncc95.pdf/bartell95learning.pdf. Last accessed Jan. 25, 2008, 9 pages.
Stefano Mizzaro. Measuring the Agreement Among Relevance Judges, MIRA Conference, 1999 http://citeseer.ist.psu.edu/cache/papers/cs/20797/http:zSzzSzwww.dimi.uniud.itzSz˜mizzarozSzpaperszSzagreement-mira.pdf/mizzaro99measuring.pdf. Last accessed Jan. 25, 2008, 13 pages.
T. Joachims. Optimizing Search Engines using Clickthrough Data. SIGKDD 02 Edmonton, Alberta, Canada. ACM 158113567X/02/0007. http://www.cs.cornell.edu/people/tj/publications/joachims—02c.pdf. Last accessed Jan. 25, 2008, 10 pages.
Chris Buckley, et al. Retrieval Evaluation with Incomplete Information. SIGIR'04, Jul. 25-29, 2004, Sheffield, South Yorkshire, UK. ACM 1-58113-881-4/04/0007. http://doi.acm.org/10.1145/1008992.1009000. Last accessed Jan. 25, 2008, 8 pages.
Bennett Paul N.
Carterette Benjamin Austin
Chickering David M.
Dumais Susan T.
Lee & Hayes PLLC
Microsoft Corporation
Nguyen Loan T
Vital Pierre
LandOfFree
Preference judgements for relevance does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Preference judgements for relevance, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Preference judgements for relevance will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-4300280