Dropping stopwords and stemming/lemmatization process on a corpus (a set of text based documents) produce the following document space: “alone, animation, best, cold, enjoy, ever, filled, film, he, horror, i, love, most, movie, night, one, outside, overrated, put, plot, really, seen, story, scary, suspense, toy, twist, unexpected, watching”. If the corresponding set of text based documents consist of: D1 = I enjoy watching movies when it’s cold outside; D2 = Toy story is the best animation movie ever; D3 = Watching horror movies alone at night is really scary; D4 = He loves film filled with suspense and unexpected plot twists D5 = This is one of the most overrated movie I’ve ever seen.
Dropping stopwords and stemming/lemmatization process on a corpus (a set of
text based documents) produce the following document space:
“alone, animation, best, cold, enjoy, ever, filled, film, he, horror, i, love, most,
movie, night, one, outside, overrated, put, plot, really, seen, story, scary,
suspense, toy, twist, unexpected, watching”.
If the corresponding set of text based documents consist of:
D1 = I enjoy watching movies when it’s cold outside;
D2 = Toy story is the best animation movie ever;
D3 = Watching horror movies alone at night is really scary;
D4 = He loves film filled with suspense and unexpected plot twists
D5 = This is one of the most overrated movie I’ve ever seen.
and if the query document Q contains key terms watching best animation movies, perform hand calculation
to determine the rank of every single document above if the retrieval process is performed using BM25
approach. Please use k1 = 1.5, b = 0,75, and to avoid IDF with negative values please slightly modify IDF
equation (AIMA 3rd Edition, p.868-869) to become: IDF(qi)=loge(1+(N−DF(qi)+0.5)/(DF(qi)+0.5)) where N is
the number of documents and DF(qi) is the number of documents in the corpus that contain term qi.
Step by step
Solved in 2 steps
