In this project, you are to create a web crawler class. See Chapter 12 in your book for an example Web Crawler to get you started. I do not cover that section of Chapter 12 in the lectures, so you will need to read that section on your own. Also I highly encourage you to use anything available to you from Maven repository, it will make this project much easier to implement. Two notes before we discuss what to do with the class: Make sure that you sleep for at least 0.05 seconds between hitting each link. This is to make sure that you do not ultimately DDOS any site that you wish to crawl. To get your program to sleep, check out this documentation from oracle: https://docs.oracle.com/javase/tutorial/essential/concurrency/sleep.htmlLinks to an external site. For the sake of not traversing advertisement sites, we will be using wikipedia links. This class needs to do two things: Have a function that traverses 1000 wikipedia links. Have a function that counts words, that is, every time you see a specific word, increment a number associated with that word. Note: Your function should count words only, not HTML elements / attributes. For example: This is a paragraph. and are HTML elements and should not be counted. id="myId", from inside the tag, is an HTML attribute and should not be counted. "This is a paragraph." is pure text output to the screen and should be counted as 4 separate words. I do not necessarily care how you implement this project so long as you have a class that, at the end of your 1000 wikipedia link traversal your class: - prints out the title of each page you've traversed (there is a HTML element that may exist on your site) - prints out a list of words encountered. Again, html elements / attributes should not be in this list (i.e. no '<p>' or '<div>'s should be in your list).

Computer Networking: A Top-Down Approach (7th Edition)

7th Edition

ISBN:9780133594140

Author:James Kurose, Keith Ross

Publisher:James Kurose, Keith Ross

Chapter1: Computer Networks And The Internet

Section: Chapter Questions

Problem R1RQ: What is the difference between a host and an end system? List several different types of end...

See similar textbooks

Related questions

Question

In this project, you are to create a web crawler class. See Chapter 12 in your book for an example Web Crawler to get you started. I do not cover that section of Chapter 12 in the lectures, so you will need to read that section on your own. Also I highly encourage you to use anything available to you from Maven repository, it will make this project much easier to implement.

Two notes before we discuss what to do with the class:

Make sure that you sleep for at least 0.05 seconds between hitting each link. This is to make sure that you do not ultimately DDOS any site that you wish to crawl. To get your program to sleep, check out this documentation from oracle: https://docs.oracle.com/javase/tutorial/essential/concurrency/sleep.htmlLinks to an external site.
For the sake of not traversing advertisement sites, we will be using wikipedia links.

This class needs to do two things:

Have a function that traverses 1000 wikipedia links.
Have a function that counts words, that is, every time you see a specific word, increment a number associated with that word. Note: Your function should count words only, not HTML elements / attributes.

For example:

This is a paragraph.

</p>

<p> and </p> are HTML elements and should not be counted. id="myId", from inside the <p> tag, is an HTML attribute and should not be counted. "This is a paragraph." is pure text output to the screen and should be counted as 4 separate words.

I do not necessarily care how you implement this project so long as you have a class that, at the end of your 1000 wikipedia link traversal your class:

- prints out the title of each page you've traversed (there is a <title> HTML element that may exist on your site)

- prints out a list of words encountered. Again, html elements / attributes should not be in this list (i.e. no '<p>' or '<div>'s should be in your list).

Expert Solution

Trending now

This is a popular solution!

Step by step

Solved in 2 steps with 3 images

SEE SOLUTION Check out a sample Q&A here

Knowledge Booster

Learn more about

Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-engineering and related others by exploring similar questions and additional content below.

Recommended textbooks for you

Computer Networking: A Top-Down Approach (7th Edi…

Computer Engineering

ISBN:

9780133594140

Author:

James Kurose, Keith Ross

Publisher:

PEARSON