(WEB SCRAPING WITH THE REQUESTS AND BEAUTIFUL SOUP LIBRARIES) Web pages are excellent sources of text to use in NLP tasks. In the following IPython session, you’ll use the requests library to download the www.python.org home page’s content. This is called web scraping. You’ll then use the Beautiful Soup library37 to extract only the text from the page. Eliminate the stop words in the resulting text, then use the wordcloud module to create a word cloud based on the text. 37. Its module name is bs4 for Beautiful Soup 4. In [1]: import requests In [2]: response = requests.get('https://www.python.org') In [3]: response.content # gives back the page's HTML In [4]: from bs4 import BeautifulSoup In [5]: soup = BeautifulSoup(response.content, 'html5lib') In [6]: text = soup.get_text(strip=True) # text without tags In the preceding code, snippets [1]–[3] get a web page. The get function receives a URL as an argument and returns the corresponding web page as a Response object. The Response’s content property contains the web page’s content. Snippets [4]–[6] get only the web page’s text. Snippet [5] creates a BeautifulSoup object to process the text in response.content. BeautifulSoup method get_text with the keyword argument strip=True returns just the text of the web page without its structural information that your web browser uses to display the web page.
12.1 (WEB SCRAPING WITH THE REQUESTS AND BEAUTIFUL SOUP LIBRARIES) Web pages are excellent sources of text to use in NLP tasks. In the following IPython session, you’ll use the requests library to download the www.python.org home page’s content. This is called web scraping. You’ll then use the Beautiful Soup library37 to extract only the text from the page. Eliminate the stop words in the resulting text, then use the wordcloud module to create a word cloud based on the text.
37. Its module name is bs4 for Beautiful Soup 4.
In [1]: import requests In [2]: response = requests.get('https://www.python.org') In [3]: response.content # gives back the page's HTML In [4]: from bs4 import BeautifulSoup In [5]: soup = BeautifulSoup(response.content, 'html5lib') In [6]: text = soup.get_text(strip=True) # text without tags
In the preceding code, snippets [1]–[3] get a web page. The get function receives a URL as an argument and returns the corresponding web page as a Response object. The Response’s content property contains the web page’s content. Snippets [4]–[6] get only the web page’s text. Snippet [5] creates a BeautifulSoup object to process the text in response.content. BeautifulSoup method get_text with the keyword argument strip=True returns just the text of the web page without its structural information that your web browser uses to display the web page.
Trending now
This is a popular solution!
Step by step
Solved in 2 steps
