comments extraction from youtube using python seleniumAsk Question


I am using Python Selenium for comments extraction from youtube

from selenium import webdriver
browser = webdriver.Firefox()
for elem in browser.find_elements_by_xpath('//body'):
print elem.text

how to get comments?


The comments are in the divs with the class comment-renderer-text-content

for elem in browser.find_elements_by_xpath('//div[@class="comment-renderer-text-content"]'):
    print elem.text

Which gives you:

great stuff man. question: why use selenium for this site when the data you're looking for is in the source code and could be scraped with requests/beautifulsoup? disclaimer: i'm commenting a year later so the source code may be different :)
Good question, if the data is in source you're right, selenium is overkill.  I use selenium when I find it quicker to not have to reverse engineer a site looking for sever calls which return json data that only exists inside the browser etc...  So the bottom line is if you're really crafty picking off JSON calls to the server and replicating that without needing to have the DOM built for you than it's a much better be to use BeautifulSoup or Python Requests.   However if you're creating for instance an automated program to automatically pin, like stuff on facebook etc... you will most likely not be able to pull that off very easily just using BeautifulSoup. 
Answered my questions very well.
Great job! I do have a questions though. What if the site is built in silverlight? Then I cannot see the Xpath of each element...
the first test was slow because of a slow loading adserver, you can see it in firefox at the bottom bar.
This is good stuff.
Clear and useful although i'm using java. Thx
thanks man

The comments are loaded dynamically so you might need to wait for the presecnce of the elements:

from import By
from import WebDriverWait
from import expected_conditions as EC

def wait(dr, x):
    element = WebDriverWait(dr, 20).until(
        EC.presence_of_all_elements_located((By.XPATH, x))
    return element

from selenium import webdriver

browser = webdriver.Firefox()

for elem in wait(browser, '//div[@class="comment-renderer-text-content"]'):
    print elem.text
标签: selenium selenium-webdriver web-scraping
© 2014 TuiCode, Inc.