Matt Thiessen » scrape http://matt.thiessen.us Learning to make the hard things easy Sun, 05 Feb 2012 04:06:14 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Using grep to scrape web pages http://matt.thiessen.us/2009/09/using-grep-to-scrape-web-pages/ http://matt.thiessen.us/2009/09/using-grep-to-scrape-web-pages/#comments Fri, 18 Sep 2009 15:38:20 +0000 matt http://matt.thiessen.us/?p=294 In preperation to scrape a number of web pages, I used grep to make a list of URLs I need to scrape.  The list of URLs was in an RSS file.

grep -P “\<link><\![CDATA\[(.*?)]” hawkeye_stories.xml > hawkeye_stories_links.txt

]]> http://matt.thiessen.us/2009/09/using-grep-to-scrape-web-pages/feed/ 0