Read the contents of an URL with Python

Python provides the urllib2 module that helps us in HTTP requests by defining functions and classes for redirecting, cookies, authentication and more.

Here’s an example to take the content of a page and store it in a string:

import urllib2

req = urllib2.Request("http://dot-maui.blogspot.it/")

response = urllib2.urlopen(req).read()

print response

With the urllib2 module you can also run a request with the POST method in Python:

import urllib
import urllib2

url    = 'http://www.my-site.com'
values = { 'id': '1','type': 'car' }
data   = urllib.urlencode(values)
req    = urllib2.Request(url, data)

response = urllib2.urlopen(req)

Or to make a GET request:

url    = 'http://www.my-site.com'
data = urllib.urlencode({"nome":"Maui"}) 
response = urllib2.Request(url + "?%s" % data)