使用urllib2和ClientForm下载表单中的文件

Question

我快被从这个网址下载CSV文件的事情搞疯了：

这个网站上有4个表单，我成功设置了正确表单上的日期，然后提交了这个表单，得到了正确的HTML响应。但是我想要下载的是CSV文件，而不是响应的HTML。我本以为需要先提交一个表单设置日期，然后再提交另一个表单选择CSV文件，但在第一次响应中我没有得到任何可以继续操作的表单。

这是我的代码：

#!/usr/bin/env python
import csv
from urllib2 import urlopen
from ClientForm import ParseResponse
import urllib2

proxy = urllib2.ProxyHandler({'http': '172.26.10.100:8080'})
# proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/2010010' \
    '1 Firefox/4.0.1',
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language':'en-us,en;q=0.5',
    'Accept-Charset':'ISO-8859-1,utf-8;q=0.7,*;q=0.7'}

# set the request
url = "http://www.opcom.ro/rapoarte/raportPIPsiVolumTranzactionat.php?lang=en"
request = urllib2.Request(url, None, headers)

try:
    response = urllib2.urlopen(request)

except urllib2.HTTPError, response:
    pass

print response.geturl()
print response.info() # headers
# print response.read() # body

# get forms from response
forms = ParseResponse(response, backwards_compat=False)
response.close()

# print "###FORMS: " ,len(forms)    
# for i in range(len(forms)):
    # print "@@@@@"
    # print forms[i]

form1 = forms[1]

#  setting a specific date in the form
form1.set_value("7", kind="text", nr=0)
form1.set_value("10", kind="text", nr=2)
form1.set_value("2011", kind="text", nr=4)
print form1

# # # SEND THE FORM
request2 = forms[1].click()  # urllib2.Request object      BIEN

try:
    response2 = urllib2.urlopen(request2)

except urllib2.HTTPError, response2:
    pass

print response2.geturl()
print response2.info() # headers
# print response2.read() # body
with open('salida.txt', 'w') as f:
    f.write(response2.read())

forms2 = ParseResponse(response2, backwards_compat=False)
response2.close()

print "###FORMS 2: " ,len(forms2)

注意，第一个表单（日期选择器）在表单数组中是forms[1]，而forms[2]是选择CSV文件或XML的下拉框。选择CSV的代码是：

# form2 = forms2[2]


# # Select CSV file in selection control
# form2.find_control("menu_sari").items[1].selected = True  # check

不过我把这部分注释掉了，因为在响应后我没有得到这个表单。

任何帮助或反馈都非常欢迎。

Answer 1

在查看源代码时，你会发现当你填写表单后，会调用一个php脚本来生成csv文件，最后会弹出一个对话框让你保存这个文件。

比如说，当你在这个网址上：

http://www.opcom.ro/rapoarte/raportPIPsiVolumTranzactionat.php?lang=en

如果你选择了日期13/1/2012并选择csv格式，接下来会调用这个网址：

http://www.opcom.ro/rapoarte/export_csv_raportPIPsiVolumTranzactionat.php?zi=13&luna=1&an=2012&limba=ro

以下的代码对我有效：

>>> import urllib2
>>> url = "http://www.opcom.ro/rapoarte/export_csv_raportPIPsiVolumTranzactionat.php?zi=13&luna=1&an=2012&limba=ro"
>>> request = urllib2.Request(url)
>>> response = urllib2.urlopen(request)
>>> print response.read()
"PIP si volum tranzactionat pentru ziua de livrare: 13/1/2012"

"","Pret mediu [lei/MWh]","Volum [MWh]"
"ROPEX_DAM_Base (1-24)","185.23","31226.488"
"ROPEX_DAM_Peak (7-22)","239.68","22773.036"
"ROPEX_DAM_Off_Peak (1-6) & (23-24)","76.32","8453.452"

"Interval","Pret de Inchidere a Pietei [lei/MWh]","Volum Tranzactionat [MWh]"
"1","29.99","876.148"
"2","70.00","1057.729"
"3","50.00","1058.868"
"4","50.00","1044.700"
"5","50.00","1061.574"
"6","61.71","1015.513"
"7","86.08","1070.586"
"8","181.00","1187.392"
"9","222.00","1434.829"
"10","230.00","1515.633"
"11","262.60","1539.495"
"12","226.00","1538.931"
"13","225.00","1559.273"
"14","271.00","1515.113"
"15","266.42","1513.220"
"16","250.00","1534.506"
"17","263.02","1481.099"
"18","298.00","1351.114"
"19","283.42","1336.266"
"20","280.00","1398.646"
"21","271.00","1450.183"
"22","219.34","1346.750"
"23","170.00","1219.772"
"24","128.88","1119.148"

>>> 

看看这个是否对你有帮助。如果你发现表单无法正常工作，你可以按照他们的网址格式自己构建网址。