我想砍人。
这么简单,我特么搞了整整一天。从昨天下午到今天11:40.
WTF!
以下是代码(目的想把爬取到的数据存入PostgreSQL):
class PostgreSQLPipeline(object):
def process_item(self,item, spider)
#sql_desc="INSERT INTO postgresql_1(fullname, username, organization, mail, joined,followers,starred,following,popular_repos,popular_repos_download,popular_repos_star,popular_repos_info,home_page)values(item['fullname'], item['username'], item['organization'], item['mail'],item['joined'],item['followers'],item['starred'],item['following'],item['popular_repos'],item['popular_repos_download'],item['popular_repos_star'],item['popular_repos_info'], item['home_page'])"
conn = psycopg2.connect(database="mypg", user="postgres", password="student", host="127.0.0.1", port="5432")
try:
cur=conn.cursor()
#self.conn.query(sql_desc)
#cur.execute("INSERT INTO ewrrw values(dict(item));")
cur.execute("""INSERT INTO postgresql_1
(fullname, username, organization, mail, joined, followers, starred, following, popular_repos, popular_repos_download, popular_repos_star, popular_repos_info, home_page)
VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);""",
(item['fullname'],
item['username'],
item['organization'],
item['mail'],
item['joined'],
item['followers'],
item['starred'],
item['following'],
item['popular_repos'],
item['popular_repos_download'],
item['popular_repos_star'],
item['popular_repos_info'],
item['home_page']),)
conn.commit()
log.msg("Data added to PostgreSQL database!",
level=log.DEBUG,spider=spider)
except Exception,e:
print 'insert record into table failed'
print e
finally:
if cur:
cur.close()
conn.close()
return item
为什么搞了一天,。。。我哭啊
心痛,因为蠢呗。
以下7个错误 一直在出现,测试了近70次!!!才成功的将爬取到的数据存入PostgreSQL(前提先在数据库mypg里创建postgresql_1表):
1.insert record into table failed...
2.LINE 2: INSERT INTO postgresql_1(fullname, username,...
3.ProgrammingError: syntax error at or near "fullname"...
4.value too long for type character varying(100)...
5.psycopg2.InterfaceError: connection already closed...
6.psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block...
7.psycopg2.ProgrammingError: column "item" does not exist
LINE 3: values(item['fullname'], item['username'], i...