将爬取到的数据(用Python)写入PostgreSQL数据库

我想砍人。

这么简单,我特么搞了整整一天。从昨天下午到今天11:40.

WTF!

以下是代码(目的想把爬取到的数据存入PostgreSQL):

class PostgreSQLPipeline(object):
    def process_item(self,item, spider)
            #sql_desc="INSERT INTO postgresql_1(fullname, username, organization, mail, joined,followers,starred,following,popular_repos,popular_repos_download,popular_repos_star,popular_repos_info,home_page)values(item['fullname'], item['username'], item['organization'], item['mail'],item['joined'],item['followers'],item['starred'],item['following'],item['popular_repos'],item['popular_repos_download'],item['popular_repos_star'],item['popular_repos_info'], item['home_page'])"
        conn = psycopg2.connect(database="mypg", user="postgres", password="student", host="127.0.0.1", port="5432")
        try:
            cur=conn.cursor()
                #self.conn.query(sql_desc)
                #cur.execute("INSERT INTO ewrrw values(dict(item));")
            cur.execute("""INSERT INTO postgresql_1
                (fullname, username, organization, mail, joined, followers, starred, following, popular_repos, popular_repos_download, popular_repos_star, popular_repos_info, home_page)
                VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);""",
                (item['fullname'],
                item['username'],
                item['organization'],
                item['mail'],
                item['joined'],
                item['followers'],
                item['starred'],
                item['following'],
                item['popular_repos'],
                item['popular_repos_download'],
                item['popular_repos_star'],
                item['popular_repos_info'],
                item['home_page']),)

            conn.commit()
            log.msg("Data added to PostgreSQL database!",
                level=log.DEBUG,spider=spider)

        except Exception,e:
            print 'insert record into table failed'
            print e

        finally:
            if cur:
                cur.close()
        conn.close()
        return item

为什么搞了一天,。。。我哭啊

心痛,因为蠢呗。

以下7个错误 一直在出现,测试了近70次!!!才成功的将爬取到的数据存入PostgreSQL(前提先在数据库mypg里创建postgresql_1表):

1.insert record into table failed...

2.LINE 2: INSERT INTO postgresql_1(fullname, username,...

3.ProgrammingError: syntax error at or near "fullname"...

4.value too long for type character varying(100)...

5.psycopg2.InterfaceError: connection already closed...

6.psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block...

7.psycopg2.ProgrammingError: column "item" does not exist
    LINE 3:                 values(item['fullname'], item['username'], i...

写这个是因为我看到这些错误我快疯了,还好我有时间 慢慢的试啊试 试啊试 终于OK了。
将爬取到的数据(用Python)写入PostgreSQL数据库_第1张图片

你可能感兴趣的:(Scrapy学习笔记)