Python Web Scraping ———08.03.2017

Postgresql database and data structure.

Just write down what I've learned about web data scraping so that I won't forget everything and start all over next time I need to use the technique. 

How to set Primary Key postgresql:

--Firstly, remove PRIMARY KEY attribute of former PRIMARY KEY 

ALTER TABLE … DROP CONSTRAINT ..._pkey;

--Then change column name of  your PRIMARY KEY and PRIMARY KEY candidates properly.

ALTER TABLE ... RENAME COLUMN … TO id;

--Lastly set your new PRIMARY KEY

ALTER TABLE … ADD PRIMARY KEY(id);

How to Insert if not exists and do nothing if exists:

INSERT INTO invoices(invoiceid,billed)SELECT'12345','TRUE'

WHERE NOT EXISTS(SELECT 1 FROM invoices WHERE invoiceid='12345');

-----------------------------

command = ("""

INSERT INTO test3(dateTime, random) SELECT %s, %s WHERE NOT EXISTS (SELECT 1 FROM test3 where random='dddddddddd') # Keep in mind no parenthesis around %s

""")

a = "dddddddddd"

try:

cur.execute(command, (dt_utc,a))

Today I have really learned a lot, including working with postgis, how to create geometry column, how to make point, add point to specific geo column.

To install the postgis extension, you might need owner privilege to do so:

CREATE EXTENSION postgis;

Therefore, you will need to grant access to the user:

GRANT ALL PRIVILEGES ON TABLE stations TO janton;

To drop primary key:

ALTER TABLE sample.public.employee DROP CONSTRAINT employee_pkey

To delete all rows from table:

DELETE FROM (tbl_name)