Great Expectations

---

Greate Expectation Website: https://greatexpectations.io/.
Greate Expectation Document: https://docs.greatexpectations.io/docs/.
Greate Expectation Github: https://github.com/great-expectations/great_expectations.

---

1.Introduction

1.1 Introduction

• 一共三点：验证，记录和分析。
  

1.2 SLACK

• 可以进slack社区提问。
  

1.3 Integrations

Integration		Notes
	Pandas	Great for in-memory machine learning pipelines!
	Spark	Good for really big data.
	Postgres	Leading open source database
	BigQuery	Google serverless massive-scale SQL analytics platform
	Databricks	Managed Spark Analytics Platform
	MySQL	Leading open source database
	AWS Redshift	Cloud-based data warehouse
	AWS S3	Cloud based blob storage
	Snowflake	Cloud-based data warehouse
	Apache Airflow	An open source orchestration engine
	Prefect	An open source workflow management system
	Other SQL Relational DBs	Most RDBMS are supported via SQLalchemy
	Jupyter Notebooks	The best way to build Expectations
	Slack	Get automatic data quality notifications!

1.4 What does Great Expectations not do

2.Demo File

• install great_expectation
  sudo pip3 install great_expectations
• 看看安装路径
  sudo python3 -m site
  /usr/local/python3/bin
• 查询版号
  /usr/local/python3/bin/great_expectations --version
• 初始化init
  /usr/local/python3/bin/great_expectations init
  
• 创建软连
  sudo ln -s /usr/local/python3/bin/great_expectations /usr/bin/great_expectations

2.1 Connect DB

• 初始化init
  /usr/local/python3/bin/great_expectations datasource new --no-jupyter
  enter option 1
  enter option 1
  
  

2.2 create expectation

• 另开窗口，继续执行
  /usr/local/python3/bin/great_expectations suite new
• select
  enter option 3
  enter option 1
  
• Enter the file name
  Name the new Expectation Suite [yellow_tripdata_sample_2019-01.csv.warning]: getting_started_expectation_suite_taxi.demo
  
• open Jupyter Notebook
  直接访问提示的地址，进入Jupyter Notebook
• run validate
  /usr/local/python3/bin/great_expectations checkpoint new my_checkpoint

3.RDB

3.1 Connect DB

• install great_expectation
  sudo pip3 install great_expectations
• 看看安装路径
  sudo python3 -m site
  /usr/local/python3/bin
• 查询版号
  /usr/local/python3/bin/great_expectations --version
• 初始化init
  /usr/local/python3/bin/great_expectations init
  
• 连接DB
  /usr/local/python3/bin/great_expectations datasource new --no-jupyter
  enter option 2
  enter option 1 =>我用的mysql
  
• 我是用python3，所以要手动执行
  sudo pip3 install psycopg2-binary
  
• 重新执行上一步
  /usr/local/python3/bin/great_expectations datasource new --no-jupyter
  
• 按照提示继续
  jupyter notebook /home/os-nan.zhao/great_expectations/uncommitted/datasource_new.ipynb --allow-root --ip 0.0.0.0
  
• 浏览器访问红框中的地址
  
• 将token输入，enter new password
  datahub@123
• 点进datasource_new.ipynd
  sudo pip3 install pymysql
  sudo pip3 install pymssql

#----第三步----
host = "YOUR_HOST"
port = "3306"/"1433"
username = "YOUR_USERNAME"
password = "YOUR_PASSWORD"
database = "YOUR_DATABASE"
schema_name = "YOUR_SCHEMA"


#----第四步----
example_yaml = f"""
name: {datasource_name}
class_name: Datasource
execution_engine:
  class_name: SqlAlchemyExecutionEngine
  credentials:
    host: {host}
    port: '{port}'
    username: {username}
    password: {password}
    database: {database}
    schema_name: {schema_name}
    drivername: mysql+pymysql/mssql+pymssql    =>对应前面的port
data_connectors:
  default_runtime_data_connector_name:
    class_name: RuntimeDataConnector
    batch_identifiers:
      - default_identifier_name
  default_inferred_data_connector_name:
    class_name: InferredAssetSqlDataConnector
    include_schema_name: True"""
print(example_yaml)

3.2 create expectation

• 另开窗口，继续执行
  /usr/local/python3/bin/great_expectations suite new
• select 2
  enter option 2
  
• Index of the table of which you want to create the suite
  enter option 10
  
• Enter the file name
  demo01
  
• 晕死，没有开8889的port
  这个datahub，真难提前开好所有port
• 编辑
  /usr/local/python3/bin/great_expectations suite edit --no-jupyter
  jupyter notebook /great_expectations/uncommitted/edit_.ipynb --allow-root --ip 0.0.0.0
• 执行
  /usr/local/python3/bin/great_expectations checkpoint new --no-jupyter
• next
  jupyter notebook /great_expectations/uncommitted/edit_checkpoint_.ipynb --allow-root --ip 0.0.0.0

3.3 Run on Jupyter Notebook

4.Waken

         在一秒钟内看到本质的人和花半辈子也看不清一件事本质的人，自然是不一样的命运。