airflow执行py文件；pyspark窗口函数、pandas_udf、udf函数；spark提交集群任务
参考：https://blog.csdn.net/hahayikeshu/article/details/103552631
1、airflow
安装
pip install apache-airflow
airflow initdb
airflow scheduler
web端口
----- start the web server, default port is 8080
airflow webserver -p 8080
启动任务
1、把执行py文件拷贝到airflow/dags/下
2、运行任务调度
airflow unpause dag_id 启动
airflow pause dag_id 暂停
unpause相当于在web页面操作点击on、off；红色为失败，黄色为重启中，绿色为执行中，深绿是成功执行完
案例，文件放在/usr/local/airflow/dags下，airflow会自动识别
主要：
1>> scheduler要确保先运行起来，不然后面识别不了taskid和调度不起来
2 >> dags文件产生新的文件保存，可能需要进行权限赋予，sudo chomd 777 dags
3 >> /usr/long.txt '必须使用绝对路径，不然会保存不了
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
import pytz
#default_args = dict()
import pendulum
tz = pendulum.timezone("Asia/Shanghai")
#tz = pytz.timezone('Asia/Shanghai')
# naive = datetime.strptime("2018-06-13 17:40:00", "%Y-%m-%d %H:%M:%S")
# local_dt = tz.localize(naive, is_dst=None)
# utc_dt = local_dt.astimezone(pytz.utc).replace(tzinfo=None)
dt = datetime(2021, 8, 13, 14, 55, tzinfo=tz)
#utc_dt = dt.astimezone(pytz.utc).replace(tzinfo=None)
default_args = {
'owner': 'airflow',
'start_date': dt,
'depends_on_past': False,
'email': ['1***[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=1)
}
#每5分钟执行一次
dag = DAG(dag_id='bash_test',default_args=default_args,schedule_interval='*/5 * * * *')
task_1 = BashOperator(
task_id='bash_1',
bash_command='echo "hello world" >> /usr/long.txt ',
dag=dag
)
#获取参数的值
cmd = """
{% for i in range(4) %}
echo 'hello'
echo '{{ds}}'
echo '{{params.my_param}}'
{% endfor %}
"""
task_2 = BashOperator(
task_id='bash_2',
bash_command=cmd,
params={'my_param':'task_2'},
dag=dag
)
task_1.set_downstream(task_2)
SparkSubmitOperator可能出问题，建议使用BashOperator方便
****使用 Jinja 作为模版,可以传参形式：
参考：https://www.thinbug.com/q/42016491
https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#templating-with-jinja
****依赖关系：
参考：https://cloud.tencent.com/developer/article/1831455
t1.set_downstream(t2)
# 这意味着 t2 会在 t1 成功执行之后才会执行
# 与下面这种写法相等
t2.set_upstream(t1)
# 位移运算符也可用于链式运算
# 用于链式关系 和上面达到一样的效果
t1 >> t2
# 位移运算符用于上游关系中
t2 << t1
# 使用位移运算符能够链接
# 多个依赖关系变得简洁
t1 >> t2 >> t3
# 任务列表也可以设置为依赖项。
# 下面的这些操作都具有相同的效果:
t1.set_downstream([t2, t3])
t1 >> [t2, t3]
[t2, t3] << t1
t2.set_upstream(t1)
t3.set_upstream(t1)
# 等价于 t1 >> [t2, t3]
****airflow cron时间表达式
schedule_interval 是任务时间设定:与Linux cron 时间是不同的
airflow cron 表达式: * * * * * * (分 时 月 年 周 秒)
0 0/1 * * * 每小时
0/50 * * * * 每50分钟
***Trigger DAG 点击是触发重新启动整个task
2、pyspark
窗口函数
参考：https://database.51cto.com/art/202101/639239.htm
https://blog.csdn.net/liuyingying0418/article/details/108025262
**窗口函数最后结果总行数不会变
# spark = SparkSession.builder.appName('Window functions').getOrCreate()
employee_salary = [
("Ali", "Sales", 8000),
("Bob", "Sales", 7000),
("Cindy", "Sales", 7500),
("Davd", "Finance", 10000),
("Elena", "Sales", 8000),
("Fancy", "Finance", 12000),
("George", "Finance", 11000),
("Haffman", "Marketing", 7000),
("Ilaja", "Marketing", 8000),
("Joey", "Sales", 9000)]
columns= ["name", "department", "salary"]
df = spark.createDataFrame(data = employee_salary, schema = columns)
df.show(truncate=False)
窗口函数格式：
from pyspark.sql.window import Window
import pyspark.sql.functions as F
windowSpec = Window.partitionBy("department").orderBy(F.desc("salary"))
df.withColumn("row_number", F.row_number().over(windowSpec)).show(truncate=False)
udf函数
udf：
# udf cid+vendor
def new_cid(col1, sidx1, res1):
if col1 == "p1":
ress = eval(res1)[int(sidx1)]
newcid = str(ress[0]) + str(ress[1])
return newcid
else:
ress = eval(res1)[int(sidx1)]
newcid = str(ress[0]) + str(ress[1])
return newcid
cid_udf = F.udf(new_cid, StringType())
df = df.withColumn("new_cid", cid_udf("pageid", "sidx", "results"))
pandas_udf
参考：https://blog.csdn.net/weixin_42902669/article/details/104677557
https://blog.csdn.net/weixin_40161254/article/details/91548469
注意：
1）pandas_udf 传入和返回都是pd series格式
2）pandas_udf 传入的是batch数据，不是一条一条的内容
from pyspark.sql.functions import pandas_udf, PandasUDFType
df = spark.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],("id", "v"))
def subtract_mean(pdf):
v = pdf.v
return pdf.assign(v=v - v.mean())
sub_spark = pandas_udf(f=subtract_mean, returnType="id long, v double", functionType=PandasUDFType.GROUPED_MAP)
df.groupby("id").apply(sub_spark).show()
from pyspark.sql.functions import pandas_udf,PandasUDFType
from pyspark.sql.types import IntegerType,StringType
slen=pandas_udf(lambda s:s.str.len(),IntegerType())
@pandas_udf(StringType())
def to_upper(s):
return s.str.upper()
@pandas_udf(IntegerType(),PandasUDFType.SCALAR)
def add_one(x):
return x+1
df = spark.createDataFrame([(1, "John Doe", 21)], ("id", "name", "age"))
df.select(slen("name").alias("slen(name)"), to_upper("name"), add_one("age")).show()
df.withColumn('slen(name)',slen("name")).show()
3、spark提交集群任务
spark-submit提交任务集群计算：
nohup spark-submit --conf spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC --conf spark.yarn.executor.memoryOverhead=6G --conf spark.debug.maxToStringFields=100 --master yarn --deploy-mode cluster --executor-memory 4G - -executor-cores 6 --driver-memory 8G User***Play.py
查看集群执行log:
yarn logs -applicationId applic******id值**0_77969