python模型转PMML

关于python模型的部署,目前有以下几种方式

  • flask等python为服务框架,无需跨语言
  • xgb4j,lgb4j等Java包,需跨语言,但只支持xgb/lgb
  • PMML,跨语言,支持所有sklearn接口的模型

综上所述,当遇到跨语言部署时,PMML是个万金油方式,可以将所有sklearn接口的模型转换为PMML文件,并用JAVA/SCALA相关的包进行解析,然而经过一番调研,网上关于python如何转为PMML的信息却极为有限,故在此总结。

1、DataFrameMapper

  • 目前DataFrameMapper支持sklearn.preprocessing中的若干类,如MinMaxScaler()、OneHotEncoder()等
  • DataFrameMapper支持自定义函数,可使用FunctionTransformer(),将自定义函数转换为类似MinMaxScaler()类的格式
  • DataFrameMapper支持单列或多列级联变换
  • sklearn.preprocessing中的函数输入为numpy.ndarray
    mapper = DataFrameMapper([
              (["Sepal.Length"],FunctionTransformer(np.abs)),
              (["Sepal.Width"],[MinMaxScaler(), Imputer()]),
              (["Petal.Length"],None),
              (["Petal.Width"],OneHotEncoder()),
            (['Petal.Length', 'Petal.Width'], [MinMaxScaler(),StandardScaler()])
    ])

2、PMMLPipeline

  • PMMLPipeline中支持整体变换类,如PCA、SelectKBest、GBDT等,只要符合sklearn接口格式,具有fit transform即可
  • 理论上支持符合规则的自定义函数
    iris_pipeline = PMMLPipeline([
        ("mapper", mapper),
        ("pca", PCA(n_components=3)),
        ("selector", SelectKBest(k=2)), #返回k个最佳特征
        ("classifier", GBDT)])
    iris_pipeline.fit(df_x, y)

3、sklearn2pmml

保存为PMML文件
sklearn2pmml(iris_pipeline, savemodel, with_repr=True)

其他注意事项

  • 由于DataFrameMapper对特征工程支持有限,特征工程可以线上线下分开单独做,也可以用 DataFrameMapper 的方式实现特征工程,导出到模型文件中,这样线上就无需再做一遍特征工程

完整代码

"""
文件说明:鸢尾花数据集
"""
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn2pmml import sklearn2pmml, PMMLPipeline
from sklearn2pmml.decoration import ContinuousDomain
from sklearn.feature_selection import SelectKBest

# frameworks for ML
from sklearn_pandas import DataFrameMapper
from sklearn.pipeline import make_pipeline

# transformers for category variables
from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import Imputer

# transformers for numerical variables
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import Normalizer

# transformers for combined variables
from sklearn.decomposition import PCA
from sklearn.preprocessing import PolynomialFeatures

# user-defined transformers
from sklearn.preprocessing import FunctionTransformer

def read_data():
    #读取鸢尾花数据
    data=load_iris()
    x=data.data
    y=data.target
    df_x = pd.DataFrame(x)
    df_x.columns = ["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"]
    return df_x,y


def all_classifiers_test(savemodel='GBDT.pmml'):
    '''
    GBDT模型
    '''
    GBDT = GradientBoostingClassifier()
    df_x,y = read_data()
    # 特征工程
    mapper = DataFrameMapper([
              (["Sepal.Length"],FunctionTransformer(np.abs)),
              (["Sepal.Width"],[MinMaxScaler(), Imputer()]),
              (["Petal.Length"],None),
              (["Petal.Width"],OneHotEncoder()),
            (['Petal.Length', 'Petal.Width'], [MinMaxScaler(),StandardScaler()])
    ])

    # mapper = DataFrameMapper([
    #         (["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"], [MinMaxScaler(),StandardScaler(),Imputer()])
    # ])

    iris_pipeline = PMMLPipeline([
        ("mapper", mapper),
        ("pca", PCA(n_components=3)),
        ("selector", SelectKBest(k=2)), #返回k个最佳特征
        ("classifier", GBDT)])
    iris_pipeline.fit(df_x, y)
    # iris_pipeline.fit(X_train.values, y_train)
    # 导出模型文件
    sklearn2pmml(iris_pipeline, savemodel, with_repr=True)

你可能感兴趣的:(机器学习and数据挖掘)