【Pyspark】报错：Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6fb1741000,

1.报错：

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6fb1741000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.

 

原因：

在使用pyspark的时候中间自己加了cache，导致内存爆炸 

解决方案：

不需要的地方不用加cache

 

 

2.

pyspark: ValueError: Some of types cannot be determined after inferring

pyspark进行pandas转dataframe的格式互相转换或者创建临时表的时候，如果数据为空，或者存在整列为null 的时候会报此错

 

解决方法：

设定好每列的数据格式

https://stackoverflow.com/questions/40517553/pyspark-valueerror-some-of-types-cannot-be-determined-after-inferring

from pyspark.sql.types import StructType, StructField, StringType
schema = StructType([StructField("col1", StringType(), True)，StructField("col2", StringType(), True)，StructField("col1", StringType(), True)])
df = spark.createDataFrame(df_, schema=schema)