天天看点

spark OutOfMemoryError (GC overhead limit exceeded)

spark 执行报告异常:

15/04/17 04:15:57 INFO scheduler.TaskSetManager: Lost task 471.0 in stage 1.0 (TID 459) on executor ip-172-1-1-2.cn-north-1.compute.internal: java.lang.OutOfMemoryError (GC overhead limit exceeded) [duplicate 2]

解决方法:

spark.executor.memory 调大参数扩大内存(默认512M,调整为2G)

继续阅读