Spark

Post thumbnail
Post thumbnail
Spark Tuning and Cluster Sizing Whether dynamic allocation is on, even with dynamic allocation, executor size needs to be determined Some core settings spark.driver.memory spark.executor.memory spark.executor.cores driver core = 1 in client mode, core can be set in cluster mode Memory Overhead executor overhead = spark.yarn.executor.memoryOverhead driver (cluster mode) =... [Read More]
Tags: Learning Spark