Apache Spark 1.3 Architecture - Module Spark-core

After spending a significant time in reading the source code in spark-core project, I can briefly draw the architecture showing the relationships and the flow (messages passed) between important components in this module:

alt

See you in my next posts for more details on them. I believe that it’s extremely important to understand following components: schedule, shuffle and storage.

Updates:

Leave a Comment