Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers . If you believe the question would be on-topic on another Stack Exchange site , you can leave a comment to explain where the question may be able to be answered.

Closed 7 years ago .

Magpie ( https://github.com/llnl/magpie ) is a package of shell scripts used for running Big Data applications (such as Spark) on top of traditional HPC clusters (such as those managed by SLURM). It manages much of the complexity of queueing jobs for you, but you may still need to modify the submission scripts to suit your own Spark application - see the file submission-scripts/script-sbatch/magpie.sbatch-spark for more information.

As an interesting side-note, here's an article about LLNL's development of Magpie and the results they obtained on their HPC cluster: computation.llnl.gov/research/project-highlights/… – Gnat May 8, 2015 at 2:06

Just use Spark's standalone mode side-by-side with whatever Slurm does. This means allocating a static subset of resources on the target machines to each system.

Implement Slurm as a cluster option for Spark. This is best for true management and sharing of resources, but much harder because of you have to figure out Spark's clustering abstractions and implement them correctly. You could start by looking at how YARN or Mesos support is implemented.