Clamor: Extending Functional Cluster Computing Frameworks with Fine-Grained Remote Memory Access
Pratiksha Thaker, Hudson Ayers, Deepti Raghavan, Ning Niu, Philip Levis, and Matei Zaharia
Published in Proceedings of the ACM Symposium on Cloud Computing (SoCC 2021), November 2021.
Abstract
We propose Clamor, a functional cluster computing framework that adds support for fine-grained, transparent access to global variables for distributed, data-parallel tasks. Clamor targets workloads that perform sparse accesses and updates within the bulk synchronous parallel execution model, a setting where the standard technique of broadcasting global variables is highly inefficient. Clamor implements a novel dynamic replication mechanism in order to enable efficient access to popular data regions on the fly, and tracks fine-grained dependencies in order to retain the lineage-based fault tolerance model of systems like Spark. Clamor can integrate with existing Rust and C++ libraries to transparently distribute programs on the cluster. We show that Clamor is competitive with Spark in simple functional workloads and can improve performance significantly compared to custom systems on workloads that sparsely access large global variables: from 5X for sparse logistic regression to over 100X on distributed geospatial queries.
Paper (1MB)
BibTeX entry
@inproceedings{thaker21-clamor, author = "Pratiksha Thaker and Hudson Ayers and Deepti Raghavan and Ning Niu and Philip Levis and Matei Zaharia ", title = "{Clamor: Extending Functional Cluster Computing Frameworks with Fine-Grained Remote Memory Access}", booktitle = "{Proceedings of the ACM Symposium on Cloud Computing (SoCC 2021)}", year = {2021}, month = {November} }





Login