I wonder if similar performance can be achieved with Spark accelerator like http... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		whoevercares on July 31, 2024 \| parent \| context \| favorite \| on: Amazon's exabyte-scale migration from Apache Spark... I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?

spott on Aug 1, 2024 [–]

It sounds like a lot of the value here was in a more appropriate level of abstraction, not in the speed of the compute itself.

Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact