Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Thanks for the reply. I'm going to look deeper at scipipe :). It's good to hear you're going to continue developing it.

What advantages does scipipe have over nextflow?



Thanks for your interest!

I should start with saying that Nextflow is a fantastic tool, with an very thriving community and lots of support and buy-in especially in the biomedical community.

But as you are asking, below are the points that I'm aware of, that motivates keeping me using and updating SciPipe:

Before going into the list though, I want to mention that at the time of publication of our paper [1], SciPipe was unique in comparison with Nextflow in allowing to create re-usable modules, which was our primary motivation for creating our own tool. Later Nextflow introduced re-usable modules in its DSL2.

Still, there are a few more factors weighing in, keeping me using and developing SciPipe:

1. SciPipe has a small and maintainable code base, between ~2k LOC last I checked. Nextflow's codebase is much larger AFAIK.

2. SciPipe has zero external dependencies, outside Go and Bash. Nextflow depends on the Java runtime, a Groovy interpreter, and the GPars library for concurrency.

3. SciPipe can compile pipelines to statically compiled binaries, for extremely simple deployment. Nextflow will always require the JVM + Nextflow to be installed.

4. SciPipe has per file audit reports. Nextflow still has per workflow reports only, AFAIK.

5. SciPipe does not require learning a new language, or installing new tooling (editors, syntax highlighting etc), apart from Go, a Go editor and related tooling.

6. SciPipe can handle more concurrent tasks. There are reports that Nextflow can not handle more than 512 concurrent tasks [2]. I have tested SciPipe with 4999 concurrent tasks (It then hit a limit of maximum subprocesses in Go, which I think might even be removed recently. Will have to test if I can go even further still).

7. Debugging SciPipe code can be done using Go debugging tools (Delve or just CGDB). Last I checked, using JVM-based debugging tools for Nextflow code was rather confusing to say the least, as you would end up deep in a stack of Groovy-parsing, and you would be pretty much left to do print-statement based debugging. I don't know if the situation has changed since then.

[1] https://doi.org/10.1093/gigascience/giz044

[2] https://www.nature.com/articles/s41598-021-99288-8




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: