I've recently released a comprehensive tool for managing config / env vars - https://dmno.dev. It is the only tool I've seen that natively understands monorepos. This means you can reuse config from the root or other services (See https://dmno.dev/docs/guides/schema/#sharing-config-between-...)

It is admittedly geared towards JS/TS stacks, but supports other languages by running commands via `dmno run -- yourcommand` which will inject the resolved config into your command as env vars. Generating types and adding helpers for other languages should not be too tough. We'll follow user demand as to which to implement first.

We are currently working on a tool that will help you run and manage your local dev services as well, but it is not ready yet. In the meantime, you can check out https://tilt.dev which is oriented around docker and k8s but does have some ability to just run arbitrary scripts. As for running scripts in general, I've had some success hacking together pnpm and turborepo, even to run scripts in non js/ts services. Lastly, I'll mention https://moonrepo.dev/ as it has more native support for polyglot repos, and its notion of task inheritance keep things fairly clean, especially in a larger repo with many services.

DMNO looks wonderful. I would love to use it for a monorepo which is using TypeScript/React for frontend services and Python for the backend. Native Python support please, `user_demand += 1`

Have just reorganized my personal monorepo to be totally based around nix: each package just has a default.nix and .envrc, with latter containing something like: "use flake .#package". The default.nix can be a regular package, npm package, python, common lisp, etc, and can reference anything else in the repo. The only housekeeping in flake.nix is adding lines like:

  packages.foo = pkgs.callPackages { };
  devShells.foo = pkgs.mkShell { inputsFrom = [ packages.foo ];

There are some special considerations around devShells: I don't need inputs on an npm package, for example, or that I want to include the asdf package itself within its own devShell for lisp. Have made a few wrapper funcs like "mkLispDevShell" and "mkNpmDevShell" to handle this.

This isnt perfect I know with all the housekeeping, but it balances itself out in terms of flexibility. For example, I can trivially create a relatively simply "assets" package/derivation that is both just a folder a files assured to be somewhere during a build that needs it, as well as be a simple lisp package that interfaces with those files. And prod builds can become almost entirely nix derivations that have inputs from across the repo and are defined by simple inline scripts: "nix run .#prod-server".

I've worked in a few monorepos, and organize my personal projects into a monorepo. The most common tool I have seen to manage them (and what I use for my personal stuff) has been bazel.

The first monorepo I worked in was a python/typescript monorepo. It largely grew organically, and was heavily iterated on. We had no tooling to manage the complexity, and constant bumped into sharp edges that made life difficult.

    - Dependencies change and silently break applications or developer environments.
    - Figuring out how to start up applications you didn't work on could be difficult because you needed to figure out the tool chains and how to invoke them. This painpoint came up everytime toolchains changed.
    - Figuring out where dependencies were used was really hard. This lead to lots of zombie code, we weren't sure if it was alive or dead.

There are a bunch of 'meta' tools for this, bazel, pants, buck, gradle, etc. Professionally I have only seen bazel used. I suspect its less to do with the tools themselves, but more to do with the ecosystem.

With that being said, bazel does seem to solve most of the sharp edges we had with our ad-hoc/no tooling, but it comes at a cost. Bazel is not trivial, and won't allow you to do some things that the underlying toolchain might have been okay with. So that may leave you making code changes to fit inside bazels constraints.

Bazel tooling also isn't equally mature across all languages. The go tooling is pretty mature, and there are great generators that 'make things work', the situation for frontend is less mature, but getting better. It will add additional friction that webdevs might not love.

At the end of the day though, I would choose to use something like bazel (or one of the alternates), and the friction it brings over not having it.

Example bazel frontend. https://github.com/bazelbuild/examples/tree/main/frontend/ne...

buck2 is arguably the state of the art currently, it is developed by meta based on the learnings from their own buck tool and bazel(/blaze), and others:

> The retroactively named Buck1 was a capable build system and is still in use today at Meta (although many users have migrated). Buck2 is a rewrite that aims to keep the best bits of Buck1 (with a high degree of target compatibility) but also borrows ideas from academic research and build systems, including Bazel, Pants, Shake, Tup, and more.

https://buck2.build/docs/about/why/

I'd say the two main tradeoffs to consider is do you want something very general-purpose, or is something more specific to certain language(s) appropriate? and do you want mature ecosystem, or bleeding edge solution?

Bazel/buck2 are very general and language agnostic, while something like nx is more focused on JS with some support for other languages. Bazel is pretty mature at this point, while with buck2 you might need to do more trailblazing yourself.

> do you want something very general-purpose, or is something more specific to certain language(s) appropriate?

This. At work, we use Bazel for the Java monorepo, and it's great (although pants2 seems like it might be better? Dunno, never used it...). Folks can convert individual classes to Kotlin, as they like to do, and it all just works.

The Go monorepo still uses standard Go tooling, with a couple makefiles and some scripts: there are certainly things that bazel/pants2 would do better, but the Go tooling is just so nice (and so fast) that there's a much higher activation energy required to give it up.

[Edit: adding...]

One note: probably the biggest thing about a monorepo is that you really need to invest in tooling to make your CI still go fast. Typically, that means (a) tracing out dependencies of a particular change, and testing only those things, and (b) partitioning the tests up intelligently and running them on lots of different machines.

IIUC, you could potentially get around having to do any partitioning or dependency tracing yourself by just using Bazel, with a bunch of Bazel workers running in the cloud, and then just do `bazed test //...` and let Bazel figure it out.

Otherwise, you're going to want to find or write something that divides your tests up.

I worked on tooling for a fairly large JDK based monorepo using vanilla gradle + some custom tooling. It worked well until we acquired companies and became more polyglot. If I were doing it again today I'd probably go with Bazel

Hmm I’ve always just done monorepos for organizational reasons, like forcing synchronicity between backend and frontend versions. The actual apps themselves have always been the same ol’ folders with the same ol’ build tools, deployed via individual commands (I personally use Dokku but I imagine all PaaS is similar). Can someone vaguely gesture at what I might be missing out on?

There's a decent (if fairly biased) comparison on monorepo tooling here: https://monorepo.tools/

Note that it's written by nx.dev (TS monorepo tool), so well worth taking the specific comparisons with a large grain of salt. Useful to understand what is out there though!

This might be helpful: https://monorepo.tools/

Although I'm not sure process management is touched on specifically. I've used docker compose for a similar purpose in the past, with source code in a volume to support live reloading.

I've been using Nx (https://nx.dev/) for a while now, but primarily on a front end set of apps currently. While it started as a TS/JS monorepo tool, it has evolved into something that (they claim) can be used for just about any language. You will probably still see it's roots as a TS/JS based tool though. It's not been without it's challenges, but overall I'm fairly happy with it.

NX works great if you are using the TS/JS stack (as we are in my company). We tried also adding our flutter repos into it and found it somewhat difficult to integrate.

Good to know, we have a Flutter repo and I have been wondering how hard it would be to integrate. That said, not sure there is a lot of benefit for having the Flutter / Web app in a monorepo, maybe reduced branch management when working on a feature that touches both?

As someone who had never used a monorepo before and wanted to set one up for the first time, https://www.pantsbuild.org/ seemed intuitive to me and of an apparently good provenance. Their Slack community also answered every question I had and righted my wrongs 100% of the time, which went a long way.

We have a very similar setup (except s/Next.js/Remix) and went with npm workspaces + Task [1], and it's working really, really well so far. This is our second iteration of a monorepo (first one was a straight React SPA bundled with Vite instead of Remix).

We have a root Taskfile that references our JS and Go taskfiles and wraps / abstracts all tasks. It's good for dependency management and task orchestration (`task lint` will run lint in both JS and Go for example, and `task test` does the same).

TypeScript sharing between packages can be a lil' funky, but so far it's super solid. Happy to share more if it would be helpful.

The nice thing is that this just relies on npm, which means setup is simple and straightforward, and we benefit from all of the stability and continual improvements npm brings to the table.

[1]: https://taskfile.dev/

Probably my best experience was when I was given a vm running somewhere. I could sync the remote folders to my own computer, but any change I made on that machine would trigger a build. I could also request ops to send traffic to that machine, and I'd get some mirrored production traffic.

I just used git in my home folder and it "just worked." Code reviews were pretty simple; the folder/file in the code review matched what was in your home folder.

It was pretty nice with a good .gitignore setup, so you could have your own dotfiles on the vm. There were also some really good scripts (you shared them by branch `git checkout jims-scripts -- ./scripts` or by commit—you didn't need review to commit new script files, but it was encouraged).

This might be more suited to monorepos where all projects are in JS/TS, but check out https://onerepo.tools

Makefile and docopt. No muss, no fuss. No "works on my machine" issues.

I'm running a hand-rolled monorepo with ansible, terraform, and several language projects (JS/TS, Swift, Ruby) that each have a makefile to standardize entrypoints for CI/CD. Docopt is nice because it's self documenting so you can better understand how all your apps are deployed.

In our projects we use Moon https://moonrepo.dev/moon extensively, it supports multiple languages and quite a few features.

For me personally I like that it updates Typescript references automatically and makes docker image setup far easier.

I really like what I’m seeing with moon, it seems very light weight but powerful. Although little immature.

I have this template project for Node.js/NPM monorepostories: https://github.com/bhouston/template-typescript-monorepo

This one I used a few times, might be useful for your case.

https://github.com/simplesurance/baur

IMO it’s language dependent.

Thinking back on my work experience I’ve worked on:

1. Pure JS/TS 2. JVM/Kotlin/Java 3. Polyglot Kotlin+Python 4. Polyglot JVM+Cpp+TS+Python

I’ll address each one and what I used at the time.

But first I’ll address my dev flow for a greenfield project. Initially after some cursory documentation I build a scaffold. Over the years my scaffold has pretty much boiled down to two main directories, packages and services. Packages expose functionality, services consume functionality from packages and distribute it over RPC. It’s served me well.As for service standup, it goes: local host to incubate it, docker compose to integrate it and develop , kubernetes to deploy it.

For the pure TS code base, at the time I used yarn workspaces and lerna. This worked well at the time and I had no complaints from what I can remember. I think I remember having to volume mount my whole repo into each individual service to get hot reloading working, which was no fun.

For Pure JVM stacks, I found gradle was the way to go. It seems like a pain in the ass at first but once you grok it, and stay on a stable version it rocks. You can orchestrate whole workflows in it.

Polyglot Kotlin and Python. So I went about this a couple ways until I settled on something I liked. First I said, ahh yeah gradle can do everything, so I wrote a poetry wrapper and had my python services pretend they were like JVM services. This ended up not working so well in the long term. Especially because python library sharing is a lot different than JVM. The I went for a hybrid approach where there were multiple layers. Gradle for JVM code, poetry for python code. This was good but switching context kinda sucked and there were too many commands to remember, and ctrl+r history and tab only gets you so far.

For the last one, Polyglot JVM, Cpp, Python, JS/TS… I think I finally have something I’m happy with (for now at least), like I feel productive and I don’t dread having to do like anything in the codebase. The final touch was, take the gradle for JVM, poetry for Python, use turbo repo with pnpm in a web directory for all your JS/TS code. I then tie everything together in a combination of gradle scripts but more importantly, a python tooling package that uses click cli and poetry. The python tooling package has logic for anything I want to script, and I expose a CLI for it. Then I have one of my modular make files expose the functionality by calling into the python package.

Seems like quite the contraption but it works.

Judging from other comments I might want to check out bazel.

Bazel (originating from Blaze inside Google) is great but very heavy/takes a lot of work to adopt. It's likely the gold standard if you're working with a large polyglot repo.

As a python dev, I've been frustrated with existing solutions, which is why we built Tach to solve some of the problems we've faced, particularly around organization and isolation - https://github.com/gauge-sh/tach

Pants is another great build system out of Twitter that has seen widespread adoption - https://www.pantsbuild.org/

The build system is a key component. If you are working primarily in one language with one set of consistent tools that work together, then you can likely get by with just using whatever tools your language offers. Apparently, people have great success in the JS/TS world with tools like pnpm/turborepo. It's totally fine and common to just put everything in one place with one build tool if that's all you really need.

Even if you only have like, 2 languages, often it is possible to stitch the tools together, if you don't go too far off the trodden path, and you understand the two well.

But that leads to the two most important questions: "Do you need to support multiple languages" (N = 3 or more I'd say) and "Do you need to support incremental caching" (including test results.) Once that happens, it becomes very difficult to properly express dependencies and cache artifacts, and it becomes increasingly difficult to expect every developer to consider all the possible interactions between the tools. That means builds take longer (no big cache), and it's easier to break them (missing dependencies) or make them slow (overly conservative dependencies).

If the answer to both of these questions is "Yes", then you need to take serious looks into solutions like Nix, Bazel, or Buck2 to manage your dependencies accurately and cache the results of your builds incrementally. All of these are, quite frankly, going to be difficult to adopt if you are not familiar with them. All of them can save you an incredible amount of pain and effort, too. It's sort of inherent to the problem space; building software is often more complex than we think -- part of the reason why simple single-language tools have so much appeal! -- and has a wide array of nuanced requirements.

Note that the common wisdom is that Nix/Buck/Bazel are often only needed at very large scale. That's true in the sense they are nearly mandatory, but there's a flip side, which is that it's often easiest to adopt those tools early on at a small scale, when there is the least amount of technical debt and smallest amount of impedance mismatch, and you can sort out those problems proactively. It's much easier than solving them all after the fact.

All of these tools will require some amount of dedicated engineering to keep them well maintained and oiled with all the modern conveniences (LSP support, packaging end-result artifacts, etc.)

my org has a pnpm and nx combo for the backend.

it works.

...but I've no idea how. i don't dare step off the beaten path. when it stops working, i call for help, loudly.

Interesting. But yeah, even with a very small number of tools, what ends up happening when you just kitbash them together is that the only way to truly understand the dependency graph and build requirements is "Be the person who wrote it in the first place" so this kind of thing where anyone stepping outside the box causes the card house to fall over, is... really common. I have feelings about this situation, but it is what it is.

I also take more of my experiences here from the open-source world, where "The build broke for me, what happened?" is often met with a response somewhere between "Works for me" to "You're on your own" or "Just rebuild everything." So, if you at least have someone dedicated who can help you when things go wrong, you're already ahead of the game!

docker-compose and shell scripts are my general go-to. I tend to push for the use of WSL for Windows environments, which helps with these kinds of tools. I've also used deno+shebang for shell scripting as well, which can work fine with the msys tooling (git-bash installed with git for windows).

For JS/TS, you can get a surprisingly capable monorepo with nothing more than pnpm + changesets.

Could you please elaborate on the tangible advantages of adopting a monorepo structure?