Senior software engineer, ~11 years' experience. Most recently at Lambda (AI inference infra). Before that, ~7 years at Solvuu building a data science platform for genomics research and data analysis. Looking for senior backend / devops roles — full-time.

Location: NYC (preferred) or Orlando, FL

Remote: Yes

Willing to relocate: NYC or Orlando only

Technologies: Python, TypeScript, Go, Rust, OCaml, Haskell, Lean, AWS, Kubernetes, Terraform, Envoy, Docker

Resume: available upon DM

GitHub: https://github.com/rdavison

LinkedIn: https://linkedin.com/in/richardndavison

Email: richardneildavison@gmail.com

To me it sounds like a tooling problem. OP seems to be trying to use probabilistic text systems as if they enforce rules, but rule enforcement should really live outside the model. My sense is that there was a failure to verify the agent's intent.

The tooling that invokes the model should really define some kind of guardrails. I feel like there's an analogy to be had here with the difference between an untyped program and a typed program. The typed program has external guardrails that get checked by an external system (the compiler's type checker).

What tooling? It's a probabilistic text generator that runs in a black box on the provider's server. What tooling will have which guardrails to make sure that these scattered markdown files are properly injected and used in the text generation?

That's the million dollar question. Maybe have systems of agents that all validate each other's work? Maybe something needs to be done at the harness level? I don't suppose that we could realistically expect 100% accuracy, but if we take 100% to be the upper limit, we could build systems that get us closer to that ideal.

This is faith in magic. "There's some magic way to make probabilistic text generator running in the cloud to never miss local files"

No no, that’s not what I’m saying. The fact that the data is stored in files is incidental. It could be in a database, in a knowledge graph, derived from so other data Regardless of where it is, something should know to include it in the context, but only when it’s relevant.

So for instance you could start by trying to classify the prompt in some way. If you use an LLM for this, you might need to get it to return a machine parsable data format. Then your harness can pattern match on the classification and use it to enrich the prompt with additional context. The challenge would be in determining how exactly you want to go about this, balancing tradeoffs such as accuracy, cost, time, etc..

For the classification step you might begin with something like "Determine whether the following prompt is a QUESTION or a STATEMENT. Respond using only one of the two words. Prompt: $PROMPT"

You could have multiple back-and-forths like this and at each round you gain more information about the prompt, and you can use that information to determine further classifications and/or context to include.

> Regardless of where it is, something should know to include it in the context,

Magic. You're talking about magic. You keep re-iterating the same faith that "There's some magic way to make probabilistic text generator running in the cloud to never miss local files", where "files" is "files, knowledge graphs, databases etc.".

It doesn't matter how data is stored. You can't know when to include something relevant in the context because the whole thing including context is running in the cloud. You are not in the driver's seat. Literally anything you include locally in the prompt can and will be ignored.

I’m not following. If I run an agent on ollama locally, it’s not in the cloud. I don’t see what cloud has anything to do with the argument.

As to your other point about anything you include in the prompt can and will be ignored. Yes, I agree. You could draw an analogy to how a teacher assigns an in-class reading assignment and follows it up with a reading comprehension quiz. If your mind wanders during the reading you may come to find that you will fail the quiz because “anything you include in the prompt can and will be ignored”. Therefore, the quiz result serves the purpose of an evaluation.

> Why would anyone settle for

The answer to such questions is always that, given their circumstances, they have no realistic choice not to.

This is very obvious, and it's frustrating to continually see people pretend otherwise.

> If folks expect someone to solve problems for them

In this type of situation, the fundamental issue is that making progress depends on many people acting in unison to increase their bargaining power, which is (a) hard to arrange even if everyone who acted this way would benefit, and (b) actually may be detrimental to some people (usually the high performers).

I agree it is nearly impossible to alter the inertia of existing firms. Most have entrenched process people that defend how things are done right up until a company enters insolvency. Fine if you sell soda or rubber tires, but a death knell for technology or media firms.

In my observations it is usually conditioned fear, personal debt-driven risk aversion, and or failure to even ask if the department above you is really necessary. These days, it is almost always easier to go to another firm if you want a promotion. =3

Feels like we're living in parallel universes. You're building Hale, and I'm building Abs XD : https://github.com/rdavison/abs

Wait, what does NFTs have to do with RAG?

Nothing, I think they're just pointing out a seeming lack of awareness of what really is or isn't dead.

They were doing an ad hominem, thats what its called

You know you could just define the verified specs in lean and if performance is a problem, use the lean spec to extract an interface and tests for a more performant language like rust. You could at least in theory use Lean as an orchestrator of verified interfaces.

Strong agree. OCaml's compiler is sofa king good at catching and preventing real bugs that the agents accidentally introduce here and there. It's the same as with humans, except the agents don't complain about the bullshit reasons humans don't like OCaml. They just crank through and produce high quality output.

It's still in beta but I repackaged Descent Raytracer (a remaster of Descent (1995) made by students at Breda University) to be launchable on macs with Apple Silicon (ray tracing reqs M3+).

https://github.com/rdavison/DXX-Raytracer-ar/releases/tag/ar...

What a weird coincidence. Literally today I started building a fully keyboard driven MIDI sequencer in Rust. I was originally going to build it as a TUI but then decided against it because I wanted to have more control over the UI, so I'm building it as a pseudo-TUI with Bevy. But the idea is very similar, I'm approaching this project as a "vim"-like editor but for MIDI editing.

Here's my take. It helps you enforce properties about your data. Didn't mean to make this response so long, but alas.

  (* 
    Quick note on notation: I will use "double quotes" when referring to _values_ and `backticks` when referring to _types_.
  *)

  (* 
    Think of this as an interface. It defines the shape of a module. Notice that the interface describes a module that defines a type called `t`, and two values: "of_string", and "to_string", and they are functions with types: `string -> t`, and `t -> string`. 
  *)
  module type ID = sig
    type t
    val of_string : string -> t
    val to_string : t -> string
  end
  
  (* 
    Below this comment is a module named "Id" that _is of type_ (in other words: it _implements the interface called_) `ID`. Due to the explicit type annotation (Id : ID), now from the perspective of anywhere else in the code, the exported interface of the module "Id" is `ID`.
  
    Modules only contain two things: `type declarations`, and "values". Values are your primitives such as 1, '<', "hello", but also composite such as (fun x -> x + 1), (Some x), f x, { foo = "bar"; baz = 42 }, and even (module Id) (yes! modules can be values too!). Type declarations tell the compiler . Anything which is a value _always_ has a type that can _usually_ be inferred.
  
    No type annotation is necessary when the compiler correctly deduces the type of your value through static analysis. For instance, in the module below, "of_string" is deduced to be of type ('a -> 'a). The ' on the symbol 'a signifies a "type variable", and it means that it can be filled in with any type. For instance (t -> t) and (string -> string), but not (t -> string) or (string -> t). For those it would have to be of type ('a -> 'b). We cannot deduce this type, however, because our implementations do nothing with their inputs besides return them. Since nothing is changed, it's always the same type. 

    Now, can you spot the pink elephant? Notice how the "ID" interface from above defines "of_string" to be of type (string -> t). How can this be possible? It's because we gave the compiler a hint when we said `type t = string`. This says that a "value" of type `t` is backed by a value of type `string`. If something type checks as `t`, it also type checks as `string`. 

    So, we could reason through and say ('a -> 'a) can be instantiated to (t -> t), but `t` is also equal to `string`, so we can mentally imagine a hypothetical intermediate type... something like ({t,string} -> {t,string}). This type and type equality is visible _inside_ the module. But when the `ID` interface was applied over the `Id` module as in (Id : ID), this has the effect of hiding the type equality (the fact that `type t = string`) because in the `ID` interface we define `t` without an equals sign: `type t`. This forces us to _choose_ a concrete type to expose externally, even though the type is less general than what the implementation sees.
  
    NOTE: OCaml doesn't use parens for function definition or application. Compare this OCaml code against its Python equivalent.
  
    > let hello_world h w = (h, w)
    > let h, w = hello_world 1 2
  
    vs.
  
    > def hello_world(h, w):
    >   return (h, w)
    > h, w = hello_world(1, 2)
  *)
  module Id : ID = struct
    type t = string
    let of_string s = s
    let to_string s = s
  end
  
  let main () =
    let s = "abc123" in
    let id = Id.of_string s in
  
    (* NOTE(type error): because the built-in "print_endline" function is of type (string -> unit) and not (Id.t -> unit) *)
    (* NOTE: if an expression returns unit, you don't need to create a let binding for it. You can simply tack a semicolon to the end of it if you need sequence another expression to follow it. *)
    print_endline id;
  
    (* okay *)
    (* STDOUT: abc123 *)
    print_endline (Id.to_string id)
  ;;
  
  main ()

You could imagine implementing this pattern of defining parsers such as "of_string", "of_bytes", "of_json", "of_int", "of_db_row", "of_request", for any piece of input data. You can think of all of these functions as static constructors in OOP... you take in some data, and produce some output value: e.g. "of_string" takes in a `string` and produces a `t`.

Now, if you have a bunch of "values" of type `t`, you know that they _only_ could have been produced by the `of_string` function, because `of_string` might be the _only_ function that ends with `-> t`. Therefore, all the values maintain the same properties enforced by the `of_string` function (similar to class constructors in OOP). With this, you can create types such as `Nonnegative.t`, `Percent.t`, `Currency.t`, `Image.t`, `ProfilePicture.t`, and parsers from another type to the newly minted type.

The compiler can help you enforce these properties by providing guardrails in the form of static compiler checks (these checks are run _before_ your code can even be compiled). If I have a value of type `Nonnegative.t`, then not only do I not need to validate that it's not negative, I also don't have to validate that it's not negative everywhere else that values of that type are used -- the validation logic is baked into the constructor. Parse, don't validate.*