RubyFlow The Ruby and Rails community linklog

×

The Ruby and Rails community linklog

Made a library? Written a blog post? Found a useful tutorial? Share it with the Ruby community here or just enjoy what everyone else has found!

I gave a generic LLM access to Ruby documentation through RAG.

by Giménez Silva Germán Alberto —

https://rubystacknews.com/2026/06/04/turning-a-generic-llm-into-a-ruby-expert-what-rag-fixed-and-what-it-didnt/

The hallucinations didn’t disappear. They changed. The most interesting failures happened when the model had the correct context and still answered incorrectly.

Context is necessary, but not sufficient.

Ruby #AI #RAG #LLM

Comments

What are you using to get hallucinations in 2026???

Marcus S —

While they do occur on SOTA models, they are still pretty common on smaller models you might run locally, especially the smaller you go. So the OP’s technique isn’t a bad idea, especially if you’re relying on some <30B model to code for you, because their raw “intelligence” and tool calling is very good now, and they will often be happy to use tools to look up knowledge rather than make stuff up.

Peter Cooper —

Because I like to try things in extremis, I rigged up the frankly useless knowledge-wise Qwen 3.5 0.8b with an eval to test its knowledge (which was terrible) and then gave it a tool it was directed to use when it didn’t know something, and it did very well. I wouldn’t use such a model for coding, but I just wanted to validate a hunch. I haven’t tried it, but I suspect with the right tooling and knowledge base around it, something like a 9B model could punch far above the raw model’s weight.

Peter Cooper —

Post a comment

You can use basic HTML markup (e.g. <a>) or Markdown.

As you are not logged in, you will be
directed via GitHub to signup or sign in