Roxas Ventures Case Study
Evaluating AI Skills Across Real Work
- Industry
- Open Source
- Role
- Sole Designer / Builder
Capabilities
- Product Design
- Engineering
- Desktop App
My agency team started writing AI skills, reusable instructions for recurring work like drafting client emails or writing in a brand’s voice. When a draft missed the mark, we’d change a rule. That could fix the marketing post in front of us and make the next client email worse. We had no repeatable way to tell.
I designed, built, and released Skill Studio, an open-source desktop app that lets you test a skill on the work it’s meant to handle, compare revisions, and review the results yourself.
Test one change across several tasks
In the first version of this tool, AI rewrote the instructions and graded the new drafts. When a draft scored well but still felt wrong, I had to explain why and ask for another pass.
Even when the next draft was better, I still had to check what the new instructions would do on other tasks.
I designed the app around tasks users can save and rerun. For a brand voice skill, you might save a marketing brief and a client email brief. Running both with and without the skill shows what the instructions add.
After editing a rule, you can run the same briefs again and compare drafts from the old and new versions. If the post sounds sharper but the email becomes too casual, you can see the tradeoff before keeping the edit.
The AI judge picked the wrong draft
Skill Studio lets users compare drafts generated by different versions of an AI skill. In one test, I asked the app’s AI judge to choose between two thank-you notes. It picked the one that broke the two-sentence rule.
The judge had seen both drafts, but the app had only asked which was better. It hadn’t given the judge the original instructions.
I rebuilt the comparison around the task. The judge now sees what the user asked for and which requirements each draft should meet.
If its confidence is too low, the app shows that it was unsure and leaves the vote out of the rankings.
I also gave users a way to rate drafts before seeing the AI’s grades. That matters when a requirement is subjective.
If the AI gives a client email high marks for tone and the person who would send it finds it too formal, they can see the disagreement before using that grade to decide which version of the skill to keep.
Give quick edits and open questions different space
Writing a rule often meant leaving the editor, pasting it into Claude or ChatGPT, and bringing a rewrite back.
Sometimes the rule’s intent was clear and the wording needed work. Other times, the conversation helped me figure out what I wanted the AI to do in the first place.
That distinction shaped three actions in development.
Auto-refine checks it and offers one edit if it finds a problem.
Suggest changes offers three rewrites to compare. Both stay beside the selected text.
Help me improve opens a side panel and asks questions before drafting a rule from the answers.
The user decides when to apply a draft, and the original rule stays in view during the conversation.
What shipped and what I’m testing next
The first release turned a tool I ran from the command line into a desktop app others could install.
The next test is whether a teammate can bring in a skill they already use, choose tasks that reflect their work, and use the results to decide which revision to keep without my help.