Hi All,

I'm starting this thread so that we can share tips, experiences and best practices how to use AI models to speed up development of AROS or software for AROS/Amiga.

If you use AI tools to augment your AROS/Amiga development, please share your experiences, what worked and what didn't work and anything that can be useful in this regard. Please keep this thread focused on AROS rather then generic AI development.

First I will present use of GitHub Copilot to analyze code and solve simple bugs. Copilot is available for anyone with Github account and provides 50 free requestes per month. After you pay $10, the amount increases to 300 requests per month. The key functionality is that you can point Copilot to a specific Github repository (like a fork of AROS repository) and have it base it answers on that codebase, rather then on its generic knownledge.

See below as I'm asking it about an issue with "RAM". The prompt is described in user terms, not so much technical terms. I didn't describe what RAM was, but AI was smart enought to figured it out from the codebase and locate the issue just by analyzing code. Note that this time it was first-time right, but it's not always the case. Once it makes a suggestion, you need to make the edit and see what is the effect. Then describe it back to AI and it will continue analysis. My longest loop so far was 10 interations until the AI found the cause of the bug.

So how do you start with Github Copilot?

First, log in. Then click on the Copilot icon next to search bar:



Then select the repository it will use for knowledge (1) select model (2) and start chatting.



Note that the model selection is actually the heart of Copilot. I really can't stress this enought. Some models will be amazing, while other models will be poor. If you are just using the 50 free requests, you can choose between GTP5-mini and Claude Haiku 4.5. In my experience, Haiku model is much better than GPT5-mini. Stick to it for now.

Copilot is a whole platform. Using the chat option is the most straighforward one to start with and there are other options for more advanced usage.

Here is another example of using Copilot. Once the issue is understood, it can quickly generate unit tests. I specifically point it to an existing unit test so that it doesn't try to be creative with code, but follows the exiting template. Then I can take such a generated file, change or remove what I don't need and can quickly increase AROS code coverage.

PS. Thanks to terminills and Kalamatee for inspiration and help on AI usage topic.

Continuing exploration of Copilot I decided to test different available models with another example reported bug. This bug is specific as it results in crash in text.datatype while the original source of the issues is located in a different, but related, module - amigaguide.datatype. It is a use-after-free bug and and is caused by caching information on text.datatype level which is manipulated (created and freed) by amigaguide.datatype.

I tried following 4 models: OpenAI GPT-5 mini, Claude Haiku 4.5, Claude Sonnet 4.6 and Claude Opus 4.6. The conversation with each of the models is attached at the end of this post.

Cost of using

GPT-5 mini and Haiku 4.5 models are available in Free tier (50 requests per month). To use Sonnet 4.6 and Opus 4.6 a subscription is need ($10/month => 300 requests per month). Each of the models has a different price, suggesting their relative capabilities. GPT-5 mini costs 0 requests, Haiku 4.5 - 0.33 requests, Sonnet 4.6 - 1 request and and Opus 4.6 - 3 requests. Speed-wise, GPT-5 mini, Haiku and Sonnent were comparable in generating answers. Opus was taking around 2x longer.

Communication style

All Claude models have a similar style - they reply is concise, they produce one or two possible answers and described reasoning behind them. I would call that style "just enough". GPT-5 mini is more verbose. It produced several possibilities when describing either the problem or a solution. Also those answers felt more theoretical opposed to more practical answers of Claude models. GPT-5 produced answers more of a type "here is what you can check yourself" rather then "here is where the problem is" of Claude models. For me personally GPT-5 mini output was "too much to read".

Locating the issue

Opus and Sonnet models were able to locate the issue and assess it as use-after-free within the first prompt reply. GPT-5 mini model gave several answers in first prompt - use-after-free was there also, but the description wasn't as clear as with Opus and Sonnet models. Haiku also suggested use-after-free problem, but it's answer felt more random and it also added another issue, which was unrelated and insisted getting back to that second issue.

Solving the issue

GPT-5 mini provided more of a description on how the issue can be solved, rather then specific code. The description contained a portion that was relevant, but a bigger portion of output was just confusing the core of the issue. In order to use GPT-5 mini, the person would already have to have a good understanding of code base.

Haiku 4.5 kept suggesting theoretical solutions that were related to codebase and gave general idea of what needs to be solved, but not how. Haiku also wasn't able to move on from text.datatype (where problem is visible) to amigaguide.dataype (where the problem is located).

Sonnet 4.6 provided a solution that is not the worst one but also not the greatest one. It correctly identified amigaguide.datatype as the location of the issue.

Opus 4.6 took it's time thinking about a solution (around a minute) but eventually located the correct place and generated a fix almost identical with my hand-made fix.

Summary

GPT-5 mini behaved more like a lecturer then a helper programmer. Haiku 4.5 behaved like a junior software engineer who wants to boast his very wide (but very shallow) knowledge. It was quick to reply, but replies were lacking depth. Though if you are using Copilot Free tier, I'd still go with Haiku 4.5 over GPT-5 as Haiku's answers are much more actionable (and they both cost 1 request in Free tier).

If you are using Copilot subscription and you have a rough idea of what you are doing, I'd go with Sonnet model. It's more expensive then Haiku, but where Sonnet located the issue and proposed a solution in 2 prompts (=2 requests), Haiku was still going in circles after 4 prompts (1,3 request).

It's also important to note that this comparison is done a single, slightly more advanced use case. If you have your own experiences with Copilot or other AI tools in AROS development, please share them here!

https://axrt.org/media/copilot_example_03.zip