I wonder how much better the captions would be with visual input. In addition to the ASCII representation, we could provide an actual image of the level/scene, and tell the LLM that the ASCII corresponds to it, before asking for a descriptive caption.
As an initial test, try doing this in the web interface for both ChatGPT and Claude and comment on the results here.
Then, the more ambitious thing to try is to automate the process in code. We can try this with Open AI, but only use a very small sample of levels, maybe 5 to start, since image processing will no doubt be very expensive.
However, open-source VLMs also exist:
https://ollama.com/library/llava:7b
https://ollama.com/library/moondream
https://ollama.com/richardyoung/smolvlm2-2.2b-instruct
I have no experience with any of these, but they are worth considering.
I wonder how much better the captions would be with visual input. In addition to the ASCII representation, we could provide an actual image of the level/scene, and tell the LLM that the ASCII corresponds to it, before asking for a descriptive caption.
As an initial test, try doing this in the web interface for both ChatGPT and Claude and comment on the results here.
Then, the more ambitious thing to try is to automate the process in code. We can try this with Open AI, but only use a very small sample of levels, maybe 5 to start, since image processing will no doubt be very expensive.
However, open-source VLMs also exist:
https://ollama.com/library/llava:7b
https://ollama.com/library/moondream
https://ollama.com/richardyoung/smolvlm2-2.2b-instruct
I have no experience with any of these, but they are worth considering.