On Expert mode, I decided to ask it a simple question but in a niche language, to see how well it can scour the internet.
How do I emit JS object literals in a ClojureScript macro?
Instead I was given an answer to a completely unrelated question and it cited some "Learn ClojureScript" website. In short, it provided the following example.
But I was looking for (1) a macro, and (2) the JS object to be generated at compile-time, not run-time. Also, the stray comma is very weird, but thankfully commas are ignored. Concretely, I was expecting something like this
which will emit a call to `f` with the JavaScript array `[1 2 3]` at compile-time.
I know what the response will be to this comment: either "Clojure is a niche language, who cares?" or "get better at prompting." But otherwise, this is on-par with ChatGPT Plus, even when presented with the possibility to crawl Clojurians Slack archives, Stack Overflow, a bunch of blog posts, etc.
Your prompt definitely takes GPT-4 on the right path. Unfortunately, CLJS is still too niche (and/or the answer is buried too deeply in search results) that its suggested macro does not work.
It wants to unquote, presumably because macros are processed by the compiler, which exists in Clojure world, not JavaScript world, so the #js literal does not exist there, but unquoting will let us emit code that CLJS is happy with. Unfortunately, the tag doesn't actually do anything!
With all of that said, this is a VERY niche question, but it does not involve any macro magic whatsoever, and I'm sure most Clojure novices don't even know doing this is possible. It essentially requires two bits of knowledge: (1) macros run at compile-time, and (2) JSValue is an object container for native JS arrays and maps.
It's still impressive that GPT-4 was able to make a guess that looks right until you decide to experiment at the REPL.
Admittedly I am not very well versed in Clojure, I can understand only a little of what you are saying. But it seems to me that throwing more training data at the model should fix the issue.
I know what the response will be to this comment: either "Clojure is a niche language, who cares?" or "get better at prompting." But otherwise, this is on-par with ChatGPT Plus, even when presented with the possibility to crawl Clojurians Slack archives, Stack Overflow, a bunch of blog posts, etc.