by unignorant on 3/5/24, 7:56 PM with 8 comments
by yorwba on 3/5/24, 9:04 PM
This doesn't indicate that Claude is unfamiliar with Circassian, only that Circassian is sufficiently rare that refusing to answer is a plausible response.
The language is not that obscure in the grand scheme of things, there's a Wikipedia article explaining the grammar https://en.wikipedia.org/wiki/Kabardian_grammar which is definitely in Claude's training set, probably alongside a few hundred linguistics papers and a bunch of monolingual data.
If you measured the performance for different numbers of initial translation examples, I suspect that there will be a sudden jump at the point where Claude stops refusing to even try, and after that additional examples will only marginally improve the output.
by dkarras on 3/5/24, 10:26 PM
first week ChatGPT was public more than a year ago, I tried the early model to make it play along with me inventing a new programming language with novel attributes in syntax. after some back and forth, it could translate my javascript samples to the new programming language paying attention to the new language's semantics, and could even simulate running simple pieces of code. Sure, it had some token errors here and there but it was working. It was understanding what I was telling it, and responding in kind.
Over the months since, things only got better. So I'm not surprised with the results of this post, but still astonished the same.
by Imnimo on 3/5/24, 11:02 PM
by derelicta on 3/6/24, 10:37 AM