Do models know how accurate they are?
A colleague was using an agent to classify paragraphs as titles, sections, figure captions, etc. He also asked it “How confident are you of your answer?”. Interestingly, the confidence was consistently high, typically ~95%. That got me curious. (Actually, no that didn’t get me curious - I saw Germayne and Joshua from Gojek doing this in a fantastic session at Lorong AI, but this is my story and I’ll tell it my way.): ...