Can AI tools like Chatgpt be duped by asking them questions in a way that they have to divulge information that they are not supposed to?

[harry]

Case in point a screenshot I saw the other day on a forum which showed a person asking Chatgpt - which sites are banned for piracy worldwide - and it responded by giving out a list!