If you are going to use AI to do fact checking in the group and post the results, please also post the prompt that you use so that others can independently repeat the fact check.
Also, please start with a clean session in the AI and explicitly ask the AI to do a fact check for scientific accuracy.
If you have previous conversions with the AI prior to doing the fact check, then you are biasing the response. Biasing the AI ahead of time is not a valid way to do objective fact checking.
You have to explicitly ask the AI to do a fact check for scientific accuracy in order to give it permission to disagree with you. Otherwise, it will just to tend agree with what you say. Again, that is not a valid way to do objective fact checking.
Guidelines for using AI to do fact checking in the group
I highly encourage the use of AI for doing fact checking in the group.
These AI tools are not perfect, but they can be very accurate when used appropriately.
They are the best tool that we currently have for doing fact checking in the group, short of consulting a human expert on every single relevant topic.
You can learn how to use AI to do fact checking here: ↪ How to fact-check posts and comments with Google Gemini.
To do these fact checks properly in an unbiased way, you need to start from a clean session with the AI. You cannot have past conversations in the same session, or use saved results from previous sessions. These past conversations bias the results.
You should specifically ask the AI to critique and fact check the text for scientific accuracy. In other words, you need to encourage the AI to disagree if scientifically warranted. If you don’t explicitly do this, the AI will not be compelled to express disagreement when appropriate.
If you post your AI-generated results in the group, then you should also disclose enough information that people in the group can independently reproduce the fact check on their own. Independent reproducibility is key.
You should state:
- The specific AI that you used.
- The specific prompt that you used.
If people can’t independently reproduce your fact-check using the same AI and prompt, then it isn’t valid.
Finally, and most important… you need to actually check the references that the AI cites to justify what it says. You cannot just blindly assume that everything that the AI said is true. Often, the AI gets confused by what it reads. It can easily misunderstand things. You need to do some basic due diligence to make sure that the AI actually understands what it is that you are trying to fact check. Ultimately, this is your responsibility.
Just like a human, the AI’s do make mistakes. These fact checks are far from perfect. Parts of them can be wrong. It’s important to read them critically, and not interpret them as always true. The quality of the response usually depends on the quality of the references that the AI uses to generate what it says. If it uses references that are junk, then it tends to produce responses which are also junk. If you are unsure what references the AI used to generate what it said, then you should followup and ask it directly to provide those references. For example, ask: “What references did you use to generate what you said?”
These are the guidelines that I personally try to follow when posting AI fact-checks in the group.
#factcheckingHD #rulesHD #aiHD #evidenceHD