How to Use the Moderation AI Agent

Also, I just had a community member reach out because they posted a link to one of our help articles but part of the link was automatically removed by the AI moderator thus breaking the link. I wouldn’t have known about this if the member hadn’t reached out.

I was under the assumption that if AI moderator flags a post, it would place the post into Pending so I have the ability to approve or edit.

Having the AI moderator automatically remove part of a link but keep the link clickable in the post knowing it’s broken, is a very poor experience.

Additionally, the member informed me that the moderator pop up happens every time he replies to a post, which adds to the poor experience.

Any way to troubleshoot this ASAP ​@Larry ? If not, I will most likely have to turn AI mod off.

​@erin.brisson  Just heard back from my team earlier today as we took a deeper dive to best understand all the scenarios in which this could happen.

In short, you are correct. This is occurring solely due to the AI Moderator enablement. This is behavior we are going to prioritize fixing within the next week. While we address this fix, you can decide whether you disable AI moderation temporarily or not, but regardless, this is an issue I’m going to have the team get started early next week.

I’ll report back here with an update soon but please add any questions or comments here so I see them!

I’ve been testing AI Moderation in the sandbox and my prompt seems to be quite good already. We do not want the AI to redact any PII information > we just don’t want it published. My prompt says

“Do not include phone numbers, customer numbers, or any personal information in your messages. Messages containing such information will be moved to the Reported folder.”

Now it seems to trash all posts containing PII, which is what I wanted, but we have a massive moderation problem which is blocking us in taking this into use.  All of our moderators have the role of “Staff Moderator” which means that they do not have access to Control home. So they’d have to separately go through the trash can in order to find the content AI trashed. That’s not really a lean process.

Could this idea be implemented?

We’re currently testing AI Moderation in the o2 Community.

As I understand, AI Moderation only chooses from the default moderator tags you’ve provided in the opening post. However, they don't match the moderation framework we've already established internally. I’d like to have the ability to define our own moderation tags as part of the AI Moderation Code of Conduct and have the AI use those tags when classifying content.

For example, instead of fixed platform tags, we'd like to be able to define:
• violation category (commercial_spam, fraud, pii, abuse, illegal_content)
• severity of violation (low, medium, high)
• confidence of the assessment (low, medium, high)

In fact, I have defined all of this in our Code of Conduct, but it’s being ignored (or the tool isn’t able to).

This is the excerpt from our Code of Conduct, FYI:

Moderator Tags

When a violation is detected, a violation category, a severity level, and a confidence level should also be assigned.

Violation categories for evaluation:
- commercial_spam = unsolicited bulk or off-topic content, or targeted advertising for products or services
- pii = personally identifiable information
- fraud = attempts at fraud, deception, or identity theft
- abuse = offensive, discriminatory, provocative, or harassing content
- illegal_content = illegal content

Severity levels for evaluation:
- low
- medium
- high
- critical
The selected violation category and severity level must correspond to the provided justification and the recommended moderation action.

Confidence Level of the Assessment:
- low
- medium
- high
The confidence level describes how certain the AI is that the selected violation category applies.
A low confidence level does not automatically mean that no violation exists. It merely indicates that the classification is based on less clear signals.
The confidence level is intended to assist moderators in prioritizing and reviewing decisions.This would make AI decisions much easier to understand for moderators, align classifications with our existing processes, and improve reporting and analysis.

It would also help a lot during testing and tuning, because we could immediately see whether the AI is classifying content according to the categories defined in our Code of Conduct. As of now, we don’t fully understand the AI’s decisions and where to adjust the Code of Conduct in order to make it do what we’re expecting it to do.