AI AGENTCONCIERGE

AI Agents & Architecture

Why the Same AI Model Can Behave Differently in Different Agent Systems

Before blaming the model, I'd check what the agent can actually see and do.

Two matching toolboxes beside different sets of job paperwork and planning materials.
The same core equipment, with different support for the job. Illustration created for AIC Team.

You use an AI model in one app, and it gets what you're asking. You try the same model somewhere else, and suddenly you're explaining the assignment for the third time.

At that point it's tempting to ask what happened to the AI. Did it forget how to do this between breakfast and lunch?

Before changing models, I'd look at the setup. What can this agent actually see? What tools does it have? And did anyone tell it how this particular job works?

We spend a lot of time talking about which model is smartest. Fair enough, the model matters. But there's a whole working environment around it, and that's where some of the difference comes from.

Same model, different working day

Let's say a customer emails your service business asking when someone can come back to finish a repair. There are notes from the first visit, a part on order, and a schedule to work around.

You give that email to two setups using the same model.

The first can read the message and write a reply. That's about it. It doesn't have the job folder, can't check the part, and has no idea what's on the calendar. Now you're the one gathering all of that information and feeding it into the conversation.

The second has connections to those records and instructions for this kind of request. Check the job, check the part, find a possible time, and put a reply together for review.

If the part is still on order, that's something you want to know before arranging another visit. With the right records, the second setup can bring that into the conversation. The first needs you to supply it.

You haven't changed the model. You've changed what it has to work with. We wouldn't send a technician out without the work order and then blame the screwdriver.

Somebody has to connect the pieces

The software around the model is often called a harness. The name sounds more complicated than the basic idea. It gets information ready, connects the tools, passes results back, and keeps the work moving from one step to the next.

When the model asks to check the calendar, that software has to make the request happen. A clear answer from the calendar gives it something useful to work with. A broken connection gives it a problem to report back to you.

Then there are the instructions. “Help with emails” is a fairly loose assignment. “Check the job notes and part status before suggesting a return visit” gives it a better starting point. A saved procedure can help with that, as long as the system brings it into the task.

And what about the note saying this customer needs the same technician? It might be sitting in a folder from last week. If this setup never retrieves it, the model is working without it. You can have an impressive collection of saved information and still end up saying, “We already talked about this.”

I wouldn't fix that by dumping the entire company drive into every conversation. I'd start with whether the agent can find the records this job actually needs.

What would I want to see?

For this return visit, I'd want the proposed reply, the information behind it, and a person making the booking decision. The booking tool should enforce that approval. Writing “ask first” in a prompt doesn't settle the matter.

That's part of choosing a setup too. How does it show you the work? Can you tell a suggested appointment from a confirmed one without digging through the whole conversation?

I'd try the same job a few times, including once with a missing record or an unavailable calendar. That's when you find out whether the agent asks for help or fills the gap with a confident answer.

If I'm still doing all the searching, explaining, and checking, I want to understand why. Maybe the model isn't up to the task. Maybe it never got the tools or information to attempt it properly. I'd rather work that out before paying for a smarter model to sit in the same unprepared office.