chore(explorer): Remove ability to call custom explorer tools

[Zylphrex]

In order to consolidate the tool definitions, this takes away the custom tool calling ability from the explorer client. Allowing custom tool definitions makes it harder to clean up. If additional tool calls are needed, we can discuss each use case and decide how to approach it.

[roaga]

In order to consolidate the tool definitions, this takes away the custom tool calling ability from the explorer client. Allowing custom tool definitions makes it harder to clean up. If additional tool calls are needed, we can discuss each use case and decide how to approach it.

Sorry to block for now, but can we discuss this first? I feel like I haven't heard what your plan actually is (it seems different than what I heard in Explorer Sync) and I'm confused why removing these capabilities is necessary. And doesn't it defeat the purpose of an "SDK" if other teams have to come to us for approval for any custom use cases?

[Zylphrex]

In order to consolidate the tool definitions, this takes away the custom tool calling ability from the explorer client. Allowing custom tool definitions makes it harder to clean up. If additional tool calls are needed, we can discuss each use case and decide how to approach it.

Sorry to block for now, but can we discuss this first? I feel like I haven't heard what your plan actually is (it seems different than what I heard in Explorer Sync) and I'm confused why removing these capabilities is necessary. And doesn't it defeat the purpose of an "SDK" if other teams have to come to us for approval for any custom use cases?

The ability to allow each consumer of the explorer client to define custom tools for each use case sounds nice in theory. But in practice, the ability to allow people to define custom tools will result in multiple implementation of similar things.

So instead of allow users to configure custom tools right immediately, we should start by seeing if all these use cases can be supported by providing the tools from the seer side.

I'm open to having this conversation again in the future once we have a few consumers of the explorer client and we find that there is a clear need to allow users to define custom tools. Until then, supporting this interface just means we have an additional surface that needs to be supported and the potential for more tools we have to consolidate.

[roaga]

In order to consolidate the tool definitions, this takes away the custom tool calling ability from the explorer client. Allowing custom tool definitions makes it harder to clean up. If additional tool calls are needed, we can discuss each use case and decide how to approach it.

Sorry to block for now, but can we discuss this first? I feel like I haven't heard what your plan actually is (it seems different than what I heard in Explorer Sync) and I'm confused why removing these capabilities is necessary. And doesn't it defeat the purpose of an "SDK" if other teams have to come to us for approval for any custom use cases?

The ability to allow each consumer of the explorer client to define custom tools for each use case sounds nice in theory. But in practice, the ability to allow people to define custom tools will result in multiple implementation of similar things.

So instead of allow users to configure custom tools right immediately, we should start by seeing if all these use cases can be supported by providing the tools from the seer side.

I'm open to having this conversation again in the future once we have a few consumers of the explorer client and we find that there is a clear need to allow users to define custom tools. Until then, supporting this interface just means we have an additional surface that needs to be supported and the potential for more tools we have to consolidate.

Hmm I wouldn't say it's just theory. The ability to define custom tools is pretty standard in agent SDKs across the industry. A good example is the Claude Code SDK, which comes with a set of built-in tools and lets you append your own custom tools as well.

For a Sentry example, if the Autopilot team wants their agent to read and edit remote instrumentation configs, that would make sense as a custom tool but definitely not as a built-in tool. As another example, maybe the Dashboards team wants Seer to do some in-depth work with widgets; it would make sense for them to own that rather than stuffing it into the general agent's tool set. And if Code Review eventually switches to the explorer agent, it would make sense for them to have a custom tool for posting review comments.

I'm also still not following what you mean by tool consolidation. All tools in use by Explorer are already in one place in the seer repo, and the plan is for the other agents to gradually be replaced by Explorer, no? And for any custom tools, it seems natural that they live with the team and feature they belong to, rather than with us?

Regardless, we're at an early stage where we're already working closely with any team using this client, so rather than deleting all this code just to add it back later, it seems easier to just advise them on best practices and document them.

[Zylphrex]

My current stance on this is that unlike the Claude Code SDK, we're not trying to build a generic agent SDK to support every possible use case. We are trying to provide a set of primitives that can be composed to build an agent that operates within the boundary of Sentry.

This means we should provide the basic building blocks to interact with various parts of Sentry such as dashoards, the github bot, etc instead of providing a raw interface to build anything they want. Think a dashboard tool that can be modified to create/edit a dashboard widget, or a github tool that can be used to post comments.

Now I see that implementing these customization will likely require some kind of interface to define them so perhaps deleting all this code is unnecessary as we may need to add something analogous back in. But I still don't think we should allow teams to define arbitrary tools right now but rather something we can explore exposing in the future.

Perhaps a middle ground right now is to change to API to only allow certain tools from the Sentry side instead of allowing arbitrary tools to be defined.

[roaga]

My current stance on this is that unlike the Claude Code SDK, we're not trying to build a generic agent SDK to support every possible use case. We are trying to provide a set of primitives that can be composed to build an agent that operates within the boundary of Sentry.

This means we should provide the basic building blocks to interact with various parts of Sentry such as dashoards, the github bot, etc instead of providing a raw interface to build anything they want. Think a dashboard tool that can be modified to create/edit a dashboard widget, or a github tool that can be used to post comments.

Now I see that implementing these customization will likely require some kind of interface to define them so perhaps deleting all this code is unnecessary as we may need to add something analogous back in. But I still don't think we should allow teams to define arbitrary tools right now but rather something we can explore exposing in the future.

Perhaps a middle ground right now is to change to API to only allow certain tools from the Sentry side instead of allowing arbitrary tools to be defined.

Definitely agree that we only want to support agents that operate within the boundary of Sentry. So to that end:

1. Since everyone who will use this client works at Sentry and is building Sentry features, isn't it pretty natural that any tool that someone defines will sit within the boundary of Sentry? Like I don't think the Dashboards team is gonna make anything other than dashboards-related tools
2. I think I disagree that we should provide the building blocks for every part of Sentry. We should handle the core data types + coding + having a generally strong agent, but it seems more natural that every feature team owns anything specific to their vertical. Like if one day Sentry adds status pages or escalation policies, those respective teams shouldn't have to nag us for support, but should be free to add AI to their features on their own time.

a dashboard tool that can be modified to create/edit a dashboard widget

I'm not really sure, what does it look like to have a tool that can be "modified"? A tool is just a prompt + json schema, so wouldn't the dev just be defining a new tool?

[Zylphrex]

From Slack:

For now, we'll keep this interface in place but work closely with other teams so we dont end up writing a bunch of custom tool calls unnecessarily.