How I Built a Slack Agent That Answers Team Questions From Internal Docs

Every ops-heavy team I've worked with has the same Slack channel. It's the one where someone asks "where's the latest version of the X template?" or "what's our policy on Y?" and a manager spends ten minutes hunting for the link. Individually these are five-minute questions. Across a week they're hours. Across a team of thirty they're a full role.

This is a build log for an agent I shipped to kill that channel. I'm going to be specific about the stack, the architecture, and the two places it nearly broke — because the whole point of a build log is that you can see the seams.

What it does

The agent lives in Slack. You @-mention it in any channel (or DM it) and ask a question in plain English. It answers in a few seconds, citing the internal doc the answer came from. If it isn't confident, it says so and tags the person who owns that area.

That last sentence is doing a lot of work. Let's unpack it.

• Answers in plain English, not a search result list. People don't want ten links; they want the answer. The agent reads the relevant chunks and synthesises.
• Cites the source. Every answer ends with a link to the doc it came from. This is non-negotiable — without it, the agent is a confident liar waiting to happen.
• Knows when to escalate. A confidence threshold gates whether the agent replies or routes to a human. Getting this right is more important than getting the model right.

The stack

I try to build with boring, owned-by-the-client tools wherever possible. No custom software where off-the-shelf works.

The model is a swappable component, not an architecture decision. Build so you can change it in a year without rewriting anything.

The pipeline

Roughly: Slack message in → embed the question → retrieve the top-k doc chunks → hand the question and chunks to the model → post the answer back with citations.

The piece that's easy to underestimate is everything before the model. Chunking your docs well, keeping the index fresh when docs change, and filtering out irrelevant chunks before retrieval — that's where the accuracy actually comes from. The prompt matters less than people think; the retrieval matters more.

Where it nearly broke

Chunk boundaries. My first version chunked every doc on a fixed character count. It was terrible. A question about the refund policy would retrieve half a refund paragraph and half a returns paragraph, stitched together. Switching to chunk-on-headings (and keeping heading + first sentence together) was a bigger accuracy win than any model upgrade.

Stale docs. The client's docs change weekly. The first version of the index only rebuilt on a cron at midnight, so Monday morning the agent would confidently cite a Friday version of a policy. Moving to event-driven re-indexing — re-embed a doc the moment it's edited — fixed it and cost almost nothing because most docs don't change often.

What I'd do differently

I'd build the escalation path earlier. I shipped the confidence-threshold-and-tag-a-human behaviour in week two after the first round of "the agent said something weird" feedback. In retrospect that's week-one work, because it's the thing that makes the agent safe to actually put in front of people. A confident wrong answer is worse than no answer; a confident wrong answer with a one-tap "ask a human" is fine.

What it did

Within a month, the "where's the file" channel was mostly quiet. The ops lead got her Monday back. The team started asking the agent things it couldn't answer yet — which is the best possible signal, because it tells you exactly what to index next.

That's the real win. Not the agent. The fact that a week after launch the team trusted it enough to push on its edges.

---

If your team has that channel — the one that's just questions someone has to go hunt for — that's the cheapest automation win I know of. Book a call and I'll tell you honestly whether your docs are ready for it.