How Quartz sees what its AI costs without waiting for the invoice

Leo runs against Quartz's live SharePoint library, roughly 3.5 TB of project documentation across four project-batch sites, answering staff questions in Teams and drafting documents from the firm's own templates. Every one of those answers is an AI call that costs money.

That cost arrived the way it usually does: on a provider console, after the fact, as an aggregate. There was no project-level or user-level view before the invoice landed, no easy way to spot an expensive or slow conversation, and no shared record across the applications different teams had built.

Watchtower sits alongside the applications rather than in front of them. It makes no model calls of its own. When Leo finishes an AI request it sends a short usage event, and Watchtower validates it, prices it against its own internal rate table, and stores it.

Because pricing is calculated inside Watchtower rather than taken from whatever the calling application reports, the numbers stay consistent across every application connected to it. A project appears on the dashboard automatically the first time an event arrives carrying its name, so nothing has to be registered in advance.

The dashboard gives rolling 30-day totals for sessions, tokens and daily activity, with cost held as an all-time figure since a project's first request. From there it splits four ways: Projects for per-project usage and model-level cost, Users for activity by requester, Errors for the requests that failed, and a drill-down from any session into the individual queries beneath it.

A session is a continuous piece of work, a conversation or a pipeline run. A request is one call inside it. Being able to move between those two levels is what turns a number on an invoice into something a manager can actually ask a question about.

Watchtower is a monitoring layer, and the current version is deliberately narrow. It does not send automated alerts by email or Slack, and dashboard access is controlled at network level rather than by a separate login. Costs are shown in USD, and a model that is not yet in the pricing table records its usage but shows a cost of $0 until the rate is added.

Deleting a project removes its sessions and queries permanently. That is the trade for a system that stays lightweight enough to sit behind every application without becoming one more thing to run.