Benchdog AI Assistant

Run benchdog assistant setup --provider anthropic --model claude-opus-5-5 and type the key when asked; it is never shown. In a script, pipe the key in with --key-stdin so it never sits on a command line or in a shell history. Settings go in assistant.json in the Benchdog folder; the key goes only in the system's secure store.

benchdog assistant status shows the provider, model and endpoint, whether the key is saved and where the settings live, never the key itself. benchdog assistant test asks the provider one small question and reports the model and the tokens it used.

Setup reads the models your key can use from the provider's own list, newest first, and lets you pick one or type another by the name the provider lists it under. benchdog assistant models prints the same list, and benchdog assistant models --provider openai another provider's. Anthropic uses claude-opus-5-5 when no model is named; OpenAI needs one named.

To change the model, run setup again or open the assistant's settings in the app; leave the key box empty to keep the saved key.

Give the endpoint in setup, or with benchdog assistant setup --endpoint https://gateway.example.com. It must start with https://, so the key and your schema never travel in the clear. Leave it empty for the provider's own address. The engine calls the same paths under it as it would at the provider, so the gateway has to speak the Anthropic or OpenAI API.

Run benchdog assistant ask "Which tables have no primary key?" --server sqlhost --database Sales --windows. The assistant is given that database's structure as its context; only the structure is read, never table data. Add --selection dbo.Customer to have one object described first and in full. Sign in with --windows, with --sql-login monitor and --remember-password, or with Entra when neither is given, and --trust-server-certificate for a server on your own network with a self-signed certificate.

Give both --server and --database, or neither; without them the assistant answers from the question alone.

Each tool gives it actions of its own, and it acts only through them. Actions that read run at once: SQL Monitor's check_server, read_queries, read_plan and read_alerts, SQL Documenter's list_objects and describe_object, Key Vault's compare_vaults and show_history by name only, and so on. An action that changes something, such as SQL Documenter's set_description, SQL Schema Compare's deploy_to_target or SQL Multi Runner's run_script, shows in the chat with This changes something. Run it, skip it, or type what to do instead. and runs only after Run.

When no action fits, the assistant says so and tells you where in the tool to do it, or the benchdog command with its options.

Run benchdog assistant forget to remove the saved key from the system's secure store, or benchdog assistant forget --provider openai for a provider other than the one set up. Add --settings to remove assistant.json too, so the assistant is set up again from the start.

Yes. The settings file and the key are shared by every Benchdog tool on that machine, in VS Code, in the desktop apps and at the command line, so set it up in one place and every tool uses it. Another machine is set up on its own.

The engine builds the context from the tool's own results, never from free text: the tool, the platform, who is signed in, the server and database, what you have picked, and the objects, differences, statistics or output that tool shows, such as a database's structure, a server's checks and a query's plan. The context is capped at 32 KB; when a list is too long it is cut and says so, so the model asks rather than guesses.

Every value is scrubbed before it leaves your computer: passwords, keys and tokens in connection strings, settings, URLs and JSON, bearer tokens and the row values SQL Server quotes in some errors. Key Vault secrets are referred to by name only and their values are never sent. SQL Documenter gives it structure and descriptions, never table data. What is sent goes only to the provider you picked, with your key.

Anthropic, with a Claude model, and OpenAI, with a GPT model, through each provider's own API over HTTPS, or through a gateway of yours that speaks it. You bring your own key and pick the model from the provider's list. Benchdog never hosts or meters the model, so what you use is between you and the provider.

The AI Assistant is still in development, and the website says so. What is built today is in the tools: the panel in every Benchdog desktop app, the view in every Benchdog VS Code extension, benchdog assistant on the command line, Anthropic and OpenAI as providers, and actions of each tool's own with Run before anything changes. Other providers, such as Azure OpenAI, are not there yet.

No. It cannot read or write files or reach a database, vault or server by itself; it acts only through the actions a tool gives it, which the tool runs as you, with your own sign-in. Any action that changes something waits for you to press Run, and a Skip, or anything you type instead, goes back to the assistant as the answer.

SQL it writes that changes data or structure is wrapped in a transaction with a comment saying what it changes, and it never writes a DROP or TRUNCATE unless you asked for that by name.

Yes. Every tool has a thirty day trial with every feature, and no card. Ask for one at benchdog.dev/trial with your name and email, and the trial licence arrives by email, as a line to paste and a benchdog.licence file. One trial per person covers every tool at once.

One licence is for one person and covers the VS Code extension, the desktop app and the benchdog command, on every machine that person works from. There is no machine limit and no server count. Buying another tool reissues your licence with that tool added.

The licence shows whose it is: your name and email appear in each tool's About and account menu.

The last version released while you were paid keeps working. Only updates stop. A licence is checked offline and never expires a version it covers, so nothing stops working the day your year ends.

The VS Code extensions run in VS Code 1.95 or later on Windows, Linux and Mac. The desktop apps and the benchdog command are single programs for Windows (x64), Mac (Apple silicon) and Linux (x64), with no .NET install needed. The SQL tools work with SQL Server 2017 or later, Azure SQL Database and Azure SQL Managed Instance.

No. The engine runs on your machine, reads your vaults and databases as you, and keeps the results with you. A tool contacts benchdog.dev only when you click something that needs it, such as asking for your licence, and never in the background. Licences are checked offline.

Every change a tool makes runs in one transaction with XACT_ABORT on, so it all happens or none of it does. A SQL Schema Compare deploy, a SQL Data Comparer sync and a SQL Data Filler fill are each one transaction, and SQL Data Scrambler scrambles each table in one. The scripts the tools write for you to run in SSMS or sqlcmd are wrapped the same way and roll back if any batch fails. SQL Multi Runner runs your own script as it is, and wraps each database in a transaction when you tick that. Before any change, a tool says what it will do and asks first: SQL Schema Compare names the tables and columns whose rows go, SQL Data Comparer says how many rows change, and SQL Data Scrambler asks you to type the database's name. SQL Schema Compare also stops if either database changed after you read the plan. Even so, nothing replaces a backup, and the SQL Server tools can take one first.

Azure calls run as you, through a browser sign-in once or through your existing az login, in any tenant you belong to. The SQL tools use your Microsoft account, your Windows sign-in or a SQL login, and every connection is encrypted. A SQL login's password is kept until the app closes, or in your system's secure store (DPAPI on Windows, Keychain on Mac, libsecret on Linux) if you tick Remember. It is never written to a settings file, a project file or a log.

The full message is The assistant is not set up. Pick a provider and a model, and save its key, with: benchdog assistant setup. Run that command, or choose Set up the assistant in the panel. The Anthropic key is not saved. Save it with: benchdog assistant setup means the settings are there but the key is not; setup keeps the settings and asks for the key.

Anthropic refused the key: ... Check it, and save it again. means the provider answered that the key is not valid; paste it again in setup. OpenAI would not allow this with the key means the key is known but may not do this, such as list models or use that model; check its permissions at the provider, and save it again if you change them. Anthropic is rate limiting this key. Wait a moment and ask again. is the provider's limit on your key, not Benchdog's.

The provider could not be reached from this machine; the rest of the message says why, with any secret taken out. Check the network, a proxy, and the endpoint if you set one with benchdog assistant status. OpenAI took too long to answer. means the provider accepted the question but no answer came in time; ask again.

The answer was cut short at the model's limit. Ask for the rest. means the model reached the most it may write in one answer; ask it to carry on. claude-opus-5-5 would not answer this. means the provider declined the question, and benchdog assistant ask exits with 1.

When the provider's list cannot be read, setup says why and lets you type the model's name as the provider lists it. The endpoint must start with https://, so the key and your schema never travel in the clear. means an http address was given. Give the OpenAI model to use. means OpenAI was picked without a model; name one, as OpenAI has no default.

The context is capped, so a long list is cut at a line and ends with a note that it is trimmed; the assistant then asks for a missing object rather than guesses. Pick the object you are asking about in the tool, or give it with --selection dbo.Customer at the command line, so it is described first and in full. Where the tool has actions, the assistant reads what the context does not show, such as describe_object in SQL Documenter or read_report in SQL Monitor.

Turn on the diagnostics log: in a desktop app, Settings, then Write a diagnostics log; in VS Code, the extension's diagnostics setting; for the benchdog command, set the environment variable BENCHDOG_DIAGNOSTICS=1. Each run writes one file to the logs folder in the Benchdog folder (%LOCALAPPDATA%\Benchdog\logs on Windows, ~/Library/Application Support/Benchdog/logs on Mac, ~/.local/share/Benchdog/logs on Linux), and the last ten are kept. The desktop apps have Open logs folder, and VS Code has the Open Diagnostics Logs Folder command.

The log records sign-in requests, Azure calls, engine requests, SQL connections, times and errors. Passwords, connection strings, tokens, secret values and your data are never written, so it is safe to send to support.

That is the crash window, for an error the app did not expect. Copy details puts the error and its details on the clipboard, Email support opens a message to support with them, and Open logs folder shows the crash report, which is written even with the diagnostics log off. If the error happened on a background thread the app has to close, and the window shows when it next starts.

Browser sign-in keeps its tokens in the desktop keyring, and that machine has none unlocked, which is common in WSL. Install and unlock a keyring such as GNOME Keyring or KWallet, or sign in with az login and turn on Use az login in Settings or pass --cli to the benchdog command. Benchdog never falls back to a plain file for tokens.

The SQL Server you connected to uses a certificate your machine does not trust, which is usual for a server on your own network with a self-signed certificate. Pick Trust the server's certificate in the picker, or add --trust-server-certificate on the command line, for servers you know. For a server you do not know, install its certificate chain instead.

In a desktop app, choose Forget saved password from the menu. On the command line, run benchdog sql forget-password --server <server> --sql-login <login>. Either removes it from the system's secure store, which is the only place it was kept.

Write to support@benchdog.dev or use the support form on this site, and a person replies. Say which tool and version you were using (Help, About in a desktop app shows both), what you did and what you saw, and attach the diagnostics log if you have one.