Benchdog Key Vault

Open the vault in the editor and click the eye button on the row, or turn on Show values for all of them. Values are read only when you show them, and are forgotten when you hide them or close the tab. In the desktop app they are also forgotten when you open another vault, sign out or minimise the window.

To copy without showing, right-click the secret and choose Copy Secret Value, or use the copy button in the editor. The engine puts the value on the clipboard, so it never passes through VS Code, and on Windows it is kept out of clipboard history.

At the command line, benchdog vault get demo-shop-prod SqlConnectionString prints the value on its own so it can be piped, and benchdog vault get demo-shop-prod SqlConnectionString --copy puts it on the clipboard instead.

In VS Code, click the diff button on the Vaults view, or right-click a vault and choose Compare Vaults, then pick two or more vaults. In the desktop app, open the Compare tab and tick the vaults. You see which secret names exist in which vaults, with only the rows that differ until you turn on Show all.

Turn on Compare values and vaults that share a value get the same letter, so A A B means prod differs. Values are compared inside the engine as fingerprints and never shown. Rules from benchdog.json apply, so a secret marked differs does not count as drift, and ignored secrets are hidden.

At the command line, run benchdog vault diff demo-shop-dev demo-shop-test demo-shop-prod --values. Add --all to see every name, --include-certificates to show certificate secrets, and --fail-on-drift to return exit code 3 when anything has drifted.

Click the pencil on a row in the vault editor. It reads the current value, with line breaks kept, and opens it for editing. Save checks the rules file, asks you to confirm, and writes a new version, so the old value stays in the history. A vault that already holds the value is skipped.

To rotate a secret without ever reading the old value, choose Replace without reading and type the new value in a password box. In the desktop app it is on the right-click menu.

At the command line, benchdog vault set SqlConnectionString --vaults demo-shop-dev,demo-shop-test asks for the value twice at a hidden prompt, shows each vault's state before, asks you to type y or yes, and shows the state after. For scripts, --value-stdin reads the value from standard input and needs --yes.

Right-click a secret and choose Show Secret History, or use the more button in the editor. Versions are listed newest first, with letters showing which versions share a value. Click a version's eye to see its value, and turn on Show who to read who made each change from the vault's audit logs. Roll back writes an earlier version's value as a new version after you confirm.

To see every change to any secret in a vault, right-click the vault and choose Show Recent Changes, then pick the last day, 7, 30 or 90 days and filter by secret or person. In the desktop app it is under Vault, Recent changes.

At the command line, run benchdog vault history demo-shop-prod SecurityCheckMode --values --who, and benchdog vault rollback demo-shop-prod SecurityCheckMode <version> with a version id from that list.

Right-click a secret and choose Copy Secret to Other Vaults..., pick the target vaults, and see each vault's state as letters before you confirm. The engine copies the value, with its content type, and skips vaults that already hold it. In the Compare view you can tick several rows, pick the vault to copy from, and choose Copy to the other vaults to bring them in line in one step.

At the command line, run benchdog vault copy SqlConnectionString --from demo-shop-dev --to demo-shop-test,demo-shop-prod. The value is never shown, and --yes skips the prompt for scripts.

Write a benchdog.json beside your code with the secrets every vault must have (required), the values a secret may hold (allowed or pattern), the secrets expected to differ per vault (differs) and the ones to hide from compare (ignore). Names may use * as a wildcard. In VS Code the file gets autocomplete, hover help and error squiggles.

Run benchdog vault check demo-shop-dev demo-shop-test demo-shop-prod --rules benchdog.json. It lists every missing required secret and every value that breaks a rule, reads only values that have a rule, and returns exit code 4 on any problem or when a value could not be checked. Without --rules, it uses benchdog.json in the current folder.

Click the search button on the Vaults view, or Find Secret in All Vaults, to search secret names across every vault at once, then open a match, copy its value, see its history, copy it to other vaults or delete it. A colon in the search also matches a double dash in a name. The desktop app has the same search on its Find tab.

In VS Code, hover over a Key Vault reference in any file, such as @Microsoft.KeyVault(SecretUri=...), VaultName=...;SecretName=... or a secret URI, to see whether that vault holds it and which others do. Find Usages in Workspace, on a secret's menu, searches your workspace for the name, including the : and __ forms of a double dash.

Delete asks Yes or No, only soft deletes, and offers Undo straight after. Right-click the vault and choose Show Deleted Secrets... to see its soft deleted secrets with the date each is removed for good, and recover one with all its versions. In the desktop app it is under Vault, Deleted secrets.

At the command line, benchdog vault deleted demo-shop-dev lists them and benchdog vault recover SqlConnectionString --vaults demo-shop-dev brings one back. Benchdog never purges a secret.

Benchdog runs as you, with your own access, and lists the vaults your account can see across your subscriptions. To list a vault's secret names you need List permission on its secrets, to show, copy or compare values you need Get as well, and to save, roll back, delete or recover you need the matching write permissions. A vault whose secrets you cannot list still shows in the tree, without a secret count, and the desktop app marks it Could not be listed.

Show who and Recent changes need the vault's audit logs sent to a Log Analytics workspace, and read access to that workspace. Without them the history still shows, with a note saying why who is missing.

No. Every save, copy, roll back, delete and recover asks first, and the command line asks you to type y or yes, or the secret's name to delete. Each save is a new Key Vault version, so the old value stays in the history. Delete only soft deletes, and nothing is ever purged. Listing, finding, comparing, history, recent changes and the rules check only read.

Values are read only when you show or copy them, and only the ones you ask for. Compare values turns each value into a fingerprint under a key made for that one comparison, so only letters leave the engine. The rules check reads only values that have a rule and checks them in memory. Values are never logged, saved or put in an error message, and the vault editor page in VS Code has no network access.

Benchdog Key Vault works with secrets. Certificate secrets, the ones Key Vault creates for managed certificates or that have a certificate content type, are hidden by default because they hold certificates rather than configuration, and --include-certificates on benchdog vault names and benchdog vault diff shows them. It does not create or manage keys or certificates.

Yes. The Apps view lists every web app and function app you can see with its slots. Right-click one and choose Restart App..., pick Production or a slot, confirm, and Benchdog waits for the app's health check path and reports the status code and time. Only the status code is kept, never the page. At the command line, benchdog app restart demo-shop-web --slot staging does the same, with --health /path, --timeout <seconds> and --no-health, and returns exit code 5 when the app is not healthy in time.

Yes. Every tool has a thirty day trial with every feature, and no card. Ask for one at benchdog.dev/trial with your name and email, and the trial licence arrives by email, as a line to paste and a benchdog.licence file. One trial per person covers every tool at once.

One licence is for one person and covers the VS Code extension, the desktop app and the benchdog command, on every machine that person works from. There is no machine limit and no server count. Buying another tool reissues your licence with that tool added.

The licence shows whose it is: your name and email appear in each tool's About and account menu.

The last version released while you were paid keeps working. Only updates stop. A licence is checked offline and never expires a version it covers, so nothing stops working the day your year ends.

The VS Code extensions run in VS Code 1.95 or later on Windows, Linux and Mac. The desktop apps and the benchdog command are single programs for Windows (x64), Mac (Apple silicon) and Linux (x64), with no .NET install needed. The SQL tools work with SQL Server 2017 or later, Azure SQL Database and Azure SQL Managed Instance.

No. The engine runs on your machine, reads your vaults and databases as you, and keeps the results with you. A tool contacts benchdog.dev only when you click something that needs it, such as asking for your licence, and never in the background. Licences are checked offline.

Every change a tool makes runs in one transaction with XACT_ABORT on, so it all happens or none of it does. A SQL Schema Compare deploy, a SQL Data Comparer sync and a SQL Data Filler fill are each one transaction, and SQL Data Scrambler scrambles each table in one. The scripts the tools write for you to run in SSMS or sqlcmd are wrapped the same way and roll back if any batch fails. SQL Multi Runner runs your own script as it is, and wraps each database in a transaction when you tick that. Before any change, a tool says what it will do and asks first: SQL Schema Compare names the tables and columns whose rows go, SQL Data Comparer says how many rows change, and SQL Data Scrambler asks you to type the database's name. SQL Schema Compare also stops if either database changed after you read the plan. Even so, nothing replaces a backup, and the SQL Server tools can take one first.

Azure calls run as you, through a browser sign-in once or through your existing az login, in any tenant you belong to. The SQL tools use your Microsoft account, your Windows sign-in or a SQL login, and every connection is encrypted. A SQL login's password is kept until the app closes, or in your system's secure store (DPAPI on Windows, Keychain on Mac, libsecret on Linux) if you tick Remember. It is never written to a settings file, a project file or a log.

The message Vault "demo-shop-prod" was not found. Run "benchdog vault list" to see the vaults you can reach. means the name did not match any vault your account can see. Names are matched against benchdog vault list, ignoring case, and a vault URI works too.

If the vault is in another tenant, set benchdog.tenant in VS Code, the tenant in the desktop app's Settings, or --tenant <id> at the command line.

Comparing values needs Get permission on the vault's secrets, not only List. A vault you can list but not read shows ? for every value, with the note value(s) could not be read. Check you have Get permission on secrets in this vault. The rest of the comparison still runs. With --fail-on-drift, a value that could not be read counts as drift, so a pipeline without Get permission never passes by accident.

The desktop app shows The vault's firewall blocks your address <address>. Add it under the vault's Networking, or use an allowed network. when Key Vault refuses your network. In VS Code and at the command line the Azure error names ForbiddenByFirewall. Add your address to the vault's firewall in the Azure portal, or connect from a network it allows, then refresh.

Who made each change comes from the vault's audit logs in Log Analytics, for up to two years back. The vault must send its audit logs to a Log Analytics workspace, and you need read access to that workspace. Otherwise the history still shows, with a note saying why who is missing. A version is matched to the audit event within two minutes of its creation time, so a version written while logging was off has no name.

A rule in benchdog.json applies. SecurityCheckMode must be one of: Off, Audit, Enforce. means an allowed rule names the values the secret may hold, and ApiUrl does not match the pattern ^https://. means a pattern rule did not match. Nothing is written when a rule fails, and the message never contains the value. Change the value, or change the rule in the file.

Benchdog could not find its engine. Set Benchdog: Engine Path to your benchdog executable. means benchdog.enginePath points somewhere that does not exist. Each extension package carries its own engine, so leave the setting empty unless you built the engine yourself. After fixing it, click Try Again in the Vaults view. If the status bar shows Benchdog: engine not reachable, turn on benchdog.diagnostics and run Benchdog: Open Diagnostics Logs Folder to see what stopped it.

Turn on the diagnostics log: in a desktop app, Settings, then Write a diagnostics log; in VS Code, the extension's diagnostics setting; for the benchdog command, set the environment variable BENCHDOG_DIAGNOSTICS=1. Each run writes one file to the logs folder in the Benchdog folder (%LOCALAPPDATA%\Benchdog\logs on Windows, ~/Library/Application Support/Benchdog/logs on Mac, ~/.local/share/Benchdog/logs on Linux), and the last ten are kept. The desktop apps have Open logs folder, and VS Code has the Open Diagnostics Logs Folder command.

The log records sign-in requests, Azure calls, engine requests, SQL connections, times and errors. Passwords, connection strings, tokens, secret values and your data are never written, so it is safe to send to support.

That is the crash window, for an error the app did not expect. Copy details puts the error and its details on the clipboard, Email support opens a message to support with them, and Open logs folder shows the crash report, which is written even with the diagnostics log off. If the error happened on a background thread the app has to close, and the window shows when it next starts.

Browser sign-in keeps its tokens in the desktop keyring, and that machine has none unlocked, which is common in WSL. Install and unlock a keyring such as GNOME Keyring or KWallet, or sign in with az login and turn on Use az login in Settings or pass --cli to the benchdog command. Benchdog never falls back to a plain file for tokens.

The SQL Server you connected to uses a certificate your machine does not trust, which is usual for a server on your own network with a self-signed certificate. Pick Trust the server's certificate in the picker, or add --trust-server-certificate on the command line, for servers you know. For a server you do not know, install its certificate chain instead.

In a desktop app, choose Forget saved password from the menu. On the command line, run benchdog sql forget-password --server <server> --sql-login <login>. Either removes it from the system's secure store, which is the only place it was kept.

Write to support@benchdog.dev or use the support form on this site, and a person replies. Say which tool and version you were using (Help, About in a desktop app shows both), what you did and what you saw, and attach the diagnostics log if you have one.