Security

CMEK and field encryption

Vigilo wraps every sensitive secret field — integration tokens, webhook secrets, SMTP passwords, MFA secrets — in a customer-managed encryption key (CMEK)…

Last updated

Overview

Vigilo wraps every sensitive secret field — integration tokens, webhook secrets, SMTP passwords, MFA secrets — in a customer-managed encryption key (CMEK) envelope (WC.11). The wrapping uses Fernet (AES-128-CBC + HMAC-SHA256) as the data-encryption-key (DEK) algorithm. The DEK itself is wrapped by a key-encryption-key (KEK) sourced from one of three backends: a local file (dev / single-host prod), HashiCorp Vault (stub today, full integration roadmap), or AWS KMS (stub today, full integration roadmap).

The on-the-wire ciphertext is prefixed cmek:v1: followed by base64-encoded data: <wrapped_dek>|<nonce>|<ciphertext>. The prefix makes encrypted values unmistakable in database dumps and allows the descriptor field to recognize legacy plaintext during migration (the cmek_encrypt_existing management command).

Why it exists

A workspace's secrets are the keys to its kingdom — a leaked Jira token can post fake tickets, a leaked Slack token can read every DM the bot has joined, a leaked SMTP password can phish from the workspace's domain. Encrypting them at the application layer (in addition to disk encryption at rest) means:

  1. A pg_dump accidentally committed to GitHub does not leak the secrets.
  2. A read-only database replica handed to analytics doesn't include readable credentials.
  3. A SELECT * from a misconfigured admin tool shows ciphertext, not plaintext.
  4. CMEK gives compliance auditors a checkbox they always want filled.

Key concepts

EncryptedTokenField (F2)

EncryptedTokenField is a Django field descriptor (the Python descriptor pattern, not just a TextField subclass). On get, it calls the CMEK helper to decrypt; on set, it encrypts and stamps the cmek:v1: prefix. A legacy plaintext value (no prefix) is returned as-is from the get path and re-encrypted on the next save — this is what enables the rolling backfill.

The field lives in vigilo_django.security.fields. It's used like a regular TextField:

class IntegrationConnection(models.Model):
    workspace = models.ForeignKey(Workspace, on_delete=models.CASCADE)
    slack_token = EncryptedTokenField(blank=True)
    ...

CMEK helpers

The module vigilo_django.security.cmek exposes two top-level helpers:

  • encrypt(plaintext: str) -> str — returns the cmek:v1:... wrapped ciphertext.
  • decrypt(ciphertext: str) -> str — returns the plaintext; raises CMEKDecryptionError on signature failure or KEK mismatch.

Both helpers are aware of the active KEK backend (resolved from settings).

KEK backends

  • Local file (local, the default) — KEK material is a 32-byte file at the path in VIGILO_CMEK_LOCAL_KEK_PATH. Useful for dev and single-host prod. The file must be chmod 600 and owned by the Django user; the helper refuses to read it otherwise.
  • HashiCorp Vault (vault, stub today) — KEK material fetched from Vault Transit. Config: VIGILO_CMEK_VAULT_ADDR, VIGILO_CMEK_VAULT_TOKEN, VIGILO_CMEK_VAULT_KEY_NAME. The stub returns NotImplementedError; full integration lands in a follow-up release.
  • AWS KMS (aws_kms, stub today) — KEK is a KMS Customer-Managed Key. Config: VIGILO_CMEK_AWS_KMS_KEY_ID, plus standard AWS_* env vars for credentials. Stub returns NotImplementedError.

Select the backend with VIGILO_CMEK_BACKEND=local|vault|aws_kms.

What is encrypted

Every secret field across the codebase. The current full list:

  • IntegrationConnection.slack_token
  • IntegrationConnection.jira_token
  • IntegrationConnection.linear_key
  • IntegrationConnection.git_token
  • IntegrationConnection.pagerduty_key
  • IntegrationConnection.twilio_auth_token
  • Webhook.secret (the HMAC signing secret sent with each delivery)
  • WebhookSigningKey.secret (per-workspace key used for signing inbound verifications)
  • Workspace.settings['smtp']['password'] (per-workspace SMTP for the contact + notifications path)
  • UserProfile.totp_secret (TOTP shared secret)
  • UserProfile.backup_codes (already bcrypt-hashed, additionally CMEK-wrapped to defend against bcrypt dictionary attack against an exfiltrated db)
  • ScimToken.token_hash (already sha256-hashed; CMEK adds a second layer)
  • ApiToken.token_hash (same shape)

Adding a new secret field to a model: change the field type to EncryptedTokenField, run makemigrations, then run cmek_encrypt_existing --model app.Model --field field_name to roll over any historical plaintext.

Backfill: cmek_encrypt_existing

The management command lives at vigilo_django/security/management/commands/cmek_encrypt_existing.py. Usage:

python manage.py cmek_encrypt_existing --model integrations.IntegrationConnection --field slack_token --batch 500

The command iterates rows in batches, skipping any value that already starts with cmek:v1:, encrypting the rest, and saving. It is idempotent — re-running after partial progress picks up where it left off. The command respects --dry-run for safety.

Run the backfill once after enabling encryption on a previously plaintext field. Once complete, every read goes through the decrypt path and every write goes through encrypt.

Production rollout

The deployment order matters. Get it wrong and old workers try to read encrypted values as plaintext.

Correct order:

  1. Deploy the code that contains the EncryptedTokenField change. All workers can now both read plaintext (legacy) and read ciphertext (newly encrypted).
  2. Run the migration (if any schema change, e.g. lengthened column). At this point the column is still plaintext-only on disk.
  3. Run cmek_encrypt_existing. Now rows on disk are ciphertext; readers handle it because step 1 already deployed.
  4. Restart workers (optional, just to clear any old in-memory caches).

Wrong order: running the backfill before the code deploy. Old workers still expect plaintext, they read ciphertext as a literal string, integrations fail, you have an incident at 2 a.m.

Common workflows

1. Enable CMEK on a fresh deploy

  1. Generate a KEK: head -c 32 /dev/urandom > /etc/vigilo/cmek.key && chmod 600 /etc/vigilo/cmek.key.
  2. Set env: VIGILO_CMEK_BACKEND=local, VIGILO_CMEK_LOCAL_KEK_PATH=/etc/vigilo/cmek.key.
  3. Restart Django. New writes encrypt automatically.
  4. Run cmek_encrypt_existing for each existing model/field combo.

2. Rotate the KEK

KEK rotation is supported by re-wrapping every DEK with the new KEK. The script python manage.py cmek_rotate_kek --old-path /etc/vigilo/cmek.key --new-path /etc/vigilo/cmek.key.new reads each cmek:v1: value, unwraps with the old KEK, re-wraps with the new, and writes back. Run it once, swap the file, restart.

3. Decrypt a value out-of-band for support

The Django shell:

from vigilo_django.security.cmek import decrypt
print(decrypt(IntegrationConnection.objects.get(pk='...').__dict__['slack_token']))

Use sparingly and only on a host the support engineer is already on — printing plaintext to a terminal log is a security event of its own.

Permissions

CMEK is invisible to end users. The application transparently encrypts on save and decrypts on read; nobody sees the ciphertext through the UI. The KEK file and the management commands are restricted to platform operators with shell access.

Troubleshooting

CMEKDecryptionError after a deploy. The KEK on the host doesn't match the one used to encrypt. Most common cause: the KEK file was regenerated by accident. Restore from backup or accept that all encrypted secrets must be re-entered.

The integration suddenly stopped working. Look in logs for CMEKDecryptionError. If present, you have a KEK mismatch. If absent, the IdP rotated the secret — re-enter it in Settings → Integrations.

cmek_encrypt_existing reports "skipped: already encrypted" for every row. Backfill already ran. Confirm with a SELECT on the column — every value should start with cmek:v1:.

Vault / KMS backend returns NotImplementedError. Those are stubs in the current release. Use local for now; switching backends requires running the rotate command above against the new backend once it ships.

Related articles