feat(AF-448): external secrets-manager integration (Vault / AWS Secrets Manager / Azure Key Vault)#606
Merged
Conversation
…service Adds the core external-secrets layer: SecretResolutionService (core/api), resolved/failed audit events, SecretReference parsing for vault:<mount>/<path>#<field> / aws:<name-or-arn>[#jsonField] / azure:<secret-name>, per-provider SecretStore implementations (spring-vault, AWS SDK v2 secretsmanager over url-connection, Azure Key Vault over the JDK HttpClient transport), opt-in provider beans with fail-fast config validation, accessflow.secrets.* configuration, i18n keys in all locales, and unit tests.
…aths Reroutes every datasource credential decrypt through SecretResolutionService: the JDBC pool factory (primary + replica), the engine catalog's CredentialDecryptor handed to all seven native engine plugins, and the admin service's test-connection / introspection / replica paths. On write, a credential matching a reference prefix is validated against the enabled providers and stored verbatim instead of encrypted. Adds DATASOURCE_SECRET_RESOLVED / _RESOLUTION_FAILED audit actions with plain @eventlistener handlers (pool-init resolves publish outside a transaction), context-less engine-lane attribution via findByCredentialReference, and 400/502 ProblemDetail mappings.
… help GET /api/v1/datasources/secret-providers (PERM_DATASOURCE_MANAGE) returns the enabled external secret-store providers. The datasource create wizard and settings form fetch it via TanStack Query and show per-provider reference syntax help (Form.Item extra) on the password, replica-password and API-key fields, with a client-side rule mirroring the backend's SECRET_PROVIDER_DISABLED check. New pure util src/utils/secretReference.ts (covered), i18n keys in all 7 locales, api-spec entry, and tests.
Live-path proof against a real HashiCorp Vault dev server (KV v2 read, field extraction, full DefaultSecretResolutionService resolve + audit event) and a real Secrets Manager API via LocalStack community edition (plain SecretString, #jsonField extraction, missing-secret failures) using the same client construction as SecretsConfiguration.
docs/07 external-secret-stores section + secrets table rows, docs/09 Secrets Manager env group, docs/05 credential-resolution subsection, docs/03 credential column semantics + audit actions, docs/12 v2.2 roadmap bullet, CLAUDE.md env rows, README feature bullet, website landing (encryption row, proxy bullet, roadmap Security group), website docs page (deploy note + datasource form note), website content-source map row.
Contributor
Contributor
Coverage Report for Frontend Coverage (frontend)
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||
Contributor
Backend Test Results4 878 tests 4 878 ✅ 16m 8s ⏱️ Results for commit bdcc781. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #448
What
Adds a pluggable external secrets-manager integration for datasource credentials: a credential column (
password_encrypted,read_replica_password_encrypted,api_key_encrypted) may now hold a secret reference —vault:<mount>/<path>#<field>,aws:<name-or-arn>[#jsonField], orazure:<secret-name>— stored verbatim and resolved through the configured store at credential-use time. Anything else keeps flowing through the existing AES-256-GCM layer (default and fallback). Detection is unambiguous: AES ciphertext is Base64 and never contains:.How
core.api.SecretResolutionServiceis the new seam every former decrypt call site goes through:DatasourcePoolFactory.buildPool(...)— JDBC primary + replica pools (resolve carries datasource/org ids for audit context)DefaultQueryEngineCatalog—secretResolutionService::resolvebecomes theCredentialDecryptorhanded to all seven native engine plugins (coversapi_keyand the DynamoDB secret key with zero plugin changes;core/api/CredentialDecryptoruntouched, so no engine re-pins)DatasourceAdminServiceImpl— test-connection, replica test, JDBC introspection; on write, a validated reference is stored verbatim instead of encryptedcore/internal/secrets/, one bean per provider, opt-in viaaccessflow.secrets.<provider>.enabled, fail-fast config validation at startup):LifecycleAwareSessionManageron a private scheduler (deliberately not aTaskSchedulerbean, which would make Boot's scheduling auto-config back off)azure-core-http-nettyexcluded so the Azure tree can't clash with Lettuce's Netty);DefaultAzureCredential(workload/managed identity) or client-secret credentialSecretReferenceResolvedEvent/SecretReferenceResolutionFailedEvent→DATASOURCE_SECRET_RESOLVED/DATASOURCE_SECRET_RESOLUTION_FAILEDrows (provider + reference, never the value). Plain@EventListener(not@ApplicationModuleListener) because pool-init resolves publish outside a transaction where AFTER_COMMIT delivery silently drops events. Context-less engine-lane resolves are attributed via the newDatasourceLookupService.findByCredentialReference(...).400 INVALID_SECRET_REFERENCE/400 SECRET_PROVIDER_DISABLEDat write time,502 SECRET_RESOLUTION_FAILEDat use time (i18n'd in all backend locales;MessagesParityTestgreen).GET /api/v1/datasources/secret-providers(documented in docs/04 first) drives per-enabled-provider syntax help (Form.Item extra) on the password / replica-password / api-key fields of the create wizard and settings page, plus a client-side rule mirroring the backend disabled-provider check (validation parity). i18n keys in all 7 locales.poolFingerprintcompares the stored strings). Rotating the secret inside the store does not restart live pools — documented with the re-save/test workaround.Placement note (deviation from the issue)
The issue suggested
security/internalfor theSecretStoreabstraction.securityalready depends oncore, and the consumers of the resolver arecore(admin service) andproxy(pool factory, engine catalog) — placing it insecuritywould create a module cycle. The feature lives incore, next to the existingAesGcmCredentialEncryptionService.ApplicationModulesTestandApiPackageDependencyTestpass.Scope notes
Tests
SecretReferenceTest,DefaultSecretResolutionServiceTest(incl. a no-caching test — two resolves → two store fetches),VaultSecretStoreTest,AwsSecretsManagerSecretStoreTest,AzureKeyVaultSecretStoreTest,SecretsConfigurationTest(ApplicationContextRunnerbean presence per flag + fail-fast validation),SecretsPropertiesTest; updates toDatasourcePoolFactoryTest(plaintext-drop + resolve context),DefaultQueryEngineCatalogTest,DatasourceAdminServiceImplTest(verbatim store, disabled-provider 400),AuditEventListenerTest(direct + lookup-attributed + no-match-skip + swallow),GlobalExceptionHandlerTest,DefaultDatasourceLookupServiceTest,DatasourceControllerIntegrationTest(endpoint 200 + 403).VaultSecretResolutionIntegrationTest— real HashiCorp Vault dev server (Testcontainers), KV v2 read + full resolve flow + audit event;AwsSecretsManagerResolutionIntegrationTest— real Secrets Manager API via LocalStack community edition 4.9.2 (the date-tagged 2026.x images are the licensed Pro build and exit without an auth token). Azure is unit-only (no official emulator).secretReferenceutil tests (coverage-included), wizard component tests for help rendering + disabled-provider validation, locale parity.e2e
No new Playwright spec: exercising the feature end-to-end needs a real external secret store, which the e2e compose stacks don't (and shouldn't) run. Existing datasource specs are unaffected — with no provider enabled (the e2e default) the forms render exactly as before (
#passwordunchanged, no extra help text), which is also asserted by a component test.Docs & website updated
docs/07-security.md— new "External secret stores" subsection + Secrets Management table rowsdocs/09-deployment.md— new#### Secrets Managerenv group (fullACCESSFLOW_SECRETS_*reference)docs/05-backend.md— "Datasource credential resolution" subsection under Connection Pool Managementdocs/03-data-model.md— credential-column semantics (ciphertext or reference) + two audit actionsdocs/04-api-spec.md—GET /datasources/secret-providers+ error codesdocs/12-roadmap.md— v2.2 bullet (AF-448)CLAUDE.md— env-var table rowsREADME.md— features bulletwebsite/index.html— Encryption spec row, proxy credential bullet, roadmap "Security" groupwebsite/docs/index.html— "External secrets managers" deploy note + datasource-form secret-reference notewebsite/README.md— content-source map rowDependencies (pinned latest stable, verified on repo1)
org.springframework.vault:spring-vault-core:4.1.0;software.amazon.awssdk:bom:2.48.0(+secretsmanagerexcl. netty-nio/apache clients,url-connection-client);com.azure:azure-sdk-bom:1.3.7(+azure-security-keyvault-secrets,azure-identityexcl.azure-core-http-netty,azure-core-http-jdk-httpclient); test:testcontainers-vault,testcontainers-localstack(Boot-BOM managed).