ATLAS-5317: Resilient bulk purge with pre-validation, enriched responses, and worker batches

[sheetalshah1007]

Attachments

1. Design specification:ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-DESIGN.pdf`
2. Implementation notes: ATLAS-5317-Resilient-Bulk-Purge-In-Atlas-IMPLEMENTATION_NOTES.docx

---

What changes were proposed in this pull request?

This PR implements resilient bulk purge per the attached design specification (Resilient Bulk Purge in Atlas Using Transaction Batching). It replaces the old single-transaction REST bulk purge with pre-validated, worker-batch processing shared by REST and scheduled (cron) purge.

Key changes

• Pre-validation before writes — REST purge GUIDs are checked for UUID format, graph existence, DELETED state, and registered type before any mutation. Invalid GUIDs are returned in failedEntities with errorCode / errorMessage.
• Request size limit — New config atlas.purge.api.max.request.size (default 1000) caps GUIDs per REST purge request.
• Shared worker-batch path — REST and cron both use PurgeBatchOrchestrator / WorkItemManager with atlas.purge.worker.batch.size (default 100) and atlas.purge.workers.count (default 2).
• Isolated batch transactions — Each worker batch runs in its own graph transaction. Lock conflicts retry at batch level; other batch failures roll back only that batch and processing continues.
• Partial success — Earlier committed batches are kept when a later batch fails permanently.
• Enriched response — EntityMutationResponse adds failedEntities and PurgeSummary (including purgedDependenciesCount / failedDependenciesCount).
• Per-batch audit — Successful batches write bounded PURGE / AUTO_PURGE audit entries (up to 10 sample GUIDs), not one large entry per REST/cron call.
• Cron alignment — Scheduled purge keeps its existing worker-batch model; REST is aligned to that path with shared orchestration, enriched accounting, batch retry, and shutdown reconciliation.
• Parallel-batch hardening — DeleteHandlerV1 tolerates vertices/edges already removed by another worker (ATLAS-4766). accumulateDeletionCandidates() returns Set<String> (GUIDs) for worker-safe enqueueing.
• Schema — AUTO_PURGE added to atlas_operation in 0010-base_model.json for scheduled purge audit on fresh installs.

Compatibility and behavior notes

• accumulateDeletionCandidates() now returns Set<String> (GUIDs) instead of Set<AtlasVertex>. Custom AtlasEntityStore implementations should update their method signature; Apache Atlas ships only AtlasEntityStoreV2.
• Missing or ineligible GUIDs appear in failedEntities on REST (no longer silently ignored).
• Completed purge responses return HTTP 200 with outcome in the body — read summary and failedEntities for the result (implementation plan Section 10).
• Large purges produce multiple audit entries (one per successful worker batch).
  For full flow, design deviations, configuration, and review focus areas, see the attached implementation plan.

---

How was this patch tested?

Unit and module tests were added or extended in this PR.

Area	Test class	Coverage
Worker orchestration	PurgeBatchOrchestratorTest, PurgeBatchExecutorTest	WIM lifecycle, batch commit, lock retry, shutdown reconciliation, per-batch audit
Store / REST purge	AtlasEntityStoreV2Test	Pre-validation, size limit, success/partial failure, dependency expansion, concurrent expansion vs worker delete, summary accounting
Scheduled purge	PurgeServiceTest	Shared worker path, REST/cron GUID overlap
Delete handler	DeleteHandlerV1Test	Parallel batch purge; skip already-removed vertices/edges
REST layer	AdminResourceTest, AtlasBaseExceptionMapperTest	No aggregate audit on REST purge; request-size-exceeded → HTTP 400
Client / model	AtlasClientV2Test, TestEntityMutationResponse	HTTP 200 + purge field deserialization; addEntity GUID dedup
End-to-end purge	AdminPurgeTest	REST purge with enriched response and summary

Detailed test mapping: implementation plan Section 11.

Suggested verification commands:

mvn test -pl repository \
  -Dtest=PurgeBatchOrchestratorTest,PurgeBatchExecutorTest,PurgeServiceTest,AtlasEntityStoreV2Test,DeleteHandlerV1Test,AdminPurgeTest \
  -Drat.skip=true
mvn test -pl webapp \
  -Dtest=AdminResourceTest,AtlasBaseExceptionMapperTest \
  -Drat.skip=true
mvn test -pl intg,client/client-v2 \
  -Dtest=TestEntityMutationResponse,AtlasClientV2Test \
  -Drat.skip=true