perf: one-transaction flush + async indexing write path, read-path fixes

[harshach]

Describe your changes:

Fixes #28726

Optimizes the OpenMetadata read and write paths for lower API latency. Reads: several repositories overrode setFields but not setFieldsInBulk, silently dropping list fields and forcing per-item queries — added bulk field-fetch paths (Directory, Spreadsheet, all *Service, KnowledgePage, ContextMemory), dropped a redundant CSV-export tag rescan, batched single-entity tag inserts, and cached Fernet keys. Writes — Stage 1: the create/update/patch flush committed 5–7 times via per-DAO autocommit (the @Transaction annotations were decorative no-ops) and left orphan rows on mid-flush failure; it is now wrapped in one real Entity.getJdbi().inTransaction(...) + DeadlockRetry, with every external side-effect (cache-L2 invalidation, RDF/SPARQL, lineage-ES, rename-cascade reindex) deferred so the held DB connection makes no network round trip. Writes — Stage 2: those externals + the ES entity index now run asynchronously off the request thread on a new per-entity-ordered OrderedLaneExecutor (durable via the search retry outbox), while cache write-through stays synchronous post-commit so GET /{id} and /name/{fqn} remain read-your-write — only /search and SPARQL become eventually consistent.

Type of change:

• Improvement

High-level design:

• Stage 1 wraps EntityRepository create/update/patch flushes in inTransaction + deadlock-retry (mirroring the existing cleanup() idiom), collapsing 5–7 commits → 1 and making writes atomic. In-flush external calls are captured into thread-local collectors and drained post-commit; EntityUpdater state is snapshotted/restored per retry attempt (incl. the 6 cascade-bearing subclass updaters).
• Stage 2 routes those drains plus the ES entity index (SearchIndexHandler.isAsync()=true) onto OrderedLaneExecutor, keyed by entity id so single-consumer lanes preserve per-entity ordering. A full lane sheds to SearchIndexRetryQueue (a fast local-DB upsert) rather than blocking the request thread; bulk create slices per-entity onto each member's own lane.
• Consistency contract: GET-by-id/by-name stay real-time (DB + sync cache); search/RDF/lineage are eventually consistent with durable retry. Redis stays opt-in (CACHE_PROVIDER=none default).
• Honest residual: under pathological lane overflow/hard-stop, the outbox rebuilds an entity's doc + lineage from committed DB, but RDF triples and rename-cascade descendant reindexes are best-effort.

Tests:

Use cases covered

• Create/update/patch is atomic (mid-flush failure → zero orphan rows) and deadlock-replay-safe, incl. glossary-term rename and data-product domain-change cascades.
• Write returns before the doc is searchable; doc converges in search; same-entity create→update converges to the final value (lane ordering).
• Previously-dropped list fields now populate (Drive directories/spreadsheets, *Service pipelines, Knowledge Center hierarchy, Context memories).

Backend integration tests

• Added/updated in openmetadata-integration-tests/.
• OneTransactionFlushAtomicityIT, AsyncSearchIndexConsistencyIT, DirectoryResourceIT, SpreadsheetResourceIT, DatabaseServiceResourceIT, ContextMemoryIT, DatabaseResourceIT, TableResourceIT, KnowledgePageHierarchyIT, ColumnSearchIndexIT/TestCaseResourceIT (Awaitility polling for async), WritePathBenchmarkIT (inert perf harness).

Unit tests

• OrderedLaneExecutorTest, EntityLifecycleEventDispatcherTest, SearchIndexHandlerTest, FernetKeyCacheTest.

Ingestion integration tests

• Not applicable (no ingestion changes).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

• Ran targeted ITs + the full IT suite (mvn verify -Ppostgres-elasticsearch, ~13.5k tests) on cache=none and cache=redis; all green except pre-existing async-workflow flakiness.
• Write-path benchmark on a durable (fsync=on) Postgres showed ~30–40% median write-latency improvement; the residual p99 tail is DB-commit/checkpoint + GC, not the OM write path.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have added tests (unit / integration) and listed them above.
• For JSON Schema changes: N/A (no schema changes).

[harshach]

Thanks for the review — both findings are addressed in 18d2d92.

Finding 1 (Bug — async indexer captures a mutable entity)
SearchIndexHandler.isAsync() is now true, so the ordered lane reads the entity off the request thread after dispatch — but the request thread keeps mutating that same instance for the HTTP response (PII masking, clearFields stripping, secret masking on service connections). The lane could therefore index a masked/stripped/partial doc. Per-entity lane ordering doesn't help here because the race is on the in-memory POJO, not on lane submission order.

Fix: executeHandler(EntityInterface, …) now takes a Consumer<EntityInterface>, and the async branch deep-copies the committed state before submitting to the lane:

EntityInterface snapshot =
    JsonUtils.readValue(JsonUtils.pojoToJson(entity), entity.getClass());
orderedLaneExecutor.submit(
    entity.getId(), laneTask(entity, operation, () -> handlerCall.accept(snapshot)));

The sync branch still binds the live entity (no copy cost when nothing is async). The EntityReference overload is unaffected (references aren't response-mutated). The six onEntity* call sites were updated from () -> handler.onX(entity, …) to e -> handler.onX(e, …).

Finding 2 (Quality — invalidateCacheForTaggedEntities(String) always returned 0)
deferOrRunSearchBackedInvalidation discarded the inline count. It now takes an IntSupplier, so the inline (non-flush) path returns the live invalidated count:

} else {
  result = invalidation.getAsInt();
}

The deferred path still returns 0 — the work runs post-commit and the count is only known then (it's logged inside searchTaggedEntitiesAndInvalidate); this is documented in the method's Javadoc.

mvn spotless:check and a service-module compile both pass.

[github-actions]

🟡 Playwright Results — all passed (16 flaky)

✅ 4264 passed · ❌ 0 failed · 🟡 16 flaky · ⏭️ 88 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	300	0	1	4
🟡 Shard 2	801	0	3	9
🟡 Shard 3	802	0	3	8
🟡 Shard 4	852	0	3	12
🟡 Shard 5	720	0	1	47
🟡 Shard 6	789	0	5	8

🟡 16 flaky test(s) (passed on retry)

• Flow/SearchRBAC.spec.ts › User without permission (shard 1, 1 retry)
• Features/DataProductRename.spec.ts › should handle multiple consecutive renames and preserve assets (shard 2, 1 retry)
• Features/DataQuality/BundleSuiteBulkOperations.spec.ts › Add test case to existing Bundle Suite (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should display correct status badge color and icon (shard 2, 1 retry)
• Features/KnowledgeCenterList.spec.ts › Knowledge Center List - Test unbookmark functionality (shard 3, 1 retry)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Features/Table.spec.ts › Table pagination with sorting should works (shard 3, 1 retry)
• Pages/CustomProperties.spec.ts › Time Interval (shard 4, 1 retry)
• Pages/CustomProperties.spec.ts › Time Interval (shard 4, 1 retry)
• Pages/DomainUIInteractions.spec.ts › Delete domain with subdomains shows warning (shard 4, 1 retry)
• Pages/ExplorePageRightPanel_KnowledgeCenter.spec.ts › Should remove user owner for knowledgeCenter (shard 5, 1 retry)
• Pages/Lineage/DataAssetLineage.spec.ts › Column lineage for apiEndpoint -> mlModel (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify Impact Analysis service filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage service type filter selection (shard 6, 2 retries)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/Lineage/PlatformLineage.spec.ts › Verify domain platform view (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[manerow]

Looks good to me. The one-transaction wrap and deferring the external side effects out of the held connection reads cleanly. Approving.

[gitar-bot]

Code Review ✅ Approved 7 resolved / 7 findings

Optimizes API latency by implementing single-transaction flushes and an ordered asynchronous indexing path for write operations. Resolved issues regarding stale entity snapshots, cache invalidation, and regression in bot owner updates.

✅ 7 resolved

✅ Quality: invalidateCacheForTaggedEntities now always returns 0

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:3217-3231 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:3282-3294
invalidateCacheForTaggedEntities(String) previously returned the live count of invalidated entities. After the refactor it delegates to deferOrRunSearchBackedInvalidation(Runnable, String), which executes the invalidation as a Runnable and discards its result, hard-coding int result = 0 on both the inline and deferred paths (EntityRepository.java:3217-3244). The actual count produced by searchTaggedEntitiesAndInvalidate is dropped.

Impact is limited because no caller uses the count for control flow — the bulk variant invalidateCacheForTaggedEntities(Collection) only aggregates it for the log line "Invalidated cache for {} entities across {} renamed tag FQNs" (EntityRepository.java:3288-3290), which will now always report 0 (and the if (total > 0) guard means it never logs at all). The inner per-tag log inside searchTaggedEntitiesAndInvalidate still reports the true count, so the regression is purely misleading/absent aggregate logging. Suggested fix: have the inline path return the real count by using a small int-returning helper instead of a Runnable, or document that the return value is no longer meaningful and drop the dead total/logging in the bulk variant.

✅ Bug: Async indexer may read entity POJO mutated after dispatch

📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/handlers/SearchIndexHandler.java:150-157 📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:328-342
SearchIndexHandler.isAsync() was flipped from false to true (SearchIndexHandler.java:150-157), so the search document is now built and written on a background OrderedLaneExecutor lane thread that captures the same EntityInterface instance by reference in the dispatched lambda (EntityLifecycleEventDispatcher.executeHandler -> laneTask). The previous synchronous behavior was deliberately chosen — the removed comment stated "Search indexing must be visible to follow-up operations in the same request flow" and indexing ran inline before the request thread continued.

With async indexing there is now a window where the request thread can continue mutating the very same entity instance after dispatch but before the lane thread reads it to build the doc — e.g. REST-layer PII masking, clearFields/field stripping for the HTTP response, or secret encryption/decryption on service-connection configs. If any such post-dispatch mutation touches the same instance, the async indexer can observe and persist a masked/cleared/partial document into the search index. Per-entity lane ordering does not protect against this because the race is on the in-memory POJO, not on lane submission order.

I was unable to fully confirm a concrete post-dispatch mutation site within this review, so treat this as a potential issue: please verify that the entity object handed to the dispatcher is either deep-copied before async submission or is never mutated by the resource/masking/encryption layers after onEntityCreated/onEntityUpdated is called. If it can be mutated, snapshot (deep-copy) the entity at dispatch time before submitting it to the lane.

✅ Bug: Unguarded snapshot serialization can fail API after commit

📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:331-343
executeHandler now performs JsonUtils.readValue(JsonUtils.pojoToJson(entity), entity.getClass()) synchronously on the request thread, before the task is submitted to the ordered lane. This call is not wrapped in any try/catch:

• The async dispatch entry points (onEntityCreated/onEntityUpdated/onEntityDeleted) are invoked from EntityRepository.postCreate/postUpdate/post-delete hooks, which run AFTER the DB transaction has committed and do not catch RuntimeExceptions escaping the dispatcher.
• If pojoToJson/readValue throws for any entity (e.g. a field that fails to round-trip, an unexpected subtype, a serialization edge case), the exception propagates to the REST request thread. The client receives a 5xx even though the entity was already committed to the DB — a confusing 'write failed but actually succeeded' outcome.
• Worse, this also defeats the durability model this PR carefully builds: because the throw happens before orderedLaneExecutor.submit(...), the enqueueLaneFailureRetry/SearchIndexRetryQueue outbox path never fires, so the search-index update is silently lost rather than retried.

Wrap the snapshot creation in a try/catch and, on failure, route the locator to the durable retry outbox (so the worker rebuilds the doc from committed DB state) instead of letting it break the request and drop the index update.

✅ Performance: Entity re-serialized once per async handler in dispatch loop

📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:326-340 📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:144-148 📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:183-187
executeHandler is called once per applicable handler inside the dispatch loops (onEntityCreated, dispatchBulkCreate, onEntityUpdated, dispatchBulkUpdate, etc.). The snapshot JsonUtils.readValue(JsonUtils.pojoToJson(entity), ...) is therefore performed once per async handler for the same entity. When multiple async handlers apply to an entity type, the same POJO is serialized+deserialized N times on the request thread; for bulk create/update this multiplies to (entities × async handlers) full round-trips on the request path — exactly the path this PR is trying to make faster. Consider computing the snapshot once per entity (e.g. lazily on first async handler) and reusing it across handlers for the same dispatch.

✅ Edge Case: Snapshot failure drops non-search async handlers silently

📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:355-360 📄 openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java:386-400
When snapshotOrEnqueueRetry fails to serialize an entity, it returns null and executeHandler skips the lane submit for every async handler (the shared/memoized supplier returns null for all of them). However, the only durable recovery path enqueued is SearchIndexRetryQueue.enqueue(...), which the worker uses to rebuild the search document. There are three async handlers dispatched through this path — SearchIndexHandler, DomainSyncHandler (task/announcement domain sync), and VectorEmbeddingHandler — and the latter two are NOT recovered by the search-index retry outbox. So on a (rare/pathological) post-commit serialization failure, domain-sync and vector-embedding side effects are silently lost with no retry, while the Javadoc only describes the search-doc recovery.

Impact is limited because serialization failures are pathological and the write itself is already committed, so this is minor. Suggested fix: either document this explicitly as a known best-effort residual (matching the RDF/rename-cascade caveats in the PR description), or enqueue the locator into the per-handler recovery mechanisms (not just the search outbox) so domain-sync/vector embeddings also converge.

...and 2 more resolved from earlier reviews

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}