Lucene optimizations

Reduced Indexing Latency

Create different types of indexable items (might need more even granularity in pubsub), so that we can priotize indexing. I.e. meta-data changes should be immediate, while content changes might be somewhat delayed for accumulation, and files might be further delayed. This would require that we can fetch content from a cache on meta-data changes and that the cache is only invalidated on content changes (i.e. if its invalidated on meta-data changes, then meta-data index updates will still incur the full indexing cost.

This may also include the ability to manually trigger immediate indexing of a document, so that professional services could prioritze certain content over other content in response to events.

ACL in lucene

Give lucene the ability to make ACL callbacks against hits Hits collection so that limit and offset are applied to filtered results and let work is required oin the Deki side to post-process results.

Changes to Cached Query Path

Optionally Invalidate cache in cached query path, so that it's more compact result set can be used in more scenarios

More flexible cache format (at least needs to support user results as well).

Re-index

Have re-index populate a separate index that can be swapped out at re-index completion. This means it has a separate queue that can be separately throttled to reduce the impact on live operations. The re-index queue would need to receive both all the re-index document items and every item that is changing in the current index since the re-index started so that it stays current.