Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
//! Large narrow rewrites use the existing native preparation/reference encoding.
use super::*;
use crate::materializer::observation::{Counters, Phase};
use lance::dataset::optimize::RewriteResult;
use lance_context_core::annotation_staging::RESULTS_PROPERTY;

Expand All @@ -9,23 +10,30 @@ pub(crate) async fn descriptors(
selection: &Selection,
storage: &Storage,
maximum: u64,
phases: &Arc<Counters>,
) -> Result<p::Descriptors> {
let mut transaction = Transaction::new_from_version(
base.version_id(),
lance_context_core::native_compaction::prepare_rewrite(
base,
result,
metadata::MAX_NATIVE_BYTES,
let rewrite = phases
.measure(
Phase::AnnotationCompactionRemap,
lance_context_core::native_compaction::prepare_rewrite(
base,
result,
metadata::MAX_NATIVE_BYTES,
),
)
.await?,
);
transaction.uuid = selection.pending.plan.transaction.to_string();
transaction.transaction_properties = base
.read_transaction()
.await?
.and_then(|t| t.transaction_properties);
validate(base, &transaction, selection).await?;
persist_transaction(base, &transaction, selection, storage, maximum).await
.await?;
phases
.measure(Phase::AnnotationCompactionMetadata, async {
let mut transaction = Transaction::new_from_version(base.version_id(), rewrite);
transaction.uuid = selection.pending.plan.transaction.to_string();
transaction.transaction_properties = base
.read_transaction()
.await?
.and_then(|t| t.transaction_properties);
validate(base, &transaction, selection).await?;
persist_transaction(base, &transaction, selection, storage, maximum).await
})
.await
}

pub(super) async fn validate(
Expand Down
4 changes: 2 additions & 2 deletions crates/lance-context-ingest/src/materializer/compaction.rs
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ pub(super) async fn prepare(
if result.row_addrs.is_some() {
return if annotations {
crate::annotation_publication::compaction::descriptors(
&dataset, result, &selection, &storage, maximum,
&dataset, result, &selection, &storage, maximum, &phases,
)
.await
} else {
Expand All @@ -291,7 +291,7 @@ pub(super) async fn prepare(
let bytes = serde_json::to_vec(&result)?;
if annotations && bytes.len() as u64 > maximum {
crate::annotation_publication::compaction::descriptors(
&dataset, result, &selection, &storage, maximum,
&dataset, result, &selection, &storage, maximum, &phases,
)
.await
} else {
Expand Down
9 changes: 8 additions & 1 deletion crates/lance-context-ingest/src/materializer/observation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,12 @@ pub enum Phase {
AnnotationMaintenancePrepare,
/// Resolving, installing and finishing the prepared maintenance publication.
AnnotationMaintenancePublish,
/// Validating a narrow rewrite and preparing its native index remaps.
AnnotationCompactionRemap,
/// Validating and persisting narrow rewrite metadata after index remapping.
AnnotationCompactionMetadata,
}
const NAMES: [&str; 51] = [
const NAMES: [&str; 53] = [
"claim",
"acquire",
"state",
Expand Down Expand Up @@ -148,6 +152,8 @@ const NAMES: [&str; 51] = [
"annotation_maintenance_select",
"annotation_maintenance_prepare",
"annotation_maintenance_publish",
"annotation_compaction_remap",
"annotation_compaction_metadata",
];
pub const PHASE_COUNT: usize = NAMES.len();

Expand Down Expand Up @@ -175,6 +181,7 @@ impl Default for Counters {
pub struct Observation {
pub phase: &'static str,
pub started: u64,
/// Finished calls, including the failed subset below; excludes cancellation.
pub completed: u64,
pub failed: u64,
pub cancelled: u64,
Expand Down
9 changes: 9 additions & 0 deletions docs/design/annotation-application.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,15 @@ catalog publication. They distinguish actual native work from the parent
`compact` scheduling checks. Each child records one admitted native call, not
one fragment, cloud request or successfully published version.

After a narrow native rewrite, `annotation_compaction_remap` measures rewrite
validation and native index remapping. `annotation_compaction_metadata` then
measures transaction assembly, validation and descriptor persistence. These
steps are outside `native_compaction`; an already completed file rewrite can
still be waiting on index reads or metadata writes before publication. These
phases cover narrow native descriptor preparation, not body-table maintenance.
For all phase counters, `completed` includes `failed`; cancellations are counted
separately. Fields are sampled independently and are not an atomic progress view.

Annotation commit resolution reports four nested phases.
`annotation_commit_base` loads the fixed predecessor;
`annotation_commit_publish` validates and publishes the prepared target when it
Expand Down
Loading