Iceberg V4 Equality Delete Deprecation and Polaris MCP Auth Token Boundaries

Alex Merced’s Apache Data Lakehouse Weekly: July 21 to July 29, 2026 rounds up what is moving in Apache Iceberg, Polaris, Parquet, Arrow, DataFusion, and Ossie.

Project	What is moving	Status
Iceberg	Deprecating equality deletes in V4	Under discussion
Iceberg	REST catalog expression representation, remote signing, incremental scan	Under discussion
Parquet	Adding ALP encoding for floating-point values to the format	Vote passed
Parquet	Inlining `parquet.thrift` into `parquet-java`	Proposal raised
Polaris	1.7.0 rc0	Withdrawn over missing license documents
Polaris	Principal properties and authentication modes for MCP	Under consideration
Arrow	Meaning of the nullability flag on nested types	Under discussion

My earlier post on the MCP 2026-07-28 spec making the protocol stateless looked at the removal of session IDs and at server/discover from the protocol side.
What Polaris is debating here is the principal that connects from an MCP process to the catalog. Whose credentials reach the catalog is decided on the catalog side, separately from the protocol going stateless.

Iceberg V4 deprecates equality deletes

In Iceberg, a discussion has started about deprecating equality deletes in V4.
An equality delete lets the writer record a deletion such as “delete the row where id is 42” without knowing which file holds the target row.
Streaming writers can process deletes without resolving the in-file position of the target rows in advance.
In exchange, readers match the delete conditions on every scan. The planner has no condition to narrow down which files a delete applies to, so it evaluates the delete against a broader set of candidates.

In the Weekly, Ryan Blue strongly supports Huaxin Gao’s proposal, laying out a plan that keeps equality deletes in existing V2/V3 tables readable after an upgrade while banning new writes in V4.
As migration paths, background conversion by Flink and generating delete vectors at write time with a future shared index are under discussion.

The proposal affects setups that feed Flink or CDC (change data capture) ingestion directly into Iceberg.
Existing deletes are resolved through conversion or compaction. Converted deletes are replaced by delete vectors, and after compaction they are reflected in the rewritten data. For writers that cannot resolve row positions, the discussion covers two paths: keep writing equality deletes in V3 while running background conversion, or generate delete vectors with a future shared index and move to V4.

flowchart TD
    EQ[equality deletes in<br/>V2/V3 tables] -->|still readable<br/>after upgrade| V4[V4 tables<br/>new writes banned]
    EQ -->|background conversion<br/>by Flink| DV[replaced by<br/>delete vectors]
    W[writers that cannot<br/>resolve row positions] -->|option 1| EQ3[keep writing equality<br/>deletes in V3 while<br/>running conversion]
    W -->|option 2| DV2[generate delete vectors with a<br/>future shared index, move to V4]

For incremental scans, there is also a discussion about pinning down implementation differences in the spec.
The current REST incremental scan API returns a set of files and handles append-only diffs.
Diffs that include deletes or updates cannot be expressed as a set of files.
For V4 change detection, the direction is to make the meaning of a diff between snapshots readable from the spec itself.

The Parquet ALP encoding vote

On the Parquet side, the proposal to add ALP encoding to the format passed its vote.
According to the Weekly, there were 11 +1 votes, 7 of them binding.
ALP stands for Adaptive Lossless floating-Point. It maps double and float values toward a decimal representation, encodes them as integers, and falls back to a different scheme for values that do not fit the pattern.

An engine that reads Parquet files cannot read a column written with ALP unless it supports the encoding.
The official ALP specification lists decimal-origin sensor values and prices as examples suited to ALP.
For other floating-point columns such as embedding components, suitability depends on the exception rate and compression ratio of the actual data.
Using ALP requires support on both the write side and the read side.

In the same Parquet week, a proposal to inline parquet.thrift into parquet-java was also raised.
Today parquet-java depends on the Thrift definition shipped inside the parquet-format JAR, so even trying out a format change on the Java side has to wait for a release of the format definition JAR.
A single source of truth keeps the definitions from drifting apart, but Java-side CI and merges cannot move until a new format definition JAR is published.

Authentication modes in polaris-tools

In Polaris, alongside the withdrawal of 1.7.0 rc0, authentication modes for polaris-tools were also discussed.
In the same week’s rc0 vote, 44 of the 269 staged JARs were missing META-INF/LICENSE and META-INF/NOTICE, a binding -1 vote came in, and the vote was cancelled.

The authentication modes under discussion for a remote shared MCP exposed over HTTP/SSE are service credentials, caller token forwarding, and OAuth token exchange.

Setup	How the token is handled
Local, single user	Use the user’s own Polaris token
Remote shared, the received token targets Polaris	Forward that token as is (caller token forwarding)
Remote shared, the server receives a token targeting the MCP server	An external authorization server issues a separate Polaris-targeted token (OAuth token exchange is the candidate)

The design items for the remote shared setup are credential precedence, where tokens and headers come from, and how to identify both the user and the service.
The open question is whether reads, permission changes, and directory or catalog creation performed by an AI agent get recorded as the user’s operation or as the service’s.
Making MCP stateless is a change that reduces session management in HTTP transport; the authentication principal for the catalog is defined on the Polaris side.

The original thread lists token audience (the target service), header provenance, trusted proxies, and service identity as design items for remote MCP.

Is Arrow’s nullability flag an observation or a contract

For Arrow’s nullability flag, there is a debate over what nullable: false guarantees.
When nullable: false is set on a nested type such as struct, list, or map, the spec can be read either way: it may cover only the field itself, or extend to nulls inside child elements.

Reading	Meaning	Downstream handling
Observation	This batch contains no nulls	Treated as transient information
Contract	No nulls will ever arrive on this stream	Grounds for optimizers and readers to skip null checks

In the Arrow format, parent and child arrays hold separate validity bitmaps, and nullable: false on a parent field does not determine whether child elements can be null.