GH-47435: [Python][Parquet] Add direct key encryption/decryption API by smaheshwar-pltr · Pull Request #49667 · apache/arrow

smaheshwar-pltr · 2026-04-06T17:24:34Z

Rationale for this change

See #47435.

What changes are included in this PR?

Adds direct encryption / decryption Python API

Are these changes tested?

Yes, see PR.

Are there any user-facing changes?

Yes, new Python bindings.

GitHub Issue: [Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

github-actions · 2026-04-06T17:24:58Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

smaheshwar-pltr · 2026-04-07T03:16:52Z

cc @ggershinsky, this is a requirement for PyIceberg encryption support, see apache/iceberg-python#3221

ggershinsky · 2026-04-07T15:03:17Z

sgtm, #47435 (comment)

github-actions · 2026-04-08T19:04:34Z

⚠️ GitHub issue #47435 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-04-08T19:05:34Z

⚠️ GitHub issue #47435 has been automatically assigned in GitHub to PR creator.

smaheshwar-pltr · 2026-04-08T19:06:01Z

+    This bypasses the KMS-based :class:`CryptoFactory` API and directly
+    constructs decryption properties from a plaintext key. This is useful
+    when the caller manages key wrapping externally (e.g. via an
+    application-level envelope encryption scheme).
+
+    For most use cases, prefer the higher-level :class:`CryptoFactory`
+    with :class:`DecryptionConfiguration`, which handles envelope
+    encryption and key rotation automatically.


Per #47435 (comment), adding this documentation

I don't think it's true that the CryptoFactory handles key rotation automatically, that part of the comment should probably be removed. It does allow key rotation if you use external key material though.

smaheshwar-pltr · 2026-04-16T20:19:12Z

Not sure who to ping here for review, @AlenkaF @raulcd @rok maybe could I please have some eyes on this?

adamreeve

Thanks for implementing this @smaheshwar-pltr! I've left a few suggestions

adamreeve · 2026-04-17T04:01:17Z

+    This bypasses the KMS-based :class:`CryptoFactory` API and directly
+    constructs decryption properties from a plaintext key. This is useful
+    when the caller manages key wrapping externally (e.g. via an
+    application-level envelope encryption scheme).
+
+    For most use cases, prefer the higher-level :class:`CryptoFactory`
+    with :class:`DecryptionConfiguration`, which handles envelope
+    encryption and key rotation automatically.


I don't think it's true that the CryptoFactory handles key rotation automatically, that part of the comment should probably be removed. It does allow key rotation if you use external key material though.

smaheshwar-pltr · 2026-04-17T05:59:36Z

@adamreeve, thank you for the great review here 🙌 . I've reworked the docs now - they are consistent with my (inexperienced) understanding.

Would appreciate your eyes on this again, please let me know what you think!

github-actions · 2026-04-19T23:56:33Z

Revision: 3e529d5

Submitted crossbow builds: ursacomputing/crossbow @ actions-4b28d0a71b

Task	Status
preview-docs

adamreeve

Thanks for the changes @smaheshwar-pltr, I just noticed one more minor issue that I missed first time around sorry.

adamreeve · 2026-04-21T00:29:06Z

@smaheshwar-pltr, can you please comment on #47435 so I can assign it to you? Eg. just a "take" comment.

Then I'll merge this, the CI failures are unrelated and also failing on main.

adamreeve · 2026-04-21T01:11:00Z

I've created #49821 as a follow-up issue to add support for per-column keys.

pitrou · 2026-04-21T16:30:03Z


+    cdef cppclass CFileDecryptionPropertiesBuilder\
+            " parquet::FileDecryptionProperties::Builder":
+        CFileDecryptionPropertiesBuilder() except +


If all these APIs can raise C++ exceptions, which kind of exceptions will be raised on the Python side?

(I don't think these specific calls actually reach a (C++) throw on the paths we hit, so this is being defensive. We do validations python-side that throw TypeError and ValueError on e.g. invalid length / algorithm.)

If an exception was raised somehow though, I think we'd be fine and are consistent with elsewhere in this file - from https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html#exceptions, the except + here would surface it as a RuntimeError.

For example, FileMetaData::AppendRowGroups is also exposed as except +

arrow/python/pyarrow/includes/libparquet.pxd

Line 403 in 640b66c

void AppendRowGroups(const CFileMetaData& other) except +

and reachably throws ParquetException

arrow/cpp/src/parquet/metadata.cc

Line 933 in 640b66c

throw ParquetException(msg);

which would Cython surfaces as RuntimeError - we even have a test for this:

arrow/python/pyarrow/tests/parquet/test_metadata.py

Line 751 in 640b66c

with pytest.raises(RuntimeError, match=prefix + expected_error):

pitrou · 2026-04-21T16:30:24Z

+        b"0123456789abcdef",
+        b"0123456789abcdef01234567",
+        b"0123456789abcdef0123456789abcdef",


Nit: reuse the KEY_ constants above?

Thank you, addressed in 640b66c (I also lifted to module scope to match the other constants)

pitrou · 2026-04-21T16:31:18Z

+        dec_props_no_aad = pe.create_decryption_properties(
+            footer_key=self.KEY_128,
+        )
+        with pytest.raises(IOError):


Can we also test something about the exception message?

Suggested change

with pytest.raises(IOError):

with pytest.raises(IOError, match='XXX...'):

Thank you, addressed in 640b66c.

pitrou · 2026-04-21T16:32:11Z

+            footer_key=self.KEY_128,
+            aad_prefix=b"wrong_prefix",
+        )
+        with pytest.raises(IOError):


Thank you, addressed in 640b66c.

pitrou · 2026-04-21T16:33:54Z

+        CFileDecryptionPropertiesBuilder* builder
+        shared_ptr[CFileDecryptionProperties] props
+
+    footer_key_bytes = tobytes(footer_key)


We shouldn't call tobytes as it will utf8-encode a str object.

Thank you, addressed in 640b66c I think with a isinstance(footer_key, bytes) check.

pitrou · 2026-04-21T16:34:09Z

+        builder.footer_key(c_footer_key)
+
+        if aad_prefix is not None:
+            c_aad_prefix = tobytes(aad_prefix)


Thanks - #49667 (comment)

pitrou · 2026-04-21T16:34:42Z

+        )
+
+    c_footer_key = CSecureString(<c_string>footer_key_bytes)
+    builder = new CFileDecryptionPropertiesBuilder()


Why are we using new? We can create a plain value, I think.

Thanks, switched to CFileDecryptionPropertiesBuilder in 640b66c as described

pitrou · 2026-04-21T16:36:23Z

+        shared_ptr[CFileEncryptionProperties] props
+        ParquetCipher cipher
+
+    footer_key_bytes = tobytes(footer_key)


Same here and other instances below.

Done, thanks

Actually, I had to revert this specifically in 72029b6 due to https://github.com/apache/arrow/actions/runs/25529723022/job/76124851585 - CI failed with "C++ class must have a nullary constructor to be stack allocated". parquet::FileEncryptionProperties::Builder only declares explicit Builder(SecureString footer_key) (no default ctor), so Cython can't stack-allocate it.

smaheshwar-pltr · 2026-05-08T00:34:01Z

Thank you for the review @pitrou 🙏 I believe all comments are now addressed, PTAL!

smaheshwar-pltr · 2026-05-13T20:50:20Z

Friendly ping @adamreeve @pitrou 🙏

smaheshwar-pltr · 2026-05-18T14:21:13Z

Friendly ping here @adamreeve @pitrou / others! Would love to get this in / reviewed!

adamreeve

The latest changes look good to me thanks @smaheshwar-pltr.

The build timing out is a problem on other branches too so looks unrelated to this change.

smaheshwar-pltr · 2026-05-19T22:17:20Z

Thanks @adamreeve I've pushed an empty commit in 5dd54ca to retrigger 🙏

adamreeve · 2026-05-20T01:31:10Z

The timeout is happening consistently but doesn't need to pass for this PR to be merged. I've raised #49998 for this.

I'll leave this PR open a while longer in case @pitrou wants to take another look.

conbench-apache-arrow · 2026-05-25T05:06:47Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 336fdb3.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.

…n API (apache#49667) ### Rationale for this change See apache#47435. ### What changes are included in this PR? Adds direct encryption / decryption Python API ### Are these changes tested? Yes, see PR. ### Are there any user-facing changes? Yes, new Python bindings. * GitHub Issue: apache#47435 Authored-by: Sreesh Maheshwar <smaheshwar@palantir.com> Signed-off-by: Adam Reeve <adreeve@gmail.com>

Support Parquet direct encryption

3a4fb52

github-actions Bot added Component: Python Component: Documentation awaiting review Awaiting review labels Apr 6, 2026

smaheshwar-pltr changed the title ~~Support Parquet direct encryption~~ [Python][Parquet] Add direct key encryption/decryption API Apr 7, 2026

Improvement: Comments

b0cfb95

smaheshwar-pltr mentioned this pull request Apr 7, 2026

POC: Encryption read support apache/iceberg-python#3221

Draft

smaheshwar-pltr changed the title ~~[Python][Parquet] Add direct key encryption/decryption API~~ GH-47435: [Python][Parquet] Add direct key encryption/decryption API Apr 8, 2026

smaheshwar-pltr commented Apr 8, 2026

View reviewed changes

smaheshwar-pltr marked this pull request as ready for review April 8, 2026 19:06

smaheshwar-pltr requested review from AlenkaF, raulcd and rok as code owners April 8, 2026 19:06

adamreeve reviewed Apr 17, 2026

View reviewed changes

github-actions Bot added awaiting review Awaiting review awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 17, 2026

Sreesh Maheshwar added 2 commits April 16, 2026 22:40

Address review comments

40ef506

Merge branch 'main' into parquet-direct-key-encryption

76c8211

github-actions Bot added awaiting review Awaiting review and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Apr 17, 2026

Nit: Unfix trailing whitespace

3e529d5

adamreeve reviewed Apr 20, 2026

View reviewed changes

Comment thread python/pyarrow/_parquet_encryption.pyx Outdated

Fixes

0959c7d

adamreeve approved these changes Apr 20, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Apr 20, 2026

adamreeve mentioned this pull request Apr 21, 2026

[Python][Parquet] Per-column keys for low-level encryption/decryption properties API #49821

Open

pitrou reviewed Apr 21, 2026

View reviewed changes

smaheshwar-pltr mentioned this pull request Apr 22, 2026

[Parquet][Python] API to decrypt parquet file using one DEK and no metadata #47435

Closed

Address review comments

640b66c

smaheshwar-pltr force-pushed the parquet-direct-key-encryption branch from 5c018f7 to 640b66c Compare May 2, 2026 01:06

Nit improvements

606d0e8

smaheshwar-pltr requested review from adamreeve and pitrou May 8, 2026 00:31

Sreesh Maheshwar added 2 commits May 16, 2026 01:01

Fix CI by reverting stack allocation on encryption side

72029b6

One line error message

3772bd7

adamreeve approved these changes May 19, 2026

View reviewed changes

empty

5dd54ca

adamreeve merged commit 336fdb3 into apache:main May 24, 2026
17 of 18 checks passed

adamreeve removed the awaiting committer review Awaiting committer review label May 24, 2026

smaheshwar-pltr mentioned this pull request May 27, 2026

Feature Request: Encryption apache/iceberg-python#3222

Open

	with pytest.raises(IOError):
	with pytest.raises(IOError, match='XXX...'):

Conversation

smaheshwar-pltr commented Apr 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Apr 6, 2026

Uh oh!

smaheshwar-pltr commented Apr 7, 2026

Uh oh!

ggershinsky commented Apr 7, 2026

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Apr 16, 2026

Uh oh!

adamreeve left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smaheshwar-pltr commented Apr 17, 2026

Uh oh!

github-actions Bot commented Apr 19, 2026

Uh oh!

adamreeve left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamreeve commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamreeve commented Apr 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented May 8, 2026

smaheshwar-pltr commented Apr 6, 2026 •

edited by github-actions Bot

Loading

adamreeve commented Apr 21, 2026 •

edited

Loading