Apply suggestions from code review

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
This commit is contained in:
wiredfool 2025-03-29 23:18:48 +00:00 committed by GitHub
parent cca80e2a23
commit ae187cacb2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 46 additions and 61 deletions

View File

@ -35,17 +35,11 @@ jobs:
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
python-version: [ "3.10", "3.11", "3.12", "3.13" ] python-version: ["pypy3.11", "pypy3.10", "3.10", "3.11", "3.12", "3.13", "3.14"]
architecture: ["x64"] architecture: ["x64"]
os: ["windows-latest"] os: ["windows-latest"]
pyarrow: ["true"]
include: include:
# Test the oldest Python on 32-bit # Test the oldest Python on 32-bit
- { python-version: "3.9", architecture: "x86", os: "windows-2019", pyarrow: "false" }
# test the non-pyarrow capable ones
- { python-version: "3.14", architecture: "x64", os: "windows-latest", pyarrow: "false" }
- { python-version: "pypy3.11", architecture: "x64", os: "windows-latest", pyarrow: "false" }
- { python-version: "pypy3.10", architecture: "x64", os: "windows-latest", pyarrow: "false" }
timeout-minutes: 30 timeout-minutes: 30
name: Python ${{ matrix.python-version }} (${{ matrix.architecture }}) name: Python ${{ matrix.python-version }} (${{ matrix.architecture }})
@ -92,10 +86,9 @@ jobs:
run: | run: |
python3 -m pip install PyQt6 python3 -m pip install PyQt6
- name: Install PyArrow Test Dependency - name: Install PyArrow dependency
if: "matrix.pyarrow == 'true'"
run: | run: |
python3 -m pip install --only-binary=:all: pyarrow python3 -m pip install --only-binary=:all: pyarrow || true
- name: Install dependencies - name: Install dependencies
id: install id: install

View File

@ -4,9 +4,7 @@ import pytest
from PIL import Image from PIL import Image
from .helper import ( from .helper import hopper
hopper,
)
@pytest.mark.parametrize( @pytest.mark.parametrize(

View File

@ -4,71 +4,72 @@
Arrow Support Arrow Support
============= =============
Arrow is an in memory data exchange format that is the spritual `Arrow <https://arrow.apache.org/>`__
successor to the numpy array interface. It provides for zero copy is an in-memory data exchange format that is the spiritual
access to columnar data, which in our case is Image data. successor to the NumPy array interface. It provides for zero-copy
access to columnar data, which in our case is ``Image`` data.
The goal with Arrow is to provide native zero-copy interop with any The goal with Arrow is to provide native zero-copy interoperability
arrow provider or consumer in the Python ecosystem. with any Arrow provider or consumer in the Python ecosystem.
.. warning:: Zero-copy does not mean zero allocation -- The internal .. warning:: Zero-copy does not mean zero allocation -- the internal
memory layout of Pillow images contains an allocation for row memory layout of Pillow images contains an allocation for row
pointers, so there is a non-zero, but significantly smaller than a pointers, so there is a non-zero, but significantly smaller than a
full copy memory cost to reading an arrow image. full-copy memory cost to reading an Arrow image.
Data Formats Data Formats
============ ============
Pillow currently supports exporting arrow images in all modes Pillow currently supports exporting Arrow images in all modes
**except** for ``BGR;15``, ``BGR;16`` and ``BGR;24``. This is due to **except** for ``BGR;15``, ``BGR;16`` and ``BGR;24``. This is due to
line length packing in these modes making for non-continuous memory. line-length packing in these modes making for non-continuous memory.
For single band images, the exported array is width*height elements, For single-band images, the exported array is width*height elements,
with each pixel corresponding to the appropriate arrow type. with each pixel corresponding to the appropriate Arrow type.
For multiband images, the exported array is width*height fixed length For multiband images, the exported array is width*height fixed-length
4 element arrays of uint8. This is memory compatible with the raw four-element arrays of uint8. This is memory compatible with the raw
image storage of 4 bytes per pixel. image storage of four bytes per pixel.
Mode ``1`` images are exported as 1 uint8 byte/pixel, as this is Mode ``1`` images are exported as one uint8 byte/pixel, as this is
consistent with the internal storage. consistent with the internal storage.
Pillow will accept, but not produce, one other format. For any Pillow will accept, but not produce, one other format. For any
multichannel image with 32 bit storage per pixel, Pillow will accept multichannel image with 32-bit storage per pixel, Pillow will accept
an array of width*height int32 elements, which will then be an array of width*height int32 elements, which will then be
interpreted using the mode specific interpretation of the bytes. interpreted using the mode-specific interpretation of the bytes.
The image mode must match the arrow band format when reading single The image mode must match the Arrow band format when reading single
channel images channel images.
Memory Allocator Memory Allocator
================ ================
Pillow's default memory allocator, the :ref:`block_allocator`, Pillow's default memory allocator, the :ref:`block_allocator`,
allocates up to a 16MB block for images by default. Larger images allocates up to a 16 MB block for images by default. Larger images
overflow into additional blocks. Arrow requires a single continuous overflow into additional blocks. Arrow requires a single continuous
memory allocation, so images allocated in multiple blocks cannot be memory allocation, so images allocated in multiple blocks cannot be
exported in the arrow format. exported in the Arrow format.
To enable the single block allocator:: To enable the single block allocator::
from PIL import Image from PIL import Image
Image.core.set_use_block_allocator(1) Image.core.set_use_block_allocator(1)
Note that this is a global setting, not a per image setting. Note that this is a global setting, not a per-image setting.
Unsupported Features Unsupported Features
==================== ====================
* Table/Dataframe protocol. We currently support a single array. * Table/dataframe protocol. We support a single array.
* Null markers, producing or consuming. Null values are inferred from * Null markers, producing or consuming. Null values are inferred from
the mode. e.g. RGB images are stored in the first three bytes of the mode. e.g. RGB images are stored in the first three bytes of
each 32 bit pixel, and the last byte is an implied null. each 32-bit pixel, and the last byte is an implied null.
* Schema Negotiation. There is an optional schema for the requested * Schema negotiation. There is an optional schema for the requested
datatype in the arrow source interface. We currently ignore that datatype in the Arrow source interface. We ignore that
parameter. parameter.
* Array Metadata. * Array metadata.
Internal Details Internal Details
================ ================
@ -76,12 +77,12 @@ Internal Details
Python Arrow C interface: Python Arrow C interface:
https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
The memory that is exported from the arrow interface is shared -- not The memory that is exported from the Arrow interface is shared -- not
copied, so the lifetime of the memory allocation is no longer strictly copied, so the lifetime of the memory allocation is no longer strictly
tied to the life of the python object. tied to the life of the Python object.
The core imaging struct now has a refcount associated with it, and the The core imaging struct now has a refcount associated with it, and the
lifetime of the core image struct is now divorced from the python lifetime of the core image struct is now divorced from the Python
image object. Creating an arrow reference to the image increments the image object. Creating an arrow reference to the image increments the
refcount, and the imaging struct is only released when the refcount refcount, and the imaging struct is only released when the refcount
reaches 0. reaches zero.

View File

@ -3210,16 +3210,9 @@ class SupportsArrowArrayInterface(Protocol):
data interface. data interface.
""" """
# Sorry, no types for you until pre-commit.ci stops changing stringy def __arrow_c_array__(
# PyCapsule type to an unimportable value type, which then fails lint. self, requested_schema: "PyCapsule" = None # type: ignore[name-defined] # noqa: F821, UP037
# PyCapsules are not importable, and only available in the C-API layer. ) -> tuple["PyCapsule", "PyCapsule"]: # type: ignore[name-defined] # noqa: F821, UP037
# def __arrow_c_array__(
# self, requested_schema: 'PyCapsule' = None
# ) -> tuple['PyCapsule', 'PyCapsule']:
# raise NotImplementedError()
# old not typed definition.
def __arrow_c_array__(self, requested_schema=None):
raise NotImplementedError() raise NotImplementedError()
@ -3312,7 +3305,7 @@ def fromarray(obj: SupportsArrayInterface, mode: str | None = None) -> Image:
def fromarrow(obj: SupportsArrowArrayInterface, mode, size) -> Image: def fromarrow(obj: SupportsArrowArrayInterface, mode, size) -> Image:
"""Creates an image with zero copy shared memory from an object exporting """Creates an image with zero-copy shared memory from an object exporting
the arrow_c_array interface protocol:: the arrow_c_array interface protocol::
from PIL import Image from PIL import Image
@ -3323,7 +3316,7 @@ def fromarrow(obj: SupportsArrowArrayInterface, mode, size) -> Image:
If the data representation of the ``obj`` is not compatible with If the data representation of the ``obj`` is not compatible with
Pillow internal storage, a ValueError is raised. Pillow internal storage, a ValueError is raised.
Pillow images can also be converted to arrow objects:: Pillow images can also be converted to Arrow objects::
from PIL import Image from PIL import Image
import pyarrow as pa import pyarrow as pa
@ -3339,13 +3332,13 @@ def fromarrow(obj: SupportsArrowArrayInterface, mode, size) -> Image:
:param size: Image size. This must match the storage of the arrow object. :param size: Image size. This must match the storage of the arrow object.
:returns: An Image Object :returns: An Image Object
Note that according to the arrow spec, both the producer and the Note that according to the Arrow spec, both the producer and the
consumer should consider the exported array to be immutable, as consumer should consider the exported array to be immutable, as
unsynchronized updates will potentially cause inconsistent data. unsynchronized updates will potentially cause inconsistent data.
See: :ref:`arrow-support` for more detailed information See: :ref:`arrow-support` for more detailed information
.. versionadded:: 11.2 .. versionadded:: 11.2.0
""" """
if not hasattr(obj, "__arrow_c_array__"): if not hasattr(obj, "__arrow_c_array__"):

View File

@ -240,14 +240,14 @@ ArrowError(int err) {
return ImagingError_MemoryError(); return ImagingError_MemoryError();
} }
if (err == IMAGING_ARROW_INCOMPATIBLE_MODE) { if (err == IMAGING_ARROW_INCOMPATIBLE_MODE) {
return ImagingError_ValueError("Incompatible Pillow mode for Arrow Array"); return ImagingError_ValueError("Incompatible Pillow mode for Arrow array");
} }
if (err == IMAGING_ARROW_MEMORY_LAYOUT) { if (err == IMAGING_ARROW_MEMORY_LAYOUT) {
return ImagingError_ValueError( return ImagingError_ValueError(
"Image is in multiple array blocks, use imaging_new_block for zero copy" "Image is in multiple array blocks, use imaging_new_block for zero copy"
); );
} }
return ImagingError_ValueError("Unknown Error"); return ImagingError_ValueError("Unknown error");
} }
void void
@ -313,7 +313,7 @@ _new_arrow(PyObject *self, PyObject *args) {
PyImagingNew(ImagingNewArrow(mode, xsize, ysize, schema_capsule, array_capsule) PyImagingNew(ImagingNewArrow(mode, xsize, ysize, schema_capsule, array_capsule)
); );
if (!ret) { if (!ret) {
return ImagingError_ValueError("Invalid arrow array mode or size mismatch"); return ImagingError_ValueError("Invalid Arrow array mode or size mismatch");
} }
return ret; return ret;
} }