Merge pull request #6947 from nulano/winbuild-ninja

.github/workflows/test-windows.yml

@@ -97,7 +97,7 @@ jobs:
     - name: Prepare build
       if: steps.build-cache.outputs.cache-hit != 'true'
       run: |
-        & python.exe winbuild\build_prepare.py -v --python=$env:pythonLocation --srcdir
+        & python.exe winbuild\build_prepare.py -v --python $env:pythonLocation
       shell: pwsh
 
     - name: Build dependencies / libjpeg-turbo

winbuild/README.md

@@ -10,7 +10,7 @@ For more extensive info, see the [Windows build instructions](build.rst).
 
 * Requires Microsoft Visual Studio 2017 or newer with C++ component.
 * Requires NASM for libjpeg-turbo, a required dependency when using this script.
-* Requires CMake 3.12 or newer (available as Visual Studio component).
+* Requires CMake 3.15 or newer (available as Visual Studio component).
 * Tested on Windows Server 2016 with Visual Studio 2017 Community, and Windows Server 2019 with Visual Studio 2022 Community (AppVeyor).
 * Tested on Windows Server 2022 with Visual Studio 2022 Enterprise (GitHub Actions).
 

winbuild/build.rst

@@ -21,10 +21,13 @@ Download and install:
   <https://visualstudio.microsoft.com/downloads/>`_
   (MSVC C++ build tools, and any Windows SDK version required)
 
-* `CMake 3.12 or newer <https://cmake.org/download/>`_
+* `CMake 3.15 or newer <https://cmake.org/download/>`_
   (also available as Visual Studio component C++ CMake tools for Windows)
 
-* x86/x64: `NASM <https://www.nasm.us/pub/nasm/releasebuilds/?C=M;O=D>`_
+* `Ninja <https://ninja-build.org/>`_
+  (optional, use ``--nmake`` if not available; bundled in Visual Studio CMake component)
+
+* x86/x64: `Netwide Assembler (NASM) <https://www.nasm.us/pub/nasm/releasebuilds/?C=M;O=D>`_
 
 Any version of Visual Studio 2017 or newer should be supported,
 including Visual Studio 2017 Community, or Build Tools for Visual Studio 2019.
@@ -35,41 +38,50 @@ Visual Studio is found automatically with ``vswhere.exe``.
 Build configuration
 -------------------
 
-The following environment variables, if set, will override the default
+Run ``build_prepare.py`` to configure the build::
-behaviour of ``build_prepare.py``:
 
-* ``PYTHON`` + ``EXECUTABLE`` point to the target version of Python.
+    usage: winbuild\build_prepare.py [-h] [-v] [-d PILLOW_BUILD]
-  If ``PYTHON`` is unset, the version of Python used to run
+                                     [--depends PILLOW_DEPS]
-  ``build_prepare.py`` will be used. If only ``PYTHON`` is set,
+                                     [--architecture {x86,x64,ARM64}]
-  ``EXECUTABLE`` defaults to ``python.exe``.
+                                     [--python PYTHON] [--executable EXECUTABLE]
-* ``ARCHITECTURE`` is used to select a ``x86``, ``x64`` or ``ARM64`` build.
+                                     [--nmake] [--no-imagequant] [--no-fribidi]
-  By default, uses same architecture as the version of Python used to run ``build_prepare.py``.
-* ``PILLOW_BUILD`` can be used to override the ``winbuild\build`` directory
-  path, used to store generated build scripts and compiled libraries.
-  **Warning:** This directory is wiped when ``build_prepare.py`` is run.
-* ``PILLOW_DEPS`` points to the directory used to store downloaded
-  dependencies. By default ``winbuild\depends`` is used.
 
-``build_prepare.py`` also supports the following command line parameters:
+    Download dependencies and generate build scripts for Pillow.
 
-* ``-v`` will print generated scripts.
+    options:
-* ``--no-imagequant`` will skip GPL-licensed ``libimagequant`` optional dependency
+      -h, --help            show this help message and exit
-* ``--no-fribidi`` or ``--no-raqm`` will skip optional LGPL-licensed dependency FriBiDi
+      -v, --verbose         print generated scripts
-  (required for Raqm text shaping).
+      -d PILLOW_BUILD, --dir PILLOW_BUILD, --build-dir PILLOW_BUILD
-* ``--python=<path>`` and ``--executable=<exe>`` override ``PYTHON`` and ``EXECUTABLE``.
+                            build directory (default: 'winbuild\build')
-* ``--architecture=<arch>`` overrides ``ARCHITECTURE``.
+      --depends PILLOW_DEPS
-* ``--dir=<path>`` and ``--depends=<path>`` override ``PILLOW_BUILD``
+                            directory used to store cached dependencies (default:
-  and ``PILLOW_DEPS``.
+                            'winbuild\depends')
+      --architecture {x86,x64,ARM64}
+                            build architecture (default: same as host Python)
+      --python PYTHON       Python install directory (default: use host Python)
+      --executable EXECUTABLE
+                            Python executable (default: use host Python)
+      --nmake               build dependencies using NMake instead of Ninja
+      --no-imagequant       skip GPL-licensed optional dependency libimagequant
+      --no-fribidi, --no-raqm
+                            skip LGPL-licensed optional dependency FriBiDi
+
+    Arguments can also be supplied using the environment variables PILLOW_BUILD,
+    PILLOW_DEPS, ARCHITECTURE, PYTHON, EXECUTABLE. See winbuild\build.rst for more
+    information.
+
+**Warning:** The build directory is wiped when ``build_prepare.py`` is run.
 
 Dependencies
 ------------
 
 Dependencies will be automatically downloaded by ``build_prepare.py``.
 By default, downloaded dependencies are stored in ``winbuild\depends``;
-set the ``PILLOW_DEPS`` environment variable to override this location.
+use the ``--depends`` argument or ``PILLOW_DEPS`` environment variable
+to override this location.
 
 To build all dependencies, run ``winbuild\build\build_dep_all.cmd``,
-or run the individual scripts to build each dependency separately.
+or run the individual scripts in order to build each dependency separately.
 
 Building Pillow
 ---------------
@@ -100,7 +112,7 @@ The following is a simplified version of the script used on AppVeyor::
 
     set PYTHON=C:\Python38\bin
     cd /D C:\Pillow\winbuild
-    C:\Python37\bin\python.exe build_prepare.py -v --depends=C:\pillow-depends
+    C:\Python37\bin\python.exe build_prepare.py -v --depends C:\pillow-depends
     build\build_dep_all.cmd
     build\build_pillow.cmd install
     cd ..

winbuild/build_prepare.py

@@ -1,3 +1,4 @@
+import argparse
 import os
 import platform
 import re
@@ -55,24 +56,28 @@ def cmd_nmake(makefile=None, target="", params=None):
     )
 
 
-def cmd_cmake(params=None, file="."):
+def cmds_cmake(target, *params):
-    if params is None:
+    if not isinstance(target, str):
-        params = ""
+        target = " ".join(target)
-    elif isinstance(params, (list, tuple)):
+
-        params = " ".join(params)
+    return [
-    else:
+        " ".join(
-        params = str(params)
-    return " ".join(
             [
                 "{cmake}",
-            "-DCMAKE_VERBOSE_MAKEFILE=ON",
-            "-DCMAKE_RULE_MESSAGES:BOOL=OFF",
                 "-DCMAKE_BUILD_TYPE=Release",
-            f"{params}",
+                "-DCMAKE_VERBOSE_MAKEFILE=ON",
-            '-G "NMake Makefiles"',
+                "-DCMAKE_RULE_MESSAGES:BOOL=OFF",  # for NMake
-            f'"{file}"',
+                "-DCMAKE_C_COMPILER=cl.exe",  # for Ninja
+                "-DCMAKE_CXX_COMPILER=cl.exe",  # for Ninja
+                "-DCMAKE_C_FLAGS=-nologo",
+                "-DCMAKE_CXX_FLAGS=-nologo",
+                *params,
+                '-G "{cmake_generator}"',
+                ".",
+            ]
+        ),
+        f"{{cmake}} --build . --clean-first --parallel --target {target}",
     ]
-    )
 
 
 def cmd_msbuild(
@@ -118,19 +123,14 @@ deps = {
             ".+(libjpeg-turbo Licenses\n======================\n\n.+)$"
         ),
         "build": [
-            cmd_cmake(
+            *cmds_cmake(
-                [
+                ("jpeg-static", "cjpeg-static", "djpeg-static"),
                 "-DENABLE_SHARED:BOOL=FALSE",
                 "-DWITH_JPEG8:BOOL=TRUE",
                 "-DWITH_CRT_DLL:BOOL=TRUE",
-                ]
             ),
-            cmd_nmake(target="clean"),
-            cmd_nmake(target="jpeg-static"),
             cmd_copy("jpeg-static.lib", "libjpeg.lib"),
-            cmd_nmake(target="cjpeg-static"),
             cmd_copy("cjpeg-static.exe", "cjpeg.exe"),
-            cmd_nmake(target="djpeg-static"),
             cmd_copy("djpeg-static.exe", "djpeg.exe"),
         ],
         "headers": ["j*.h"],
@@ -156,25 +156,13 @@ deps = {
         "filename": "xz-5.4.2.tar.gz",
         "dir": "xz-5.4.2",
         "license": "COPYING",
-        "patch": {
-            r"src\liblzma\api\lzma.h": {
-                "#ifndef LZMA_API_IMPORT": "#ifndef LZMA_API_IMPORT\n#define LZMA_API_STATIC",  # noqa: E501
-            },
-            r"windows\vs2019\liblzma.vcxproj": {
-                # retarget to default toolset (selected by vcvarsall.bat)
-                "<PlatformToolset>v142</PlatformToolset>": "<PlatformToolset>$(DefaultPlatformToolset)</PlatformToolset>",  # noqa: E501
-                # retarget to latest (selected by vcvarsall.bat)
-                "<WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>": "<WindowsTargetPlatformVersion>$(WindowsSDKVersion)</WindowsTargetPlatformVersion>",  # noqa: E501
-            },
-        },
         "build": [
-            cmd_msbuild(r"windows\vs2019\liblzma.vcxproj", "Release", "Clean"),
+            *cmds_cmake("liblzma", "-DBUILD_SHARED_LIBS:BOOL=OFF"),
-            cmd_msbuild(r"windows\vs2019\liblzma.vcxproj", "Release", "Build"),
             cmd_mkdir(r"{inc_dir}\lzma"),
             cmd_copy(r"src\liblzma\api\lzma\*.h", r"{inc_dir}\lzma"),
         ],
         "headers": [r"src\liblzma\api\lzma.h"],
-        "libs": [r"windows\vs2019\Release\{msbuild_arch}\liblzma\liblzma.lib"],
+        "libs": [r"liblzma.lib"],
     },
     "libwebp": {
         "url": "http://downloads.webmproject.org/releases/webp/libwebp-1.3.0.tar.gz",
@@ -215,9 +203,11 @@ deps = {
             },
         },
         "build": [
-            cmd_cmake("-DBUILD_SHARED_LIBS:BOOL=OFF"),
+            *cmds_cmake(
-            cmd_nmake(target="clean"),
+                "tiff",
-            cmd_nmake(target="tiff"),
+                "-DBUILD_SHARED_LIBS:BOOL=OFF",
+                '-DCMAKE_C_FLAGS="-nologo -DLZMA_API_STATIC"',
+            )
         ],
         "headers": [r"libtiff\tiff*.h"],
         "libs": [r"libtiff\*.lib"],
@@ -229,10 +219,7 @@ deps = {
         "dir": "lpng1639",
         "license": "LICENSE",
         "build": [
-            # lint: do not inline
+            *cmds_cmake("png_static", "-DPNG_SHARED:BOOL=OFF", "-DPNG_TESTS:BOOL=OFF"),
-            cmd_cmake(("-DPNG_SHARED:BOOL=OFF", "-DPNG_TESTS:BOOL=OFF")),
-            cmd_nmake(target="clean"),
-            cmd_nmake(),
             cmd_copy("libpng16_static.lib", "libpng16.lib"),
         ],
         "headers": [r"png*.h"],
@@ -244,10 +231,7 @@ deps = {
         "dir": "brotli-1.0.9",
         "license": "LICENSE",
         "build": [
-            cmd_cmake(),
+            *cmds_cmake(("brotlicommon-static", "brotlidec-static")),
-            cmd_nmake(target="clean"),
-            cmd_nmake(target="brotlicommon-static"),
-            cmd_nmake(target="brotlidec-static"),
             cmd_xcopy(r"c\include", "{inc_dir}"),
         ],
         "libs": ["*.lib"],
@@ -325,9 +309,9 @@ deps = {
             }
         },
         "build": [
-            cmd_cmake(("-DBUILD_CODEC:BOOL=OFF", "-DBUILD_SHARED_LIBS:BOOL=OFF")),
+            *cmds_cmake(
-            cmd_nmake(target="clean"),
+                "openjp2", "-DBUILD_CODEC:BOOL=OFF", "-DBUILD_SHARED_LIBS:BOOL=OFF"
-            cmd_nmake(target="openjp2"),
+            ),
             cmd_mkdir(r"{inc_dir}\openjpeg-2.5.0"),
             cmd_copy(r"src\lib\openjp2\*.h", r"{inc_dir}\openjpeg-2.5.0"),
         ],
@@ -346,10 +330,7 @@ deps = {
             }
         },
         "build": [
-            # lint: do not inline
+            *cmds_cmake("imagequant_a"),
-            cmd_cmake(),
-            cmd_nmake(target="clean"),
-            cmd_nmake(target="imagequant_a"),
             cmd_copy("imagequant_a.lib", "imagequant.lib"),
         ],
         "headers": [r"*.h"],
@@ -361,10 +342,11 @@ deps = {
         "dir": "harfbuzz-7.1.0",
         "license": "COPYING",
         "build": [
-            cmd_set("CXXFLAGS", "-d2FH4-"),
+            *cmds_cmake(
-            cmd_cmake("-DHB_HAVE_FREETYPE:BOOL=TRUE"),
+                "harfbuzz",
-            cmd_nmake(target="clean"),
+                "-DHB_HAVE_FREETYPE:BOOL=TRUE",
-            cmd_nmake(target="harfbuzz"),
+                '-DCMAKE_CXX_FLAGS="-nologo -d2FH4-"',
+            ),
         ],
         "headers": [r"src\*.h"],
         "libs": [r"*.lib"],
@@ -377,9 +359,7 @@ deps = {
         "build": [
             cmd_copy(r"COPYING", r"{bin_dir}\fribidi-1.0.12-COPYING"),
             cmd_copy(r"{winbuild_dir}\fribidi.cmake", r"CMakeLists.txt"),
-            cmd_cmake(),
+            *cmds_cmake("fribidi"),
-            cmd_nmake(target="clean"),
-            cmd_nmake(target="fribidi"),
         ],
         "bins": [r"*.dll"],
     },
@@ -455,7 +435,7 @@ def extract_dep(url, filename):
     import urllib.request
     import zipfile
 
-    file = os.path.join(depends_dir, filename)
+    file = os.path.join(args.depends_dir, filename)
     if not os.path.exists(file):
         ex = None
         for i in range(3):
@@ -496,12 +476,12 @@ def extract_dep(url, filename):
 
 
 def write_script(name, lines):
-    name = os.path.join(build_dir, name)
+    name = os.path.join(args.build_dir, name)
     lines = [line.format(**prefs) for line in lines]
     print("Writing " + name)
     with open(name, "w", newline="") as f:
         f.write(os.linesep.join(lines))
-    if verbose:
+    if args.verbose:
         for line in lines:
             print("    " + line)
 
@@ -570,11 +550,14 @@ def build_dep(name):
 def build_dep_all():
     lines = ["@echo on"]
     for dep_name in deps:
+        print()
         if dep_name in disabled:
+            print(f"Skipping disabled dependency {dep_name}")
             continue
         script = build_dep(dep_name)
         lines.append(rf'cmd.exe /c "{{build_dir}}\{script}"')
         lines.append("if errorlevel 1 echo Build failed! && exit /B 1")
+    print()
     lines.append("@echo All Pillow dependencies built successfully!")
     write_script("build_dep_all.cmd", lines)
 
@@ -593,56 +576,91 @@ def build_pillow():
 
 
 if __name__ == "__main__":
-    # winbuild directory
     winbuild_dir = os.path.dirname(os.path.realpath(__file__))
+    pillow_dir = os.path.realpath(os.path.join(winbuild_dir, ".."))
 
-    verbose = False
+    parser = argparse.ArgumentParser(
-    disabled = []
+        prog="winbuild\\build_prepare.py",
-    depends_dir = os.environ.get("PILLOW_DEPS", os.path.join(winbuild_dir, "depends"))
+        description="Download dependencies and generate build scripts for Pillow.",
-    python_dir = os.environ.get("PYTHON")
+        epilog="""Arguments can also be supplied using the environment variables
-    python_exe = os.environ.get("EXECUTABLE", "python.exe")
+                  PILLOW_BUILD, PILLOW_DEPS, ARCHITECTURE, PYTHON, EXECUTABLE.
-    architecture = os.environ.get(
+                  See winbuild\\build.rst for more information.""",
+    )
+    parser.add_argument(
+        "-v", "--verbose", action="store_true", help="print generated scripts"
+    )
+    parser.add_argument(
+        "-d",
+        "--dir",
+        "--build-dir",
+        dest="build_dir",
+        metavar="PILLOW_BUILD",
+        default=os.environ.get("PILLOW_BUILD", os.path.join(winbuild_dir, "build")),
+        help="build directory (default: 'winbuild\\build')",
+    )
+    parser.add_argument(
+        "--depends",
+        dest="depends_dir",
+        metavar="PILLOW_DEPS",
+        default=os.environ.get("PILLOW_DEPS", os.path.join(winbuild_dir, "depends")),
+        help="directory used to store cached dependencies "
+        "(default: 'winbuild\\depends')",
+    )
+    parser.add_argument(
+        "--architecture",
+        choices=architectures,
+        default=os.environ.get(
             "ARCHITECTURE",
+            (
                 "ARM64"
                 if platform.machine() == "ARM64"
-        else ("x86" if struct.calcsize("P") == 4 else "x64"),
+                else ("x86" if struct.calcsize("P") == 4 else "x64")
+            ),
+        ),
+        help="build architecture (default: same as host Python)",
     )
-    build_dir = os.environ.get("PILLOW_BUILD", os.path.join(winbuild_dir, "build"))
+    parser.add_argument(
-    sources_dir = ""
+        "--python",
-    for arg in sys.argv[1:]:
+        dest="python_dir",
-        if arg == "-v":
+        metavar="PYTHON",
-            verbose = True
+        default=os.environ.get("PYTHON"),
-        elif arg == "--no-imagequant":
+        help="Python install directory (default: use host Python)",
-            disabled += ["libimagequant"]
+    )
-        elif arg == "--no-raqm" or arg == "--no-fribidi":
+    parser.add_argument(
-            disabled += ["fribidi"]
+        "--executable",
-        elif arg.startswith("--depends="):
+        dest="python_exe",
-            depends_dir = arg[10:]
+        metavar="EXECUTABLE",
-        elif arg.startswith("--python="):
+        default=os.environ.get("EXECUTABLE", "python.exe"),
-            python_dir = arg[9:]
+        help="Python executable (default: use host Python)",
-        elif arg.startswith("--executable="):
+    )
-            python_exe = arg[13:]
+    parser.add_argument(
-        elif arg.startswith("--architecture="):
+        "--nmake",
-            architecture = arg[15:]
+        dest="cmake_generator",
-        elif arg.startswith("--dir="):
+        action="store_const",
-            build_dir = arg[6:]
+        const="NMake Makefiles",
-        elif arg == "--srcdir":
+        default="Ninja",
-            sources_dir = os.path.sep + "src"
+        help="build dependencies using NMake instead of Ninja",
-        else:
+    )
-            msg = "Unknown parameter: " + arg
+    parser.add_argument(
-            raise ValueError(msg)
+        "--no-imagequant",
+        action="store_true",
+        help="skip GPL-licensed optional dependency libimagequant",
+    )
+    parser.add_argument(
+        "--no-fribidi",
+        "--no-raqm",
+        action="store_true",
+        help="skip LGPL-licensed optional dependency FriBiDi",
+    )
+    args = parser.parse_args()
 
-    # dependency cache directory
+    arch_prefs = architectures[args.architecture]
-    os.makedirs(depends_dir, exist_ok=True)
+    print("Target architecture:", args.architecture)
-    print("Caching dependencies in:", depends_dir)
 
-    if python_dir is None:
+    if args.python_dir is None:
-        python_dir = os.path.dirname(os.path.realpath(sys.executable))
+        args.python_dir = os.path.dirname(os.path.realpath(sys.executable))
-        python_exe = os.path.basename(sys.executable)
+        args.python_exe = os.path.basename(sys.executable)
-    print("Target Python:", os.path.join(python_dir, python_exe))
+    print("Target Python:", os.path.join(args.python_dir, args.python_exe))
-
-    arch_prefs = architectures[architecture]
-    print("Target Architecture:", architecture)
 
     msvs = find_msvs()
     if msvs is None:
@@ -650,35 +668,47 @@ if __name__ == "__main__":
         raise RuntimeError(msg)
     print("Found Visual Studio at:", msvs["vs_dir"])
 
-    print("Using output directory:", build_dir)
+    # dependency cache directory
+    args.depends_dir = os.path.abspath(args.depends_dir)
+    os.makedirs(args.depends_dir, exist_ok=True)
+    print("Caching dependencies in:", args.depends_dir)
+
+    args.build_dir = os.path.abspath(args.build_dir)
+    print("Using output directory:", args.build_dir)
 
     # build directory for *.h files
-    inc_dir = os.path.join(build_dir, "inc")
+    inc_dir = os.path.join(args.build_dir, "inc")
     # build directory for *.lib files
-    lib_dir = os.path.join(build_dir, "lib")
+    lib_dir = os.path.join(args.build_dir, "lib")
     # build directory for *.bin files
-    bin_dir = os.path.join(build_dir, "bin")
+    bin_dir = os.path.join(args.build_dir, "bin")
     # directory for storing project files
-    sources_dir = build_dir + sources_dir
+    sources_dir = os.path.join(args.build_dir, "src")
     # copy dependency licenses to this directory
-    license_dir = os.path.join(build_dir, "license")
+    license_dir = os.path.join(args.build_dir, "license")
 
-    shutil.rmtree(build_dir, ignore_errors=True)
+    shutil.rmtree(args.build_dir, ignore_errors=True)
-    os.makedirs(build_dir, exist_ok=False)
+    os.makedirs(args.build_dir, exist_ok=False)
     for path in [inc_dir, lib_dir, bin_dir, sources_dir, license_dir]:
         os.makedirs(path, exist_ok=True)
 
+    disabled = []
+    if args.no_imagequant:
+        disabled += ["libimagequant"]
+    if args.no_fribidi:
+        disabled += ["fribidi"]
+
     prefs = {
         # Python paths / preferences
-        "python_dir": python_dir,
+        "python_dir": args.python_dir,
-        "python_exe": python_exe,
+        "python_exe": args.python_exe,
-        "architecture": architecture,
+        "architecture": args.architecture,
         **arch_prefs,
         # Pillow paths
-        "pillow_dir": os.path.realpath(os.path.join(winbuild_dir, "..")),
+        "pillow_dir": pillow_dir,
         "winbuild_dir": winbuild_dir,
         # Build paths
-        "build_dir": build_dir,
+        "build_dir": args.build_dir,
         "inc_dir": inc_dir,
         "lib_dir": lib_dir,
         "bin_dir": bin_dir,
@@ -687,6 +717,7 @@ if __name__ == "__main__":
         # Compilers / Tools
         **msvs,
         "cmake": "cmake.exe",  # TODO find CMAKE automatically
+        "cmake_generator": args.cmake_generator,
         # TODO find NASM automatically
         # script header
         "header": sum([header, msvs["header"], ["@echo on"]], []),
@@ -699,4 +730,6 @@ if __name__ == "__main__":
 
     write_script(".gitignore", ["*"])
     build_dep_all()
+    if args.verbose:
+        print()
     build_pillow()