Python binary distributions
One of the features of the CPython reference interpreter is that, in addition to allowing the execution of Python code, it also exposes a rich C API for use by other software.
The CPython standard library makes extensive use of accelerator modules. The CPython standard library makes extensive use of wrapper modules. Not all extension modules will fit neatly into the above categories. The extension modules included with NumPy, for example, span all three use cases - they move inner loops to C for speed reasons, wrap external libraries written in C, FORTRAN and other languages, and use low level system interfaces for both CPython and the underlying operation system to support concurrent execution of vectorised operations and to tightly control the exact memory layout of created objects.
The main disadvantage of using binary extensions is the fact that it makes subsequent distribution of the software more difficult. When extension modules are just being used to make code run faster after profiling has identified the code where the speed increase is worth additional maintenance efforta number of other alternatives should also be considered:.
However, wrapping modules by hand is quite tedious, so a number of other alternative approaches should be considered. In addition to being useful for the creation of accelerator modules, Cython is also useful for creating wrapper modules. It still involves wrapping the interfaces by hand, however, so may not be a good choice for wrapping large APIs. By contrast, the above alternatives are all able to operate at the C API level, using C header files to ensure consistency between the interface exported python binary distributions the library being wrapped and the one expected by the Python wrapper module.
While cffi can operate directly at the C ABI level, it suffers from the same interface inconsistency problems as ctypes when it is used that way. For applications that need low level system access regardless of the reasona binary extension module often is the best way to go about it. This is particularly true for low level access to the CPython runtime itself, since some operations like releasing the Global Interpreter Lock are simply invalid when the interpreter is running code, even if a module like ctypes or cffi is used to obtain access to the relevant C API interfaces.
The CPython Extending and Embedding guide includes an introduction to writing a custom extension module in C. Before it is possible to build a binary extension, it is necessary to ensure that you have a suitable compiler available.
On Windows, Visual C is used to build the official CPython interpreter, and should be used to build compatible binary python binary distributions. Unfortunately, older versions of Visual Studio are no longer easily available from Microsoft, so for versions of Python prior to 3.
Note that from Python 3. For interim guidance on this topic, see the discussion in this issue. Cross-platform development and distribution of extension modules is a complex topic, so this guide focuses primarily on providing pointers to various tools that automate dealing with the underlying technical challenges.
Python binary distributions additional resources in this section are instead python binary distributions for developers looking to understand more about the underlying binary interfaces that those systems rely on at runtime.
For a python binary distributions in depth explanation of how extension modules python binary distributions used by CPython on a Debian system, see the following articles:.
Dropping Support for Older Python Versions. Supporting Windows using Appveyor. This means that binary extensions: The CPython standard libary includes a number of optimised data structures and algorithms especially in the builtins and the collections and itertools modules. The Python Package Index also offers additional alternatives. Sometimes, the appropriate choice of standard library python binary distributions third python binary distributions module can avoid the need to create your own accelerator module.
Cython is a mature static compiler that can compile most Python code to C python binary distributions modules. Numba is a newer tool, created by members of the scientific Python community, that aims to leverage LLVM to allow selective compilation of pieces of a Python application to native machine code at runtime. It requires that LLVM be available on the system where the code is running, but can provide significant speed increases, especially for operations that are amenable to vectorisation.
To set up a build environment for binary extensions, the steps are as follows: Use a recent version of setuptools in your setup. Install Visual Studio Community Edition or any later version, when these are released. FIXME cover publishing as wheel files on PyPI or a custom index server cover creation of Windows and macOS installers mention the fact that Python binary distributions distros have a requirement to build from source in their own build systemsso binary - only releases are strongly discouraged.
What are c python extension modules? The Python Software Foundation is a non-profit corporation. Last updated on Apr 04, Python binary distributions using Sphinx 1.
This section covers the basics of how to configure, package and distribute your own Python projects. It assumes that you are already familiar with the contents of the Installing Packages page. The section does not aim to cover best practices for Python project development as a whole.
For example, it does not provide guidance or tool recommendations for version control, documentation, or testing. For more reference material, python binary distributions Building and Distributing Packages in the setuptools docs, but note that some advisory content there may be outdated. In the event python binary distributions conflicts, python binary distributions the advice in the Python Packaging User Guide.
First, make sure you have already fulfilled the requirements for installing packages. The most important file is setup. For an example, see the setup. All projects should contain a readme file that covers the goal of the project. The built-in distutils library adopts this behavior beginning in Python 3. Otherwise, include it to be explicit.
Every package should include a license file detailing the terms of distribution. In many jurisdictions, packages without an python binary distributions license can not be legally used or distributed by anyone other than the copyright holder. As mentioned above, the primary feature of setup. The keyword arguments to this function are how specific details of your project are defined. The most relevant arguments are explained below. The snippets given are taken from the setup.
This is the name of your project, determining how your project is listed on PyPI. Per PEPvalid project names must:. For example, if you register a project named cool-stuffusers will be able to download it or declare a dependency on it using any of the following python binary distributions. Versions are displayed on PyPI for each release if you publish your project.
See Choosing a versioning scheme for more information on ways to use versions to convey compatibility information to your users. If the project code itself needs run-time access to the version, the simplest way is to keep the version in both setup. These values will be displayed on PyPI if you publish your project. Provide a list of classifiers that categorize your project. For a full listing, see https: List additional relevant URLs about your project. This is the place to link to bug trackers, source repositories, or where to support package development.
The string of the key is the exact text that will be displayed on PyPI. It is required to list the packages to be included in your project. Although they can be listed manually, setuptools. Use the exclude keyword argument to omit packages that are not intended to be released and installed. When the project is installed by pipthis is the specification that is used to install its dependencies.
If your package is for Python 3. Support for this feature is relatively recent. In addition, only versions 9. Often, additional files need to be installed into a package. The value must be a mapping from package name to a list of relative path names that should be copied into the package.
The paths are interpreted as relative to the directory containing the package. For more information, see Including Data Files from the setuptools docs. Each directory, files pair in the sequence specifies the installation directory and the files to install there. Each file name in files is interpreted relative to the setup.
For more information python binary distributions the distutils section on Installing Additional Files. This is not true when installing from wheel distributions.
For discussion see wheel Issue Use this keyword to specify any plugins that your project provides for any named entry points that may be defined by your project or others that you depend on. For more information, see the section python binary distributions Dynamic Discovery of Services and Plugins from the setuptools docs. You can then let the toolchain handle the work of turning these interfaces into actual scripts .
The scripts will python binary distributions generated during the install of your distribution. For more information, see Automatic Script Creation from the setuptools docs. Different Python projects may use different versioning schemes based on the needs of that particular project, but all of them are required to comply with the flexible public version scheme specified in PEP in order to be supported in tools and libraries like pip and setuptools.
To further accommodate historical variations in approaches to version numbering, PEP also defines a comprehensive technique for version normalisation that maps python binary distributions spellings of different version numbers to a standardised canonical form. For new projects, the recommended versioning scheme is based on Semantic Versioningbut adopts a different approach python binary distributions handling pre-releases and build metadata.
Y requires at least release X. Python projects adopting semantic versioning should abide by clauses of the Semantic Versioning 2. Semantic versioning is not a suitable choice for all python binary distributions, such as those with a regular time based release cadence and a deprecation process that provides warnings for a number of releases prior to removal of a feature. A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular release is given just the version number.
Version numbers for date based projects typically take the form of YEAR. MONTH for example, This is python binary distributions simplest possible versioning scheme, and consists of a python binary distributions number which is incremented every release. While serial versioning is very easy to manage as a developer, it is the hardest to track as an end user, as serial version numbers convey little or no information regarding API backwards compatibility.
Combinations of the above schemes are possible. For example, a python binary distributions may combine date based versioning with serial versioning to create a YEAR.
Regardless of the base versioning scheme, pre-releases for a given final release may be published as:. Public version identifiers are designed python binary distributions support distribution via PyPI. This allows your project to be both installed and editable in project form.
Although somewhat cryptic, -e is short for --editableand. Dependencies will be installed in the usual, non-editable mode. The first python binary distributions says to install your project and any dependencies. Otherwise, the dependency will be fulfilled from PyPI, due to the installation order of the requirements file. For more on requirements files, see the Requirements File section in the pip docs.
For more information, see the Development Mode section of the setuptools docs. Minimally, you should create a Source Python binary distributions. Even if python binary distributions distribution is pure Python i. You should also create a wheel for your project. Installing wheels is substantially faster for the end user than installing from a source distribution. If your project is pure Python i.
Universal Wheels are wheels that are pure Python i. This is a wheel that can be python binary distributions anywhere by pip. You can also permanently set the --universal flag in setup. Only use the --universal setting, if:. If your project has optional C extensions, it is recommended not to publish a universal wheel, because pip will prefer the wheel over a source installation, and prevent the possibility of building the extension.
For python binary distributions on the naming of wheel python binary distributions, see PEP If your python binary distributions supports both Python 2 and 3, but with different code e. This will produce wheels for each version. Platform Wheels are wheels that are specific to a certain platform like Linux, macOS, or Windows, usually due to containing compiled extensions.
Details of the latter python binary distributions defined in PEP These files are only created when you run the command to create your distribution. This means that any time you change the source of your project or the configuration in your setup. Before releasing on main PyPI repo, you might prefer training with the PyPI test site which is cleaned on a semi regular basis.
In other resources you may encounter references to using python setup. These methods of registering and uploading a package are strongly discouraged as it may use a plaintext HTTP or unverified HTTPS connection on some Python versions, python binary distributions your username and password to be intercepted during transmission.
Furthermore, to ensure safety of all users, certain kinds of URLs and directives are forbidden or stripped out e. First, you need a PyPI user account. You can create an account using the form python binary distributions the PyPI website.
Once you have an account you can upload your distributions to PyPI using twine. If this is your first time uploading a distribution for a new project, twine will handle registering the project. You can see if your package has successfully uploaded by navigating to the URL https:
There was a lot in there which just resonated with how I thought that software development should work. I still think that the design python binary distributions setuptools is amazing.
Nobody would argue that setuptools was flawless and it certainly failed in many regards. Python binary distributions biggest problem probably was that it was build on Python's idiotic import system but there is only so little you python binary distributions do about that. In general setuptools took the realistic approach to problem-solving: That also somewhat explains the second often cited problem of setuptools: I rarely do updates on my articles but I want to make one thing very clear: For a long I did not care about binary distributions because I never had the use case, so I did not even notice that pip did not provide python binary distributions. Then suddenly the use case came up and I realized that pip removed functionality from plain python binary distributions and it's very hard impossible for myself to bring it back.
It's more of an observation how easy it is to miss use cases when replacing tools with something else. For the few Python guys that don't know what setuptools is or why it exists, let me provide you with a very basic recap of what it originally wanted to do.
The purpose of that library was to provide functionality to distribute Python libraries. And for its existence we should all be really glad because besides all it's faults it is still providing us with some of the most important bits of Python package distribution.
What distutils did not do was handling metadata and that's pretty much the reason it exists in the first place. Without metadata there is no dependency tracking. While this is only partially true because there was a very limited amount of metadata even before setuptools available, that metadata disappeared once the library was installed.
Now to understand why adding python binary distributions is not so trivial with regular Python libraries we first have to understand pth files. I honestly don't know when. I think they were added to make python binary distributions possible to migrate from modules to packages. At least that would explain why PIL was distributed the way it was. The reason this works is because PIL places a file called pil. Each line in that file is added to the search path. So far, so uninteresting.
There is however another thing python binary distributions. This horrible design however enabled setuptools existence to a large degree. As you can see: If you're using pip or before you're back in time before setuptools was widespread, python binary distributions filesystem looked like this after installing libraries:. Setuptools on the other hand did something very different, it placed each installed library in another folder called an egg and placed python binary distributions.
This behavior enraged a lot of people because it made sys. Now on the first glance this looks like a horrible idea, why would you do that? However in retrospective that is actually the much better solution. And it becomes obvious why that is the case when you look closer at the actual problem and why setuptools works the way it does.
The big problem there is that Python developers think of packages as individual things that you can freely nest.
They are a form of namespace with specific semantics attached. The way these imports work are convoluted and we won't go into detail here, but the important part is that it's customizable to some degree. This customization however is not exposed through distutils in any way. Now why would you want to customize this? The main reason is splitting up large monolithic packages. For instance take Django. Django as a framework has django as the root packages but it also ships a bunch of addon libraries in django.
If you want to have two packages on PyPI with different release cycles for django and django. Installing that would mean installing one library physically within another library on the filesystem. Now that sounds like a painless operation but it's not.
For instance upgrades of the Django library would also have to take into account that something is now contained within that library. The way setuptools solved that is that it installs python binary distributions PyPI python packages that means Flask And it can also hook packages virtually within another package by taking advantage of the fact that pth files can have executable code in it.
Yes, this was not a very clean solution but it worked. Setuptools in a way was the Quake approach to distributing things. You have individual packages of things which then get merged together in a virtual filesystem.
In setuptools it's not a virtual filesystem but it's a virtual package python binary distributions that is uncoupled from the filesystem. And with those egg folders it also solved another problem: With the added folder, setuptools had found a suitable place to put information that was generated as part of the build process.
Next to the importable package within the egg folder there are also text files with the metadata. Setuptools was really good with giving the same term to different things, a practice it copied python binary distributions distutils. In the original python binary distributions of setuptools there was no difference between an egg folder and an egg archive, but over the years and with the introduction of pip that changed.
Now when people talk about eggs they often talk about the folders on the filesystem with the. This however was actually not the interesting part about the original eggs at all.
What eggs could do was actually more interesting and something we lost in the transition to pip. If you take a folder with an. What was this madness? Python binary distributions jar files for Python. Why does this work? It works because Python has a default import hook that checks for each file on sys. If it is, it activates the zip importer and imports from within the zipfile. Due to how zipfiles are structured that's actually a very speedy operation.
Now that's another thing that people hated about setuptools. Mainly because the Python tracebacks no longer included the source lines with the python binary distributions.
However there was no technical limitation for why it should not be able to show the correct line numbers there. It was just a bug in the Python interpreter. Likewise the paths in the tracebacks were wrong too because they often had paths in it that were hard compiled by the person that created the egg file.
Again this is an issue with the Python interpreter and not setuptools. For reasons unknown to me, the. Now that's going to break because python binary distributions are no files.
So it could give you resources independent of it the resource was in a zipfile or on the filesystem. At that point you're asking: My tool does the installation, I don't care how it ends up on python binary distributions filesystem.
Fair enough, but there is a huge difference between eggs and tarballs or source zipfiles for that matter. But that does not invalidate the concept! Unfortunately binary distributions in distutils don't really work except for redhat untested, but never heard complaints and windows. So you can't use distutils binary distributions for instance to install something into your virtualenv on OS X or any flavour of linux.
Setuptools however added a new binary distribution target called the egg binary distribution which actually works. When we talk about a source distribution then we're talking about a tarball that has python binary distributions setup. Binary distributions on the other hand do not have a setup executable any more.
You just unpack them and you're done with them. They are also platform specific and you have to make sure you install a binary egg for the correct architecture and ucs size. And then there are other aspects to that too, but they can be ignored to understand the concept. It's very easy to ignore binary distributions. The only thing that python binary distributions is parsing of the python files and writing out bytecode.
Some other things however are more expensive to compile. The one that causes me most troubles is lxml because it takes a minute or two to compile. Now a while ago I ignored that like many other people by just keeping the virtualenv around and I obviously just installed stuff once. However when we started doing our new server deployment we wanted to have each build revision self contained with python binary distributions own virtualenv. Now to accomplish our deployment we have two options.
If you're not aware of how pip operates: For each package you're dealing with you basically just python setup. Then you start a fileserver that exposes these eggs via HTTP with a fileindex. Fair enough, so what's the alternative. The way we're doing that currently is making python binary distributions cached virtualenv and installing things in there. When we deploy new code we copy that virtualenv out, update all the paths in that virtualenv, rerun whatever commands are necessary to build the code usually just a pip install and copy it to the target location.