Packaging and distributing projects#

Page Status:

Outdated

Last Reviewed:

2023-12-14

This section covers some additional details on configuring, packaging and distributing Python projects with setuptools that aren’t covered by the introductory tutorial in Packaging Python Projects. It still assumes that you are already familiar with the contents of the Installing Packages page.

The section does not aim to cover best practices for Python project development as a whole. For example, it does not provide guidance or tool recommendations for version control, documentation, or testing.

For more reference material, see Building and Distributing Packages in the Setuptools docs, but note that some advisory content there may be outdated. In the event of conflicts, prefer the advice in the Python Packaging User Guide.

Requirements for packaging and distributing#

  1. First, make sure you have already fulfilled the requirements for installing packages.

  2. Install “twine” [1]:

    python3 -m pip install twine
    
    py -m pip install twine
    

You’ll need this to upload your project distributions to PyPI (see below).

Configuring your project#

Initial files#

setup.py#

The most important file is setup.py which exists at the root of your project directory. For an example, see the setup.py in the PyPA sample project.

setup.py serves two primary functions:

  1. It’s the file where various aspects of your project are configured. The primary feature of setup.py is that it contains a global setup() function. The keyword arguments to this function are how specific details of your project are defined. The most relevant arguments are explained in the section below.

  2. It’s the command line interface for running various commands that relate to packaging tasks. To get a listing of available commands, run python3 setup.py --help-commands.

setup.cfg#

setup.cfg is an ini file that contains option defaults for setup.py commands. For an example, see the setup.cfg in the PyPA sample project.

README.rst / README.md#

All projects should contain a readme file that covers the goal of the project. The most common format is reStructuredText with an “rst” extension, although this is not a requirement; multiple variants of Markdown are supported as well (look at setup()’s long_description_content_type argument).

For an example, see README.md from the PyPA sample project.

Note

Projects using Setuptools 0.6.27+ have standard readme files (README.rst, README.txt, or README) included in source distributions by default. The built-in distutils library adopts this behavior beginning in Python 3.7. Additionally, Setuptools 36.4.0+ will include a README.md if found. If you are using setuptools, you don’t need to list your readme file in MANIFEST.in. Otherwise, include it to be explicit.

MANIFEST.in#

A MANIFEST.in is needed when you need to package additional files that are not automatically included in a source distribution. For details on writing a MANIFEST.in file, including a list of what’s included by default, see “Using MANIFEST.in”.

However, you may not have to use a MANIFEST.in. For an example, the PyPA sample project has removed its manifest file, since all the necessary files have been included by Setuptools 43.0.0 and newer.

Note

MANIFEST.in does not affect binary distributions such as wheels.

LICENSE.txt#

Every package should include a license file detailing the terms of distribution. In many jurisdictions, packages without an explicit license can not be legally used or distributed by anyone other than the copyright holder. If you’re unsure which license to choose, you can use resources such as GitHub’s Choose a License or consult a lawyer.

For an example, see the LICENSE.txt from the PyPA sample project.

<your package>#

Although it’s not required, the most common practice is to include your Python modules and packages under a single top-level package that has the same name as your project, or something very close.

For an example, see the sample package that’s included in the PyPA sample project.

setup() args#

As mentioned above, the primary feature of setup.py is that it contains a global setup() function. The keyword arguments to this function are how specific details of your project are defined.

Some are temporarily explained below until their information is moved elsewhere. The full list can be found in the setuptools documentation.

Most of the snippets given are taken from the setup.py contained in the PyPA sample project.

See Versioning for more information on ways to use versions to convey compatibility information to your users.

packages#

packages=find_packages(include=['sample', 'sample.*']),

Set packages to a list of all packages in your project, including their subpackages, sub-subpackages, etc. Although the packages can be listed manually, setuptools.find_packages() finds them automatically. Use the include keyword argument to find only the given packages. Use the exclude keyword argument to omit packages that are not intended to be released and installed.

py_modules#

py_modules=["six"],

If your project contains any single-file Python modules that aren’t part of a package, set py_modules to a list of the names of the modules (minus the .py extension) in order to make Setuptools aware of them.

install_requires#

install_requires=['peppercorn'],

“install_requires” should be used to specify what dependencies a project minimally needs to run. When the project is installed by pip, this is the specification that is used to install its dependencies.

For more on using “install_requires” see install_requires vs requirements files.

package_data#

package_data={
    'sample': ['package_data.dat'],
},

Often, additional files need to be installed into a package. These files are often data that’s closely related to the package’s implementation, or text files containing documentation that might be of interest to programmers using the package. These files are called “package data”.

The value must be a mapping from package name to a list of relative path names that should be copied into the package. The paths are interpreted as relative to the directory containing the package.

For more information, see Including Data Files from the setuptools docs.

data_files#

data_files=[('my_data', ['data/data_file'])],

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.

Each (directory, files) pair in the sequence specifies the installation directory and the files to install there. The directory must be a relative path (although this may change in the future, see wheel Issue #92), and it is interpreted relative to the installation prefix (Python’s sys.prefix for a default installation; site.USER_BASE for a user installation). Each file name in files is interpreted relative to the setup.py script at the top of the project source distribution.

For more information see the distutils section on Installing Additional Files.

Note

When installing packages as egg, data_files is not supported. So, if your project uses Setuptools, you must use pip to install it. Alternatively, if you must use python setup.py, then you need to pass the --old-and-unmanageable option.

scripts#

Although setup() supports a scripts keyword for pointing to pre-made scripts to install, the recommended approach to achieve cross-platform compatibility is to use Creating executable scripts entry points (see below).

Choosing a versioning scheme#

See Versioning for information on common version schemes and how to choose between them.

Working in “development mode”#

You can install a project in “editable” or “develop” mode while you’re working on it. When installed as editable, a project can be edited in-place without reinstallation: changes to Python source files in projects installed as editable will be reflected the next time an interpreter process is started.

To install a Python package in “editable”/”development” mode Change directory to the root of the project directory and run:

python3 -m pip install -e .

The pip command-line flag -e is short for --editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your project) in editable mode. This will also install any dependencies declared with install_requires and any scripts declared with console_scripts. Dependencies will be installed in the usual, non-editable mode.

You may want to install some of your dependencies in editable mode as well. For example, supposing your project requires “foo” and “bar”, but you want “bar” installed from VCS in editable mode, then you could construct a requirements file like so:

-e .
-e bar @ git+https://somerepo/bar.git

The first line says to install your project and any dependencies. The second line overrides the “bar” dependency, such that it’s fulfilled from VCS, not PyPI.

If, however, you want “bar” installed from a local directory in editable mode, the requirements file should look like this, with the local paths at the top of the file:

-e /path/to/project/bar
-e .

Otherwise, the dependency will be fulfilled from PyPI, due to the installation order of the requirements file. For more on requirements files, see the Requirements File section in the pip docs. For more on VCS installs, see the VCS Support section of the pip docs.

Lastly, if you don’t want to install any dependencies at all, you can run:

python3 -m pip install -e . --no-deps

For more information, see the Development Mode section of the Setuptools docs.

Packaging your project#

To have your project installable from a Package Index like PyPI, you’ll need to create a Distribution (aka “Package”) for your project.

Before you can build wheels and sdists for your project, you’ll need to install the build package:

python3 -m pip install build
py -m pip install build

Source distributions#

Minimally, you should create a Source Distribution:

python3 -m build --sdist
py -m build --sdist

A “source distribution” is unbuilt (i.e. it’s not a Built Distribution), and requires a build step when installed by pip. Even if the distribution is pure Python (i.e. contains no extensions), it still involves a build step to build out the installation metadata from setup.py and/or setup.cfg.

Wheels#

You should also create a wheel for your project. A wheel is a built package that can be installed without needing to go through the “build” process. Installing wheels is substantially faster for the end user than installing from a source distribution.

If your project is pure Python then you’ll be creating a “Pure Python Wheel” (see section below).

If your project contains compiled extensions, then you’ll be creating what’s called a *Platform Wheel* (see section below).

Note

If your project also supports Python 2 and contains no C extensions, then you should create what’s called a Universal Wheel by adding the following to your setup.cfg file:

[bdist_wheel]
universal=1

Only use this setting if your project does not have any C extensions and supports Python 2 and 3.

Pure Python Wheels#

Pure Python Wheels contain no compiled extensions, and therefore only require a single Python wheel.

To build the wheel:

python3 -m build --wheel
py -m build --wheel

The wheel package will detect that the code is pure Python, and build a wheel that’s named such that it’s usable on any Python 3 installation. For details on the naming of wheel files, see PEP 425.

If you run build without --wheel or --sdist, it will build both files for you; this is useful when you don’t need multiple wheels.

Platform Wheels#

Platform Wheels are wheels that are specific to a certain platform like Linux, macOS, or Windows, usually due to containing compiled extensions.

To build the wheel:

python3 -m build --wheel
py -m build --wheel

The wheel package will detect that the code is not pure Python, and build a wheel that’s named such that it’s only usable on the platform that it was built on. For details on the naming of wheel files, see PEP 425.

Note

PyPI currently supports uploads of platform wheels for Windows, macOS, and the multi-distro manylinux* ABI. Details of the latter are defined in PEP 513.

Uploading your Project to PyPI#

When you ran the command to create your distribution, a new directory dist/ was created under your project’s root directory. That’s where you’ll find your distribution file(s) to upload.

Note

These files are only created when you run the command to create your distribution. This means that any time you change the source of your project or the configuration in your setup.py file, you will need to rebuild these files again before you can distribute the changes to PyPI.

Note

Before releasing on main PyPI repo, you might prefer training with the PyPI test site which is cleaned on a semi regular basis. See Using TestPyPI on how to setup your configuration in order to use it.

Warning

In other resources you may encounter references to using python setup.py register and python setup.py upload. These methods of registering and uploading a package are strongly discouraged as it may use a plaintext HTTP or unverified HTTPS connection on some Python versions, allowing your username and password to be intercepted during transmission.

Tip

The reStructuredText parser used on PyPI is not Sphinx! Furthermore, to ensure safety of all users, certain kinds of URLs and directives are forbidden or stripped out (e.g., the .. raw:: directive). Before trying to upload your distribution, you should check to see if your brief / long descriptions provided in setup.py are valid. You can do this by running twine check on your package files:

twine check dist/*

Create an account#

First, you need a PyPI user account. You can create an account using the form on the PyPI website.

Now you’ll create a PyPI API token so you will be able to securely upload your project.

Go to https://pypi.org/manage/account/#api-tokens and create a new API token; don’t limit its scope to a particular project, since you are creating a new project.

Don’t close the page until you have copied and saved the token — you won’t see that token again.

Note

To avoid having to copy and paste the token every time you upload, you can create a $HOME/.pypirc file:

[pypi]
username = __token__
password = <the token value, including the `pypi-` prefix>

Be aware that this stores your token in plaintext.

For more details, see the specification for .pypirc.

Upload your distributions#

Once you have an account you can upload your distributions to PyPI using twine.

The process for uploading a release is the same regardless of whether or not the project already exists on PyPI - if it doesn’t exist yet, it will be automatically created when the first release is uploaded.

For the second and subsequent releases, PyPI only requires that the version number of the new release differ from any previous releases.

twine upload dist/*

You can see if your package has successfully uploaded by navigating to the URL https://pypi.org/project/<sampleproject> where sampleproject is the name of your project that you uploaded. It may take a minute or two for your project to appear on the site.