Python has been around for so long that there hasn't been a de facto standard project management and build tool, which has led to a variety of ways in which Python projects are structured and built. This may be a manifestation of Python's free will.
Unlike Java, which went through the initial manual builds, to the semi-automated Ant, to Maven is basically the de facto standard. In the meantime, Maven has been challenged by other Gradle (Android project), SBT (mainly Scala project), Ant+Ivy, Buildr, etc., but none of them have been able to shake Maven's status, and the others have pretty much followed Maven's directory layout.
Back in Python, there were package management tools like pip, pipenv, conda, but there was no convention on the directory layout of the project.
A lot of the build is still the continuation of the traditional Makefile method, and then add setup.py and build.py to install and build with program code. Regarding the project directory layout, there are project templates, and then tools are made to apply project templates.
Here's a quick look at how to use the four tools
- CookieCutter
- PyScaffold
- PyBuilder
- Poetry
CookieCutter is a classic Python project directory structure
$ pip install cookiecutter
$ cookiecutter gh:audreyr/cookiecutter-pypackage
# 以 github 上的 audreyr/cookiecutter-pypackage 为模板，再回答一堆的问题生成一个 Python 项目
......
project_name [Python Boilerplate]: sample
...... The final project template generated by the cookiecutter looks like this:
$ tree sample
sample
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── docs
│ ├── Makefile
│ ├── authors.rst
│ ├── conf.py
│ ├── contributing.rst
│ ├── history.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── make.bat
│ ├── readme.rst
│ └── usage.rst
├── requirements_dev.txt
├── sample
│ ├── __init__.py
│ ├── cli.py
│ └── sample.py
├── setup.cfg
├── setup.py
├── tests
│ ├── __init__.py
│ └── test_sample.py
└── tox.ini
3 directories, 26 files This is probably the main framework of the current popular directory structure, and the main elements are:
$ tree sample
sample
├── Makefile
├── README.rst
├── docs
│ └── index.rst
├── requirements.txt
├── sample
│ ├── __init__.py
│ └── sample.py
├── setup.cfg
├── setup.py
└── tests
├── __init__.py
└── test_sample.py Duplicate the sample directory in the project sample directory to place the Python source file, the tests directory is the test file, and add a docs directory to put the document, README.rst, and other setup, setup.cfg and Makefile files for the build.
This is actually a very classic Python project structure, and the next build is to use the make command, and you will see the instructions defined in the Makefile file when you type make
$ make
clean remove all build, test, coverage and Python artifacts
clean-build remove build artifacts
clean-pyc remove Python file artifacts
clean-test remove test and coverage artifacts
lint check style
test run tests quickly with the default Python
test-all run tests on every Python version with tox
coverage check code coverage quickly with the default Python
docs generate Sphinx HTML documentation, including API docs
servedocs compile the docs watching for changes
release package and upload a release
dist builds source and wheel package
install install the package to the active Python's site-packages In order to use the above build process, you need to install the appropriate packages, such as tox, wheel, coverage, sphinx, flake8, which can all be installed via pip. After that, you can make test, make coverage, make docs, make dist, etc. Among them, make docs can generate a beautiful web document.
PyScaffold creates a project
PyScaffold, as the name suggests, is a tool used to create scaffolding for Python projects, install and use:
$ pip install pyscaffold
$ putup sample This creates a Python project with a directory structure similar to the template selected by the cookiecutter above, except that it places the source files in the src directory instead of the sample directory.
$ tree sample
sample
├── AUTHORS.rst
├── CHANGELOG.rst
├── CONTRIBUTING.rst
├── LICENSE.txt
├── README.rst
├── docs
│ ├── Makefile
│ ├── _static
│ ├── authors.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── contributing.rst
│ ├── index.rst
│ ├── license.rst
│ ├── readme.rst
│ └── requirements.txt
├── pyproject.toml
├── setup.cfg
├── setup.py
├── src
│ └── sample
│ ├── __init__.py
│ └── skeleton.py
├── tests
│ ├── conftest.py
│ └── test_skeleton.py
└── tox.ini The whole project will be built using tox. tox is an automated test and build tool that creates a Python virtual environment during the build process, which allows for a clean environment for testing and builds.
tox -av displays all the tasks defined in tox.ini:
$ tox -av
default environments:
default -> Invoke pytest to run automated tests
additional environments:
build -> Build the package in isolation according to PEP517, see https://github.com/pypa/build
clean -> Remove old distribution files and temporary build artifacts (./build and ./dist)
docs -> Invoke sphinx-build to build the docs
doctests -> Invoke sphinx-build to run doctests
linkcheck -> Check for broken links in the documentation
publish -> Publish the package you have been developing to a package index server. By default, it uses testpypi. If you really want to publish your package to be publicly accessible in PyPI, use the `-- --repository pypi` option. 要执行哪个命令便用 tox -e build, tox -e docs 等
In my experience with the tox command, every step seems to be slow, and it should take some time to create a virtual machine.
PyBuilder
It's best to take a look at another build tool, PyBuilder, which creates a directory structure that is very close to Maven, so let's take a look
$ pip install pybuilder
$ mkdir sample && cd sample # 项目目录需手工创建
$ pyb --start-project # 回答一些问题后创建所需的目录和文件 Let's take a look at its directory structure:
$ tree sample
.
├── build.py
├── docs
├── pyproject.toml
├── setup.py
└── src
├── main
│ ├── python
│ └── scripts
└── unittest
└── python The build process is still done with the pyb command, you can use pyb -h to view the help, pyb -t lists all the tasks, the tasks of PyBuilder are added as plugins, and the plugins are configured in the build.py file.
$ pyb -t sample
Tasks found for project "sample":
analyze - Execute analysis plugins.
depends on tasks: prepare run_unit_tests
clean - Cleans the generated output.
compile_sources - Compiles source files that need compilation.
depends on tasks: prepare
coverage - <no description available>
depends on tasks: verify
install - Installs the published project.
depends on tasks: package publish(optional)
package - Packages the application. Package a python application.
depends on tasks: compile_sources run_unit_tests(optional)
prepare - Prepares the project for building. Creates target VEnvs
print_module_path - Print the module path.
print_scripts_path - Print the script path.
publish - Publishes the project.
depends on tasks: package verify(optional) coverage(optional)
run_integration_tests - Runs integration tests on the packaged application.
depends on tasks: package
run_unit_tests - Runs all unit tests. Runs unit tests based on Python's unittest module
depends on tasks: compile_sources
upload - Upload a project to PyPi.
verify - Verifies the project and possibly integration tests.
depends on tasks: run_integration_tests(optional)
$ pyb run_unit_tests sample PyBuilder also creates virtual environments before building or testing, and from version 0.12.9 onwards you can skip creating virtual environments with the --no-venvs parameter. With --no-venvs, the Python code will be executed in the current Python environment running pyb, and the required dependencies will be installed manually.
The project's dependencies are also defined in the build.py file
@init
def set_properties(project):
project.depends_on('boto3', '>=1.18.52')
project.build_depends_on('mock') The above dependencies are then installed when pyb is executed to create a virtual environment, where tests and builds are run.
Poetry
The last Poetry, I feel like it's a more mature and active Python build, it has a more powerful trust management function, you can add dependencies with poetry add boto3, poetry show --tree shows the dependency tree. Take a look at how to install and create a project
$ pip install poetry
$ poetry new sample It creates projects that are simpler than all above
$ tree sample
sample
├── README.rst
├── pyproject.toml
├── sample
│ └── __init__.py
└── tests
├── __init__.py
└── test_sample.py 如果给 poetry new 带上 --src 参数，那么源文件目录 sample 会放在 src 目录下，即 sample/src/sample.
Poetry Init generates a pyproject.toml file in the current directory, and the generation of directories, etc., needs to be done manually.
It doesn't pay attention to the generation of documentation, the checking of code specifications, and the lack of code coverage. Its project configuration is more centralized, all in the pyproject.toml file, what is toml?it is a configuration file format Tom's Obvious, Minimal Language (https://github.com/toml-lang/toml).
pyproject.toml 有些类似 NodeJS 的 package.json 文件，比如 poetry add, poetry install 命令的行
# 往 pyproject.toml 中添加对 boto3 的依赖并安装(add 还能从本地或 git 来安装依赖 ),
poetry add boto3
# 将依照 pyproject.toml 文件中定义安装相应的依赖到当前的 Python 虚拟环境中
# 比如在 <test-venv>/lib/python3.9/site-packages 目录中，安装好模块后也可让测试用例使用
poetry install Other major ones
1. poetry build # 构建可安装的 *.whl 和 tar.gz 文件
2. poetry shell # 会根据定义在 pyproject.toml 文件中的依赖创建并使用虚拟环境
3. poetry run pytest # 运行使用 pytest 的测试用例，如 tests/test_sample.py
4. poetry run python -m unittest tests/sample_tests.py # 运行 unittest 测试用例
5. poetry export --without-hashes --output requirements.txt # 导出 requirements.txt 文件, --dev 导出含 dev 的依赖，或者用 poetry export --without-hashes > requirements.txt Poetry Run can execute any system command, except that it will be executed in the virtual environment it is intended to do. So it's conceivable that poetry projects must use poetry run ... command to support sphinx, coverage or flake8.
Create a file my_module.py in the sample directory (at the same level as the pyproject.toml file) with the contents
def main():
print('hello poetry') Then write it in pyproject.toml
[tool.poetry.scripts]
my-script="sample.my_module:main" Execute again
$ poetry run my-script 就会输出 "hello poetry"。
Through the understanding of the above four tools, the complexity of the project structure is reduced by cookiecutter-pyproject -> PyScaffold -> PyBuilder -> Poetry, and the difficulty of using it is roughly the same order
![Efficient project management: Hidden features make a difference[fig]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)