Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.4-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3b0f7ff5701c81bd4650dca3ec375e57f336218ea23808b3b0a408d087513cd |
|
MD5 | 9cf771120661aec55d23b1075039278e |
|
BLAKE2b-256 | 46b5aed475da88dc6d5a93130d264eefff4701a152a106388721eb9169dd8545 |
Hashes for blosc2-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0674702ff600768eedbee3c1c38fdf10b2df88429fd8dda7733c9c879745514f |
|
MD5 | 7939e0c47aa29f151f20020da904805d |
|
BLAKE2b-256 | f49b8c9ed843014bac6efaa2de26d4fe431ef932cb43c685293669f411fb5a4b |
Hashes for blosc2-2.2.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 313fd8be4c6b57b4a4d312828e1df5c16e98b01e690580e7d0bc56755e49d597 |
|
MD5 | ebc3dbedd42e06bdaa02ba4f381b3f27 |
|
BLAKE2b-256 | 2a426348c272389e14739aa996b0148b97f71e39da2be14d8b5513ec7a82ecf2 |
Hashes for blosc2-2.2.4-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf39bd1557dc8e7922c59ffd006503bf3b3159a08b367077b5f076ba4f430e70 |
|
MD5 | bdbdf6dea17f2a652920db61934943e0 |
|
BLAKE2b-256 | a884f56b1ba8d6ccd5557f2f8603a8391b2379a0d912fca9d46a065aecc1dcff |
Hashes for blosc2-2.2.4-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f919918dbf59cf51fde242d95e2d794a2a45dc61dd8b45f0f50f112909d855a |
|
MD5 | d4f614dfbc965e270baba3034425b00a |
|
BLAKE2b-256 | 1068eddaa549b970e73129faa1049c0ec828192c6789d6291d8ab4dec19ed2d4 |
Hashes for blosc2-2.2.4-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53e7d77b6f44c820df06357ada2a6eb6411d84de9d7a306ac44ae5590c7b3586 |
|
MD5 | 1970867c9828ede720635a1cd4d47ad8 |
|
BLAKE2b-256 | 8f600139c0e7498624b659890a52c861b8751e0a08e5011631ace04aacac5a8c |
Hashes for blosc2-2.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f61436249aa4d688a9c49a6f53f1a6254e41c2a90b672f9fb13d189813f6bde |
|
MD5 | bc47d603e4f6fd8a95059d1814b2925b |
|
BLAKE2b-256 | 31a0a030bcd89f47ebe44fad4968220334a634493f6ff2ad3ec02b1499032175 |
Hashes for blosc2-2.2.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cccee8442223e8583a5b06a67295e761166b6fcd817cfe236d3788426806e8a3 |
|
MD5 | f1d9ba8815e1b57e4d44fef57963a195 |
|
BLAKE2b-256 | 921ee0f1787cadf75a1203b56e3cb2964cb4c8421367dece18062d6fe15207c3 |
Hashes for blosc2-2.2.4-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 014d8dddc70488ecb8c27d3022991fef3b4cb5682dbd64d9042813406c9dc562 |
|
MD5 | 3e9b472c8c82fc9eb28b0284a37ca262 |
|
BLAKE2b-256 | d39b48c0ffd2073ee1ad25e78e5baac207f2a60c3f93087f290593467d36347f |
Hashes for blosc2-2.2.4-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2dd5d471a248886f2174c5073960cc62ef979197cf5c0c6d7cecced843fd3d1a |
|
MD5 | a50501c73d2aedead7aaa287f7e9dbbd |
|
BLAKE2b-256 | ceed85d48ff2e87cff31930e3e9137d30de006e70e82ca20cfb6f77133f0f341 |
Hashes for blosc2-2.2.4-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ff1c0767d233be98182138f8ef7a7f1cdaf15fd1e047bafe20ffaf86632cd13 |
|
MD5 | 1ac04bf93515a4d97245081a784572b3 |
|
BLAKE2b-256 | c849a45e8fa8d88a25ee3efbdf21edc7498a2de3c67577e87a49bcdc3a55fe14 |
Hashes for blosc2-2.2.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ad1d2a7a68676e7aad713ca753ddff52f9994599661b560ac4ef1c4f257c8f6 |
|
MD5 | e1050f39acdebd012caa0df00ddaba93 |
|
BLAKE2b-256 | bc998a239dfa731c5512674d997842e15176fb7c16fc3685700f5a81828aab82 |
Hashes for blosc2-2.2.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 218bd0f1b5916e0b3bc92f49bb6971d9b6e5af20cf25c1ffc8fcc2994dbbf465 |
|
MD5 | f14a59ef7fa057f5464de8bc55488e8c |
|
BLAKE2b-256 | 59107ab9d3af68b107b1265eb180dd3f24ce0e47dde4b4ee804f49d8a047e754 |
Hashes for blosc2-2.2.4-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2314112a4407c2cdc933eba724282329fe43d0cc1766ee302db312fe00d78784 |
|
MD5 | 2699de5c66e87ea26c0a7598aad5000b |
|
BLAKE2b-256 | b16e263c01f7963b20f5d1d72fe60367e504df662523e050bab502a6115e40fa |
Hashes for blosc2-2.2.4-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2dc020f57699a98794a66dffbe5bf4b25e90d8d53efe97f777ffda1f8790a4f8 |
|
MD5 | 0fad29c9db3a722f5894f61b6fe40830 |
|
BLAKE2b-256 | d60280d5d232c9187fe4ced7ce32d1f5124aa5dbfa1c60a85f02fca62bca947f |
Hashes for blosc2-2.2.4-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a9e69814dca35117a1397a619037119cfd6fc308287c765ba93d2e878b48ab6 |
|
MD5 | 46cb521d4b5653f8c68263c5b61141ed |
|
BLAKE2b-256 | 9c217223c19c87ec9d1c8c22f185a69930f0ed7f962b84d62aecd007736dcdc9 |
Hashes for blosc2-2.2.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50605061851b1c652ad7dd427639fd8f06f824bcd7d14651acb2cb9b35688fef |
|
MD5 | ae947c47c3dc0a07a74827a5fdd75fed |
|
BLAKE2b-256 | 756490e426682e18dfac7e84b51d13098ab98c49a10637681181ff05cd237d88 |
Hashes for blosc2-2.2.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3929960c256969d7d298d2c004df1dabc68a0da8b12742af23fff41bcf5bec8f |
|
MD5 | 406827f7bd697d1dda4242eea718d3ad |
|
BLAKE2b-256 | c34ef7e574f1c35f65fa4b1dc87614f78de5521a8bb755bd0410ebb604e05b67 |
Hashes for blosc2-2.2.4-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7fd26a6122edf67fff8fe00365fa85effc6f846278d39d2240c9111baccd9406 |
|
MD5 | 4c70fafcb030f4b72733a4574ac0308c |
|
BLAKE2b-256 | 1f99997ad197d0b6e55f10152939f8a0ba8164c6ed1a3bc8c3c8df21eddd6964 |
Hashes for blosc2-2.2.4-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 652219132a69aa4c76fb280824d497672e31c0652ee740095e8ebbc4527a0a40 |
|
MD5 | 65547eeab91a7cc9a8038cd851e44764 |
|
BLAKE2b-256 | e392c02cabbe19ad790a7f3cd369bc9fe4b2387f1f99b0c81e36d156625ee986 |