Python wrapper for the C-Blosc2 library.
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Enjoy data!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05643c03786b64082b36f40d9b4a7151669494ba439b9765a768105665c00a72 |
|
MD5 | 04d12faabd1fca3c2c55d9ef4f0bb489 |
|
BLAKE2b-256 | 545ded012d730126631011caab51577f235c734278f3e209dd75c63bf27d7d67 |
Hashes for blosc2-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01062444e6cce13987b32b3f605c1e678c1f77e1ddce91c2e7492d4bea828697 |
|
MD5 | 3a595da95cc58d977c5f0c76377a7d71 |
|
BLAKE2b-256 | 4a5c5a15393fe80503a0b5a0f4c1b880d80d19d8579b217b0103a21f6f070a55 |
Hashes for blosc2-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c092e2d997001e18119d0ee0bed1ccfb6c1be67473752ce6f6de445d8b0ec16 |
|
MD5 | dbc5c7aec3a4756a4e3a0653a88ee503 |
|
BLAKE2b-256 | 25998bc182a15bbb1fc93d23beb209e0c1ee8f75da0bf7b32a1f06378f1d08bc |
Hashes for blosc2-2.2.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4faba91c7ffa9f7b0c595eb0008ac045d0b85c0b213d6f63297a8cca29faf08 |
|
MD5 | 11a7a13bde24eeed9e2870416bd79900 |
|
BLAKE2b-256 | 9ae0d66174f51b22fb8e9ebbcf17d78b87ebf108f517fc51899ea7e171401189 |
Hashes for blosc2-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5017c709f9c1eae09206bab41249bad141f7a1a1295f8bd16d4d7a40fcbb9773 |
|
MD5 | c3afda0628174ce318e6d33a6f15953a |
|
BLAKE2b-256 | 174560263d9027972f80671ab6515a0a4b02a7ce3e850c176dea0c4d0a23b994 |
Hashes for blosc2-2.2.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 677b0c6a69860f8ca56070956d7510e5c6c43a7379a6f7320d573c541f58211a |
|
MD5 | 3aaac83ce599fccb2f00a6ee36bb677a |
|
BLAKE2b-256 | 43ad2ab01148d488e7b179b1ecfe178e08fda0c4add5c90de0aff80726616a7b |
Hashes for blosc2-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4824e3263f6462a08df7efa1d3e1de8b04d4c96a2529ecd6278498bc440125e1 |
|
MD5 | 01d31b53d02fb82dd0097d71291de8dd |
|
BLAKE2b-256 | 82b169d21c8a48a3fa771c490c80c0eafb77c59a6861596f140e8cc7c84bde98 |
Hashes for blosc2-2.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a22eda1c2723ce97a5395150592b30681e3edc32409b97e0500230101a8ea760 |
|
MD5 | aa1ebc115a82f82bf39dec35a36845f4 |
|
BLAKE2b-256 | 5292a9cc2926e5361a88363d33f77936dafc95299c4236d66fd45ce4a7ea9472 |
Hashes for blosc2-2.2.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d31434048adc813b9e921ba581d0cb960ca245a4d1ef116342b2bf3d27b9bbb |
|
MD5 | 62c6b0fc250070cc1a65b70c9cbcdde4 |
|
BLAKE2b-256 | b29b715f3b09d2e3e5bd390fe53c85bd68ce838277de5f6fa2e12f024eca63a1 |
Hashes for blosc2-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24dedc40b3e42a1b6b47a72901ab2c9ecffd6047ad4f5344e9041f871804ec2b |
|
MD5 | d839e3f276c72eba22b5a06c375fad90 |
|
BLAKE2b-256 | cb2b38400c0a2c49ba3e1580ddb6fafd93a4c20189bab9c0a28bd334314fde45 |
Hashes for blosc2-2.2.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66fe439152c7ec761b461515167f6226bf78812a88f6eb291c289b6e0342962e |
|
MD5 | 20d3359759cdf7eda9203af3beacee96 |
|
BLAKE2b-256 | f390eaf2a677727ae0498ef73858cc83054acc726fa42f63479b79f306f33a05 |
Hashes for blosc2-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b549a80cc52b2fca846d5c9b0fc80a4949b483ca484117e6e95b5ac7f7d874e7 |
|
MD5 | 09b5a20fc08481ba65db5b2abff9c33d |
|
BLAKE2b-256 | f99175ac6e90fd648d9f1b59faa38a2edd4fd4b9e07f305738ce9790b0d1f4cf |
Hashes for blosc2-2.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19169c5deff308986e0e7140a90f2b0c48fc7fb54aa41960c66257bc562d3101 |
|
MD5 | e4ad3431406070acff08cb3fc4ae5c77 |
|
BLAKE2b-256 | e14468509628134daf1788f91f9907466956f964711520f9666f6707afe3bb60 |
Hashes for blosc2-2.2.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf1c7a958881a6a1963fe8d79dfb54eee4373bb2bd5c863378edfa1c93f2967b |
|
MD5 | c87e158f838064688702022f36552bdb |
|
BLAKE2b-256 | d8dbb59a85474a91b77786a7db42d3ed50690c113da7608da569133906db420b |
Hashes for blosc2-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1feaa6b931eb4501a608313d26470cc9a7de0578fe463df7b521a124af89f27f |
|
MD5 | c9bf6b3b5ff7b00f1615848ca0aaf46d |
|
BLAKE2b-256 | 659d6e07863f9018ca5c6c95df597b0ca65776c29d533f5b5587b502c25dc7ea |
Hashes for blosc2-2.2.1-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cdc2ffec7fde3ea0b61e0ef02c7dc160e73a33db7274314dfc75e973f3c1262 |
|
MD5 | e66c3c7e0fc91851809fc3ba41e5f3bb |
|
BLAKE2b-256 | 133d8e7b15a691d1e3c998785fc791810e4236a584bb3020c67888fc9bd1feaa |
Hashes for blosc2-2.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e299b1877207ebe40b64ed6d341bcc9c6f7cf980cf72b08f6f91e81804c98ed |
|
MD5 | 92bf331c1a2bfa408a8df11ad20334bf |
|
BLAKE2b-256 | ffe1eb2324081ee63dfa94bb80f4ecc28461da69ee6182f3688c12d25a184d57 |
Hashes for blosc2-2.2.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02834e4a545c7937b4dea324b15b7ab0768c05622ab42eab46a3605305ea57f4 |
|
MD5 | b45f01768a246379092b8f60acef6e1b |
|
BLAKE2b-256 | 76d5586f2c0ba3fdd1bc099ddaef9f47593a17d503086899c3ed44f75f8d07da |
Hashes for blosc2-2.2.1-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f7cbd18bd26ed2241f0c92c98d4a8c5ad2c030efc2e2c47095474f354c6b479 |
|
MD5 | 6e50002a95d96c1df2ed67948103f952 |
|
BLAKE2b-256 | a17dc82ec9e9ae7c78d0bb60d1dd008c83fa1347922ce125b588e7a75ba3d75f |
Hashes for blosc2-2.2.1-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96fb47707c0b4b38e1a8905c477541cd7903ddae86e4c869bacd810c97813b68 |
|
MD5 | 3fde80372e63ecc4626f5790b2e8a764 |
|
BLAKE2b-256 | de4ea6ac6f7d12ebb0c5b1af3ab2600049ad509a46823782144808b3c47ad9d3 |