Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.3.2-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54b10b61c30da90cdd0e32c35346700512324e47120de303267fbcc99bec34b6 |
|
MD5 | 540df264bcd736012aaff795ed33b4f7 |
|
BLAKE2b-256 | 9fc8136adee1eb955daa25375f069881c1aa9f21e96867e74dfa01424bee3f20 |
Hashes for blosc2-2.3.2-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a6d726c3958e3b66c4182d67af0fe44f3823b568bcba2f59a0cd1f03934ca1f |
|
MD5 | ca9b0ed011eb07064554e11fcc6f4aaf |
|
BLAKE2b-256 | 409b03829395e0dc75d01da4967338a103e7b45ab7d73fb30cbddea0435d86f0 |
Hashes for blosc2-2.3.2-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa94b426d812be22a7a9515f72e89d7818a630adad2da35cedc0830bd472f083 |
|
MD5 | d00a9fd491c2aa9ebadcdb711cd255b2 |
|
BLAKE2b-256 | 3f5190b8e5323d520527683752846dc4255401e090675d2a5b196003b22a96f0 |
Hashes for blosc2-2.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c53cc7cb7a80001cecdfa49ac137f7298fcd9b409a93205b5845c8c87dadfaac |
|
MD5 | 1089b7914b0689cc57e4111ab9a3c831 |
|
BLAKE2b-256 | e2c3e5daaa634314a4c721af4884e9e6e4d811d7f132b045e3dd3cdec8b5ca56 |
Hashes for blosc2-2.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 005f0573a592b4805b4be30d7f4450d2dfd56911ae9be7421fb1caeaa76a0c24 |
|
MD5 | 59c2574d2ac5bbe967c08648b338d03b |
|
BLAKE2b-256 | a5a4d897899ff908c27fe987e7cc65ff645f8ae893c3f83a5002937137f9d17e |
Hashes for blosc2-2.3.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b502604e313e47187202b83cbb4cfdb53987574cc052a4903d99ef5029bf5145 |
|
MD5 | 31fb431d266ee8b1b7a37f7634f55776 |
|
BLAKE2b-256 | 2f317d501e58761979ea45677243f7fe550cfe56a2620048b6fcf8d56c01c237 |
Hashes for blosc2-2.3.2-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6b7963712c5c52836bcd46d0c1393d6a57140586d13106b13869bd2b4a19b29 |
|
MD5 | c94d171d9ad846914da32fc505a46d34 |
|
BLAKE2b-256 | 417e522bb0a1f5491381e264284aa97252913e588b71dbe222de651b41ee5cd7 |
Hashes for blosc2-2.3.2-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a347dc04d61b151f1c3c500f3614b411a7cfb2556463fd3e63ffd846c1645e2 |
|
MD5 | 96f6a5a1d8a70120b725b60005e6bd98 |
|
BLAKE2b-256 | 6a157ccf6afaac9bb0d36e396e37c0214a39947122d2e2176f7573718eaf635c |
Hashes for blosc2-2.3.2-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f67917f087cbb6f22eeb79acdbfaba5deeca764223d1b649055d6ea39143a1d |
|
MD5 | 709f52b1d54f295cc011f549ad1e8831 |
|
BLAKE2b-256 | ec12b2d7515d39f7926ae438081bb002675744b2182aa0ed7e2fed39de14a66f |
Hashes for blosc2-2.3.2-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82f9c0d60199bb496569a55bd3a61c67a216e4001a4849427df096d7ab0e315e |
|
MD5 | ae5c371eecb972eee3bc315e557569e6 |
|
BLAKE2b-256 | 6f9254e9b3aeb48b9d3dcc11d6e1e945c093a7e1251c1ff540b7e6222c4e936e |
Hashes for blosc2-2.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 244fde8a89a4e37ab76116e66c820e5483179e31a936943860a9fa2869aa118a |
|
MD5 | d95eb63c9d5ce518023f15a2da557e40 |
|
BLAKE2b-256 | 83d06d062782190a724ae0559b1eced438ad819f7e1c91f6ca274a25c39d3549 |
Hashes for blosc2-2.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7fa7fdcba465d490422d061bd02b8f267607324952aab495dec84b6963802968 |
|
MD5 | 6e6cfbc184a32772b3e89350e847ac68 |
|
BLAKE2b-256 | 202e38ac3228fd0bc937a2994520b4ba10fea0439da30922d1a9d03ed5a1ddab |
Hashes for blosc2-2.3.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe9094f9187f76172ae09fe55d8b3e167defe1d37a50e902707d60be681cb02d |
|
MD5 | 448f7e7bd0f74c430fb9870d6ddfa10b |
|
BLAKE2b-256 | d4f6498281ab5e6cefcd4349ab3d6f146ace38d435b265faa29d5b03436926a9 |
Hashes for blosc2-2.3.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d37f745456c7db29a830a68cc54a006c6058f091b609455cd399c347d007987 |
|
MD5 | bfdf7d3e12c8e5cd23f75bd450a114e3 |
|
BLAKE2b-256 | d1e025cf697151a94884d662fa363f087a979b146ff33215e934ef2ddd4e9058 |
Hashes for blosc2-2.3.2-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 694b9fd1f4351d2de8a9e648f56c6f582da781cdcd2b18a1e6a0bde0662ff31e |
|
MD5 | 0ec297ccd26a6ad879e3b2d575cdbce2 |
|
BLAKE2b-256 | 2bf69d84452da69b222963b911eafe135bc7bc0202a0945a09fdf2e82a4769c1 |
Hashes for blosc2-2.3.2-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ebe61f053e3c429284249f7b840014f2a5353cd42ce40be5a9cacfb9bdcf24b |
|
MD5 | b6863ee84bfa4ad420a16be7ece248ad |
|
BLAKE2b-256 | 6cc42e939a30fc2acca90a1fa6f990c37fed4fa764ebb21969be739df0a23422 |
Hashes for blosc2-2.3.2-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0207cf180fb7fe9e513a016b122c97410061abca294e60ee9a80d1c928159763 |
|
MD5 | 76b1e7277ac97d5ea98b57833acdea3f |
|
BLAKE2b-256 | d1b3bad6db11dded706645727b500b3b4191aa77bda077dc86c3db7920c737c5 |
Hashes for blosc2-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd63f139a1d28d1ea334fe9c85aa180ef69fe037c470c94aedcaf94beaf3b1f6 |
|
MD5 | 47b0dbb3766e1c23291d394f1c18438e |
|
BLAKE2b-256 | 22c8b071292cae16e8edb4c51008267ac74332a1dc095611d13e17f9ae8d8daa |
Hashes for blosc2-2.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 748c4d681c00c1fe1c89490c3224b9ecf161b8da86b3440b82a9cfdae35b54af |
|
MD5 | bf006b72617e4009072a04e566844c2e |
|
BLAKE2b-256 | 81dbbf99a34b61c06eabcf25e73d20e7498f482806545fdb7bb5cd0a9682eb93 |
Hashes for blosc2-2.3.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a54290721d5eae04913111ddc6fcfa866bab530849b3bc403fcdf1a233d1dc0a |
|
MD5 | a711dd952440c96562a4ab3969edb2a0 |
|
BLAKE2b-256 | 6151ce729571442ce7ae5a51f08382eefb62d8bcdfbd0eba6628469a31524173 |
Hashes for blosc2-2.3.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfbd9cf8e804d1fb4c44d8e8d5851177e0bb6f4ae145ddc6ba7670e65b652853 |
|
MD5 | 71a1be192006c251aced9572ce671e96 |
|
BLAKE2b-256 | 0f85f8be60fc10fa2f089678f6289775f4851ead811102d18e8a146fcdbd4da6 |
Hashes for blosc2-2.3.2-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 618c29cd31aa34cc4491fa46b4144e3e70149edbd7fa4551b54b02e568b3f652 |
|
MD5 | 3f1abd23470310f2381bf493268eea70 |
|
BLAKE2b-256 | 56b90e17a4876f2d40f2816d0f6d372c3afafbcd16ec54342997af3ae0252814 |
Hashes for blosc2-2.3.2-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba0c9b44e7aa7322863dbb48e91edc9b01cfe302b6e4b3ca4e052cbb47835fd3 |
|
MD5 | 6652d86ca07c513beec2e196e301963a |
|
BLAKE2b-256 | 900c8ac11fa9ce898861fb7ab7f1d2b10ce6e9f9711f529c7e20cb676622693a |
Hashes for blosc2-2.3.2-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 659bfecbcaf37a69588c60a61eed4efa0d710a425f3b15ca4cc9f40430afceef |
|
MD5 | 723eb50566e71b20c1061eb45dc32d86 |
|
BLAKE2b-256 | ed57eee18c45141d5d07669613518dc318b0a898720d14cde3f91089e63c5c56 |
Hashes for blosc2-2.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bbdb3adaaa3533dab4d4b9c91ebaa3c4e72cae43a3bc1b9ef109e67825d45c0 |
|
MD5 | 3a4a4503e981222b5e49c5513107b386 |
|
BLAKE2b-256 | 7b7fe3de8ce032c5f03c426ee0bc2d74a1ea6739068f2d7f94afc9f8f68fa413 |
Hashes for blosc2-2.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38a0d09174344a4601adf94894729d00767bea859729115611c45dd1927c8fa5 |
|
MD5 | 70e99e822356e8753b11dcb0f6c6a4f0 |
|
BLAKE2b-256 | 6c915d2dc8baa5b82505a32e9941652621c2e9331af5ada45a846e4b0444c1c4 |
Hashes for blosc2-2.3.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e2479b3966a46adfd8aca269f5835ee28a0182605ae94023eebcc9c7fb98643 |
|
MD5 | 1695198d7f0e1b91360b9c9bd27d323b |
|
BLAKE2b-256 | 4243f85a4b02c1400ea8a6a4cc0ae4587890d9c03af55940a6eeee4d9f52fca8 |
Hashes for blosc2-2.3.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04c18acdd9e88a27549580c3b6ead5e86ca5708581c21bc398a522878132191d |
|
MD5 | b7079a06325ed3b72706dfa20161396c |
|
BLAKE2b-256 | b5aa603aa836266a62ea9626fa598c06b135af1ebab0148ef26e1b2911e6147e |