A Python interface to the Rust object_store crate, providing a uniform API for interacting with object storage services and local files.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for devseed from gravatar.com devseed

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

obstore

PyPI

Simple, fast integration with object storage services like Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compliant APIs like Cloudflare R2.

  • Sync and async API.
  • Streaming downloads with configurable chunking.
  • Streaming list, with no need to paginate.
  • Support for conditional put ("put if not exists"), as well as custom tags and attributes.
  • Automatically supports multipart uploads under the hood for large file objects.
  • Optionally return list results as Arrow, which is faster than materializing Python dict/list objects.
  • Easy to install with no required Python dependencies.
  • The underlying Rust library is production quality and used in large scale production systems, such as the Rust package registry crates.io.
  • Support for zero-copy data exchange from Rust into Python in get_range and get_ranges.
  • Simple API with static type checking.
  • Helpers for constructing from environment variables and boto3.Session objects

Installation

pip install obstore

Documentation

Full documentation is available on the website.

Usage

Constructing a store

Classes to construct a store are exported from the obstore.store submodule:

  • S3Store: Configure a connection to Amazon S3.
  • GCSStore: Configure a connection to Google Cloud Storage.
  • AzureStore: Configure a connection to Microsoft Azure Blob Storage.
  • HTTPStore: Configure a connection to a generic HTTP server
  • LocalStore: Local filesystem storage providing the same object store interface.
  • MemoryStore: A fully in-memory implementation of ObjectStore.

Example

import boto3
from obstore.store import S3Store

session = boto3.Session()
store = S3Store.from_session(session, "bucket-name", config={"AWS_REGION": "us-east-1"})

Configuration

Each store class above has its own configuration, accessible through the config named parameter. This is covered in the docs, and string literals are in the type hints.

Additional HTTP client configuration is available via the client_options named parameter.

Interacting with a store

All methods for interacting with a store are exported as top-level functions (not methods on the store object):

  • copy: Copy an object from one path to another in the same object store.
  • delete: Delete the object at the specified location.
  • get: Return the bytes that are stored at the specified location.
  • head: Return the metadata for the specified location
  • list: List all the objects with the given prefix.
  • put: Save the provided bytes to the specified location
  • rename: Move an object from one path to another in the same object store.

There are a few additional APIs useful for specific use cases:

All methods have a comparable async method with the same name plus an _async suffix.

Example

import obstore as obs

store = obs.store.MemoryStore()

obs.put(store, "file.txt", b"hello world!")
response = obs.get(store, "file.txt")
response.meta
# {'path': 'file.txt',
#  'last_modified': datetime.datetime(2024, 10, 21, 16, 19, 45, 102620, tzinfo=datetime.timezone.utc),
#  'size': 12,
#  'e_tag': '0',
#  'version': None}
assert response.bytes() == b"hello world!"

byte_range = obs.get_range(store, "file.txt", offset=0, length=5)
assert byte_range == b"hello"

obs.copy(store, "file.txt", "other.txt")
assert obs.get(store, "other.txt").bytes() == b"hello world!"

All of these methods also have async counterparts, suffixed with _async.

import obstore as obs

store = obs.store.MemoryStore()

await obs.put_async(store, "file.txt", b"hello world!")
response = await obs.get_async(store, "file.txt")
response.meta
# {'path': 'file.txt',
#  'last_modified': datetime.datetime(2024, 10, 21, 16, 20, 36, 477418, tzinfo=datetime.timezone.utc),
#  'size': 12,
#  'e_tag': '0',
#  'version': None}
assert await response.bytes_async() == b"hello world!"

byte_range = await obs.get_range_async(store, "file.txt", offset=0, length=5)
assert byte_range == b"hello"

await obs.copy_async(store, "file.txt", "other.txt")
resp = await obs.get_async(store, "other.txt")
assert await resp.bytes_async() == b"hello world!"

Comparison to object-store-python

Read a detailed comparison to object-store-python, a previous Python library that also wraps the same Rust object_store crate.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for devseed from gravatar.com devseed

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

This version

0.2.0

0.2.0b2 pre-release

0.1.0b1 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

obstore-0.2.0-pp310-pypy310_pp73-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ x86-64

obstore-0.2.0-pp310-pypy310_pp73-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ i686

obstore-0.2.0-pp310-pypy310_pp73-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-pp310-pypy310_pp73-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ ARM64

obstore-0.2.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

obstore-0.2.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ s390x

obstore-0.2.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ i686

obstore-0.2.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-pp39-pypy39_pp73-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ x86-64

obstore-0.2.0-pp39-pypy39_pp73-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ i686

obstore-0.2.0-pp39-pypy39_pp73-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-pp39-pypy39_pp73-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded PyPy musllinux: musl 1.2+ ARM64

obstore-0.2.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

obstore-0.2.0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ s390x

obstore-0.2.0-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ i686

obstore-0.2.0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-cp312-none-win_amd64.whl (3.4 MB view hashes)

Uploaded CPython 3.12 Windows x86-64

obstore-0.2.0-cp312-cp312-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ x86-64

obstore-0.2.0-cp312-cp312-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ i686

obstore-0.2.0-cp312-cp312-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-cp312-cp312-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ ARM64

obstore-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

obstore-0.2.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ s390x

obstore-0.2.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ i686

obstore-0.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-cp312-cp312-macosx_11_0_arm64.whl (3.1 MB view hashes)

Uploaded CPython 3.12 macOS 11.0+ ARM64

obstore-0.2.0-cp312-cp312-macosx_10_12_x86_64.whl (3.3 MB view hashes)

Uploaded CPython 3.12 macOS 10.12+ x86-64

obstore-0.2.0-cp311-none-win_amd64.whl (3.4 MB view hashes)

Uploaded CPython 3.11 Windows x86-64

obstore-0.2.0-cp311-cp311-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ x86-64

obstore-0.2.0-cp311-cp311-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ i686

obstore-0.2.0-cp311-cp311-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-cp311-cp311-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ ARM64

obstore-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

obstore-0.2.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ s390x

obstore-0.2.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ i686

obstore-0.2.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-cp311-cp311-macosx_11_0_arm64.whl (3.1 MB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

obstore-0.2.0-cp311-cp311-macosx_10_12_x86_64.whl (3.3 MB view hashes)

Uploaded CPython 3.11 macOS 10.12+ x86-64

obstore-0.2.0-cp310-none-win_amd64.whl (3.4 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

obstore-0.2.0-cp310-cp310-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ x86-64

obstore-0.2.0-cp310-cp310-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ i686

obstore-0.2.0-cp310-cp310-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-cp310-cp310-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ ARM64

obstore-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

obstore-0.2.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ s390x

obstore-0.2.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ i686

obstore-0.2.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-cp310-cp310-macosx_11_0_arm64.whl (3.1 MB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

obstore-0.2.0-cp39-none-win_amd64.whl (3.4 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

obstore-0.2.0-cp39-cp39-musllinux_1_2_x86_64.whl (3.7 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ x86-64

obstore-0.2.0-cp39-cp39-musllinux_1_2_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ i686

obstore-0.2.0-cp39-cp39-musllinux_1_2_armv7l.whl (3.6 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ ARMv7l

obstore-0.2.0-cp39-cp39-musllinux_1_2_aarch64.whl (3.7 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ ARM64

obstore-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

obstore-0.2.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ s390x

obstore-0.2.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (3.6 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64le

obstore-0.2.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl (3.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ i686

obstore-0.2.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.3 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARMv7l

obstore-0.2.0-cp39-cp39-macosx_11_0_arm64.whl (3.1 MB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft Sir Raditya Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page