Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bluesky_kafka-0.9.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7b9a744b7e4ce0e7a0845371fb02ec3b57e376c9d527d3671ed61a7dfe88395 |
|
MD5 | 4198123985787ea363e0c377ee1f5720 |
|
BLAKE2b-256 | 75d838fbd8a8ada1412624a7d06b982dc3710e800fe185ffb8551cd553a1852a |