Unleashing the Power of Memphis.dev: A Guide for Developers and Data Engineers
You might be wondering, "What is Memphis?" In this post, we'll go into great detail about it and some of the incredible capabilities it has to offer to both developers and data engineers, as well as what they can accomplish with it.
Prerequisites
Knowledge of command-line tools: Memphis is installed and configured using the command line, therefore readers should be comfortable using it.
A fundamental understanding of any of the following programming languages and frameworks: Go, JavaScript, Python, Typescript, etc. This will make it easier for you to understand the idea presented in this post, but don't worry—I've got you covered.
What exactly is a message broker?
You might be asking what these "messages" are because a message broker is a middleman who makes it easier for producers and consumers to exchange messages. These fundamental data packets are transmitted between producers and consumers. In general, messages in Memphis are an important part of the model for data streaming pipelines employed by the tool, which is necessary for creating reliable and adaptable data pipelines and event-driven systems and is frequently referred to as Message Queue
What exactly is Memphis Broker?
Memphis is a cloud-native message broker that is simple, robust, and durable. When they say it, they are just stating the fact that Memphis is a next-generation replacement to existing message brokers. Combining Rapid Development, stability, performance, and efficiency all at once, it delivers a positive developer experience. How frequently does a message broker do that?
Memphis also includes a UI and CLI to assist you to speed up development and let you concentrate on what is important.
Memphis is currently supported by NodeJS, TypeScript, NestJS, Go, and Python, with support for additional languages coming soon.
Why is Memphis significant in data engineering and why?
Memphis differs from other message brokers, such as Apache Kafka and RabbitMQ, in that it requires less coding on the part of developers to set up. Memphis allows you to embed processing an application and perform activities while it's inside, in addition to ingestion and data collection. Memphis uses extremely few lines of code and Schema management is a key component of what it does.
Here are some explanations on why message brokers like Memphis are crucial in data engineering and development.
Stability: It might be challenging to make sure that all of your systems can handle the load as the amount and complexity of data increase. You may spread the load across several systems and make sure that your pipelines can scale up or down as necessary by employing a message broker.
Microservices async communication: In the Kubernetes context, it offers a potent tool for managing asynchronous communication between microservices. By separating your systems and services using a message broker, you can add, remove, or modify components without affecting the pipeline as a whole. Your pipelines may be simpler to maintain and develop as a result.
Effectiveness and Performance: Making use of a message broker can help to ensure that data is not lost in transit and that messages are delivered consistently. In mission-critical systems, where data loss can have negative effects, this can be very crucial.
Real-time processing: Instead of waiting for batch processing, you may process messages as they come in via a message broker. This can be especially crucial in systems that depend on quick processing, like fraud detection or real-time analytics
Memphis installation and configuration
Memphis may now be set up and configured in one of two ways: using Kubernetes, which is primarily for production, or using Docker to run it locally on your workstation. In this article, Docker will be used. A group of platform-as-a-service solutions known as Docker provide software in packages known as containers using OS-level virtualization. Containers can communicate with one another through well-defined channels and are insulated from one another while bundling their software, libraries, and configuration files. Docker must be installed and operational before you can install Memphis.
Run this command on your terminal to install Memphis.
curl -s https://memphisdev.github.io/memphis-docker/docker-compose.yml -o docker-compose.yml && docker compose -f docker-compose.yml -p Memphis up
Once Memphis is installed, you may choose the language you want to use to create and consume messages. Memphis queues are referred to as stations, just like RabbitMQ has queues and Kafka has topics.
Once Memphis has started running in your terminal, you can access the UI by navigating to "http://localhost:9000/." Memphis has a stunning user interface that makes it the best available and is friendly for beginners.
When Memphis is operating, you can see this in your terminal.
2023-03-13 23:25:36 UI/CLI/SDK root username - root 2023-03-13 23:25:36 UI/CLI root password - memphis 2023-03-13 23:25:36 SDK connection token - memphis Use the standard login credentials of "root" and "memphis" to log in. When you install Memphis using Docker, these are the "root" user's default login credentials.
I'll be using stock feed as my station name and factory name for this project, but you may choose whatever name you choose to depend on the project you're working on.
What will we be constructing?
For this project, I'll create a straightforward program that downloads a set of firms' random stock values and delivers them to the "stock prices" station. By subscribing to the same station, the consumer receives and prints out the stock prices for their separate clients. The "price producer" and the "price consumer" are the only two functions this app will have. This allows you to create any complex software you want, which is what Memphis is for.
Let's get going
I'll be utilizing Python to create this application.
Make a new, empty directory and give it any name you like; I'm calling my stock feed.
mkdir stock_feed
cd stock_feed
as soon as you enter the directory, launch the Python virtual environment
python -m venv .venv
Use the pip command to install Memphis in your project.
pip3 install --upgrade memphis-py
You can now get going and begin writing your code.
Let's first create the producer, which we'll call "price producer.py," from which all signals flow.
import asyncio
import random
from memphis import Memphis, Headers, MemphisError, MemphisConnectError, MemphisHeaderError, MemphisSchemaError
async def main():
try:
# Create a Memphis instance and connects to the Memphis server
memphis = Memphis()
await memphis.connect(host="localhost", username="stock_feed", connection_token="memphis")
# Create a producer for the stock_prices station and set headers
producer = await memphis.producer(station_name="stock_prices", producer_name="stock_price_producer")
headers = Headers()
while True:
# Continuously sending the prices to the Memphis server
company = random.choice(["META", "MSFT", "AAPL", "GOOG", "FB", "AMZN", "TSLA"])
price = round(random.uniform(100, 500), 2)
headers.add("company", company)
print(f"Producing: {company} - ${price}")
await producer.produce(bytearray(f"{price}", 'utf-8'), headers=headers)
await asyncio.sleep(1)
except (MemphisError, MemphisConnectError, MemphisHeaderError, MemphisSchemaError) as e:
print(e)
finally:
await memphis.close()
if __name__ == '__main__':
asyncio.run(main())
When the aforementioned code is executed using "python price_producer.py," it transmits stock feed messages to the Memphis station continually while it waits for a user to access them.
Second, we'll create the consumer—call let's it "price_consumer.py"—from where all messages originate. The consumer can then access these messages.
import asyncio
from memphis import Memphis, MemphisError, MemphisConnectError, MemphisHeaderError
# Define the asynchronous main function for the price consumer
async def main():
async def msg_handler(msgs, error, context):
try:
for msg in msgs:
company = msg.get_headers().get("company", "UNKNOWN")
price = msg.get_data().decode('utf-8')
print(f"Received: {company} - ${price}")
await msg.ack()
if error:
print(error)
except (MemphisError, MemphisConnectError, MemphisHeaderError) as e:
print(e)
return
try:
# Create a Memphis instance and connect to the Memphis server
memphis = Memphis()
await memphis.connect(host="localhost", username="stock_feed", connection_token="memphis")
# Create a consumer for the "stock prices" station
consumer = await memphis.consumer(station_name="stock_prices", consumer_name="stock_price_consumer", consumer_group="stock_price_group")
consumer.set_context({"key": "value"})
consumer.consume(msg_handler)
# Keep your main thread alive so the consumer will keep receiving data
await asyncio.Event().wait()
except (MemphisError, MemphisConnectError) as e:
print(e)
finally:
await memphis.close()
if __name__ == '__main__':
asyncio.run(main())
Now, the messages that are already on the Memphis server may be accessed by the consumer when it is launched using python price_consumer.py.
Memphis has a stunning UI that allows you to view each packet of data sent and received, along with its payload, all with only a few lines of code, boosting productivity and allowing developers to concentrate on what counts.
Conclusion
Memphis is a potent distributed messaging system that gives programmers and data engineers a very dependable, scalable, and effective way to exchange messages across programs, services, and systems. Teams may create and scale large-scale applications using Memphis' messaging technology, which can be utilized to create distributed systems and microservices architectures.
The ease of use, scalability, and simplicity of Memphis are three of its main advantages. The Memphis API is straightforward for developers to use from the get-go since it is clear and simple to understand. Developers may send and receive messages across various apps, systems, and languages with just a few lines of code. Memphis is a flexible option for teams that use a variety of technologies because it supports a wide range of programming languages, including Python, TypeScript,.NET, and Go, among others.
In conclusion, Memphis is a fantastic option for programmers and data engineers who require a strong, scalable, and trustworthy messaging system for their distributed systems.
Buy me a coffee ☕
Use the button below to purchase a coffee for me if you enjoyed and found value in my essay. In ahead, I appreciate it.
Please like and follow for more material if you can. This is my first in-depth technical essay on a project, so if you have any questions or find any mistakes, just let me know.