PyPy has been on my radar for a while but I have never really brought myself to the point of actually trying it. Recently, a notification caught to my attention stating PyPy 2.0.2 was released and had support to run Gevent. Good news! I was looking at speed improvements for my Wishbone library and decided to spend some effort into getting PyPy with Gevent up and running. The information available explaining how to setup Gevent in PyPy is rather sparse which might be a bit time consuming when you have to figure out the bits and pieces. After some tinkering I got PyPy with Gevent up and running within virtualenv on a Fedora host. Here are my notes, It might be useful for you to and save some previous time:
Submit Nagios metrics to Graphite with ModGearman and MetricFactory
When it comes down to monitoring Nagios is still the weapon of choice for many. I would have abandoned it if there weren’t projects like Livestatus, Mod_Gearman and Thruk which to my opinion should never be missing from any Nagios setup. Mod_Gearman, the framework which makes Nagios scalable, has a feature which stores the performance data produced by the Nagios plugins into a Gearman queue. Graphing that performance data with Graphite is a straightforward job with Metricfactory.
Testing Graphite with MetricFactory
Graphite is great. Not only because it’s a great piece of software but also because of the community around it which brings forth all kinds of metrics goodness. Although it’s pretty straightforward to get Graphite up and running on one node, it gets a bit more complex to get it up and running in a clustered/sharded/federated mode. I found Jeff Forcier’s Clustering Graphite and Richard Crowley’s Federated Graphite to be very helpful. Once you have your clustered Graphite setup up and running you might want to test its behavior and get acquainted with its different settings and modules before going to production. That’s where Metricfactory might help you out.
Consume, process and produce data with Wishbone and RabbitMQ
When I first came in touch with the messaging concept it was some kind of a revelation. Ever since many solutions I work on involve message brokers one way or the other. Because of that, a design pattern occurred hence the need for a Python framework allowing me to easily write solutions to interact with messaging based setups. Wishbone is a Python framework which simplifies writing coroutine event pipelines. Wishbone is a framework in which you can load and connect multiple modules in order to come to a tailored solution.
Although Wishbone’s scope is larger than that, I would like to demonstrate how easy it is to write a daemon which consumes and processes messages from RabbitMQ.
Using MetricFactory to get Hadoop metrics into Graphite.
Without metrics we’re flying blind and that’s very much the case with Hadoop. Hadoop is a well known framework to build reliable, scalable and distributed computing clusters. The Hadoop framework is a complex environment which “out of the box” hardly offers any metrics oversight on how the different components are performing.
Graphite is king of the hill when it comes to graphing metrics with open source. Hadoop doesn’t support Graphite’s data format and way to submit metrics, however it does natively support the Ganglia graphing framework. That is something we are going to use to our advantage.
MetricFactory is a modular tool which allows you to build servers which can do “stuff” with metrics. In this case “stuff” means accepting Ganglia metrics, convert and submit them to the Graphite framework.
The information this blog post is just another way of doing things which might suit your needs better than any other available options.
New Year Python Meme 2012
Following Tarek Ziadé’s idea for a 2012 New Year Python Meme here’s mine
After a long hiatus
Finally after finishing a whole bunch of private “projects”, I’m able to spend some more time again to the various open source projects I’ve been working on until now. I have gathered a whole bunch of new ideas, processed good and bad feedback, experienced that some ideas just don’t work out while others are quite encouraging.
Working with Moncli part2: Creating a simple request
In our previous post we have covered how to create an example plugin which allows you to generate the size of directories as a metric. A plugin on itself doesn’t do that much at all. When a plugin is executed it returns the values of that very moment. Executing a plugin is done by Moncli itself and we want to have control over that as much as possible.
moncli_request
A request is a JSON document which is submitted to the messagebroker on which the Moncli clients are listening for incoming requests. Creating and submitting a request could be done Read More →
Moncli 0.2.5
A new version of Moncli is released:
This release fixes 1 important bug and introduces a new feature:
- Fixed bug which resulted in a growing number of never closing Moncli processes.
- Writing to a logfile is replaced by writing to syslog only.
It is advised to upgrade to this version.
For support please post a message to the mailing list or submit a bug report.
Working with Moncli part1: Creating a plugin
Producing data
In the previous 2 posts we had an introduction about Moncli and we have seen how to consume Moncli data (reports) from the Message broker with Krolyk and process them. Now its time to let Moncli generate some data we can work with. In the next couple of articles we will go through the process of building a new plugin and use it to generate metrics and perform different kind of evaluations.
Creating a simple plugin
One of the goals for Moncli was to create a framework which allows people to easily create their own plugins. Although Moncli already comes with a set of plugins which provide basic system metrics they will not cover all your needs, so we’ll create one as an example.
Read More →