Jason's Site

Today I have had a new Nagios plug-in I’ve been working on published. The plug-in is check_asterisk_ami and is capable of checking asterisk VOIP server for number of active calls/channels and also the number of SIP & IAX Peers in use.

The plug-in can be found here:

http://exchange.nagios.org/directory/Plugins/Telephony/Asterisk/check_asterisk_ami/details

and any feedback is welcome.

I already have a few people using this with success, Inspiration for this plug-in came from our asterisk system being hacked and calls routed through our server; fortunately I managed to catch it before it got too bad, and this is now on 24hour SMS support.

Over the years at work I’ve toyed with a few monitoring solutions. It all started with a few issues on the network and Bash scripts to check on them, and notify me via XMPP when there were issues, this grew into something somewhat unmanageable, and eventually I turned to Nagios. I went through and configured a few hosts and services, and wrote an XMPP plug-in (that I’m still yet to publish – and for that matter, finish!) and an SMS plug-in that worked with my VOIP Provider at the office. so now I get full notifications on XMPP and SMS for the more critical systems.

Next Management want to see some nice graphs, so I threw something up with RRD and it was OK, I looked at a few graphing solutions for Nagios, including Cacti, nagiosgrapher, and nagiosgraph, I’ve also looked at swapping out the Nagios install for something else, of which I have tried Groundwork Open Source, Centreon, and more recently thanks to a recommendation, Opsview.

All of these tools appear very good, and are all built around Nagios, but each have their own issues, but also advantages. I’m going to go through a few of them here.

NAGIOS

Ok, First up, Pure Nagios, this is configured from text files, and the way you arrange them is really up to you, so for me I had directories of “Servers”, “Development”, “Switches”, etc. within these I had a file for each host, that contained the host information and services associated with it. for me this made it very easy to add a new host as I just created a new file for the host, and added the information. There’s no Web configuration on Nagios (though there are 3rd party ones that I didn’t really look into), and there’s no graphing, without 3rd party apps to do that too, though, there’s nothing wrong with doing one yourself with RRD and linking to the graphs from within Nagios.

Groundwork Open Source

Next I tried Groundwork, This is a very good application, and has a very nice web interface for configuring hosts, including a “Auto Discovery” tool that will go and find all hosts within a range of IP address’s, Limitations of this are that it only does IPv4 address’s, though at least it gives you a start point on your network. Adding hosts and services on this is manually also very easy, it sets a few graphs up for you, and creating more or customizing the graphs is also very easy. Unfortunately, it seems the Groundwork team are not doing any work on their Open-Source version anymore, which is a little disheartening as they’re using Open-Source software under the hood to do most of it, it’s also very heavily orientated with Java, and requires a bit of CPU grunt to do a lot of the processing.

Centreon

Next I tried Centreon, again this is a very good tool, with very nice web GUI for configuring hosts, it lacks an Auto-Discovery, but it does allow you to import your configuration from Nagios, and for me that worked perfectly, I didn’t have to configure much outside of that other than the graphing data. Adding new hosts and services is very easy with the web GUI, and Nagios is still accessible along-side Centreon so you can still access the views that Nagios has should you wish to. The downside I have found with Centreon is the graphing data. For the most part, ping times, etc it’s absolutely fine, however, when it comes to network traffic, that’s a whole different ball-game, Here’s what it does with Network Statistics.

As you can see, that’s not the most useful data one could get. this is due to settings within RRD tool, and I have not managed to yet find out a way to change this within Centreon, Keep in mind, I don’t want to change scripts, this should be do able from within the web interface. MRTG is very good at these, and I don’t mind plugging the MRTG graphs into this, however, it would be nice if there was a single point to get all of this data. The graphs that are in-place also appear to show far more information than they need to, and again I’ve not yet worked out how to solve this. Everything else works well, but the graphing seems to feel somewhat unfinished.

Opsview

The last few days I’ve been playing with Opsview thanks to a recommendation from a fellow geek. The first stumble I hit was the fact that in Nagios I’m monitoring upwards of 100 hosts and around 1500 services including CPU Load, Memory usage, HTTP Response times, etc, each network service is on both IPv4 and IPv6, and the entire config can be a bit of a nightmare. Opsview does have a tool to import Nagios configurations, however, this was not easy, it complained about custom plug-ins that I have written and it didn’t know about, easy fix for that, just copy them in-place, it complained about a few other things, too, and eventually I bailed out of doing this and just started to add hosts manually. Opsview have taken a very different approach to your plug-ins or commands, instead of having a service, that points to a command, that points to the actual script, you have a service that points to the script with the arguments assigned, well, I use things like $USER7$ for my SNMP Community, could I work out where you can set these? no. it also means you can’t have say a single command with 2 services assigned for things like “local-ping” and “long-distance-ping”, though you can still do this, it’s a different way of doing it, and this is what broke a lot of my Nagios importing, and why I took the Manual route. It does make sense the way they’ve chosen to do it, it’s just a knot in the head when you’re used to the other way. As for the Graphing, it plugs into MRTG for the network interface stats, which is good, though currently mine is saying “No Data” – I’ll look into this at some point, I know how to configure MRTG, so it might involve a small amount of tinkering under the hood, the rest of the graphs are great, clean and tidy, only display the information that you want, and does what it says on the tin. Other than the learning curve of differences and things that are not quite working out of the box (MRTG) it’s looking good. I’ll stick with it for a while and see if I can make sense of the broken bits. I also need to import my XMPP and SMS notification scripts, this should be fun, as the notification system is rather different too, I’ll have to do some working out on this.

Conclusion

The conclusion I’ve come up with, is Nagios is very very good at monitoring your network, it does exactly what you tell it to do, but if you want easier configurations, graphing, etc, there’s a lot of options, and I’m still yet to find one that ticks all the boxes, Hopefully once I’ve configured Opsview a little more it will tick those missing boxes. Groundwork did in-fact tick all the boxes, but the fact that they’re not publishing the Open-Source version anymore bothers me. I understand companies have to make money, and I’m happy to support them, but don’t call yourself “Open-Source” when you’re not.

Hopefully with Opsview, other than the initial configuration of devices, the search will be over, Maybe I’m just attacking it wrong and should have Nagios for the monitoring and something else for the graphing (Cacti?) and leave it be, let each do their own job, We’ll see when I’ve played with Opsview a little more.

I really really hate RPM Based systems, I cringe every time I step near one. Granted, no where near as much as I do every time I use a windows system, but it pains me more and more every time. First of all, I feel for anyone coming from windows that thinks they wish to try Linux, and decides “Redhat is the way to go” because that’s what business’s use. it’s wrong, Please don’t do it, pick something like ubuntu, mint, or almost anything based on Debian (even Debian itself is quite friendly in my opinion). Business’s use Redhat because there’s a support contract and they can get bugs fixed fast, that’s the only reason I can see.

So, on to my rant about RPM systems, why do I hate them? Well, I run monitoring software called Nagios, this has a remote package called NRPE, this allows me to run scripts on remote systems for monitoring and works very well. This package is found in the repository called “EPEL” (Extra Packages for Enterprise Linux) hosted by the folks at Fedora.

I am unable to update my system because RPM system is broken. I commonly have an issue with installing something like webmin on a fresh Debian install, it would be nice if it was a little better, but it’s ok, I do the following:

dpkg -i install webmin*.deb

This results in an error that I do not have packages installed that it requires, no problem, the following command is what I want (and it tells me!)

apt-get install -f

This downloads and installs the packages required, and then installs my webmin package, great.

Now, lets see about RHEL, we have YUM, ok, I want to update my system, so I run “yum update” this chugs away and eventually comes back with:

Error: Missing Dependency: nagios-plugins = 1.4.15-2.el5 is needed by package nagios-plugins-disk
Error: Missing Dependency: nagios-plugins = 1.4.15-2.el5 is needed by package nagios-plugins-load
Error: Missing Dependency: nagios-plugins = 1.4.15-2.el5 is needed by package nagios-plugins-users
Error: Missing Dependency: nagios-plugins = 1.4.15-2.el5 is needed by package nagios-plugins-procs

Here, I grumble. but ok, quick search for nagios-plugins 1.4.15-2.el5 on google and I find the package, but interestingly the package I downloaded was from the EPEL repository, so, it’s there! why did it not find it? Anyway, ignoring that issue, I try installing it, I alerady have an older nagios-plugins installed, so I run the following:

rpm -Uvh nagios-plugins-1.4.15-2.el5.x86_64.rpm

This results in the following:

nagios-common is needed by nagios-plugins-1.4.15-2.el5.x86_64

Was it not by the older package? or does it mean I have an older version? who knows. Anyway, I run “yum install nagios-common” and that’s now installed, from EPEL, great. So we try rpm -Uvh nagios-plugins-1.4.15-2.el5.x86_64.rpm again, and this time I get the follwing:

error: Failed dependencies:
nagios-plugins = 1.4.13-11.el5 is needed by (installed) nagios-plugins-disk-1.4.13-11.el5.x86_64
nagios-plugins = 1.4.13-11.el5 is needed by (installed) nagios-plugins-load-1.4.13-11.el5.x86_64
nagios-plugins = 1.4.13-11.el5 is needed by (installed) nagios-plugins-users-1.4.13-11.el5.x86_64
nagios-plugins = 1.4.13-11.el5 is needed by (installed) nagios-plugins-procs-1.4.13-11.el5.x86_64

But, wait I was trying to update those packages, it just couldn’t find the one I’m trying to install here. Despite it being in the same place in the repositories! By this time I can feel my blood pressure starting to rise, which is never a good thing. I also keep saying to myself “on Debian, I just run apt-get install … and it does it” anyway, I have to run RHEL for work purposes, all my servers at work are actually running Debian, however, for building we run RHEL to be “compatible” with our customers. Except, we can’t be because it won’t update! So now I have to download the other packages manually, and install them, manually, because it can’t find them.

I’m starting to get annoyed with this, and if it was my own system that I did not need for work, it would have been changed to Debian long ago.

All I can say, is that I feel for anyone that “Tries” Linux and goes for an RPM solution. This is not the first time I have had this issue, and I have even had the issue within the main repositories with packages. It’s annoying, It’s frustrating, and I can’t get rid of it. I just HATE it!

Given a choice of package managers, I would opt for Portage followed by Apt, both are incredibly good and I have very few issues with them.

Well, apparently 2GB isn’t enough memory anymore, with Windows wanting at least 4GB to run well, linux is still happy with 128MB, but with compiz you really want 4GB again.

When it comes to servers however, it’s a whole new ballpark. The software written by the company I work for deals with incredibly large computational arrays requiring a fair amount if memory at times. While I write this I am waiting for a pre-boot memory check to complete do I can get on and check the system is working, it’s taking a while, and here’s a extensor as to why…

That’s 96GB Memory in this system, What’s even more interesting about a system with this much memory, is that it will be used. and this is not a virtualised system, this is a single OS (though dual-boot) and is used for testing…. oh yeah, not to mention the fact that it has 2 quad-core CPU’s…. this system really is a beast. and I would quite like it to be my desktop machine, though, I might have to invest in some ear protectors, as the cooling is rather loud :-)

Right, I haven’t posted for a while mostly because I’ve been busy with work, social life and moving house, however I’m completely fedup with the idiocy from some ISP’s, media, and people.

IANA have handed out the last /8′s of IPv4. and it couldn’t come sooner in my opinion. I’ve been using IPv6 for many years now, have IPv6 when I’m at home, this is native. I have it at work, this is currently tunnelled, but only because of Mikrotik ROS not supporting native IPv6 over PPPoE on 4.x builds. I even have it on my laptop when I’m sat in a pub. I have helped friends and family get IPv6, and everything on it just works.

Why is it, then, that ISP’s “have no plans” to implement IPv6 still. stop complaining that IPv4′s have run out and you don’t know what to do, IPv6 has been around since 1996, it works well and I have systems that _only_ have IPv6 address’s. Don’t use “there’s no routers on the market that do it” because there is. Billion have one, ZyXEL have one, Comtrend have one. they’re all priced at or under £50. what’s not consumer about them? if you want more, then OpenWRT and DD-WRT DO have IPv6 support, and can be flashed onto many routers, if you want to put the boat out a little, Cisco support IPv6, as do Routerboard.

it is of my opinion that there is absolutely no excuse for not offering IPv6 to consumers, if the consumers don’t want to use it, that’s up to them. Ipv4 is LEGACY it is the OLD Internet. IPv6 is the CURRENTLY used Internet Protocol. so, come-on people, pull your fingers out and get it done. I will not be buying any devices that do not support IPv6 anymore. this includes Routers, VOIP Phones, Printers and even Set-Top Boxes – yes, they have ethernet for their “Interactive” services. if they don’t do IPv6, they will not be used.

For now, back to work…

Well, one of the office machines has just has a new hard disk installed, so clean install of Windows XP x64, and start running updates. you now get this “Browser choice” thing on your desktop (which is frustrating when you’ve been using a machine for years and it suddenly pops up, but anyway…) for this system, we’ll be using IE8, well. That’s what I hoped anyway. However, Microsoft appear to fail to accept that Windows XP x64 Professional is actually a windows operating system.

Here’s a screenshot of what I got when I tried to put IE8 on the system:

If Microsoft don’t class Windows XP x64 as a Windows Operating system, should everyone else do the same?
well, I guess we’ll use a standards-compliant browser instead on this system… now, where can I get one of those :-)

A few months ago, I asked an internet provider,Timico, with which I have a co-location server with if they would supply IPv6 on their network. They told me there were currently no plans to implement it. OK, fine, I’ll use another method, Tunnelling. This is fine, there was a way around the problem, and everything worked fine.

Today, however there was a BBC Article explaining that IPv4 was going to run out soon. well, this isn’t news to me, but reading further into the article, there was news. From the BBC Article: “Unless more ISPs and others start to adopt IPv6 those delays could start to hit general web browsing, fears Mr Davies.” Now. Mr. Davies is Trefor Davies, the Technical Director for Timico. Hold on a moment. Timico told me they had no plans for IPv6. so what’s he talking about?

I have done a small amount of digging, and on Trefor’s personal blog, on the 25th February (only a few weeks after I was told that there were no plans to implement) he said that IPv6 is almost running on the Timico core network. So, why is it not available to their customers, and why have they delayed it this much. I have been using IPv6 for some time now, long before my blog stream started. To start with it was Tunnelled and now I have Native at home, though I’m still Tunnelled at the office, mostly due to some firewall restrictions.

Anyway, My problem with this is that Trefor is quoted saying that people need to move to IPv6, well, they do, but why should he be saying this when the ISP that he is a director of, does not offer it to their customers. My Chosen ISP at home (AAISP) does offer IPv6, they also offer it tunnelled or native, and have done so for a very long time. I have emailed Trefor, and Hopefully he can let in some light about his comments and why he thinks he should be able to say what people should be using when he doesn’t offer what he says they should.

possible updates depending on Trefors response. :-)

I am no longer a “Subscriber” to the internet. I am instead a “Communications Provider” but, what does this mean?

Well, it means I provide some form of Communications to other people at no cost to them and with no contract. In my case I offer Open Public Wifi access for free. It has users using it, and all woks well. I have also made my ISP aware that I am a Communications Provider (infact, it was their idea). There are a number of reasons to do this, the biggest is just to prove how stupid laws can be. The New DEB pretty much makes you guilty of downloading before you’ve even thought about doing it but only as a “Subscriber” (which I now am not). Now, I haven’t downloaded music or video’s, etc for many years. and neither have I bought any. The thought of going to the shop to buy a disk with music on when I’ve not heard any of it and don’t know if I will like it is ludicrous. 5 years ago I downloaded music, if I liked it I went and bought it, if I didn’t like it I quietly discarded it, I don’t really see anything wrong with this. would you go and buy a car without test driving it? would you buy a house without viewing it? no, you wouldn’t. Anyway, I don’t agree with a “guilty until proven innocent” and I dislike being declared “Guilty” on an assumption. I have a network with 10 PC’s why do you think it’s me that’s downloading? could be any of the other people. It is for this reason that I have moved to be a Communications Provider.

I don’t intend on doing any downloading, and maybe I’m making myself automatically guilty by putting myself down as a Communications Provider, but they’re going to have to really prove that *I* was doing the downloading. Given I host public access wifi, good luck with that one. What I’m actually doing is sticking up my middle finger at Labour, and showing them why the law doesn’t work, and why they shouldn’t rush these things through just because there’s an election going on. There’s more important things to worry about.

I think what annoys me the most about this whole thing is the Music Industry, By which I mean the record labels, think they’re losing money, well the artists I saw complaining about people downloading had such shite music I wouldn’t even use it to wipe the dogs arse with. The artists that appears to agree that downloading music could be OK if you could some how monitor it, or limit it, and that it was “The way of the future” are the ones that have good music. The pattern, ah, that’s right, if you write good music people will pay you anyway, if you’re shit people wont. I’m betting everyone that downloaded the music of the complaining artist probably quickly discarded it straight after.

Anyway, back to the point, there are ways around the DEB, and I’m only mentioning one of them here. I’m not the only one doing this, and some ISP’s are advising people do. keep in-mind that an ISP isn’t going to want to send you letters telling you they’re going to disconnect your internet, this would be complete corporate suicide. I advise anyone that can to put themselves down as a Communications Provider, and show the government what’s wrong with their system. Note you can be a Communications Provider even if you just offer the internet to anyone in your own house that is not the “subscriber”, you don’t have to open a network up for public access.

Ok, so I’ve been slacking again. Actually, I’ve been busy. Recently our VOIP server was receiving a SIP registration attack. the source IP was one from Amazons EC2 Network. having blocked them on the firewall at my end-point, the attack continued to try and send data to my system. I followed protocol, and sent an Abuse report to Amazon EC2. The abuse report contained a graph of the on-going data, seen here:

It also contained a cut-down of the logs, showing which IP from their network was attacking our system and an explanation of what was happening. This was also CC’d to my ISP, I don’t normally CC them in on abuse reports, as when sending them for SSH attacks there’s alot of them, however, this isn’t the first time it’s happened from the Amazon network, and the data usage was incredibly large. and persisted even after blocking on the firewall. Fortunately for me, My ISP (Andrews and Arnold) give me a lot of control over my lines, including routing tables specific to IP’s that I have allocated and in this instance it took un-routing the subnet from my lines before the traffic stopped (though, according to someone at my ISP, the attack continued for some hours after un-routing the subnet).

Anyway, I received a response from Amazon today, they quoted the IP Address of my server that the attack was going to and had this to say:

Thank you for submitting your abuse report. There was no single customer using the source IP address(es) during the time you provided. This may be due to the fact that we do not own the IP address(es), the time or time zone you provided was incorrect, or there were multiple customers with instances running during the time and IP address(es) you specified. You may try re-submitting your report with a different time if you wish.

What that reads to me is “I didn’t actually look that closely to the logs and ignored most of the information that told me the time-zone in which your network is using, I also don’t know how to read logs, and assumed the IP address was a different one from what you had quoted” I have responded, telling them of their mistake. I have told them again which time-zone the logs are in, and I have told them again which IP Address they should be looking at in the logs. Today, the entire Amazon EC2 network has blocked access to my VOIP Server. What this means is that if there is anyone using Amazon EC2 legitimately for a VOIP server, they can not directly call our numbers.  I doubt this actually happens very often anyway but the least I expect from a company like Amazon when sending in an abuse report is that they actually give it to someone that has more than a single brain cell and doesn’t know what a computer is.

We’ll see what happens with this, but I’m not hopeful, and will never recommend Amazons EC2 service to anyone.

J

Over the past few months myself along with some other residents of Basingstoke, The council, both Basingstoke & Dean and Hampshire County and our M.P. Maria Miller have been in meetings and discussions with some rather large telecomms companies regarding broadband speeds in and around Basingstoke. (More so around, really). This came about with BT releasing that they were going to deploy FTTC to Basingstoke, which has turned out that it’s actually 30% of Basingstoke. A Meeting with them went “OK” it wasn’t great, but it wasn’t bad, we offended them on a number of points, but showed some good statistics from door knocks about where the demand was. Things like 95% of people have internet and only 19% of those were actually happy with the service, Also that 70% of people would change providers and 60% would pay more to get a faster service.

All interesting information, so you’d think, but BT really didn’t appear all that interested, we have given them dates to get back to us and maybe they’ll surprise us with some good news. However, today we had a meeting with Virgin Media. to start with, it was much more friendly, with the occasional laugh coming from our meeting room. There was very positive feedback about the deployment of Virgin Cable in the area’s that BT have blindly ignored (that is most area’s outside of town).

Currently the eHampshire website only has information regarding FTTC, as this is a new deployment. If, and we hope they do, Virgin Media start a new deployment in Basingstoke, the information will also be in-putted into the eHampshire website.but I advice anyone in Baingstoke that hasn’t already to register their details on the site at http://www.ehampshire.org/ so that we can easily see the area’s in which deployments should take place.

I’m personally looking forward to a deployment from Virgin Media, as this will really show BT that they don’t have the better game, I personally believe with the bad press BT have currently got in and around Basingstoke, a lot of people will be wanting to flee away from them. Come-on Virgin, do us all a favour!

Jason