Monday, October 22, 2007

VOIP Monitoring Project

Initially we deployed VOIP as a interoffice communication savings. With the presence in India we have, and its growth it has some huge savings attached to it. We then decided to modernize the call centers that we have deployed. That meant we were deploying Cisco solutions that were very network centric to function using call manager, and the rest of the suite.

In the next phase we realized that we were spending a huge amount of money on conference calling services. We use a LOT from every vendor you can think of. We even max out some of our vendors when we run conference calls for earnings periods. We started to implement Cisco Meetingplace. The product has been working well, and we are monitoring the same way we monitor any set of cisco devices, and a few windows machines.

One of our CEOs was on a conference call and it started to have voice quality issues. That they brought up the fact that we are missing a whole bunch of ways to debug and fix issues relating to voice and overall network quality. We came up with the following criteria, and set of vendors we are looking at:


 

Metrics

These metrics must apply across anywhere that KPI is used : Jitter Delay, R-Factor, MOS, Cocec Transcoding, QOS (load, cpu, memory)

Monitoring and Packet Capture

  

Network General

NetQOS

Wildpackets

General

Must Provide 7 days at 4gb/s of buffered capture of full packet data

  

  

  

Monitoring

Provide network and service monitoring based on metrics defined above

  

  

  

Operations

Provide Real-time network packet capture to monitor and troubleshoot problems quickly

  

  

  

Design and Planning

Ability to conduct VOIP quality testing and simulate traffic to measure and fix potential problems

  

  

  

Design and Planning

Provide MOS scores as well as R-Factor values for each call.

  

  

  

Operations

Analyze and track against PKIs to ensure voice quality and performance

  

  

  

Operations

Ability to provide per call analysis and support multiple signaling protocols for voice

  

  

  

Operations

Analyze packet level details of RTP, RTCP, MGCP, SIP, H323 streams

  

  

  

Operations

Evaluates packet delay variations, packet loss, jitter,

  

  

  

Operations

Provide Application Response Time (ART) analysis

  

  

  

Operations

Provide full 7-layer decodes, alarms, triggers

  

  

  

Design and Planning

Provide call playback or replay of captured data to hear and see VOIP impairments

  

  

  

Operations

Ability to extend monitoring and troubleshooting at remote locations

  

  

  

Operations

Perform analysis of mission critical applications both in real-time and post capture.

  

  

  

Operations

Provide Efficient per-call filtering to expedite problem resolution and minimizes packet transfer

  

  

  

Operations

Provide clear identification marking of VOIP calls, video conference calls and other services

  

  

  

Operations

Utilize standard SNMP and RMON probes. Ability to monitor relevant OIDs

  

  

  

Operations

Capable of importing or exporting file to standard formats

  

  

  

Monitoring

Monitor CISCO and other telephony devices from AVAYA or NORTEL.

  

  

  

Operations

Analyze call setup protocol of SCCP

  

  

  

Design and Planning

Measure network performance of RTP, MGCP, H323.

  

  

  

Display Web Reporting \ Dashboard view of information and events

  

  

  

  

Executive

Reports generated in HTTP web format for easy viewing. No use of plug-ins or Java on the client.

  

  

  

Executive

Reports can be formatted or imported into a dashboard style display

  

  

  

Executive

Provide Canned and Custom Reports

  

  

  

Executive

Historical data archived or saved for 1 year, and exportable via automated manner to standard DB platforms, MSSQL, Oracle, or MySQL

  

  

  

Executive

Able to import or export files from third party or freeware monitor tools to create reports

  

  

  

Executive

Report on Key performance indicators and measurements

  

  

  

Executive

Provide measurement of SLA levels

  

  

  

  

  

  

  

  

Traffic Aggregation

We are having somewhat of an issue getting visibility properly across the network traffic. Netflow is useful, but limited in the data it can give you. There are more and more products which really need to see the traffic in order to work properly. I came up with a business case on why we need to do this back in June.

  1. Coradiant monitoring of the traffic

    We seem to miss pockets, and we cannot get a proper view of the backend web server response and quality by looking at data on the edge of the datacenter.

  2. VOIP monitoring

    This will be the topic of my next post

  3. Sniffing and debugging issues
  4. IDS deployment

    We currently have scattered IDS deployments, which we need to centralize.

It's finally gotten to the point where we want to implement it. One of our network guys came up with a nice design using the Apcon switches, and cisco switches to help with multiplexing the data more easily. This will allow us to implement monitoring and spanning during the day without reconfiguring production switches. Its something we really need and will help immensely.

Homegrown Tools

Due to recent organizational changes, I am working closely with other parts of my business. At the same time we are buying another large company. We now have 3 strategies involved in the company that I find very interesting and conflicting at the same time. I know that I feel open for change regardless of direction. It just needs to make sense financially and have a solid technological foundation. It needs to meet the needs of the consumers of our tools internally and externally.

One of our divisions moved off IBM tools for server monitoring, to modified open source tools. The products are cost effective in terms of deployment to many machines, but they need full time developers on staff. My team uses mostly HP OV products on the systems, but we use a lot of other tools for other areas. We have a very good deal from HP on the agent pricing. The company we are buying is a big IBM shop, all they use is IBM. This means we have 3 camps:

  1. Open/Free
  2. Closed/Diverse
  3. Closed/Single Vendor

Let's see which will win over the next couple years J

Appliances and Upgrades

We have a bunch of devices which are "appliances" the vendor has slowly refreshed the hardware. Our devices are 2.5 years old, and the new devices came out earlier this year. Now the new revisions of the software are not going to have all of the features on the "older" appliances. The machines are just older CPUs and less memory.

They want to charge me something, I haven't seen the quote yet, to upgrade the appliances. I think they will charge me more than just the cost of the new hardware. I'm kind of upset about the strategy to get some sales for the end of the year…. We still pay maintenance, which should include all of the upgrades we need…

Not nice!