A high-end router manufacturer was looking for an efficient "performance management solution" that would provide them with meaningful reports of the flow processing that was performed by their NE. Since the NE was capable of supporting millions of flows per minute, it meant that gigabytes of data were going to be generated every few minutes. Designing a performance management module for their EMS posed quite a few challenges some of which are outlined below:
- In this situation, since gigabytes of data was transferred every few minutes, the protocol used for performance management had to have very low overhead.
- Number of NE's that could be managed in parallel by the EMS was limited because of the "amount of processing" that has to be done by the EMS to support a single NE.
- "Generating desired reports from the vast amount of data collected" required efficient data management.
Initial attempts were made to re-use the SNMP based data collection engine; however it became apparent that SNMP couldn't be used in this case because converting each of the million records to SNMP is a "time consuming process" which could end up choking the system. Also, if SNMP was used, the EMS will end up lagging behind the NE when it comes to processing the performance statistics.
A different approach, as explained below, was taken using FTP.
- Raw performance data was dumped into a file - a simple text file - with comma separated values.
- "File compression tools" were used to reduce the size of the text file - reduced the transfer time by 90%!
- "Compressed raw data file" was transferred to the EMS via FTP (or) SCP - most suited approach for transferring voluminous information.
- File was decompressed by the EMS and the raw data was recovered.
The above approach allowed the data to be transferred from the NE to the EMS in less than a couple of minutes (depending on the size) which enabled the EMS to complete the processing of data within a few minutes; this in turn allowed data to be collected more frequently.
In order to generate desired reports, it was essential to have all the flow records in the database. Adding millions of records to the database by reading one line at a time from the file and inserting it into the database is very time consuming and not very efficient. Therefore bulk data insertion was performed by using the tools that were supported by the databases. This allowed to quickly store all the records in the database and once all the flow data was available in the database, "generating the desired reports" was easily done using NetMan reporting tools.