2006-11-26

Color Scheme Generator for Charts

Sometimes one needs to show many data series on one chart. In such cases, usually distinction between those data series is achieved by using different colors for each data set. Most charting libraries leave colors selection to developer. This can be problem when there is need to choose more than six colors which can be easily distinguished by most people. Below you can find Python program which can be helpful.

Program generates palette of colors where each color can be easily distinguished from other colors in its neighborhood. Difference between colors is computed using W3C's formula for color contrast. Minimum contrast (brightness and color), palette size, neighborhood diameter and maximum number of iterations can be easily set in first few lines of program code.

New colors are created using pseudo-random generators and tested for minimum contrast. If contrast test fails, the color is dropped and new one is created. Besides of minimum required contrast, the size of neighborhood has great impact on probability that test will fail. Too big neighborhood diameter could make it impossible to create complete neighborhood and lock the program. To prevent this, there is limit for number of iterations.
#!/usr/bin/python

from random import randint

class Color:
   __c = [0, 0, 0]
   __cB = None
   def __init__(self, r=-1, g=-1, b=-1):
       if r == -1:
           r = randint(1,255)
       if g == -1:
           g = randint(1,255)
       if b == -1:
           b = randint(1,255)
       self.__c = [r, g, b]
   def __getitem__(self, key):
       return self.__c[key]
   def __setitem__(self, key, value):
       self.__c[key] = value
       self.__cB = None
   def getBritness(self):
       if self.__cB:
           return self.__cB
       else:
           self.__cB = ((self.__c[0] * 299) +
                       (self.__c[1] * 587) +
                       (self.__c[2] * 114)) / 1000
           return self.__cB
   def __sub__(self, other):
       return ((max(self.__c[0], other[0]) - min(self.__c[0], other[0])) +
               (max(self[1], other[1]) - min(self[1], other[1])) +
               (max(self[2], other[2]) - min(self[2], other[2])))
   def __str__(self):
       return "%02x%02x%02x" % (self.__c[0], self.__c[1], self.__c[2])

colors = []
num = 25
comparedNum = 5

c = Color()
colors.append(c)
stopper = 100000

minBrit = 35
minColDiff = 100

print "searching...",
for n in range(num)[1:]:
   while stopper:
       newC = Color()
       passed = 0
       for back in range(comparedNum):
           testedNum = n-(back+1)
           if testedNum < 0:
               passed += 1
               continue
           c = colors[testedNum]
           bDiff = abs(c.getBritness() - newC.getBritness())
           cDiff = c - newC
           if  cDiff >= minColDiff and  bDiff > minBrit:
               passed += 1
           else:
               break
       if passed == comparedNum:
           colors.append(newC)
           print "found:", newC,
           break

       stopper -= 1

if not stopper > 0:
   print "searching failed"
else:
   print "finished"


html = """
< head>

< /head>
< body>
""" % (800/n)

python = "colors = ("

for c in colors:
   html += '
\n' % c python += '"%s", ' % c python = python[:-2] + ")" html += "< /body>" out = file("colors.html", "w") print >>out, html print python out.close()

2006-11-21

Synchronizing EMF and GMF Generated Editors

Frequently one does not want graphical representation for objects of all classes of semantic model in a graphical diagram editor, but graphical objects usually have references to non-graphical and there is still need for creating and editing all of them. In that case it would be nice to reuse EMF generated editor for non-graphical elements as a helper for diagram editor, so there would be no need for creation of custom edit forms. Full article is here.

2006-11-16

Regular Expression To Match IPv4 Addresses

I needed a regex pattern to match IPv4 addresses today. I thought that this is quite common thing and it will be easy to find one on the Net. I have found a lot of them indeed. Unfortunately no one was good enough. Most of them was too simple and could match a string which is not IPv4 address. On the other hand, those more specific didn't match some fully legal IPv4 addresses (for example the one from regexlib.com). So I have developed my own:
((2[0-5]{2}|2[0-4]\d|1\d{2}|[1-9]\d|\d)\.){3}(2[0-5]{2}|2[0-4]\d|1\d{2}|[1-9]\d|\d)
(/(3[012]|[12]\d|\d))?
This pattern matches all (I hope) IPv4 addresses in dot-decimal notation (first line) and network blocks in CIDR notation (when you join second line).

2006-11-02

Visualization of Progress in MUDding using Python and RRDTool

I like skill-based, level-less MUDs, so this will be about one of such MUDs, but provided here scripts probably can be adopted to any MUD system or even to any log file. Skill-based MUDs send special message to player when he archives progress in character development. Such message can look like this:
 "You learned a trick of casting snake rune" 
Let's try to write some code which can visualise progress of character development on time axis. As input data we will use log of all mud sessions. Saving mud session to log-file can be configured in most mud clients. First we have to extract progression marks and their essence. This can be easily done using following *nix commands:
grep 'You learned a trick of ' atp_log_0 | \
sort | \
sed -e 's/You learned a trick of casting \([a-zA-Z ]\+\).*/\1/'
As a result we will get something like this:
...
snake rune
stoneskin
stoneskin
stoneskin
summon
summon
...
Now the harder part, we have to count occurrences of the same strings and put them to some database. To see changes in time we have to do this regularly, for example every hour. In this case rrdtool would be very good as database engine. Following Python script will do everything what we need:
#!/usr/bin/python

import os
import rrdtool
from sys import argv
from sys import exit

if len(argv) != 2:
    print "USAGE: %s prepared_log" % argv[0]
    exit(1)

lines = open(argv[1], "r").readlines()

counter = {}

spell = ""
for line in lines:
    line = line[:-1]
    if spell == line:
        counter[line] += 1
    else:
        counter[line] = 1
    spell = line

for spell in counter.keys():
    print spell + ": " + str(counter[spell])
    rrdFileName = spell + ".rrd"
    if not os.path.exists(rrdFileName):
        # create rrd file
        rrdtool.create(rrdFileName, '-s', '3600',
            'DS:data:COUNTER:86400:0:50000',
            'RRA:AVERAGE:0.5:1:5000')
        print "created: " + rrdFileName
    rrdtool.update(rrdFileName, 'N:' + str(counter[spell]))
Now we have to play MUD for few hours and collect data. Finally we can attempt to perform visualisation. Manual generation of image[s] with charts from all .rrd files can be tiring, so we will use following Python script:
#!/usr/bin/python

import os

RRD_EXT = ".rrd"

colors = ("B81818","5CDE00","0089DE",
    "1A1F14","DEC400","486B20","1216A1",
    "432A96","913F64","9A3C17","F4A179","4B7B47",
    "7E6C67","51D9F4","559E7B","4B4D35","D8980E","A32245")

defs = ""
lines = ""
num = 0;
numc = 0;
files = os.listdir(".")
for file in files:
    base,ext = os.path.splitext(file)
    if ext == RRD_EXT:
        color = colors[numc]
        defs += 'DEF:d%d="%s":data:AVERAGE ' % (num, file)
        lines += 'AREA:d%d#%s:"%s":STACK ' % (num, color, base)
        num += 1
        if len(colors) == numc:
            numc = 0;
        else:
            numc += 1

command = 'rrdtool graph test.png --end now --start end-58h ' + \
'--height 400 --width 600 ' + defs + ' ' + lines

k = os.system(command)
if k == 0:
    os.system("eog test.png")
else:
    print "error code: ", k
"eog" is a standard Gnome image viewer. Of course you can put here your favourite one.

2006-10-31

Cisco/Catalyst Vlans on Ports - final solution

With great help from Cisco NMS EMEA TAC, I have finally found solution to map tagged and untagged vlans to physical ports on the Cisco Catalyst switches, without using SNMP Community String Indexing (CSI). With CSI one can select entity/instance of multi-instanced MIBs, but it is quite awkward and not well supported on most NMSs because you have to use separate community strings for every instance. On Catalysts, BRIDGE-MIB is a multi-instanced MIB, instances of which exist separately for every VLAN. Its dot1dBaseTable is the only way to map bridge port numbers to interfaces from ifTable. Some SNMP tables use bridge port numbering to point physical ports, so to bind data from that tables to something more tangible than bridge port number (like interface from IF-MIB or physical port from ENTITY-MIB) one have to scan dot1dBaseTable 3000 times , in case of having 3000 vlans!


I came up against BRIDGE-MIB and CIS in Vlan->Port mapping case because of the port numbering used in vmMembershipSummaryTable (CISCO-VLAN-MEMBERSHIP-MIB). I have been using that table to get access ports (ports with only one untagged vlan) for given vlan. CISCO-VLAN-MEMBERSHIP-MIB has another table - vmMembershipTable - which looked promising, but due to some kind of blackout of my mind, I thought that vmMembershipTable is write only and I have to use that damn vmMembershipSummaryTable. Now I know that vmMembershipTable can be read, is indexed in the the same manner as interfaces in ifTable and has one very useful column: vmVlan.


So for tagged vlans use vlanTrunkPortTable and for untagged - vmMembershipTable. Booth contain links to interfaces in ifTable, which can be easily linked to physical ports in ENTITY-MIB, then.

2006-10-20

Beryl on Xgl

Finally I have Beryl running on my desktop at home. If you want to better feel how it works (this is very dynamic gadget) then check this video. I don't like this wavy animation of window [un]minimization, so I've changed it to something else. Beryl setting manager allows you to change almost everything without pain in the ass. Usually all changes take affect immediately and you don't have to restart window manager, but in random cases it doesn't work.

JGrups on Linux With Multiple Network Interfaces

Few hints for those, who want to run JBoss Cache in clustered mode ([a]synchronous replication/invalidation) on Linux machine with multiple network interface cards (NICs) and using UDP/multicasting in OSI layers 4/3 . Here is situation overview: We need multicast communication through eth1 to other JBoss Cache nodes.
JBoss Cache uses JGroups for network communication. And I'm going to focus on configuration of this service (Cache configuration is very easy). First of all, for device with multiple NICs, you need to specify a bind address in configuration file (of course if use xml file to configure services) in section ClusterConfig/UDP. AFAIK this can't be 0.0.0.0. Probably you can use any address from your NICs, but I would suggest using the one from eth1 (this will be logical but it won't change process of packet generation and sending). Without this parameter service won't start throwing an exception.


Second thing is to specify outgoing NIC. On Linux platform process run by non-root user can't specify source address and outgoing interface for it's packets. The only way to control this is through routing table. Without special entry in routing table for your cache's multicast address, all packets will go outside of machine using DEFAULT route, which in case from above picture will mean eth0. This is not what we want to archive. You can correct it like this:

$ ip r a 228.1.2.3/24 dev eth0
The last thing is packet fragmentation issue. Maybe only I had this problem, but I spend a lot of time trying to solve it. Every replication of bigger amount of data resulted in timeout exception. To get things working I had to set timeout to 120 seconds! Of course that wasn't right solution. Finally I've changed max_xmit_size and frag_size parameters to MTU minus approximate size of all headers. This is about 1400 B.


If you still have problems then check your Ethernet switch connected to eth1. For example if it has IGMP SPAN mode enabled then check if it works correctly. I had problems with it on 3Com SuperStack 3 Switch 3870 with software version 1.01.

2006-10-17

Cisco MIBs horror

Suppose you want to know what vlans are configured/allowed on individual ports in one of recent Cisco Catalyst switches - quite basic information. SNMP seems to be right protocol for this task. Structure of most Catalyst switches can be discovered using ENTITY-MIB and that's good news. In entPhysicalTable you can find ports, modules, chassis and many other parts with defined relationships between them. Now you want to find vlans for discovered ports and this is where scary part starts. Since port can have many vlans and vice versa, such mapping can be done in SNMP using one column table with row index build by combination of two numbers (portID.vlanID). As value of this row could be:
  • 1 meaning native vlan
  • 2 meaning tagged vlan
Lack of port-vlan combination in table would mean that this port isn't present in given vlan. Simple? Yes, but too simple for Cisco engineers.

Cisco found another solution. In order to get via SNMP vlans per port information for Cat2950, Cat2970, Cat4500 and Cat6500 (with recent IOSes) you have to correctly combine following tables:

ifTable from IF-MIB entPhysicalTable from ENTITY-MIB, entAliasMappingTable from ENTITY-MIB, vtpVlanTable from CISCO-VTP-MIB, vlanTrunkPortTable from CISCO-VTP-MIB, vmMembershipSummaryTable from CISCO-VLAN-MEMBERSHIP-MIB, dot1dBasePortTable from BRIDGE-MIB

Some of those tables are 3D, have thousands of columns (one bit per column) and all of them use different, non-continuous indexes. But that would be still to easy. To get mapping from ports in vmMembershipSummaryTable to ports in ifTable you have to query dot1dBasePortTable using separate SNMP community for every vlan. Thank you Cisco!

Does anyone have any other idea how to solve this problem?

2006-10-15

E-mail Posting Test

Currently I try to combine Eclipse Platform (OSGi, EMF, GMF) with Spring Framework (as Beans container and factory) and JBoss Cache (synchronous distributed cache) and many other useful pieces of software (like SNMP4J) in order to build network management application.

All this software is very good. The problem is that Eclipse and JBoss suffer lack of documentation for newly developed stuff while I'm mostly interested in newly developed stuff.

Some measures for today: Execution time analysis for cached/direct access to following SNMP tables: VlanTrunkPortTable(from Cisco MIBS) and IfTable combined with InterfacesTable:

  • direct run time: 1308
  • cached run time: 2684 (empty cache)
  • cached run time: 6 (full cache)

So building and propagating cache takes: 1376 milliseconds. That's not good.

About This Blog

As always my first choice wasn't good. I had to immediately change blog template to completely different. The old one could break your eyes. I like new things, so I've converted my blog to http://beta.blogger.com. It's one way process, but I have nothing to loose. Let's see how this all works. In the mean time I'll worm up my tea in microwave. This blog will be mostly about me versus resistance of some cutting edge software development technologies (usually open-source Java and Python frameworks, libraries and tools).

Let's start

World comes to the an end. I have a blog.