Charting the web with Cewolf/JFreeChart - Producing Time Series plots

I recently had an opportunity to use the Cewolf 1.0 at work for some Time Series plots (a variant of the XY Chart, shown in the figure above, where the X-axis is for time values). This blog is about Cewolf and how to create time series plots with it.

Cewolf is a JSP tag library which uses JFreeChart for rendering the charts. It comes with a controller servlet which is used for interpreting the parameters passed through the JSP tag and accordingly generating the chart image in-memory (no files created on the file system of the server) and embeds the image as tag in the HTML output to the client response stream.

IMO, Cewolf/JFreeChart is the best free charting package for a web application required to draw charts and being developed in Java EE. It supports several different types of charts and one of them was the Time Series plots. Here is some code which can produce a simple time series plot (using cewolf).

1. To install Cewolf you just need to copy the jars from its lib/ path (which includes the JFreeChart jar too) to WEB-INF/lib of your web application.

2. We need to write a data producer which gets the data set (in {time, value} pairs for the time series plot). A typical time series data producer is given below:

//~--- non-JDK imports --------------------------------------------------------

import de.laures.cewolf.DatasetProduceException;
import de.laures.cewolf.DatasetProducer;

import org.jfree.data.time.Minute;
import org.jfree.data.time.TimeSeries;
import org.jfree.data.time.TimeSeriesCollection;

//~--- JDK imports ------------------------------------------------------------

import java.io.Serializable;

import java.util.Date;
import java.util.Map;

/**
 * A sample data producer for the time series plot.
*/
public class MyDataProducer implements DatasetProducer, Serializable
{
    public MyDataProducer()
    {
    }

    public Object produceDataset(Map map) throws DatasetProduceException
    {
        /*
         *   To this time series collection we can add more than one time series
         *   where each time series will be represented by its own line on a
         *   plot.
         */
        TimeSeriesCollection ts = new TimeSeriesCollection();

        try {
            String[] allSeries = { "series1", "series2" };

            // Loop through all series and add the data to series and series to
            // timeseries collection (ts).
            for (int i = 0; i < allSeries.length; i++) {

                // Get data for series from some kind of datasource
                MyDataSet[] myDataSet = GetDataForSeries(allSeries[i]);
                TimeSeries  mySeries  = new TimeSeries("My Data Series " + i, Minute.class);

                // Add data to series
                for (MyDataSet data : myDataSet) {
                    mySeries.add(new Minute(new Date(data.getTimestamp().getTime())), data.getYValue());
                }

                // Add the series to the collection
                ts.addSeries(mySeries);
            }
        } catch (Exception e) {
            e.printStackTrace();

            throw new DatasetProduceException();
        }

        return ts;
    }

    public boolean hasExpired(Map map, Date date)
    {
        return false;
    }

    public String getProducerId()
    {
        return "My Data Producer";
    }

    private MyDataSet[] GetDataForSeries(String string)
    {
        // Get data from DB or some data source
        // return an array of time/value pairs (for instance, as an array
        // of MyDataSet instances.       
    }
}

MyDataSet class is:

import java.sql.Timestamp;

/** A sample time/value pair data. An array/list of this type will constitute
 * the data set for the plot.
*/
public class MyDataSet
{
    private Timestamp timestamp; // You can use other date/time types in Java SE here.
    private double    yValue;

    public MyDataSet()
    {
    }

    public Timestamp getTimestamp() {
        return timestamp;
    }

    public void setTimestamp(Timestamp timestamp) {
        this.timestamp = timestamp;
    }

    public double getYValue() {
        return yValue;
    }

    public void setYValue(double yValue) {
        this.yValue = yValue;
    }
}

In the JSP page you include the chart now:

<jsp:usebean id="myPlotData" class="com.mycompany.MyDataProducer">

<cewolf:chart
    id="MyChart"
    type="timeseries"
    title="My Plot Title"
    xaxislabel="Time"
    yaxislabel="My Data Value">
    <cewolf:data>
    <cewolf:producer id="myPlotData" usecache="false">
    </cewolf:data>
</cewolf:chart>
<cewolf:img chartid="MyChart" renderer="/cewolf" width="1000" height="400">

Lastly, one needs to configure the CewolfServlet in the web.xml:

<servlet>
   <servlet-name>CewolfServlet</servlet-name>
   <servlet-class>de.laures.cewolf.CewolfRenderer</servlet-class>
</servlet>

<servlet-mapping>
   <servlet-name>CewolfServlet</servlet-name>
 <url-pattern>/cewolf/*</url-pattern>
</servlet-mapping>

The generated charts on tomcat does require one to increase the JVM heap size to at least 256MB from the default 64MB.

The pros of using Cewolf/JFreeChart in a Java EE web application:
1. Compared to the other open source packages, JFreeChart happens to be the best in terms of the look and feel of the plots and the ease of use of the API.
2. Cewolf contributes to the glory by making the JFreeChart available as JSP tag library. And as far as i know, there isn't any other better option for plotting in the open source Java world.

The duo of Cewolf/JFreeChart are lacking in a few important features:
1. AJAX support for real-time plots. So if we want the server to be able to asynchronously (or by the virtue of some background polling from client) refresh the chart in (near) real-time then its not something supported today in Cewolf/JFreeChart. If this feature is required then JViews Charts or Chart Director are the two commercial offerings that i know of that can do AJAX based live charts.
2. Zoom and Pan interactions are not easily supported by the API.

Working with JMaki

I recently had an opportunity to use some of the JMaki UI components and it took me some googling to figure out how to pass data dynamically (which is what most of the time you will want and unfortunately all examples use some static data in the JSON format) to the UI components. JMaki's integration with Netbeans makes it really simple to have those nice Web UI components working for you in a jiffy (like grid, tree, menu, captcha, autocomplete etc). Though i am a big fan of DWR (having used the reverse ajax in DWR 2.0 for an event browser application to show events in real-time) for Ajax support in my work, i did like the Ajax-enabled UI Components that come with JMaki. Another nice thing about JMaki is, it provides a common data model for multiple implementations of a certain UI component. For example, you have a Yahoo UI Tree and a dojo toolkit tree component. Since JMaki provides the abstraction by keeping the data models same for both these tree components, so we have the option to switch between these implementations with almost no change to code.

Now going back to the point (the reason i am writing this post after all) ... the JMaki components accept data dynamically in JSON format and though one can create the JSON format string to pass as value attributes to the widgets, it becomes cumbersome for widgets like trees or grid to escape the quotes and construct the strings. To make our lives easy in constructing the JSON formatted dynamic data, JMaki comes bundled with org.json.* classes (JSONObject, JSONArray etc) using which one can create the data to pass in an elegant and maintainable way. You will need to convert the JSONObject or JSONArray types to their Object literal form using the following code which Greg Murray (JMaki project manager) released in reply to one post on JMaki users forum:

    /**
     * Converts a JSON Object to an Object Literal
     *
     *
     * @param jo
     * @param buff
     *
     * @return
     *
     * @throws JSONException
     */
    public static String jsonToObjectLiteral(JSONObject jo, StringBuffer buff)
            throws JSONException
    {
        if (buff == null) {
            buff = new StringBuffer("{");
        } else {
            buff.append("{");
        }

        JSONArray names = jo.names();

        for (int l = 0; (names != null) && (l < names.length()); l++) {
            String key   = names.getString(l);
            String value = null;

            if (jo.optJSONObject(key) != null) {
                value = key + ":";
                buff.append(value);
                jsonToObjectLiteral(jo.optJSONObject(key), buff);
            } else if (jo.optJSONArray(key) != null) {
                value = key + ":";
                buff.append(value);
                jsonArrayToString(jo.optJSONArray(key), buff);
            } else if (jo.optLong(key, -1) != -1) {
                value = key + ":" + jo.get(key) + "";
                buff.append(value);
            } else if (jo.optDouble(key, -1) != -1) {
                value = key + ":" + jo.get(key) + "";
                buff.append(value);
            } else if (jo.opt(key) != null) {
                Object obj = jo.opt(key);

                if (obj instanceof Boolean) {
                    value = key + ":" + jo.getBoolean(key) + "";
                } else {
                    value = key + ":" + "'" + jo.get(key) + "'";
                }

                buff.append(value);
            }

            if (l < names.length() - 1) {
                buff.append(",");
            }
        }

        buff.append("}");

        return buff.toString();
    }

    /**
     * Converts a json array to string.
     *
     *
     * @param ja
     * @param buff
     *
     * @return
     *
     * @throws JSONException
     */
    public static String jsonArrayToString(JSONArray ja, StringBuffer buff)
            throws JSONException
    {
        if (buff == null) {
            buff = new StringBuffer("[");
        } else {
            buff.append("[");
        }

        for (int key = 0; (ja != null) && (key < ja.length()); key++) {
            String value = null;

            if (ja.optJSONObject(key) != null) {
                jsonToObjectLiteral(ja.optJSONObject(key), buff);
            } else if (ja.optJSONArray(key) != null) {
                jsonArrayToString(ja.optJSONArray(key), buff);
            } else if (ja.optLong(key, -1) != -1) {
                value = ja.get(key) + "";
                buff.append(value);
            } else if (ja.optDouble(key, -1) != -1) {
                value = ja.get(key) + "";
                buff.append(value);
            } else if (ja.optBoolean(key)) {
                value = ja.getBoolean(key) + "";
                buff.append(value);
            } else if (ja.opt(key) != null) {
                Object obj = ja.opt(key);

                if (obj instanceof Boolean) {
                    value = ja.getBoolean(key) + "";
                } else {
                    value = "'" + ja.get(key) + "'";
                }

                buff.append(value);
            }

            if (key < ja.length() - 1) {
                buff.append(",");
            }
        }

        buff.append("]");

        return buff.toString();
    }

So, after you have your dynamic data put in JSONObject or JSONArray, you can invoke the corresponding conversion method stated above to get the String form of your JSON data ready to be passed to the component.

For instance, in your tree builder code, you will need to do the following (the example is from this post where the solution was posted by Greg Murray):

public static JSONObject buildTreeData(AuthorizedTeams ateams)
throws JSONException {

 JSONObject retValue = new JSONObject();
 JSONObject root = new JSONObject();
 root.put ("title", "Organizations");
 root.put ("expanded", true);
 JSONArray data = new JSONArray();

 Team[] teams = ateams.getTeams();

 for (int i=0; i<teams.length; i++) {
   JSONObject teamObj = new JSONObject();
   teamObj.put("title", teams[i].getTeamName());
   teamObj.put("expanded", true);

   JSONArray children = new JSONArray();

   User[] teamUsers = teams[i].getMembers();
   for (int j=0; j<teamUsers.length; j++) {
     JSONObject childObj = new JSONObject();
     childObj.put("title",teamUsers [j].getUserName());
     children.put(childObj);
   }
   teamObj.put("children", children);
   data.put(teamObj);
 }
 root.put ("children", data);
 retValue.put ("root", root);

 return jsonToObjectLiteral(retValue, new StringBuffer());
}

Here is the JSP snippet:

 <jsp:useBean id="teams"
   class="com.myapp.assignment.AuthorizedTeams"
   scope="request"/>
 <a:widget name="dojo.tree" value="${teams.teamsData}">

JavaServer Faces Part 1 - Introduction

This is first in the series of blogs on JSF.

JSF = JavaServer Faces.

It’s a web framework. The 3 independent elements that make up a usable JSF component in a page are:

1. UIComponent class – defines behavior of component. Eg. UISelectOne
2. Renderer class – provides specific renderings of component. For eg, a UISelectOne can be rendered in HTML as either a group of radio buttons or a select menu.
3. A JSP tag – which associates a Renderer with a UIComponent and makes them usable in JSP as a single tag, eg <h:selectOneMenu>

JSF UI components are bound to server-side Java beans (which are registered as Managed Beans in faces-config.xml). In the JSP pages, the UI components are bound to Managed Beans using the JSF Expression Language (which in JSF 1.2 is same as JSTL 2.1’s EL and is now called Unified EL). Once bound, updating bean properties or invoking bean methods from a web interface is handled automatically by JSF request processing lifecycle. This ability to automatically synchronize server-side Java Bean properties to a hierarchical set of components that are based on UI presented to the client user is a major advantage of JSF over other web frameworks like Struts.

JSF Request Processing Lifecycle

1. When a JSP page with JSF components is requested first time, then JSF runtime creates an in-memory components tree on server side.
2. In between requests, when nothing is happening in application, the component tree is cached on server.
3. Upon a subsequent request, the component tree is reconstituted, and if form input values are sent in request, they are processed and validations are executed.
4. Upon successful validation, server-side managed bean properties are updated.
5. Once all event processing and updates are over, the response is sent to client.

To enable JSF support in a Java EE web application, following needs to be done:

1. An entry for Faces Servlet in web.xml and mapping of this servlet to *.faces or /faces/* etc. (A request that uses the appropriate faces URL pattern can be considered a faces request and when received by faces controller, it processes the request by preparing an object known as the JSF context, which contains all accessible application data and routes the client to appropriate view page based on the navigation rules as defined in the faces-config.xml.)
2. A JSF configuration file – faces-config.xml in WEB-INF/ path.
3. Following jar files in WEB-INF/lib path:
   1. JSF jars – jsf-api.jar and jsf-impl.jar
   2. Apache commons jars – commons-beanutils.jar, commons-collection.jar, commons-digester.jar, and commons-logging.jar.
   3. JSTL jars – standard.jar and jstl.jar

For a JSP page to be JSF enabled,

• we need to include at least the following taglibs from Sun’s JSF RI (you may also use Apache MyFaces implementation of JSF spec):

   1: <%@taglib uri=”http://java.sun.com/jsf/core” prefix=”f”%>

   2: <%@taglib uri=”http://java.sun.com/jsf/html” prefix=”h”%>

• In the JSP page body, we must add <f:view> tag which becomes the base UI component of component tree in memory on server side when the page is requested for viewing.


• If page processes form input, then we can add <h:form> tag.

Example code:

inputname.jsp – shows a form to user to enter name

If outcome is “greeting” then show greeting.jsp to user

The input name between the two pages is stored in memory in PersonBean’s personName field. The personName is registered as managed bean and JSF’s EL is used in the JSP pages to access the PersonBean’s personName field values.

   1: inputname.jsp:

   2:  

   3: <%@ taglib uri="http://java.sun.com/jsf/html" prefix="h" %>

   4: <%@ taglib uri="http://java.sun.com/jsf/core" prefix="f" %>

   5: <f:loadBundle basename="jsfks.bundle.messages" var="msg"/>

   6:  

   7: <html>

   8:  <head>

   9:   <title>enter your name page</title>

  10:  </head>

  11:  <body>

  12:    <f:view>

  13:      <h1>

  14:       <h:outputText value="#{msg.inputname_header}"/>

  15:      </h1>

  16:      <h:form id="helloForm">

  17:       <h:outputText value="#{msg.prompt}"/>

  18:       <h:inputText value="#{personBean.personName}" />

  19:       <h:commandButton action="greeting" value="#{msg.button_text}" />

  20:      </h:form>

  21:    </f:view>

  22:  </body>

  23: </html>

Where, message bundle is defined in a message.properties file (which needs to be put in WEB-INF/classes path in your web applications WAR) as,

 

   1: inputname_header=JSF KickStart

   2: prompt=Tell us your name:

   3: greeting_text=Welcome to JSF

   4: button_text=Say Hello

   5: sign=!

We bind a PersonBean to the inputText filed in helloForm. To do so we also need to register the PersonBean as managed bean in faces-config.xml. We also need to define the navigation rule from :

   1: <?xml version="1.0"?>

   2: <!DOCTYPE faces-config PUBLIC

   3:   "-//Sun Microsystems, Inc.//DTD JavaServer Faces Config 1.1//EN"

   4:   "http://java.sun.com/dtd/web-facesconfig_1_1.dtd">

   5:  

   6: <faces-config>

   7:   <navigation-rule>

   8:    <from-view-id>/pages/inputname.jsp</from-view-id>

   9:     <navigation-case>

  10:      <from-outcome>greeting</from-outcome>

  11:      <to-view-id>/pages/greeting.jsp</to-view-id>

  12:    </navigation-case>

  13:   </navigation-rule>

  14:  

  15:   <managed-bean>

  16:     <managed-bean-name>personBean</managed-bean-name>

  17:     <managed-bean-class>jsfks.PersonBean</managed-bean-class>

  18:     <managed-bean-scope>request</managed-bean-scope>

  19:   </managed-bean>

  20: </faces-config>

And here’s what the greeting.jsp is:

   1: <%@ taglib uri="http://java.sun.com/jsf/html" prefix="h" %>

   2: <%@ taglib uri="http://java.sun.com/jsf/core" prefix="f" %>

   3: <f:loadBundle basename="jsfks.bundle.messages" var="msg"/>

   4:  

   5: <html>

   6:   <head>

   7:    <title>greeting page</title>

   8:   </head>    

   9:   <body>

  10:      <f:view>

  11:          <h3>

  12:       <h:outputText value="#{msg.greeting_text}" />,

  13:       <h:outputText value="#{personBean.personName}" />

  14:          <h:outputText value="#{msg.sign}" />

  15:         </h3>

  16:      </f:view>

  17:  </body>    

  18: </html>

And the managed bean PersonBean.java:

   1: package jsfks;

   2:  

   3: public class PersonBean {

   4:  

   5:    String personName;

   6:     

   7:    /**

   8:    * @return Person Name

   9:    */

  10:    public String getPersonName() {

  11:       return personName;

  12:    }

  13:  

  14:    /**

  15:    * @param Person Name

  16:    */

  17:    public void setPersonName(String name) {

  18:       personName = name;

  19:    }

  20: }

This completes the short introduction to JSF 1.1.

Understanding Cable Broadband Technology

Below is an introduction to the terms, concepts and summary of the features of DOCSIS standard versions which i have compiled from several sources in an effort to learn about the cable broadband technology.

Cable Modem: is a device that is designed to bridge customer's home computing network to an external network, usually the Internet. This is accomplished by using the preexisting coaxial cable network, originally designed for the cable TV infrastructure, known as Community Antenna Television (CATV).

Coaxial Cable (RG-6 type): Many video channels, each carried at a specific frequency, are superimposed by the cable provider onto a single carrier medium - a standard coaxial cable. This process modulates each channel so that it is exactly 6 MHz (8MHz in Europe) away from the previous channel, and the frequency range available for a CATV provider to use typically runs from 42 to 850 MHz. When a user is watching a channel, the TV is tuned to the frequency that represents the channel and so displays only the part of the cable signal that corresponds to that channel. The legacy CATV infrastructure was designed as a one-way communication network.

ADSL: As demand for faster home internet service increased, cable companies began using their existing coax cable networks to offer digital internet connectivity. At the same time, telcos (phone companies) started using their existing copper two-wire phone lines to offer a similar service known as ADSL (Asynchronous Digital Subscriber Line), where the downstream connection is faster than the upstream connection. Unlike dialup, DSL uses a sophisticated frequency-modulation method to transmit data through copper line wires without disrupting the regular phone service over the line.

Quick comparison between Cable modem and ADSL broadband technologies:

Cable Modem	ADSL
	DSL is decent for browsing web, sending emails, sending and receiving pictures and downloading music but it usually lacks bandwidth for anything having to do with video.
Almost insensitive to distance between CMTS and CM as fiber optic cables can support digital data transmission over longer distances.	DSL is distance sensitive: the signal decreases with increasing distance between the modem and the network service provider, which results in a loss of data throughput. As a result, DSL modem may achieve only a fraction of the advertised data speeds.
Cable service operates on a coax cable which has a higher informational density and is physically thicker than phone wire. This provides a cleaner signal and allows you to modulate more data at higher frequencies with fewer errors.
Coax cable is a shared medium, meaning every house in the area around a local hub of coax (known as drop) is physically connected to the same coax cable.	A DSL home line is a dedicated connection that connects the home user directly with the service provider (the phone company).
Cable modems can upload faster than DSL modems can (max download speed being 38 Mbps and max upload speed being 30 Mbps) but the upstream bandwidth is usually limited by the ISP to a much slower rate.

In short, i think the one can decide which of the two technologies to choose based on which one works out cheaper and/or reliable in their area as both these technologies are capable and can coexist as means to achieve broadband internet connection for home users. I used to use ADSL from airtel in India (Bangalore) and am using Cable modem from comcast in USA (Maynard, MA) and have noticed no significant difference in the service quality during video chats on internet (viz a scenario where high usage of upstream and downstream is made).

A Cable network:

A cable coax network is a bus topology - ie all service nodes (cable modems) are connected to a common medium, the coax bus. Each modem connected to a bus shares this line with every other modem when sending and receiving data. But generally cable modem networks use a technology called hybrid fiber coax (HFC), which incorporates both optical fiber along with coaxial cable to create a broadband network.

A fiber optic node has a broadband optical transmitter and receiver capable of converting the downstream optically modulated signal coming from the headend to an electrical signal going to the homes as well as electrical signals from the home into optical signals in the reverse path. Today, this downstream electrical output is a radio frequency modulated signal that ranges from 50 MHz to 1000 MHz. Fiber optic cables connect the optical node to a distant headend or hub in a point-to-point or star topology or in some cases, in a protected ring topology. The fiber optic node also contains a reverse path transmitter that sends communication from the home back to the headend. In the United States, this reverse signal is a modulated radio frequency ranging from 5 to 42 MHz while in other parts of the world, the range is 5 to 65 MHz.

The coaxial portion of the network connects 25 to 2000 homes (500 is typical) in a tree-and-branch configuration. Radio frequency amplifiers are used at intervals to overcome cable attenuation and passive losses caused by splitting or "tapping" the cable. Trunk coaxial cables are connected to the optical node and form a coaxial backbone to which smaller distribution cables connect. Trunk cables also carry AC power which is added to the cable line at usually either 60V or 90V by a power supply and a power inserter. The power is added to the cable line so that trunk and distribution amplifiers do not need an individual, external power source.

From the trunk cables, smaller distribution cables are connected to a port of the trunk amplifier to carry the RF signal and the AC power down individual streets. If needed, line extenders, which are smaller distribution amplifiers, boost the signals to keep the power of the television signal at a level that the TV can accept. The distribution line is then "tapped" into and used to connect the individual drops to customer homes. These taps pass the RF signal and block the AC power unless there are telephony devices that need the back-up power reliability provided by the coax power system. The tap terminates into a small coaxial drop using a standard screw type connector known as an “F” connector. The drop is then connected to the house where a ground block protects the system from stray voltages. Depending on the design of the network, the signal can then be passed through a splitter to multiple TVs and a cable modem.

A single downstream 6 MHz television channel may support up to 27 Mbps of downstream data throughput from the cable headend using 64 QAM (quadrature amplitude modulation) transmission technology. Speeds can be boosted to 36 Mbps using 256 QAM. Upstream channels may deliver 500 Kbps to 10 Mbps from homes using 16QAM or QPSK (quadrature phase shift key) modulation techniques, depending on the amount of spectrum allocated for service. This upstream and downstream bandwidth is shared by the active data subscribers connected to a given cable network segment, typically 500 to 2,000 homes on a modern HFC network.Most cable modem systems rely on a shared access platform, much like an office LAN. Because cable modem subscribers share available bandwidth during their sessions, there are concerns that cable modem users will see poor performance as the number of subscribers increases on the network. “Common sense dictates that 200 cable data subscribers sharing a 27-Mbps connection would each get only about 135 Kbps of throughput -- virtually the same speed as a 128-Kbps ISDN connection -- right? Not necessarily (Crockett, 99)”.

Unlike circuit-switched telephone networks where a caller is allocated a dedicated connection, cable modem users do not occupy a fixed amount of bandwidth during their online session. Instead, they share the network with other active users and use the network's resources only when they actually send or receive data in quick bursts. “So instead of 200 cable online users each being allocated 135 Kbps, they are able to grab all the bandwidth available during the millisecond they need to download their data packets -- up to many megabits per second (Fitzgerald, 99)”.

If congestion does begin to occur due to high usage, cable operators have the flexibility to add more bandwidth for data services. A cable operator can simply allocate an additional 6 MHz video channel for high-speed data, doubling the downstream bandwidth available to users. Another option for adding bandwidth is to subdivide the physical cable network by running fiber-optic lines deeper into neighborhoods. This reduces the number of homes served by each network segment, and thus, increases the amount of bandwidth available to end-users.

The cable modem access network operates at Layer 1 (physical) and Layer 2 (media access control/logical link control) of the Open System Interconnect (OSI) Reference Model. Thus, Layer 3 (network) protocols, such as IP traffic, can be seamlessly delivered over the cable modem platform to end-users. A cable modem has atleast two MAC addresses, one for the coax interface (aka HFC MAC) and one for the Ethernet interface (aka CMCI MAC for cable modem to CPE MAC).

The DOCSIS Standard:

Almost all cable modems available in retail stores are DOCSIS-certified, which means they can work on the network of any Internet service provider that supports DOCSIS (Data Over Cable Service Interface Specification). DOCSIS is a widely agreed-upon standard developed by a group of cable providers (MSOs - Multiple Service Operators). The company CableLabs runs a certification program for hardware vendors who manufacture DOCSIS-compatible equipment.

The physical hardware of a cable modem includes, a CPU, chipset, RAM and flash memory. There are only a few DOCSIS-compatible microcontrollers in the market - major manufacturers being Broadcom and Texas Instruments.

The DOCSIS standard covers every aspect of cable modem infrastructure - from CM (cable modem at customer premise) to operator's headend equipment (CMTS). This specification details many of the basic functions of the customer's cable modem,

• including how frequencies are modulated on the coax cable,
• how the SNMP protocol applies to the cable modem,
• how data is sent and received,
• how the modem should network with CMTS, and
• how privacy is initiated.

Due to this standardization, consumers can purchase off-the-shelf retail modems for use with many different service providers, and cable operators can deploy newer and more innovative services to consumers.

As QAM level increases, the points that represent symbols have to be placed closer together and are then more difficult to distinguish from one another because of line noise, which creates a higher error rate. Cable modems use an entire TV channel's worth of bandwidth (6MHz for NTSC) for their downstream data. Because of the combined upstream noise from ingress (the distortion created when frequencies enter a medium), the upstream symbol rate is less than the downstream, which has no combined ingress noise issues.

To detect and troubleshoot network problems, cable engineers examine packet error statistics. Each time a cable modem detects a packet error, it will record it. By comparing the total number of received packets with the erroneous ones, the cable modem will produce what's known as the codeword error rate (CER).

NonErr - docsIfCmtsCmStatusUnerroreds

CorrErr - docsIfCmtsCmStatusCorrecteds
UnCorr - docsIfCmtsCmStatusUncorrectables

CER (%) = 100*(UnCorr/(NonErr+CorrErr+UnCorr))

Error ratios higher than 1% should trigger CM maintenance. Formula as mentioned here.

How modems register online?

DOCSIS specification details the procedure a modem should follow in order to register on the cable network - called provisioning process. Across DOCSIS versions, the registration process is same.

1. Tune: When a modem is powered on for the first time, it has no prior knowledge of the cable system it may be connected to. It creates a large frequency scan list for the region for which the modem was designated, which is also known as frequency plan (There are 4 major regions - North America, Europe, China, Japan and each use different channel frequencies so the modem only needs to have a list of frequencies of its intended region of use). With the list retrieved, modem begins to search for a downstream frequency from the list to connect to.

A modem scans for frequencies until it locks on to one. Since a single coax cable can contain multiple digital services, it is up to the CMTS to determine if the new device (the modem performing frequency scan) is supposed to access that particular frequency. This is accomplished by checking the modem's MAC address. Once a modem has locked on to the download channel, it proceeds to obtain the upstream parameters by listening for special packets known as UCDs (Upstream channel descriptors), which contain the transmission parameters for the upstream channel.

2. Range: Once both downstream and upstream channels are synched, the modems makes minor ranging adjustments. Ranging is the process of determining network latency (the time it takes for the data to travel) between cable modem and CMTS. A ranging request (RNG-REQ) must be transmitted from cable modem to CMTS upon registering and periodically there-after. Once the CMTS receives a ranging request, it sends the cable modem a ranging response (RNG-RSP) that contains timing, power, and frequency adjustment information for the cable modem to use. Ranging offset is the delay correction applied by the modem to help synchronize its upstream transmissions.

3. Connect: Next the cable modem must establish IP Connectivity. To do this, it sends a DHCP discover packet and listens for a DHCP offer packet. A DHCP server must be set up at the headend to offer this service, such as Cisco Network Registrar (CNR) or similar. The DHCP offer packet contains IP setup parameters for the cable modem, which includes the HFC IP address, the TFTP service IP address, the boot file name (aka the TFTP config) and the time server's IP address.

4. Configure: After this is done, the modem can (optionally) use the IP protocol to establish the current time of day (TOD) from a Unix type time server running at the headend.

Now the modem must connect to the TFTP server and request the boot file. The boot file contains many important parameters, such as downstream and upstream speed settings (DOCSIS 1.0 only), SNMP settings, and various other network settings. The TFTP server is usually a service that runs in the CMTS.

5. Register: Once the modem downloads the config file, it processes it. It then sends an exact copy of the config file back to the CMTS server, a process known as transferring the operational parameters. This part of registration process is also used to authenticate the modem. If the modem is listed in the CMTS database as valid, the modem receives a message from the CMTS that it has passed registration.

At this stage, the modem has been authenticated and is allowed to initialize its baseline privacy, an optional step that permits modem to initiate privacy features that allow it to encrypt and decrypt its own network traffic to and from the CMTS. The encryption is based on the private digital certificate (X.509 standard) that is installed on the modem prior to registration.

Finally the modem connects to cable operator's internet backbone, and is allowed to access the Web. The cable modem is now operational.

Versions of DOCSIS:

DOCSIS 1.0 key features:

1. 10Mbps upstream capability
2. 40Mbps downstream capability
3. Bandwidth efficient through use of variable packet lengths
4. Class of service (CoS) support
5. CMTS upstream and downstream limitations
6. Extensions for security (BPI)
7. QPSK and QAM modulation formats
8. SNMP v2c

DOCSIS 1.1 key features: Focused more on security. No hardware requirement changed so many DOCSIS 1.0 certified modems were able to use this 1.1 versio with just a simple firmware upgrade.

1. Baseline privacy interface plus (BPI+)
2. MAC collision detection to prevent cable modem cloning
3. Service flows that allow for tiered services
4. SNMP v3
5. VoIP support

DOCSIS 2.0 key features: focuses more on data-over-coax technology. Using Advanced TDMA (A-TDMA), this spec allows cable modems to be upstream capable of up to 30Mbps while previous releases allow up to 10Mbps only. This higher bandwidth allows providers to offer consumers two-way video services, such as video phone service. However, this new standard requires consumer modem upgrade because earlier modem hardware is not capable of this faster upload speed.

DOCSIS 3.0 focuses on data speed improvements to both upstream and downstream channels, as well as many innovations for services other than Internet. These enhancements are accomplished by bridging multiple channels together at the same time, also known as channel bonding. Thus a bandwidth of 200Mbps for downstream and 100Mbps for upstream will be possible. Additional features include support for IPv6.

So that's in short about the cable broadband technology. The DOCSIS 3.0 standard is slated to capture only 60% of the cable market by 2011. The first few products which comply with 3.0 will be released this year 2007 sometime. I have collected the above information from various sources noteable among them being wikipedia and "Hacking the Cable Modem: What Cable companies don't want you to know" book by DerEngel, a very readable work on the cable modem internals.