Wednesday, November 26, 2008

Extending Net-SNMP 5.4.2.1 Agent

This article describes how to extend Net-SNMP agent toolkit to instrument the EtherLike-MIB (RFC 3635) with hardcoded defaults. This article will describe the setup on windows (I used XP SP3) and Linux (I used openSUSE 11.0) required to develop this agent.

The following instructions were carried out for net-snmp source version 5.4.2.1. This article assumes the following default paths:

  1. Net-SNMP un-archived on windows in C:\net-snmp and ~/net-snmp on Linux.
  2. Net-SNMP installed in C:\usr on Windows and /usr/local on Linux.

1.1. Setup the development environment

  1. Download and install MSVC++ Express 2005 from http://www.microsoft.com/express/2005/download/default.aspx. There is some compilation issue of the net-snmp code with MSVC++ 2008 (Orcas) and i did not bother to fix it but used the 2005 instead.
  2. Download and setup platform sdk as described in http://www.microsoft.com/express/2005/platformsdk/default.aspx. Be sure to setup the development environment using the PSDK as described in the link step-wise.
  3. Download net-snmp source from http://www.net-snmp.org/download.html. The net-snmp version 5.4.2.1 can be downloaded from there.

1.2. Build and install net-snmp from source on windows

  1. Un-archive the net-snmp source to say c:\net-snmp.
  2. Follow the README.win32 to build the net-snmp code (Read the section Microsoft Visual C++ - Workspace – Building).
    1. We need to first load the winsdk.dsw in the MSVC++ 2005 express. Below is the screenshot of all projects that make up the win32sdk workspace in MSVC++.

clip_image002

Figure 1 - Projects in win32sdk.dsw

    1. Modify the win32\netsnmp\netsnmpconfig.h file to add:
               #define HAVE_WIN32_PLATFORM_SDK 1






    1. Build in order, the following project for (Release | Debug) configuration.





i. libagent



ii. libhelpers



iii. libnetsnmptrapd



iv. libsnmp



v. netsnmpmibsdk






    1. Go to Build > Batch Build… and select the (Debug | Release) configuration of all projects except for the ones you built in step c above.





  1. If everything is okay with setup, then build should go fine too.




  2. Using cmd line on windows, go to c:\net-snmp and run the command: win32\install_netsnmp.bat to install net-snmp you built above to location c:\usr.




  3. Set the c:\usr\bin to PATH environment variable for easy access to the command line snmp client and agent executables.




1.3. Build and install net-snmp from source on Linux





  1. Un-archive the net-snmp source to say ~/net-snmp.




  2. Follow the INSTALL to build the net-snmp code. Run the following commands on shell prompt:




    1. cd ~/net-snmp




    2. ./configure




    3. make




    4. make install (run as root)






The make install step will install the net-snmp binaries by default in /usr/local path.



1.4. Configuring the snmpd agent (snmpd.conf)





  1. Configure snmp agent by running snmpconf command (see man page) or copy the C:/net-snmp/Example.conf and modify.




    1. snmpconf –g basic_setup






  2. I used the following configuration. Copy-paste the text below into a file named snmpd.conf. The agent is configured for snmp v2c and with rwcommunity string of public.







com2sec local     localhost       public


 


group MyRWGroup    v1         local


group MyRWGroup    v2c        local


group MyRWGroup    usm        local


 


 


view all    included  .1                               80


 


 


#                context sec.model sec.level match  read   write  notif


access MyROGroup ""      any       noauth    exact  all    none   none


access MyRWGroup ""      any       noauth    exact  all    all    none


 


 


syslocation Right here, right now.


syscontact Me <me@somewhere.org>







  1. Save the file in path :




    1. C:\usr\etc\snmp\snmpd.conf (on Windows)




    2. /etc/snmp/snmpd.conf (on Linux)






1.5. Load the MIB module and generate the C code





  1. The EtherLike-MIB has 5 tables (columnar variables).




  2. For mib2c code generation I used linux host. Run the following commands on shell prompt:







mkdir ethmib_src


cd ethmib_src 


export MIBS=ALL 





EtherLike-MIB was part of the net-snmp distribution so all we need is to tell mib2c to use it by setting the environment variable MIBS.






mib2c –c mib2c.create-dataset.conf dot3StatsTable 





This will generate dot3StatsTable.c and dot3StatsTable.h files.



Similarly run the similar command for the other tables in the mib.






mib2c –c mib2c.create-dataset.conf dot3PauseTable 


mib2c –c mib2c.create-dataset.conf dot3HCStatsTable 


mib2c –c mib2c.create-dataset.conf dot3CollTable 


mib2c –c mib2c.create-dataset.conf dot3ControlTable 





Use the ethmib_src and use it for windows agent extension too.



1.6. Use the generated source files to instrument the mib on Windows





  • On Windows, the above generated source files can be copied to c:\net-snmp\agent\mibgroups. You may also create a folder names EtherLike-MIB and sub-folders for each table like dot3StatsTable etc, and keep the respective table *.c and *.h files in the folder with the table name.




  • Add the generated sources in the MSVC++ netsnmpmibssdk project as shown below:




clip_image004



Figure 2 - EtherLike-MIB files added to the netsnmpmibssdk project.





  • Add the dot3StatsTable.h and dot3StatsTable.c files to your 'netsnmpmibssdk' project in VC++.




  • Next edit the '<sourcedir>\win32\mib_module_includes.h' file to add an include to your .h file.







#include "mibgroup/EtherLike-MIB/dot3StatsTable/dot3StatsTable.h"


#include "mibgroup/EtherLike-MIB/dot3HCStatsTable/dot3HCStatsTable.h"


#include "mibgroup/EtherLike-MIB/dot3CollTable/dot3CollTable.h"


#include "mibgroup/EtherLike-MIB/dot3ControlTable/dot3ControlTable.h"


#include "mibgroup/EtherLike-MIB/dot3PauseTable/dot3PauseTable.h"





 





  • Next edit the '<sourcedir>\win32\mib_module_inits.h' file to add code to call your initialize function.







if (should_init("dot3StatsTable")) init_dot3StatsTable();


if (should_init("dot3HCStatsTable")) init_dot3HCStatsTable();


if (should_init("dot3CollTable")) init_dot3CollTable();


if (should_init("dot3ControlTable")) init_dot3ControlTable();


if (should_init("dot3PauseTable")) init_dot3PauseTable();





 



1.7. Instrument the EtherLike-MIB



This section only instruments the dot3StatsTable generated source to return some default data. Other table instrumentation can be done in the similar manner.





  1. Edit the dot3StatsTable.c file as shown below to return some hardcoded data. The changes to the generated code have been highlighted.







/*


 * Note: this file originally auto-generated by mib2c using


 *        : mib2c.create-dataset.conf 9375 2004-02-02 19:06:54Z rstory $


 */


 


#include <net-snmp/net-snmp-config.h>


#include <net-snmp/net-snmp-includes.h>


#include <net-snmp/agent/net-snmp-agent-includes.h>


#include "dot3StatsTable.h"


 


/** Initialize the dot3StatsTable table by defining its contents and how it's structured */


void


initialize_table_dot3StatsTable(void)


{


    static oid dot3StatsTable_oid[] = {1,3,6,1,2,1,10,7,2};


    size_t dot3StatsTable_oid_len = OID_LENGTH(dot3StatsTable_oid);


    netsnmp_table_data_set *table_set;


    


    // variables declared – wrajnees


    netsnmp_table_row *row;


    static int _max_cols = 21; // there are 18 columns in this table.


    int column = 0;


    int index = 1;


    int val = 20;


    static oid      objid_etherchipset[] = { 0 };     /* ethernetChipset vendor oid */


 


    // end variables declared.


 


    /* create the table structure itself */


    table_set = netsnmp_create_table_data_set("dot3StatsTable");


 


    /* comment this out or delete if you don't support creation of new rows */


    


    table_set->allow_creation = 1;


 


    /***************************************************


     * Adding indexes


     */


    DEBUGMSGTL(("initialize_table_dot3StatsTable",


                "adding indexes to table dot3StatsTable\n"));


 


    netsnmp_table_set_add_indexes(table_set,


                           ASN_INTEGER,  /* index: dot3StatsIndex */


                           0);


 


    


    DEBUGMSGTL(("initialize_table_dot3StatsTable",


                "adding column types to table dot3StatsTable\n"));         


    netsnmp_table_set_multi_add_default_row(table_set,


                                            /*COLUMN_DOT3STATSINDEX, ASN_INTEGER, 0,


                                            NULL, 0,*/


                                            COLUMN_DOT3STATSALIGNMENTERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSFCSERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSSINGLECOLLISIONFRAMES, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSMULTIPLECOLLISIONFRAMES, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSSQETESTERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSDEFERREDTRANSMISSIONS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSLATECOLLISIONS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSEXCESSIVECOLLISIONS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSINTERNALMACTRANSMITERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSCARRIERSENSEERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSFRAMETOOLONGS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSINTERNALMACRECEIVEERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSETHERCHIPSET, ASN_OBJECT_ID, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSSYMBOLERRORS, ASN_COUNTER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSDUPLEXSTATUS, ASN_INTEGER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSRATECONTROLABILITY, ASN_INTEGER, 0,


                                            NULL, 0,


                                            COLUMN_DOT3STATSRATECONTROLSTATUS, ASN_INTEGER, 0,


                                            NULL, 0,


                              0);


 


 


    


    /* registering the table with the master agent */


    /* note: if you don't need a subhandler to deal with any aspects


       of the request, change dot3StatsTable_handler to "NULL" */


    netsnmp_register_table_data_set(netsnmp_create_handler_registration("dot3StatsTable", NULL,


                                                        dot3StatsTable_oid,


                                                        dot3StatsTable_oid_len,


                                                        HANDLER_CAN_RONLY),


                            table_set, NULL);


    // Add code - wrajnees


 


    /*


     * create the a row for the table, and add the data 


     */


    row = netsnmp_create_table_data_row();


    /*


     * set the index to the IETF WG name "snmpv3" 


     */


    netsnmp_table_row_add_index(row, ASN_INTEGER, (u_char*)&index, sizeof(index));


    /*


     * set the column 2 and above


     */


    for (column = 2; column <= _max_cols; column++) {


        // Following columns are not valid.


        switch(column) {


            case 12: // INVALIDs


            case 14:


            case 15:


                break;


            case 17: // OID


                netsnmp_set_row_column(row, column, ASN_OBJECT_ID,


                               (u_char*)objid_etherchipset, 1*sizeof(oid));


                break;


            case 19: // INTEGER


            case 21:


                netsnmp_set_row_column(row, column, ASN_INTEGER,


                               (u_char*)&val, sizeof(val));    


                break;


            case 20: // TRUTH_VALUE


                netsnmp_set_row_column(row, column, ASN_INTEGER,


                               (u_char*)&val, sizeof(val));    


                break;


            default: // COUNTER


                netsnmp_set_row_column(row, column, ASN_COUNTER,


                               (u_char*)&val, sizeof(val));    


                break;


        }


 


 


    }


    /*


     * add the row to the table 


     */


    netsnmp_table_dataset_add_row(table_set, row);


 


    /*


     * Finally, this actually allows the "add_row" token it the


     * * snmpd.conf file to add rows to this table.


     * * Example snmpd.conf line:


     * *   add_row netSnmpIETFWGTable eos "Glenn Waters" "Dale Francisco"


     */


    netsnmp_register_auto_data_table(table_set, NULL);


 


    // End add code - wrajnees


}


 


/** Initializes the dot3StatsTable module */


void


init_dot3StatsTable(void)


{


 


  /* here we initialize all the tables we're planning on supporting */


    initialize_table_dot3StatsTable();


}







  1. Now rebuild the netsnmpmibssdk project and then snmpdsdk project, in order.




  2. Run the following command to re-install the modified agent with the dot3StatsTable changes.







cd c:\net-snmp 


win32\install_netsnmp.bat 







  1. Start the new agent as follows:







cd c:\usr\bin 


snmpd.exe –f –Lo –V 







  1. Open a MIB Browser to walk the dot3StatsTable instrumentation:




clip_image006



1.8. Instrumentation on Linux





  1. Copy the generated source to the path ~/net-snmp/agent/mibgroups and run the following commands:







./configure –with-mib-modules=”dot3StatsTable dot3PauseTable dot3HCStatsTable dot3CollTable dot3ControlTable” 


make 


make install (run as root)







  1. Start the snmpd as:







/usr/local/sbin/snmpd –f –Lo –V 







  1. Run a command line snmpwalk to test it.







$ snmpwalk -v 2c -mAll -c public localhost dot3StatsTable 


 


EtherLike-MIB::dot3StatsAlignmentErrors.1 = Counter32: 20 


EtherLike-MIB::dot3StatsFCSErrors.1 = Counter32: 20 


EtherLike-MIB::dot3StatsSingleCollisionFrames.1 = Counter32: 20 


EtherLike-MIB::dot3StatsMultipleCollisionFrames.1 = Counter32: 20 


EtherLike-MIB::dot3StatsSQETestErrors.1 = Counter32: 20 


EtherLike-MIB::dot3StatsDeferredTransmissions.1 = Counter32: 20 


EtherLike-MIB::dot3StatsLateCollisions.1 = Counter32: 20 


EtherLike-MIB::dot3StatsExcessiveCollisions.1 = Counter32: 20 


EtherLike-MIB::dot3StatsInternalMacTransmitErrors.1 = Counter32: 20 


EtherLike-MIB::dot3StatsCarrierSenseErrors.1 = Counter32: 20





2. Co-existance with MS Windows SNMP Agent



We have 2 approaches to getting the net-snmp agent to co-exist with the Microsoft provided SNMP agent:





  • As of Net-SNMP 5.4, the Net-SNMP agent is able to load the Windows SNMP service extension DLLs by using the Net-SNMP winExtDLL extension. In this scenario, MS SNMP agent is installed but disabled. This is required because winExtDLL extension and existing windows extensions use Windows SNMP API from snmpapi.dll. The limitations of this approach are:





    • linkUp/Down generic traps are not received for some unknown reason.




    • sysUpTime.0 does not report the correct uptime for the agent. This is because the Windows extension checks for the uptime of the SNMP service, which is not running when Net-SNMP is running.





  • Alternatively, Net-SNMP agent can run as a proxy SNMP agent and will proxy for the MS SNMP agent (running at a non-default port) for those MIBs that MS SNMP agent instruments. For all other MIBs, Net-SNMP agent can service the SNMP requests. This approach does not suffer from the winExtDLL approach’s limitation, but then we need to run two snmp agents (net-snmp and MS snmp agents).




This sums up in short, the development setup required to get started with extending the net-snmp agent.

Sunday, August 17, 2008

Vim quick reference

The following content is excerpted from http://linuxconfig.org/Vim_Tutorial. It covers most of the essentials that one needs while editing code. One thing which i know and use and is not covered is :e < filename > to open multiple files without exiting vim and then navigating among the open files using :b < num > , where num is file buffer number for the open files . Other than that, there were quite a few points in the tutorial that were really good and i did not use/know them before. Also see other good links to learn more on vi/vim at http://delicious.com/rwatsh/vi.

Vim Tutorial Summary

VIM novice level Summary

  • Moving around with cursor:
    h key = LEFT, l key = RIGHT, k key = UP, j key = DOWN
  • Exiting vim editor without saving:
    press ESC to get into command mode, enter :q! to exit.
  • Deleting characters in vim command mode:
    delete with x key
  • Inserting / appending text:
    Press i or a in command mode and type
  • Saving changes and exit:
    in command mode :wq or SHIFT+zz

VIM Operators and Motions summary

  • Deleting words:
    delete word with d operator and w or e motion
  • Deleting to the end of the line:
    delete to th end of the line with d operator and $ motion
  • Using operators, motions and counts:
    beginning of th line 0, end of the line $, end of the 2nd word 2e beginning of the 4th word 4w
  • Deleting multiple words:
    to delete 3 words you would use d3w
  • Deleting lines:
    to delete single line dd, delete n lines ndd
  • Undo changes:
    undo changes with u

VIM apprentice user summary

  • Paste command:
    paste your cache memory with p command
  • Replace characters:
    rt replace current character with t
  • Change characters:
    ce to change single word, c$ to change to the end of the line

VIM experienced user summary

  • Advanced Navigation:
    end of the file G, begging of the file gg or 1G, to get on line n use nG
    instruct vim display file information CTRL+g
  • Search text with vim:
    search forward /, search backward ?, next search n , previous search N
  • Vim Substitution :
    first occurrence single line :s/bash/perl/
    all occurrences single line :s/bash/perl/g
    first occurrence between line range: :23,100s/bash/perl/
    all occurrences between line range: :23,100s/bash/perl/g
    first occurrence in whole text: :%s/bash/perl/
    all occurrences whole text: :%s/bash/perl/g

VIM veteran user summary

  • Execute external commands on shell from vim:
    :!ls will execute ls command on your shell
  • Writing to files advanced::w saves current file without quit, :w bash.sh whites to file bash.sh
  • Highlight text ans save to different file:highlight text with v operator and save it with :w
  • Retrieve text from different file::r will retrieve content of file

VIM expert user summary

  • Using o operator:
    :o insert line bellow you cursor, O inserts line above your cursor
  • Copy and paste:
    yank line with y and paste it with p
  • Customize vim's environment:
    edit ~/.vimrc file to customize vim's environment

Friday, April 18, 2008

Free Java Programmer's Test



I stumbled on the site http://www.betterprogrammer.com today and found that it offers an open book programming test meaning you can use all reference (like one can do at work) but complete 5 programming exercises pertaining to use of core Java language for writing algorithms. It requires lots of Java collections framework usage and writing some recursion algorithms, writing tree traversal code, calculating prime numbers and stuff like that. The time limit is not very strict and it even allows you to take a break and then continue with the test. In the end you are given a free certificate based on percentile (which it states will keep changing - read may improve - with time, as more people take this free test). The questions were good quality and if you have some free time and like to write code then its quite a good avenue to spend some time on.

My certificate is here: http://www.betterprogrammer.com/certificate/BP1Q9GB5

They claim to maintain this certificate for you in their database and that you can retake the tests to improve (not sure if the questions remain the same - if they dont change then thats a bummer) but anyway take it once and you will love it. If you have taken this test already then feel free to leave your comments on how you liked it or discuss the questions you liked/found interesting.

Wednesday, March 19, 2008

Windows Vista SP1 Released Today

Get it from http://www.microsoft.com/downloads/details.aspx?FamilyID=b0c7136d-5ebb-413b-89c9-cb3d06d12674&DisplayLang=en.

I had the RC1 installed and so when i tried installing the SP1 release then it gave me an error that SP1 is already installed.

So, you must remove all previous version of SP1 before upgrading to the next.

Control Panel > Programs and Features > Installed Updates (upper left corner
under Tasks) > Highlight "Service Pack KB936330" > Uninstall

It will take awhile, there will be 2-3 reboots. Then you can install the
new version.

Follow the above steps and then go to the given link above (for MS download center for Vista SP1) and download and install the SP1 here. To know the features of SP1 please refer to this article.

Tuesday, March 18, 2008

Using SAP Memory Analyzer for Java memory leak detection


[Click to watch video of using SAP Memory Analyzer]

I recently had opportunity to use the SAP Memory Analyzer tool for analyzing an HPROF heap dump generated on OutOfMemory error in code. I found that its an excellent and user friendly tool for job of memory leak analysis with good documentation. Though i generally preferred to use JHAT for heap walking but i realized that this SAP tool is much more intuitive and memory leak detection in code was a breeze. The above video tutorial talks about using the version 1.1.1 (which happens to be the latest at this time). Highly recommended.

Thursday, March 13, 2008

Emma vs Cobertura

The 2 open source Java code coverage tools that are the best among the lot are cobertura and emma. Both have their own pros and cons.

Cobertura - http://cobertura.sourceforge.net/
vs
Emma - http://emma.sourceforge.net/

Points borrowed from: http://raibledesigns.com/rd/entry/emma_vs_cobertura_for_code
Video : http://video.google.com/videoplay?docid=820584080702226910

Emma:
1. Stats on both class and method coverage
2. Partial/fractional line coverages is unique trait - shown in yellow when there are multiple conditions in a conditional block like if (x < 0 and x > 10) and say x > 10 never gets executed this is shown in yellow. This is important feature which lets us determine if the tests cover all conditions of such more than one conditions conditional blocks.
3. Not being actively developed.
4. Stricter code coverage.
5. Integration with Eclipse available - http://www.eclemma.org/
6. Better documentation than cobertura.
7. Instrumentation process is faster than cobertura.
8. Standalone library and does not have any external dependencies.
9. Common public license 1.0 friendlier that GPL.

Cobertura: (since 2002)
1. GPL'd version of JCoverage (which is commercial). Project older than Emma.
2. Prettier reports.
3. Actively developed.
4. Branch/block and line coverages only - no class or method level coverage.
5. How many times a line has been executed - unique about cobertura.
6. <cobertura-check> where one can specify percentage of coverage that's a MUST or else build fails.
7. Data merge feature - good for QA labs... for merging coverage data to prepare historical trend graphs. Emma also supports it now but it seems its better with cobertura. Project long coverage collection possible.
8. Depends on other third party libraries.

Common factors in both of these code coverage tools:
1. bytecode instrumentation.
2. reports are filterable so you can tell what needs to be evaluated for code coverage.
3. offline instrumentation (most recommended approach) - separate instrument/execute/report tasks – this is what we adopted. The other approach is on-the-fly instrumentation.
4. ant integration.
5. testng integration.

The above information can be used in deciding about the right tool for your project. We went for Emma as it seemed to have good enough reports and was fast. I did not get a chance to experiment with Cobertura but will surely try it out soon.

Using TestNG with Emma for automating test code coverage report generation

TestNG and Emma can be used together to automate the generation of code coverage report after every test run in the ANT builld script.
Following ant build script snippet shows how both tools are used together for this important metrics collection.

To run it:
$ ant emma test

This will generate the code coverage report for the TestNG tests in ./coverage directory (where, basedir = ".").

<!-- output directory used for EMMA coverage reports: -->

<property name="coverage.dir" value="${basedir}/coverage" />

<!-- directory that contains emma.jar and emma_ant.jar: -->
<property name="emma.dir" value="${lib}/emma" />

<!-- path element used by EMMA taskdef below: -->
<path id="emma.lib">
<pathelement location="${emma.dir}/emma.jar" />
<pathelement location="${emma.dir}/emma_ant.jar" />
</path>

<!-- this loads <emma> and <emmajava> custom tasks: -->
<taskdef resource="emma_ant.properties" classpathref="emma.lib" />


<property environment="env" />
<path id="classpath">
<fileset dir="${lib}">
<include name="**/*.jar" />
</fileset>
</path>
<!-- 6. CODE COVERAGE initialization. -->
<target name="emma" description="turns on EMMA's instrumentation/reporting">
<property name="emma.enabled" value="true" />

<!-- this property, if overriden via -Demma.filter=<list of filter specs>
on ANT's command line, will set the coverage filter; by default,
all classes found in 'run.classpath' pathref will be instrumented:
-->
<property name="emma.filter" value="" />
</target>

<!-- 7. TEST the application. -->
<taskdef name="testng" classpathref="classpath" classname="org.testng.TestNGAntTask" />


<!-- EMMA ANT tasks are implemented as pseudo-nested tasks: <emma>
container task can contain an arbitrary sequence of <instr>,
<report>, and <merge>. Both the container tag and each of the nested
elements support an optional boolean 'enabled' attribute: setting it
to 'false' will no-op the element. This is convenient for
sandwhiching normal build tasks between EMMA tasks such that coverage
instrumentation and reporting could be enabled on demand. -->

<target name="test" description="execute testng tests" depends="dist">

<emma enabled="${emma.enabled}" verbosity="verbose">
<instr instrpath="${build}" mode="overwrite" metadatafile="${coverage.dir}/metadata.emma">

<!-- note that coverage filters can be set through nested <filter>
elements as well: many of EMMA setting are 'mergeable' in the
sense that they can be specified multiple times and the result
is a union of all such values. Here we are not merging several
filters together but merely demonstrating that it is possible:
-->
<filter value="${emma.filter}" />
</instr>
</emma>

<javac srcdir="${test.src.dir}" destdir="${build}" classpathref="classpath" deprecation="${compile.deprecation}" />

<testng classpathref="test.classpath" outputDir="${testng.report.dir}" sourcedir="${test.src.dir}" haltOnfailure="true">
<xmlfileset dir="${test.src.dir}" includes="testng.xml" />
<jvmarg value="-Demma.coverage.out.file=${coverage.dir}/coverage.emma" />
<jvmarg value="-Demma.coverage.out.merge=false" />
</testng>

<!-- if enabled, generate coverage report(s): -->
<emma enabled="${emma.enabled}">
<report sourcepath="${src}" sort="+block,+name,+method,+class" metrics="method:70,block:80,line:80,class:100">
<!-- collect all EMMA data dumps (metadata and runtime)
[this can be done via nested <fileset> fileset elements
or <file> elements pointing to a single file]:
-->
<fileset dir="${coverage.dir}">
<include name="*.emma" />
</fileset>

<!-- for every type of report desired, configure a nested
element; various report parameters
can be inherited from the parent <report>
and individually overridden for each report type:
-->
<txt outfile="${coverage.dir}/coverage.txt" depth="package" columns="class,method,block,line,name" />
<xml outfile="${coverage.dir}/coverage.xml" depth="package" />
<html outfile="${coverage.dir}/coverage.html" depth="method" columns="name,class,method,block,line" />
</report>
</emma>

</target>

Wednesday, January 09, 2008

Experience using apache commons EqualsBuilder class

Following are my learnings on how to use the EqualsBuilder class in apache commons lang library.

C:\Work\EqualsProto\src\equalsproto\Main.java


package equalsproto;

//~--- non-JDK imports --------------------------------------------------------

import java.util.Arrays;
import org.apache.commons.lang.builder.EqualsBuilder;


class A
{
private String s = "Watsh";
private int x = 10;
private float y = 20.2f;

//~--- constructors --------------------------------------------------------

A() {}

A(int i, int i0, String string)
{
this.x = i;
this.y = i0;
this.s = string;
}

//~--- methods -------------------------------------------------------------

/**
* Method description
*
*
* @param obj

*
* @return
*/
@Override
public boolean equals(Object obj)
{
if (obj instanceof A == false) {
return false;
}

if (this == obj) {
return true;
}

A rhs = (A) obj;

/** Note:

* Do not use appendSuper when the super class is java.lang.Object as

* default implementation of equals in Object class will return true only

* when two references are pointing to the same object instance and hence

* the effect is not desirable.
*/
return new EqualsBuilder().append(s, rhs.s).append(x,
rhs.x).append(y, rhs.y).isEquals();
}
}



class B
{
private String z = "Rajneesh";
private A a;
private A[] array;

//~--- constructors --------------------------------------------------------

B(String z, A a, A[] array)
{
this.z = z;
this.a = a;
this.array = array;
}

@Override
/**
* Learning:
* 1. appendSuper() should not be used as it then calls super.equals()

* for java.lang.Object class which will return true only when both lhs and

* rhs references point to the same object instance and hence will return false

* when the 2 object instances are different but meaningfully equivalent.

*
* 2. To compare arrays, you will either need to use the

* EqualsBuilder.reflectionEquals() approach or if you are using the

* EqualsBuilder.append() approach then append(array1, array2) calls

* array1.equals(array2) which will only do a shallow comparison for the

* 2 arrays involved. So in such a case, you must use Arrays.deepEquals() for

* all array members of your class and once that equality is met you can use

* EqualsBuilder.append() for rest of the non-array instances.

*
* NOTE: I have not tested for how this approach works for Collection classes.

*/
/*public boolean equals(Object obj)
{
if (obj instanceof A == false) {
return false;
}

if (this == obj) {
return true;
}

B rhs = (B) obj;

return new EqualsBuilder().append(z, rhs.z).append(a,

rhs.a).append(array, rhs.array).isEquals();
}*/
public boolean equals(Object obj) {
return EqualsBuilder.reflectionEquals(this, obj);
}
}



/** Testing the use of EqualsBuilder.
*
* @author wrajnees

*/
public class Main
{

/**
* @param args the command line arguments

*/
public static void main(String[] args)
{
A a1 = new A(2, 43, "xxx");
A a2 = new A(2, 43, "xxx");
if (a1.equals(a2)) {
p("a1 equals a2");
} else {
p("a1 not equals a2");
}

A[] array1 = new A[2];
A[] array2 = new A[2];

array1[0] = a1;
array1[1] = new A();

array2[0] = a2;
array2[1] = new A();

// comparing arrays

if (Arrays.deepEquals(array1, array2)) {
//if (array1.equals(a2)) { // -- does not work
p("arrays are equal");
} else {
p("arrays arent equal");
}

// comparing more complex object with containment and array

B b1 = new B("Test", a1, array1);
B b2 = new B("Test", a2, array2);
if (b1.equals(b2)) {
p("b1 equals b2");
} else {
p("b1 not equals b2");
}
}

private static void p(String s) {
System.out.println(s);
}
}



Tuesday, January 08, 2008

Detecting memory leaks in Java SE

Today i have learned about a nice approach to detecting memory leaks in the Java SE applications using the jmap and jhat (Java Heap Analysis Tool). The process to be followed is described below:

1. Run your application.

2. Run the command jps to know the process id of the J2SE application you ran.
% jps
1234 MyApp
...

3. Perform those actions in your application which you feel will cause the memory leak. You can observe the real time heap usage plot in jconsole. Launch jconsole and select your application in it to connect to.
% jconsole

4.Then run the command jmap to dump the heap.
% jmap -dump:file=myapp.bin 1234

This will produce a heap dump in myapp.bin file with the heap profile.

4. Run the JHAT (Java Heap Analysis Tool) as follows:
% jhat -J-mx512m heap.bin

The above command starts a small Http server at port 7000 by default.

5. Browse to http://localhost:7000 and you will have you the heap browser - a hyperlinked set of pages from where you can trace every object allocated and who all reference the object at the point at which the heap dump was created.

6. The important pages to browse to are:
http://localhost:7000/histo/ - to see the histogram of heap usage.
http://localhost:7000/showInstanceCounts or http://localhost:7000/showInstanceCounts/includePlatform/ to see biggest types with most object allocations (ie instances).

and some advanced features of using SQL to query values of instance members:
http://localhost:7000/oql/

7. So browse to http://localhost:7000/showInstanceCounts/. Investigate "Instances" and not "Classes". Use “Reference Chains from Rootset” (Exclude weak refs!!!) to see who’s holding the instance. This tip i found in one of the links below in the reference section and it really was what was required to find the memory leaking code.

Some good references are:
  1. Memory leaks in Java program
  2. Using Mustang's jmap/jhat to profile Glassfish
  3. Finding Memory leaks in Java Program

Monday, December 03, 2007

TestNG versus JUnit4

Comparing JUnit 4 and TestNG 5.7

Excerpts from http://www.ibm.com/developerworks/java/library/j-cq08296/ by Andy Glover.

JUnit is geared more towards unit testing - testing an object class in isolation.
TestNG provides more features and flexibility to facilitate its use not only for unit but integration, regression, functional, acceptance testings etc.

1. The setup method (annotated with @BeforeClass) needs to static and public with JUnit 4 but thats not required by TestNG. Thus TestNG is more flexible of the two.

2. Dependency testing:
Unlike JUnit, TestNG welcomes test dependencies through the dependsOnMethods attribute of the Test annotation. With this handy feature, you can easily specify dependent methods, which will execute before a desired method. What's more, if the dependent method fails, then all subsequent tests will be skipped, not marked as failed.

In JUnit 4, you can specify test orders using fixtures but if one test A fails then a test B that depends on test A will also be marked as failed.

TestNG's trick of skipping, rather than failing, can really take the pressure off in large test suites. Rather than trying to figure out why 50 percent of the test suite failed, your team can concentrate on why 50 percent of it was skipped! Better yet, TestNG complements its dependency testing setup with a mechanism for rerunning only failed tests.

3. Fail and rerun:
The ability to rerun failed tests is especially handy in large test suites, and it's a feature you'll only find in TestNG. In JUnit 4, if your test suite consists of 1000 tests and 3 of them fail, you'll likely be forced to rerun the entire suite (with fixes). Needless to say, this sort of thing can take hours.

Anytime there is a failure in TestNG, it creates an XML configuration file (testng-failed.xml) that delineates the failed tests. Running a TestNG runner with this file causes TestNG to only run the failed tests. So, in the previous example, you would only have to rerun the three failed tests and not the whole suite.

This feature doesn't seem like such a big deal when you're running smaller test suites, but you quickly come to appreciate it as your test suites grow in size.

4. Parametric testing:
By placing parametric data in TestNG's XML configuration files, you can reuse a single test case with different data sets and even get different results. This technique is perfect for avoiding tests that only assert sunny-day scenarios or don't effectively verify bounds.

JUnit testers often turn to a framework like FIT in this case because it lets you drive tests with tabular data. But TestNG provides a similar feature right out of the box.

This feature not only facilitates reuse of the test case code but also allows non-programmers to specify test data (since test data is in xml file).

public class TestWebServer {
@Test(parameters = { "number-of-times" })
public void accessPage(int numberOfTimes) {
while (numberOfTimes-- > 0) {
// access the web page
}
}
}



5. Advanced Parametric testing:
While pulling data values into an XML file can be quite handy, tests occasionally require complex types, which can't be represented as a String or a primitive value. TestNG handles this scenario with its @DataProvider annotation, which facilitates the mapping of complex parameter types to a test method.

Example:

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
return new Object[][] {
{ "Cedric", new Integer(36) },
{ "Anne", new Integer(37)},
};
}

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
System.out.println(n1 + " " + n2);
}



6. Groups:

You can define groups at the class level and then add groups at the method level. You can also specify groups and methods to be included and excluded.

@Test(groups = { "checkin-test" })
public class All {

@Test(groups = { "func-test" )
public void method1() { ... }

public void method2() { ... }
}


and then in testng.xml:


<test name="Simple example">
<groups>
<run>
<include name="checkin-test"/>
<exclude name="broken"/>
</run>
</groups>

<classes>
<class name="example1.Test1">
<methods>
<include name="testMethod" />
</methods>
</classes>
</test>

Saturday, December 01, 2007

TestNG - java testing framework

Recently i got introduced to TestNG (version 5.7) at work. I was familiar to JUnit from the past and i kind of knew about the existance of TestNG and that it had improvements over JUnit but i never thought that it will gain so much traction that i will be made to use it soon. Here are some of the features:
  • JDK 5 Annotations (JDK 1.4 is also supported with JavaDoc annotations).
  • Flexible test configuration - using multiple testng XML configuration files one per test suite.
  • Support for data-driven testing (with @DataProvider).
  • Support for parameters - you can pass parameters to test methods from the testng.xml file.
  • Allows distribution of tests on slave machines - support for parallel execution of tests and methods.
  • Powerful execution model (no more TestSuite) - test classes are annotated POJOs and don't have to extend any class or implement interface to have test methods.
  • Supported by a variety of tools and plug-ins (Eclipse, IDEA, Ant, Maven, etc...).
  • Embeds BeanShell for further flexibility.
  • Default JDK functions for runtime and logging (no dependencies).
  • Dependent methods for application server testing. - one can specify the dependsOnMethods attribute to the @Test annotation to specify a list of methods that should execute before a certain test method executes. This is a powerful feature and is required for any kind of dependent testing. If a dependent method fails, then all subsequent tests will be skipped, not marked as failed (unlike JUnit).
A good article stating improvements in TestNG over JUnit 4 is found at http://www.ibm.com/developerworks/java/library/j-cq08296/.

I used TestNG today for the first time and found the framework very easy to use and within a day i had it integrated into our build system and made a presentation to the team about its usage in our project. In this post, i am detailing the steps i performed to start using TestNG:

1. Wrote a class using just the 3 basic TestNG annotations to start with:
  • @BeforeClass
  • @Test (groups = {"xyz.groupname"}) - at class level which gets inherited by all public methods in the class.
  • @AfterClass
See http://testng.org/doc/documentation-main.html#annotations for complete list of supported annotations.

2. Wrote a master testng.xml which included all project suite-files and was referenced from the ant build script. Also wrote a testng-regression.xml which was imported in the master testng.xml. In the testng-regression.xml, defined the test runs in the suite. Each such testng-xxx.xml file is for xxx named test suite. Each suite can have one or more test runs. Each test runs identifies the class(es) or package(s) to lookup for annotated test methods. Each test run also identifies filter criteria based on groups to include and exclude in the test run. See http://testng.org/doc/documentation-main.html#testng-xml for more on testng.xml.

One powerful feature i found was the group names could be specified in dot separated (java package name like) notations and follow a hierarchy akin to the Log4j Logger naming hierarchy. So you can use wildcards in the testng.xml to not only include all classes of a group but also include classes from child groups. For example, i could just say xyz.* to include xyz.abc and xyz.def group classes.

3. Lastly, used the ant build file to call the testng ant task and pass the testng.xml location to it so that testng can execute the tests we wanted. We can have multiple targets defined for different types of tests that we may want to automate. See http://testng.org/doc/ant.html for examples.

In the latest releases of JUnit 4, it too uses JDK5 annotations and thus makes up for some of the shortcomings that led Cedric Beust to develop TestNG framework.

If you have not had a chance to explore TestNG so far, then i hope after reading this post you will have the good sense to do so now :).


Saturday, November 10, 2007

Case for Web services with JSON RPC

I have recently been working on developing JSON RPC based web services (over https) and using Java client. The server side JSONRPC services were developed using the JSON-RPC-Java and later also using the JSON-RPC C libraries.

The only client side JSON RPC stack in Java that is available at the time of this writing is http://code.google.com/p/json-rpc-client/. It supports JSON RPC over http (using apache commons httpclient library). It was easily extensible to support JSON RPC over https. In this post, i am going to put down my experiences of using JSON RPC.

  1. JSON is a fat-free XML. (Read more at http://json.org/xml.html).
  2. JSON RPC is an alternative RPC mechanism over http (or https).
  3. JSON RPC is simpler to learn and implement than SOAP. The stacks are much less lines of code compared to SOAP stacks.
  4. JSON RPC is simple as it does not include an Interface Definition Language like WSDL for SOAP based web services. So there is no contract definition between client and server in a IDL rather contract is defined on paper and then implememted in respective languages of server-side and client-side.
  5. JSON RPC spec is very loosly written and hence leaves alot of room for vendors to come up with their own solutions. Like metaparadig folks have their proprietary way of implementing class hinting (viz the way to identify the class type to the other end so that JSON message can be mapped to a class type and an instance of the class can be created with the passed in values in the JSON stream).
  6. The interoperability between JSON RPC C/C++ service and Java client is limited in following aspects:
    1. No Java collections can be used. This is same for even SOAP web services. The root cause for this limitation is that the pre JDK 1.5 Java had no generics and hence all collection classes (like ArrayList) could have held more than one Object types so it was hard to tell the type of the element held in the collection. This is solved by proprietary class hinting ismplementations when both client and server are in Java but across languages this becomes an issue. So the solution is to use arrays instead.
    2. Enum types are not supported by the metaparadigm JSON-RPC-Java stack at present as its a newer JDK 1.5 feature. So use int instead.
  7. Security: Though several approaches may be possible but the simplest solution is to implement JSON RPC over https with basic authentication for client. You may have a self-signed certificate for the web service to keep the deployments simple. But if you really want the most security possible then go for a trusted CA signed certificate for the web service but then you will require a certificate signing infrastructure in place to be able to create a certificate for each instance of web service installed.
  8. JSON RPC spec does not have anything to say about intermediary message handlers but it is easy to think of creating JSON RPC intermediary nodes although the spec does not have provisions for extensible message control headers like SOAP spec has. So JSON RPC is pretty much limited to being used between two nodes (the client and the server) - the message source and the message destination or end point. Its not really meant for "document" style messaging for which SOAP is used in B2B applications.
So if you want to build a robust, fat-free (read faster) distributed RPC infrastructure then you can base it on JSON RPC.

JSON RPC makes most sense in web applications where the client is in Javascript language as JSON maps directly to Javascript objects and hence you dont need to parse the message and extract the data, its done automatically. But other than AJAXing your web pages, you can also use it for straight forward RPC architectures where SOAP may be an overkill. You will have a working JSON RPC solution much sooner and it is of course much easier to comprehend and implement than SOAP. So when you are using SOAP web services with RPC style then think twice as you have an more able alternative approach in JSON RPC.

Let me know your thoughts by leaving your comments.

Tuesday, November 06, 2007

Using Basic authentication and HTTPS (w/ self-signed certificates) in Java

1. Client Authentication is in practice only used for B2B type applications.
2. In some cases we may even be okay with not authenticating the server on the client end during SSL handshake, for sake of:
o simplicity (no certificate signing infrastructure is required) and
o performance (we only use SSL for encryption and not for server authentication).

This approach is of self-signed certificate which the server can sign for itself and client will by-pass server authentication.

3. We first need to configure web server for SSL. Tomcat currently operates only on JKS, PKCS11 or PKCS12 format keystores.
4. We can use the JDK keytool to generate self-signed certificate for the host running tomcat as shown below:

$ keytool -genkey -alias tomcat -keyalg RSA -keystore example.keystore
Enter keystore password: secret
Re-enter new password: secret
What is your first and last name?
[Unknown]: localhost
What is the name of your organizational unit?
[Unknown]:
What is the name of your organization?
[Unknown]: <My Company Name>
What is the name of your City or Locality?
[Unknown]: <City>
What is the name of your State or Province?
[Unknown]: <State>
What is the two-letter country code for this unit?
[Unknown]: <Country Code>
Is CN=localhost, OU=Unkown, O=<My Company Name>, L=<City>, ST=<State>, C=<Country> correct?
[no]: yes

Enter key password for
(RETURN if same as keystore password): <Enter>


The example.keystore is then generated and is in JKS (Java Key Store) format.

5. Copy it to the Tomcat root directory say C:\Program Files\Apache Software Foundation\Tomcat 6.0 path.

6. The final step is to configure your secure socket in the $CATALINA_HOME/conf/server.xml file, where $CATALINA_HOME represents the directory into which you installed Tomcat 6.

<Connector protocol="org.apache.coyote.http11.Http11Protocol"
port="8443" minSpareThreads="5" maxSpareThreads="75"
enableLookups="true" disableUploadTimeout="true"
acceptCount="100" maxThreads="200"
scheme="https" secure="true" SSLEnabled="true"
keystoreFile="./example.keystore" keystorePass="secret"
clientAuth="false" sslProtocol="TLS"/>

NOTE: You can refer to the http://tomcat.apache.org/tomcat-6.0-doc/ssl-howto.html for more configuration options.

With the above settings you can verify that browsing to https://localhost:8443 returns the splash page of tomcat home.

7. We will also make sure that tomcat has a role named "manager" and some user associated with the role. We can edit the tomcat_users.xml for that:

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
<role rolename="manager"/>
<user username="admin" password="admin" roles="manager"/>
</tomcat-users>

8. Now, we can enforce that a certain URL pattern for our web application always requires https access. To do this, we need to edit the web.xml of the web application:


<security-constraint>
<display-name>some name for service</display-name>
<web-resource-collection>
<web-resource-name>My Service</web-resource-name>
<description/>
<url-pattern>/secure/XYZ/*</url-pattern>
<http-method>GET</http-method>
<http-method>POST</http-method>
<http-method>HEAD</http-method>
<http-method>PUT</http-method>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
<http-method>DELETE</http-method>
</web-resource-collection>
<auth-constraint>
<role-name>manager</role-name>
</auth-constraint>
<user-data-constraint>
<description/>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>MY_SECURE_REALM</realm-name>
</login-config>
<security-role>
<description>manager api can use this role.</description>
<role-name>manager</role-name>
</security-role>

With the above configuration, we have Basic authentication and HTTPS enabled for all resources accessed by the URL pattern /secure/XYZ/*.

So even if you try to access the resource at /secure/XYZ/* using http then tomcat will redirect you to the page using https scheme and thus enforce secure use. Since we also use the Basic authentication so browser client will prompt you entering user credentials.

9. If you are using API based http client access from say a J2SE client (using apache commons httpclient 3.x) then you will need to set the credentials for the realm MY_SECURE_REALM (which defines the Authentication Scope on the web server) in the Http header.

HttpState state = new HttpState();
state.setCredentials(new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT,
"MY_SECURE_REALM"), new UsernamePasswordCredentials(user, passwd));

Also you will need to use the org.apache.commons.httpclient.contrib.ssl.EasySSLProtocolSocketFactory to be able to by-pass the agent authentication on client side. Apache commons httpclient comes with this contrib code which is included with the source distro but is not bundled in the jar file. So you will need to pull the source out from contrib/ssl path and use it in your project.

Basically you will need to check if the uri in use has scheme type of https then associate the EasySSLProtocolSocketFactory as the protocol handler for the scheme.

if (uri.getScheme().equals("https")) {
Protocol easyhttps = new Protocol(uri.getScheme(), new EasySSLProtocolSocketFactory(), uri.getPort());

Protocol.registerProtocol("https", easyhttps);
}

The way it works is, EasySSLProtocolSocketFactory in turn uses the EasyX509TrustManager (again from contrib/ssl) to just do a agent certificate validity from and to time validation (so that the ceritificate is not expired and is not before the validity start date). As long as the certificate in use by the agent is valid the EasyX509TrustManager will be okay to bypass doing any authentication for the self-signed certificate for the agent.

That completes the simple discourse on how to use Basic authentication with HTTPS (using self-signed certificate for the server end).

Friday, October 26, 2007

Salient points about log4j

0. Log4j has three main components: loggers, appenders and layouts.

1. Loggers are named entities which follow hierarchical naming.
2. A logger is said to be an ancestor of another logger if its name followed by a dot is a prefix of the descendant logger name. A logger is said to be a parent of a child logger if there are no ancestors between itself and the descendant logger. For example, the logger named "com.foo" is a parent of the logger named "com.foo.Bar".
3. The root logger resides at the top of the logger hierarchy. It is exceptional in two ways:

1. it always exists,
2. it cannot be retrieved by name.
Invoking the class static Logger.getRootLogger method retrieves it.
4. Loggers may be assigned levels. The set of possible levels, that is:

TRACE,
DEBUG,
INFO,
WARN,
ERROR and
FATAL

are defined in the org.apache.log4j.Level class.

5. The inherited level for a given logger C, is equal to the first non-null level in the logger hierarchy, starting at C and proceeding upwards in the hierarchy towards the root logger.

6. Here are the basic Logger class methods:

package org.apache.log4j;

public class Logger {

// Creation & retrieval methods:
public static Logger getRootLogger();
public static Logger getLogger(String name);

// printing methods:
public void trace(Object message);
public void debug(Object message);
public void info(Object message);
public void warn(Object message);
public void error(Object message);
public void fatal(Object message);

// generic printing method:
public void log(Level l, Object message);
}

7. A logging request is said to be enabled if its level is higher than or equal to the level of its logger.

A log request of level p in a logger with (either assigned or inherited, whichever is appropriate) level q, is enabled if p >= q.

This rule is at the heart of log4j. It assumes that levels are ordered. For the standard levels, we have DEBUG < INFO < WARN < ERROR < FATAL.

// get a logger instance named "com.foo"
Logger logger = Logger.getLogger("com.foo");

// Now set its level. Normally you do not need to set the
// level of a logger programmatically. This is usually done
// in configuration files.
logger.setLevel(Level.INFO);

Logger barlogger = Logger.getLogger("com.foo.Bar");

// This request is enabled, because WARN >= INFO.
logger.warn("Low fuel level.");

// This request is disabled, because DEBUG < INFO.
logger.debug("Starting search for nearest gas station.");

// The logger instance barlogger, named "com.foo.Bar",
// will inherit its level from the logger named
// "com.foo" Thus, the following request is enabled
// because INFO >= INFO.
barlogger.info("Located nearest gas station.");

// This request is disabled, because DEBUG < INFO.
barlogger.debug("Exiting gas station search");


8. In fundamental contradiction to biological parenthood, where parents always preceed their children, log4j loggers can be created and configured in any order. In particular, a "parent" logger will find and link to its descendants even if it is instantiated after them.

9. Log4j makes it easy to name loggers by software component. This can be accomplished by statically instantiating a logger in each class, with the logger name equal to the fully qualified name of the class. This is a useful and straightforward method of defining loggers. As the log output bears the name of the generating logger, this naming strategy makes it easy to identify the origin of a log message. The developer is free to name the loggers as desired.Nevertheless, naming loggers after the class where they are located seems to be the best strategy known so far.

10. Log4j allows logging requests to print to multiple destinations. In log4j speak, an output destination is called an appender.

Currently, appenders exist for the console, files, GUI components, remote socket servers, JMS, NT Event Loggers, and remote UNIX Syslog daemons. It is also possible to log asynchronously.

11. More than one appender can be attached to a logger.

The addAppender method adds an appender to a given logger.

12. Each enabled logging request for a given logger will be forwarded to all the appenders in that logger as well as the appenders higher in the hierarchy.

In other words, appenders are inherited additively from the logger hierarchy. For example, if a console appender is added to the root logger, then all enabled logging requests will at least print on the console. If in addition a file appender is added to a logger, say C, then enabled logging requests for C and C's children will print on a file and on the console.

It is possible to override this default behavior so that appender accumulation is no longer additive by setting the additivity flag to false.

13.The layout is responsible for formatting the logging request according to the user's wishes, whereas an appender takes care of sending the formatted output to its destination.

The PatternLayout, part of the standard log4j distribution, lets the user specify the output format according to conversion patterns similar to the C language printf function.

See the conversion characters to use in the link below:
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html

For example, the PatternLayout with the conversion pattern "%r [%t] %-5p %c - %m%n" will output something akin to:

176 [main] INFO org.foo.Bar - Located nearest gas station.

The first field is the number of milliseconds elapsed since the start of the program.
The second field is the thread making the log request.
The third field is the level of the log statement.
The fourth field is the name of the logger associated with the log request.
The text after the '-' is the message of the statement.

14. To use log4j by reading the configuration from a properties file:

import com.foo.Bar;

import org.apache.log4j.Logger;
import org.apache.log4j.PropertyConfigurator;

public class MyApp {

static Logger logger = Logger.getLogger(MyApp.class.getName());

public static void main(String[] args) {


// BasicConfigurator replaced with PropertyConfigurator.
PropertyConfigurator.configure(args[0]);

logger.info("Entering application.");
Bar bar = new Bar();
bar.doIt();
logger.info("Exiting application.");
}
}

And a sample configuration file:

log4j.rootLogger=DEBUG, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout

# Print the date in ISO 8601 format
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

# Print only messages of level WARN or above in the package com.foo.
log4j.logger.com.foo=WARN

Example output:

2000-09-07 14:07:41,508 [main] INFO MyApp - Entering application.

This will only print warn, error or fatal but not info, debug or trace messages for all components inheriting from com.foo logger name hierarchy.

Example using multiple appenders:

log4j.rootLogger=debug, stdout, R

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log

log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1

log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n

Example output:

INFO [main] (MyApp2.java:12) - Entering application.
DEBUG [main] (Bar.java:8) - Doing it again!
INFO [main] (MyApp2.java:15) - Exiting application.


15. Under certain well-defined circumstances however, the static inializer of the Logger class will attempt to automatically configure log4j.

The exact default initialization algorithm is defined as follows:

1. Setting the log4j.defaultInitOverride system property to any other value then "false" will cause log4j to skip the default initialization procedure (this procedure).
2. Set the resource string variable to the value of the log4j.configuration system property. The preferred way to specify the default initialization file is through the log4j.configuration system property. In case the system property log4j.configuration is not defined, then set the string variable resource to its default value "log4j.properties".
3. Attempt to convert the resource variable to a URL.
4. If the resource variable cannot be converted to a URL, for example due to a MalformedURLException, then search for the resource from the classpath by calling org.apache.log4j.helpers.Loader.getResource(resource, Logger.class) which returns a URL. Note that the string "log4j.properties" constitutes a malformed URL. See Loader.getResource(java.lang.String) for the list of searched locations.
5. If no URL could not be found, abort default initialization. Otherwise, configure log4j from the URL. The PropertyConfigurator will be used to parse the URL to configure log4j unless the URL ends with the ".xml" extension, in which case the DOMConfigurator will be used. You can optionaly specify a custom configurator. The value of the log4j.configuratorClass system property is taken as the fully qualified class name of your custom configurator. The custom configurator you specify must implement the Configurator interface.

Under Tomcat 3.x and 4.x, you should place the log4j.properties under the WEB-INF/classes directory of your web-applications. Log4j will find the properties file and initialize itself. This is easy to do and it works.

Generally one would want to have the flexibility to choose between different logging implementations. In that case, one can use apache commons logging which provides similar interface as Log4J's described above (only it calls it Logger as Log and uses a LogFactory.getLog() to get the named Log instance) but comes with adapters for several other logging implementations in Java (Avalon, JDK's Logging etc).

Monday, September 03, 2007

EJB 3.0 with JBoss 4.2.1 GA and Netbeans IDE 5.5.1

I have started to read O'reilly's EJB 3.0 5th Edition by Bill Burke and Richard Monson-Haefel. The book covers EJB 3.0 and Java Persistence 1.0 in detail. It comes with a JBoss workbook for JBoss 4.0.3 release. I installed the current stable release JBoss 4.2.1 GA for my practice. Unlike the 4.0.x releases, the JBoss 4.2.x release has the EJB 3 enabled by default. I used Netbeans 5.5.1 IDE for development. It supports JBoss 4.x and even 5.x (which is still in beta). It readily recognized my JBoss 4.2.1 installation in the server manager. I created a project as Enterprise Application (with both web and ejb modules). The persistence configuration was simple and i used the default datasource HSQL DB 1.8.

There were some gotchas before i could get chapter 4 "Developing your first bean" examples working:
1. one has to change this "DefaultDS" to "java:/DefaultDS" in the META-INF/persistence.xml of your ejb module.
2. also, in the client application, you will need to reference the "TravelAgentBean/remote" as "<Your EAR application name>/TravelAgentBean/remote". To be sure as to where in the jndi tree has your session bean got registered, you can browse to jmx-console (http://localhost:8080/jmx-console) and look for "service=JNDIView". Click on the link and in the following page, invoke the method list() to see the list of names in the JNDI tree. From there you can know for sure what name to use for lookup at client end for your session facade.
3. lastly, i had mistakenly had my entity bean's id annotated as @GeneratedValue in which case when i used to call the setId() method in my session facade bean, then it used to throw an exception while persisting using EntityManager's persist() that the entity instance is detached one. So, i could either remove the annotation of generated value or don't set the id.
4. at client end, there is no need for PortableRemoteObject.narrow() method anymore. You can simply use Java casting.

For my simple example, i did not have to pass the JNDI bootstrap params as properties instance to InitialContext() constructor.

Eclipse 3.3 (Europa) has a very nice OR mapping tool called Dali, but i am yet to figure out how to make it work for me (basically if i don't have the corresponding table already created in the DB, then Eclipse is unable to map the Entity type's members to columns in the database table and so far i dont know how to turn that error off. With Netbeans i did not get that issue and when i first ran my application the table was automatically created by hibernate as i declared it to do so by setting hibernate property hibernate.hbm2ddl.auto with value "update"). Europa release supports JBoss 4.2.x release. Red Hat is also developing a comprehensive IDE solution for JBoss in partnership with Exadel to facilitate easier Rich web application development.

Popular micro services patterns

Here are some popular Microservice design patterns that a programmer should know: Service Registry  pattern provides a  central location  fo...