Sunday, October 11, 2015

Setting up OpenStack python SDK on Oracle Enterprise Linux 6

To setup openstacksdk on Oracle VM Server 3.3.3 or Oracle Enterprise Linux 6:

Oracle Enterprise Linux 6 comes with python 2.6.6 by default.

OpenStackSDK requires at least python 2.7 or 3.3.

Following are the steps to upgrade local python to 2.7.10 and 3.5.0 (the latest as of this writing) and install openstacksdk for development.

It is assumed that OpenStack is hosted on the same host where the test program is being executed.

$ sudo su -
# cd /etc/yum.repos.d
# rm -f *
# yum clean all
# yum makecache
# yum install gcc

The above step may take quite some time and is only required to install gcc. If one already has gcc installed then skip to the following step.

# yum groupinstall "Development tools"
# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
# cd /usr/src

Get ez_setup to install easy_install for python interpreter.

# tar xzf Python-2.7.10.tgz
# tar xzf Python-3.5.0.tgz

Install Python-2.7.10
# cd Python-2.7.10
# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
# make && make altinstall
# python2.7 -V
# python2.7
# easy_install-2.7 pip


Install Python 3.5.0
# cd Python-3.5.0
# ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
# make && make altinstall
# python3.5 -V
# python3.3
# easy_install-3.3 pip

Setup Virtualenv for your project:
# cd /files/project
# pip2.7 install virtualenv
# virtualenv -p /usr/local/bin/python2.7 venv

# python3.5 -m venv venv

Activate virtualenv
# source venv/bin/activate

Now install the SDK:
# pip2.7 install openstacksdk
# pip3.5 install openstacksdk

# Run the following test program ( as:
# python3.5 ./
# OR
# python2.7 ./

from openstack import connection
from openstack import utils
import logging
import sys

# Enable requests logging
logger = logging.getLogger('requests')
formatter = logging.Formatter(
    '%(asctime)s %(levelname)s: %(name)s %(message)s')
console = logging.StreamHandler(sys.stdout)

# Enable logging to console
utils.enable_logging(debug=True, stream=sys.stdout)

# Get connection
conn = connection.Connection(auth_url="http://localhost:5000/v2.0",

# Get network API
network ="watshnet")
if network is None:
    network ="watshnet")

If everything went fine so far then you are all set to begin writing your extensions for OpenStack. Examples using the sdk are available here -

Friday, October 02, 2015

Integrating Morphia and Dropwizard

Recently i had the pleasure of using mongodb's document object mapper framework morphia and the REST webservice framework dropwizard (well it is more than simply for REST but that is the primary purpose of it).

Dropwizard comes with the following essential thirdparty libraries:

  1. Jackson - for JSON parsing
  2. Jetty - as embedded light weight Java EE web container
  3. Jersey - JAX-RS 2.0 implementation
  4. and some more... 
Morphia has annotations for POJO model classes that can be persisted in mongodb and is a nice framework that simplifies the development of a mongodb client application. At the same time we can use the same POJO model classes for REST endpoint implementations as well. So there is no need to have separate DTO and entity classes. The same POJO model classes are annotated as entities (using morphia's annotations) and as DTO (using Jackson's annotations).

A simple example for such a model class is given below:

@Entity(value = "students" , noClassnameStored = true, concern = "SAFE")
public class Student extends BaseModel {
    @Id    private String id = new ObjectId().toHexString();

    @Reference    private User user;

    @Reference    private List courseRefs;

    Date lastUpdated = new Date();

    @PrePersist    void prePersist() {
        lastUpdated = new Date();

    public Student(User user) {
        this.user = user;
        this.courseRefs = new ArrayList<>();

    public Student() {

    @JsonProperty    public Date getLastUpdated() {
        return lastUpdated;

    public void setLastUpdated(Date lastUpdated) {
        this.lastUpdated = lastUpdated;

    @JsonProperty    public List getCourseRefs() {
        return courseRefs;

    public void setCourseRefs(List courseRefs) {
        this.courseRefs = courseRefs;

    @JsonProperty    public User getUser() {
        return user;

    public void setUser(User user) {
        this.user = user;

    @JsonProperty    public String getId() {
        return id;

    public void setId(String id) { = id != null ? new ObjectId(id).toHexString() : new ObjectId().toHexString();

    @Override    public String toString() {
        return "Student{" +
                "id=" + id +
                ", user=" + user +
                ", courseRefs=" + courseRefs +
                ", lastUpdated=" + lastUpdated +

Morphia annotations are:

Jackson annotations used are:

Additionally with Dropwizard we also get the bean validations through Hibernate Validator framework which can be used in the POJO above to annotate the fields in the model with @NotNull or @MinLength... etc.

One important learning in this integration was to map the mongodb _id (ObjectID BSON type) to a string id in the model but set it to the hex string value from the _id field. Getting this to work without the use of frameworks like MongoJack or Katharsis keeps the code simple to comprehend and yet meets the requirements.

Using VNC with Ubuntu on VirtualBox

  1. Create an Ubuntu 64 bit VM on VirtualBox with following specs:
    1. CPU - 1
    2. Memory - 1GB
    3. HDD - 20GB (VDI, Dynamic allocation)
    4. Network - NAT with Port forwarding - TCP, 5900 (on local host) forwarded to 5900 (on VM)
  2. Download Ubuntu desktop 15.04 and install it on the VM.
  3. Once installation completes, search for Desktop Sharing applet. 
  4. Enable sharing, provide password.
  5. Now launch a VNC viewer and specify the IP of the parent host on which VirtualBox is installed with port 5900.
    1. If connecting from same host where virtual box is installed then you may use localhost:5900. 
  6. It will prompt for password then connect.

Tuesday, September 22, 2015

AngularJS Introduction

AngularJS 1.5

Directive is a marker on an HTML tag that tells Angular to run or reference some Javascript code.

HTML page:
<html ng-app="studentModule">
    <link rel="stylesheet" type="text/css" href="css/bootstrap.min.css" />
    <script type="text/javascript" src="js/angular.min.js"></script>
    <script type="text/javascript" src="js/app.js"></script>
<body class="container" ng-controller="StudentController as s">
    <ul class="list group">
        <li class="list group item" ng-repeat="student in s.students">
                {{student.firstName}} {{student.lastName}}
                <em class="pull-right">{{student.emailId}}</em>
(function() {
  var app = angular.module('studentModule', []);

  app.controller('StudentController', ['$http', function($http){
    var store = this;
    store.students = [];

    $http.get("/api/students").success(function(data) {
        store.students = data;

Here ng-controller is the directive.

Modules: is where we write pieces of our Angular application and define dependencies for our app. Modules can use other modules (or depend on other modules).

Var app = angular.module('store', []) -- define a new module named 'store' and with no dependencies (so empty array) -- this goes into a separate app.js

-- ng-app directive will run the module named "store" when the document loads. Ng-app also makes the entire html page an angular application which means the contents within can now use the features of the angular framework - like writing expression such as:

I am {{4 + 6 }}

Will translate to:

I am 10

Similarly we can have string expressions: {{ "hello" + " world" }}

We can have filters in the expression by using pipe to get the o/p of expression as input of a filter and thus achieve filtered result:
{{ product.price | currency }}

Or in general: {{ data* | filter: options*}}





Sample 1:

Controllers: where we define our app's behavior.



The purpose of the controller is to get the JSON data (from gem object in this case) over to html page block where the ng-controller is defined. This way we can get the JSON data into a javascript client side object in some way (say by querying REST endpoint) and then populate the div block in the html page with the data - and this is facilitated by the controller's property ("product" in this case) which is set to JSON javascript object (gem in this case).


Built-in validations for common input types:
Email, url, number (max, min),


Monday, September 21, 2015
1:27 PM
Attach application module to a page
Attach a controller function to the page
Repeat a section of the page for each item in an array
Display or not display a section based on an expression
Will set to the classname when condition specified is true.
Ng-class="active:tab === 1"
Ng-model - binds the form elements to properties -- this is how we create a live preview.
Ng-submit - specify method to invoke when form is submitted.

Ng-include - include html snippet

Custom Directives - makes html more readable.. Same can be achieved using ng-include directive by including the duplicating snippet of html code from another html file. But custom directives will be like new tags -


Tuesday, September 22, 2015
12:02 AM
App.js - root module "store" is defined here that is referred in our single page app's html page (with ng-app directive). It will have dependencies on all other sub-modules.

var app = angular.module("store", ["products-directives"])

Products.js - can be another module consisting of custom directives (which also might encapsulate controllers) pertaining to products in the store.

And so on, there could be other modules which are referred by products module or by the main store module.

Tuesday, September 22, 2015
12:06 AM
Services give controllers additional functionality.
  1. Fetching JSON from webservice - $http
  2. Logging message for Javascript console - $log
  3. Filtering an array - $filter


$http({ method: 'GET',  url: '/products.json' });
$http.get('/products.json', {apiKey: 'myApiKey'… other query params});

Returns a promise object: .success() or .error()

$'/api/students',  {param: 'value'});


Monday, September 21, 2015

Flask introduction

Flask is a python microframework for building web applications or for rapid prototyping one.

sudo easy_install virtualenv
python3 -m venv venv --- create a venv directory for the project
source venv/bin/activate -- activate the virtual env.
pip install flask -- install flask

The idea for virtual environment is to import packages like flask only in a copy of the python
Virtual environment and not pollute the global python installation.

An example flask application:

from flask import Flask, render_template
app = Flask(__name__)

def index():
    return render_template('index.html')

def user(name):
    return render_template('user.html', name=name)

if __name__ == '__main__':


Hello, {{ name }}!


It will start the web server

It should o/p Hello, watsh in browser.

Sunday, September 20, 2015

Mac OSX: can't start Git - Intellij IDEA complains

The solution is: in terminal, run 'sudo xcodebuild -license' and then agree to the terms.

Friday, September 18, 2015


MongoDB is a NoSQL document-oriented database which stores data as BSON (Binary Script Object Notation) documents.

Salient features of MongoDB are:
  1. Document Oriented - aggregates the data in minimal number of documents.
  2. Ad hoc queries - like regular expression search, by ranges or field is supported.
  3. Indexing - any field in the document can be indexed.
  4. Replication - high availability is supported by maintaining replicas of data in more than one replica set member. The data is eventually consistent between the replica members.
  5. Load Balancing - uses sharding (a shard is a master with one or more slaves) to distribute the data split into ranges (based on shard key) between multiple shards.
  6. File storage - supports storing a file not as a single document but split across multiple shards - GridFS feature of MongoDB comes built in and is used by NGNIX and lighthttpd.
  7. Aggregation - is similar to SQL GROUP BY clause.
  8. Capped collections - can be used to store data in insertion order and once specified size is reached, behaves like a circular queue. Similar to RRD.
  9. Server side Javascript execution - is supported.

MongoDB is one of the top performing NoSQL database. Benchmarks have reported MongoDB performance to be better than some other NoSQL DBs by as much as 25x.
Eventual Consistency - eventually all nodes in the cluster of NoSQL DB will have the same data as data may not be propagated if the network breaks down or the node goes down but eventually when node is up and the network is working then data will be consistent across all shards.
1. Collections - set of documents
2. Document - BSON document with dynamic schema - means that documents in the same collection do not need to have the same set of fields or structure, and common fields in a collection's documents may hold different types of data.
Important mongo shell commands:
>show dbs
>show collections
>{"name":"tutorials point"}) --- will insert a movie document in movie collection.
>db.createCollection("mycol", { capped : true, autoIndexID : true, size : 6142800, max : 10000 } )
RDBMS Where Clause Equivalents in MongoDB
To query the document on the basis of some condition, you can use following operations
OperationSyntaxExampleRDBMS Equivalent
Equality{:}db.mycol.find({"by":"tutorials point"}).pretty()where by = 'tutorials point'
Less Than{:{$lt:}}db.mycol.find({"likes":{$lt:50}}).pretty()where likes < 50
Less Than Equals{:{$lte:}}db.mycol.find({"likes":{$lte:50}}).pretty()where likes <= 50
Greater Than{:{$gt:}}db.mycol.find({"likes":{$gt:50}}).pretty()where likes > 50
Greater Than Equals{:{$gte:}}db.mycol.find({"likes":{$gte:50}}).pretty()where likes >= 50
Not Equals{:{$ne:}}db.mycol.find({"likes":{$ne:50}}).pretty()where likes != 50
AND condition:
>db.mycol.find({key1:value1, key2:value2}).pretty()
OR condition:
      $or: [
      {key1: value1}, {key2:value2}
'where likes>10 AND (by = 'tutorials point' OR title = 'MongoDB Overview')'
>db.mycol.find("likes": {$gt:10}, $or: [{"by": "tutorials point"}, {"title": "MongoDB Overview"}] }).pretty()
>db.mycol.update({'title':'MongoDB Overview'},{$set:{'title':'New MongoDB Tutorial'}})

Replace old document with new data:
      "_id" : ObjectId(5983548781331adf45ec7), "title":"Tutorials Point New Topic", "by":"Tutorials Point"
>db.mycol.remove({'title':'MongoDB Overview'})
>db.COLLECTION_NAME.remove(DELETION_CRITERIA,1) -- removes the first match
>db.mycol.remove() - remove all documents in the collection.

Projection: selectively display fields and not all fields of a document.
>db.mycol.find({},{"title":1,_id:0}) -- will display title and not _id (which is always displayed otherwise)
Sort: 1 is used for ascending order while -1 is used for descending order.
Index: Here key is the name of filed on which you want to create index and 1 is for ascending order. To create index in descending order you need to use -1.
> db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$sum : 1}}}])
select by_user, count(*) from mycol group by by_user -- how many tutorials are written by each user (grouped by a user)

There is a list available aggregation expressions.
$sumSums up the defined value from all documents in the collection.db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$sum : "$likes"}}}])
$avgCalculates the average of all given values from all documents in the collection.db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$avg : "$likes"}}}])
$minGets the minimum of the corresponding values from all documents in the collection.db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$min : "$likes"}}}])
$maxGets the maximum of the corresponding values from all documents in the collection.db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$max : "$likes"}}}])
$pushInserts the value to an array in the resulting document.db.mycol.aggregate([{$group : {_id : "$by_user", url : {$push: "$url"}}}])
$addToSetInserts the value to an array in the resulting document but does not create duplicates.db.mycol.aggregate([{$group : {_id : "$by_user", url : {$addToSet : "$url"}}}])
$firstGets the first document from the source documents according to the grouping. Typically this makes only sense together with some previously applied “$sort”-stage.db.mycol.aggregate([{$group : {_id : "$by_user", first_url : {$first : "$url"}}}])
$lastGets the last document from the source documents according to the grouping. Typically this makes only sense together with some previously applied “$sort”-stage.db.mycol.aggregate([{$group : {_id : "$by_user", last_url : {$last : "$url"}}}])
Replication: client application always interact with primary node and primary node then replicate the data to the secondary nodes.
  • All write operations goes to primary
  • Reads can happen from any of primary or secondary nodes.
  • Replication - is keeping multiple copies of the same data for HA and failover. One node is primary and others are secondary. Minimum 3 nodes needed to form a replica set.
  • Writes are always written to primary node in a replica set.
  • Within the replica there will be some delay when the writes to the primary node gets replicated to the secondary. In your application you may want to wait for the writes to be replicated. This is controlled by w flag - or write to replica flag. This is set to w:1 by default in drivers.
  • Mongodb keeps the data in memory and flushes it to disk periodically. If application wants to wait for the data to be written to disk then it can set J:1 (where, j is journal written to disk). W:1, J:1  is default for drivers.
  • Another setting is w:majority, which means write should propagate to majority nodes in the replica set. These w and j settings are called write concerns.
  • An application can set its read preference to read from secondary. Secondary's data may be stale so this is not recommended.

  • Following is how mongodb replica set is created:

    mkdir -p /data/rs1 /data/rs2 /data/rs3
    mongod --replSet m101 --logpath "1.log" --dbpath /data/rs1 --port 27017 --smallfiles --oplogSize 64 --fork

    mongod --replSet m101 --logpath "2.log" --dbpath /data/rs2 --port 27018 --smallfiles --oplogSize 64 --fork
    mongod --replSet m101 --logpath "3.log" --dbpath /data/rs3 --port 27019 --smallfiles --oplogSize 64 --fork
    mongo --port 27017
    Now you will create the replica set. Type the following commands into the mongo shell:
    config = { _id: "m101", members:[
              { _id : 0, host : "localhost:27017"},
              { _id : 1, host : "localhost:27018"},
              { _id : 2, host : "localhost:27019"} ]


    Ps -ef | grep mongod -- will show all the replica sets created on the localhost.
Setting up replica set:
mongod --port "PORT" --dbpath "YOUR_DB_DATA_PATH" --replSet "REPLICA_SET_INSTANCE_NAME"
Backup/Restore MongoDB data: 

Sharding - is a solution for horizontal scaling of mongodb. More shards (which are in turn made up of replica sets) can be added depending on the load on the system. A shard key needs to be an indexed key in the collection and should be present in all documents. The key need not be unique. It is used to identify the right shard to send the data to for persistence. Within the shard, replica sets will create copies of the data on replica set member nodes. So sharding helps in splitting the data based on a shard key in the document. Its like storing data in a hashmap. The better the key selection, the better the data will be divided among the shards.

Sharding is controlled via mongos router. Application connects to the mongos router which will listen to the 27017 port for example, and will know based on the shard key on which shard to insert the data to. For read/find operation mongos will query the primary node (of the replica set) in each shard and collate the result.

  1. Easy to setup and develop against in multiple languages, like Java, python etc. Good to build POC applications.
  2. Stable enough.
  3. Querying is very powerful.
  4. Pretty performant in querying and inserts. If the schema has embedded documents mostly then it is worth using it.
  5. JSON structures can model quite complex objects.
  6. Schema-less DB can be useful (at least it appears to be so in theory) as it can reduce the pain in migration, though you still need to write the migration scripts nevertheless but not have to worry about alter-ing the schema (as there is no DDL or schema definition in MongoDB, all schema gets realized at runtime and documents within the same collection can be quite dissimilar).


  1. By design, no referential integrity is supported:
    1. So no cascade deletion - we need to handle it in application.
  2. Very hard to design with only embedded documents in the schema. Most of the time we end up with having documents with References or links which is akin to relations in the RDBMS. This comes at a price that to build a transaction with rollback is too much work in the application. MongoDB only guarantees atomicity within a document's boundary. So when you have too most documents having references instead of embedded documents in them then consider using RDBMS.
  3. Hard to model many-to-many relations.