Sunday, September 30, 2012

Free Private Git Hosting

Today I have spent some time looking for Git hosting for my new pet project. Last time I did such kind of research about three years ago. I was looking for SVN hosting, and I ended up with Unfuddle.
Currently they do not offer a free option any more, and my interests are shifted to Git. My current requirements are very simple:
  • functional free starter option, with no-hassle switch to payed services when needed;
  • both ssh and https access to repository;
  • tracking/wiki option (low priority);
  • private hosting - no requirement to open-source your code like on github;
This is my shortlist:
I had to drop both ProjectLocker and  Springloops because of no https support for Git (only ssh is available). I open accounts on both Assembla and CloudForge. Both of them do not have Trac or any other bugtracking/wiki option for free, but it is not critical on this stage. Registration on Assembla was smooth, in a couple of minutes I committed and pushed a sample project and was able to see it from web ui. Another system (CloudForge) had some glitch during registration: the confirmation email came into my mailbox with about 15-hour delay. During this period the admin module has been annoying me about unfinished registration every time I logged in, asking to provide a confirmation number in a special input field (otherwise my account will be disabled in 24 hours). When email has finally arrived, there was no number, only a registration link to click.
I decided to stay with Assembla for now, but keep an eye on CloudForge in the same time. I like their Publisher option - ability to configure your deployment for Joyent, Amazon WS, Salesforce, Google App Engine, CloudFoundry of custom FTP or SSH option. I had not tried it yet, but from description in sounds useful and time-saving.

Update (next day): I've tried BitBucket this morning, and I think it fits pretty well into all my requirements: free, private, https, wiki/issues tracking, web access, integration with other systems. It allows to get more than one user account to the repository, which also might be useful. I'll stay with BitBucket for now.

Wednesday, September 12, 2012

Teiid and HornetQ Integration

In my previous article "Simpe JMS Provider on JBoss Teiid" you can learn about how to create a Teiid translator which will "convert" your SQL commands into messages and put them into queue. This article talks about opposite approach: how to retrieve messages from queue using SQL SELECT statements, so our integration pattern will be complete. The schema below explains the integration logic.
Our goal is to pass a specific command (CALL... or INSERT...) to Teiid with some parameters. This parameters will be converted in Teiid by our custom translator (JMS Producer) into the message, which will be immediately placed into queue. New player is another custom translator (JMS Consumer). Client can issue a Continuous Execution to Teiid, which will automatically repeat itself with some time interval. Each round of execution will retrieve messages from queue and pass back to client as execution results. Client gets notified by Teiid when result is available - client obtains a callback object from Teiid when the execution initiated. The StatementCallback.onRow() method gets executed when data is passed back to client.
The process can be further improved, if instead of retrieving messages periodically we will listen queue and retrieve message back to client right after it will appear. So instead of polling we will implement pushing approach.

Implementation Details

The custom translator code for JMS Consumer should implement MessageListener interface. The provided onMessage(Message msg) method will be called each time the message will appear on subscribed queue. Translator also implements ResultSetExecution (so it will appear as a view in Teiid Virtual Database), and ReusableExecution interfaces. The DataNotAvailableException.NO_POLLING will be thrown from the translator next() method, to put translator "into sleep". This is a non-blocking process and part of Continuous Execution functionality. To bring translator back to life, we will call ExecutionContext.dataAvailable() from onMessage() method.
The complete code of JMS Consumer Translator (MessageReceiveExecution.java) is attached, and below are the most important methods:
      public void execute() throws TranslatorException {  
           this.callNext = DataNotAvailableException.NO_POLLING;  
           this.callNext.setStrict(true);  
      }  
      public List<?> next() throws TranslatorException, DataNotAvailableException {  
           if (callNext != null) {  
                DataNotAvailableException e = callNext;  
                callNext = null;  
                throw e;  
           }  
           if (msgRow != null) {  
                List<Object> results = msgRow;  
                msgRow = null;  
                return results;  
           }  
           return null;  
      }  
      public void dispose() {  
           try {  
                qreceiver.close();  
                qsession.close();  
                qcon.close();  
           } catch (JMSException e) {  
                // TODO Auto-generated catch block  
                e.printStackTrace();  
           }  
      }  
      public void onMessage(Message msg) {  
           try {  
                String messageContent;  
                if (msg instanceof TextMessage) {  
                     messageContent = ((TextMessage) msg).getText();  
                } else {  
                     messageContent = msg.toString();  
                }  
                System.out.println("!!!!!! MSG:" + messageContent);  
                setMessage(messageContent);  
           } catch (JMSException jmse) {  
                jmse.printStackTrace();  
           }  
           this.context.dataAvailable();            
      }  

To make sure it works I used a JBoss with HornetQ configuration from the previous article, and made a sample java servlet which connects to Teiid and executes a SELECT statement continuously. Also I have another SQL client application (SQuirreL or Eclipse Database Explorer) connected to Teiid, which I use to produce messages (CALL JMS Producer, like in my previous article).
Testing approach:
  • start JBoss/HornetQ/Teiid with deployed translator jar module and war file with servlet;
  • open browser, call servlet to issue a continuous execution;
  • run call statement from SQL client to send a message;
  • check servlet logs that message successfully retrieved by translator and passed back to servlet;
 MessageReceiveExecution (Google Docs).

Monday, September 3, 2012

Simple JMS Producer on JBoss Teiid

With it's Continuous Execution feature, new Teiid 8.1 came closer to functionality which reminds me SQL Server Service Broker. In particular, it might be possible to write Teiid extensions which can serve as integration points between relational data from VDB and asynchronous data available in message-oriented middleware.
For example, Teiid translator can be a JMS client which can send messages into some queue. Another Teiid translator running in Continuous Execution mode can retrieve messages from queue.
Below is results of my experiments with first approach: to have a stored procedure which will act as JMS producer.

Configure JMS in JBoss

The  JBoss 7.1.1 has built-in JMS support with HornetQ, so it is only a matter of properly editing config files. The HornetQ stuff included in standalone-full.xml profile, and Teiid configuration is in standalone-teiid.xml, so I took simplest approach and merged all JMS-related parts from -full.xml into *-teiid.xml one. Also I defined a test queue "MyQueue":
 ....  
      <address-setting match="jms.queue.MyQueue">  
        <dead-letter-address>jms.queue.MyDLQ</dead-letter-address>  
        <redelivery-delay>0</redelivery-delay>  
        <max-delivery-attempts>3</max-delivery-attempts>  
        <max-size-bytes>10485760</max-size-bytes>  
        <address-full-policy>BLOCK</address-full-policy>  
        <message-counter-history-day-limit>10</message-counter-history-day-limit>  
      </address-setting>  
 ....  
  <jms-destinations>  
    <jms-queue name="MyQueue">  
      <entry name="java:jboss/exported/MyQueue"/>  
      <durable>false</durable>  
    </jms-queue>  
 ....  


It is worth to write a small producer and consumer java classes to test the queue and make sure all configured correctly. The JBoss Management console and this post might be helpful.

Write Teiid Translator

I made a simple translator which is a stored procedure, implements the ProcedureExecution and acts as a JMS producer. Queue name and message passed as stored procedure IN parameters.

 package com.rokhmanov.test.teiid.translator.async;  
   
 import java.util.ArrayList;  
 import java.util.List;  
   
 import javax.jms.JMSException;  
 import javax.jms.Queue;  
 import javax.jms.QueueConnection;  
 import javax.jms.QueueConnectionFactory;  
 import javax.jms.QueueSender;  
 import javax.jms.QueueSession;  
 import javax.jms.Session;  
 import javax.jms.TextMessage;  
 import javax.naming.Context;  
 import javax.naming.InitialContext;  
 import javax.naming.NamingException;  
   
 import org.teiid.language.Argument;  
 import org.teiid.language.Call;  
 import org.teiid.translator.DataNotAvailableException;  
 import org.teiid.translator.ExecutionContext;  
 import org.teiid.translator.ProcedureExecution;  
 import org.teiid.translator.TranslatorException;  
   
 public class MessageSendExecution implements ProcedureExecution {  
        
      public final static String CNN_FACTORY="/ConnectionFactory";  
        
      private String queueName;  
      private String messageContent;  
      private Call cmd;  
        
      private QueueConnectionFactory qconFactory;  
      private QueueConnection qcon;  
      private QueueSession qsession;  
      private QueueSender qsender;  
      private Queue queue;  
      private TextMessage msg;  
        
      public MessageSendExecution(Call command,  
                ExecutionContext executionContext, Object connection) {  
           this.cmd = command;  
             
     final List<Argument> procArguments = new ArrayList<Argument>(this.cmd.getArguments());  
     this.queueName = procArguments.get(0).getArgumentValue().getValue().toString();  
     this.messageContent = procArguments.get(1).getArgumentValue().getValue().toString();   
      }  
   
      public List<?> next() throws TranslatorException, DataNotAvailableException {  
           return null;  
      }  
   
      public void close() {  
           // TODO Auto-generated method stub  
      }  
   
      public void cancel() throws TranslatorException {  
           try {  
                qcon.close();  
           } catch (JMSException e) {  
                throw new TranslatorException(e);  
           }  
      }  
   
      public void execute() throws TranslatorException {            
           InitialContext ic = getInitialContext();  
           initJMS(ic, queueName);  
           sendMsg(messageContent);  
      }  
   
      public List<?> getOutputParameterValues() throws TranslatorException {  
           return null;  
      }  
   
        
      private InitialContext getInitialContext() throws TranslatorException  
      {  
           try {  
                return new InitialContext();  
           } catch (NamingException e) {  
                throw new TranslatorException(e);  
           }  
      }  
        
        
      public void initJMS(Context ctx, String queueName) throws TranslatorException  
      {  
           try {  
                qconFactory = (QueueConnectionFactory) ctx.lookup(CNN_FACTORY);  
                qcon = qconFactory.createQueueConnection();  
                qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);  
                queue = (Queue) ctx.lookup(queueName);  
                qsender = qsession.createSender(queue);  
                msg = qsession.createTextMessage();  
                qcon.start();  
           } catch (NamingException e) {  
                throw new TranslatorException(e);  
           } catch (JMSException e) {  
                throw new TranslatorException(e);  
           }  
      }  
        
        
      private void sendMsg(String userMessage) throws TranslatorException  
      {  
           try {  
                msg.setText(userMessage);  
                qsender.send(msg);  
                qcon.close();  
           } catch (JMSException e) {  
                throw new TranslatorException(e);  
           }  
      }  
 }  
   

This is the important parts from the Execution Factory:

      @Override  
      public ProcedureExecution createProcedureExecution(Call command,  
                ExecutionContext executionContext, RuntimeMetadata metadata,  
                Object connection) throws TranslatorException   
      {  
           return new MessageSendExecution(command, executionContext, connection);  
      }  
   
        
      @Override  
      public void getMetadata(MetadataFactory metadataFactory, Object conn)  
                throws TranslatorException   
      {      
     final Procedure messageSender = metadataFactory.addProcedure("messageSender");  
     metadataFactory.addProcedureParameter("queue", TypeFacility.RUNTIME_NAMES.STRING, Type.In, messageSender);  
     metadataFactory.addProcedureParameter("message", TypeFacility.RUNTIME_NAMES.STRING, Type.In, messageSender);  
      }  
   

To post the message, use this SQL statement:

 call Times.messageSender('java:jboss/exported/MyQueue', 'My Message');  

Then monitor JBoss Management console or use your sample java JMS client to retrieve a messsage from queue:

 Sep 03, 2012 10:34:45 PM org.xnio.Xnio <clinit>  
 INFO: XNIO Version 3.0.3.GA  
 Sep 03, 2012 10:34:45 PM org.xnio.nio.NioXnio <clinit>  
 INFO: XNIO NIO Implementation Version 3.0.3.GA  
 Sep 03, 2012 10:34:45 PM org.jboss.remoting3.EndpointImpl <clinit>  
 INFO: JBoss Remoting version 3.2.3.GA  
 JMS Ready To Receive Messages (To quit, send a "quit" message from QueueSender.class).  
   
       My Message  

Note: this translator code is not optimized for performance or thread-safety, I wrote it just for illustration purposes. Areas of potential improvement:
  • implement UpdateExecution to make a translator handle INSERT SQL statement (more intuitive approach);
  • reuse connection to Queue;