tag:blogger.com,1999:blog-36825025650342704912024-03-21T09:02:35.672-05:00Useful Bits and BytesAndriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-3682502565034270491.post-30785209273876679542017-01-10T16:54:00.001-06:002017-01-12T11:59:48.716-06:00Amazon Echo for Devops<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0001158/" style="color: #70579d; text-decoration: none;"><b>Dave Bowman</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: Hello, HAL. Do you read me, HAL? </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0706937/" style="color: #70579d; text-decoration: none;"><b>HAL</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: Affirmative, Dave. I read you. </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0001158/" style="color: #70579d; text-decoration: none;"><b>Dave Bowman</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: Open the pod bay doors, HAL. </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0706937/" style="color: #70579d; text-decoration: none;"><b>HAL</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: I'm sorry, Dave. I'm afraid I can't do that. </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0001158/" style="color: #70579d; text-decoration: none;"><b>Dave Bowman</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: What's the problem? </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0706937/" style="color: #70579d; text-decoration: none;"><b>HAL</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: I think you know what the problem is just as well as I do. </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0001158/" style="color: #70579d; text-decoration: none;"><b>Dave Bowman</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: What are you talking about, HAL? </span></span><br />
<span style="font-family: "verdana" , sans-serif;"><i style="background-color: white; color: #333333; font-size: 13px;"><a href="http://www.imdb.com/name/nm0706937/" style="color: #70579d; text-decoration: none;"><b>HAL</b></a></i><span style="background-color: white; color: #333333; font-size: 13px;">: This mission is too important for me to allow you to jeopardize it.</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">For those who is not familiar with the quote above - this is from a Stanley Kubrick's<b> "<a href="http://www.imdb.com/title/tt0062622/">2001: A Space Odyssey</a>"</b> masterpiece, a famous dialog between the astronaut Dr Dave Bowman and an artificially-intelligent HAL 9000 computer. </span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">Taking apart the controversy around the HAL's evil behavior, the whole idea of having the voice-controlled smart helper would be extremely beneficial. Especially if you are an IT individual spending your day with system administration tasks or support and maintenance activities.</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">I am not talking about the lip reading, automated reasoning, interpreting emotional behaviors or other smarts which still distinguish the real person from their personal computer. Even basic reporting of system vital characteristics (like memory or storage usage) might be a small step or a huge leap, depends of how busy your hands (and how cool you want to look between your office colleagues). </span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; font-size: 13px;">With the appearance of <a href="https://developer.amazon.com/alexa">Amazon Echo</a> building the smart helper became a trivial task - develop a custom skill for Amazon Echo which will represent our virtual helper. The Amazon Echo activated by voice command "Alexa". I</span></span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"> followed the Hollywood theme and called our little helper "Skynet".</span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"> My</span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"> goal was to have a voice interaction like the one below:</span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; font-size: 13px;"><br /></span></span>
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; font-size: 13px;"><i style="color: black; font-family: Verdana, sans-serif;"><span style="color: #0b5394;"><b>You</b></span></i><span style="color: #333333; font-family: "verdana" , sans-serif;">: Alexa, ask Skynet what servers are supported?</span></span></span><br />
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; font-size: 13px;"><i style="color: black; font-family: Verdana, sans-serif;"><span style="color: #0b5394;"><b>Alexa</b></span></i><span style="color: #333333; font-family: "verdana" , sans-serif;">: Supported are server1, server2 and server3. </span><span style="font-family: "verdana" , sans-serif;"><br /></span></span></span>
<span style="background-color: white; font-family: "verdana" , sans-serif; font-size: 13px;"><i><span style="color: #0b5394;"><b>You</b></span></i><span style="color: #333333;">: Alexa, ask Skynet what is the disk usage on server2 ? </span></span><br />
<span style="background-color: white; font-family: "verdana" , sans-serif; font-size: 13px;"><i><span style="color: #0b5394;"><b>Alexa</b></span></i><span style="color: #333333;">: The root mount on server2 is occupied on 74 percent.</span></span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">To make this happen, obviously I need to have an Amazon Echo (or Echo Dot), the <a href="https://developer.amazon.com/alexa">Amazon Developer account</a> and <a href="https://developer.amazon.com/alexa-skills-kit">Alexa Skills Kit</a>. The most important is to find a way to retrieve the actual disk usage on specified server. Most of my servers are AWS EC2, and one of them has the <a href="http://rundeck.org/">Rundeck</a> instance. I have a Job defined in Rundeck, which once executed, takes the server name as a parameter, and runs the "disk usage" shell script on that server (basically, the df -h). The result of script execution passed back as output. Rundeck has a REST API, which allows me to execute the Job remotely as a service and obtain the JSON output with results.</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">The actual Skill development is trivial, I took one of the Java examples from Alexa Skills Kit as a basis and added some REST-calling and result-parsing functionality. Then I created new AWS Lambda definition in AWS us-east-1 region and configured Alexa Skills Kit as a trigger. Lambda has to be configured with VPC support, you might need to supply a proper Role to it and depends of your network configuration add specific Subnets and Security Groups. The new Skill has to be added on Amazon Developer under "Alexa" section, also nothing specific - all steps explained in Alexa Skills Kit step-by-step guide. I did not publish my Skill to public, since I did not want to let everybody in the world to figure out how much space is available on my servers. My Skill stays at "Test" stage in Amazon Developer console, this way only me (or everyone in the room where the actual Echo installed) can use this functionality.</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">The most interesting part was defining the actual voice dialog. See the Sample Utterances and Intent Schema below. </span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;"><br /></span>
<span style="background-color: white; color: #333333; font-size: 13px;"><span style="font-family: "courier new" , "courier" , monospace;">IntentSchema.json:</span></span><br />
<span style="background-color: white; color: #333333; font-size: 13px;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span></span>
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;">{</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intents": [</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intent": "DiskUsageIntent",</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "slots": [</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "name": "server",</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "type": "LIST_OF_SERVERS"</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> }</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> ]</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> },</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intent": "SupportedServersIntent"</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> },</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intent": "AMAZON.HelpIntent"</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> },</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intent": "AMAZON.StopIntent"</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> },</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> {</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> "intent": "AMAZON.CancelIntent"</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> }</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;"> ]</span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; font-size: 13px;"></span></span><br />
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 13px;">}</span></span><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SampleUtterances.txt:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent get disk usage for {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent get the disk usage for {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent get a disk usage for {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent what is the disk usage on {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent disk usage {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DiskUsageIntent {server}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent what servers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent what servers are supported</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which servers are supported</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which servers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which servers do you know</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent what computers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent what computers are supported</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which computers are supported</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which computers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SupportedServersIntent which computers do you know</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">As you can see, the logic and implementation is very simple. We are still quite apart from the actual HAL or Skynet functionality, no need (yet) to worry much about the AI takeover or machine dominance. </span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif; font-size: 13px;">The Alexa Skill code with my modifications <a href="https://github.com/rokhmanov/skynet-alexa/">can be found on Github</a>, so the future of humankind is in your hands.</span>Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-47519431665981049912015-03-01T13:36:00.001-06:002015-03-01T13:36:44.867-06:00Multichannel Professional Data Logger on Raspberry Pi - Part 3See the <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger_1.html">Part 1</a> for the overall description and <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger.html">Part 2</a> for the hardware implementation.<br />
<br />
<h3>
Software</h3>
<br />
The software for Data Logger based on a super handy <a href="https://code.google.com/p/webiopi/">WebIOPi</a> Internet of Things framework. I would suggest to check their website for more details. We will use version 0.7 of the framework.<br />
The WebIOPi basically consists from a python-based web server, a set of native libraries to communicate with Raspberry Pi and REST interface. Additionally, the WebIOPi supports a variety of analog and digital converters, port expanders and sensors. Installed "from the box", it would allow you to communicate with Raspberry Pi from web, update and retrieve data from your devices, and code a basic UI using the javascript extensions and python macros. Also it has a friendly Apache 2.0 license. Our Data Logger custom UI will rely on WebIOPi REST interface to periodically retrieve data from our 16 ports over two ADC devices. We will have a set of python macros to supply the proper REST endpoints.<br />
<br />
I will omit all WebIOPi installation details (see their website). From the WebIOPi config standpoint, we need to configure two GPIO ports to automatic pull-up in OUT mode. This two ports will be used in low-power detection logic. The state of this two ports will be periodically analyzed, and if port 18 is "0" - the Data Logger will start playing an annoying alert beeps (means the battery is low), and right after the port 23 will get to "0" (means the battery is almost dead) - the Data Logger will loudly announce the "initiate of shutdown sequence" and power off.<br />
Also we need to configure and enable two ADC devices. The reference voltage is 2.5 V.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[GPIO]<br /># Initialize following GPIOs with given function and optional value<br /># This is used during WebIOPi start process<br />18 = OUT 2<br />23 = OUT 2</span><br />
<span style="font-family: "Courier New",Courier,monospace;">...</span><br />
<span style="font-family: "Courier New",Courier,monospace;">... </span><br />
<span style="font-family: "Courier New",Courier,monospace;">[DEVICES]</span><br />
<span style="font-family: "Courier New",Courier,monospace;">adc0 = MCP3008 chip:0 vref:2.5<br />adc1 = MCP3008 chip:1 vref:2.5</span><br />
<br />
We will supply our own custom python script which runs by WebIOPi framework and register it in the WebIOPi config in "[SCRIPTS]" section. The script implements several mandatory methods: setup(), destroy() and loop() which are called by WebIOPi at script loading, at server shutdown, or periodically. Also we have to add in this script a several REST endpoints as a python macros (annotated as @webiopi.macro) in code:<br />
<ul>
<li>getAvailableSpace() - return a free space in MB on Raspberry Pi;</li>
<li>toC() - serialize a user configuration settings on file system;</li>
<li>getDate() - return a local date and time on Raspberry Pi;</li>
<li>getTempSimple() - return a current set of values from Temperature ADC in JSON format;</li>
<li>getVISimple() - return a current set of values from Voltage/Current ADC in JSON format;</li>
<li>startRead() - enable reading and serialization of values from both ADCs;</li>
<li>stopRead() - disable reading and serialization;</li>
</ul>
The REST endpoints were called by <a href="http://jquery.com/">jQuery</a> handlers on UI side. Below is a screenshot of Configuration screen. Notice the familiar <a href="http://getbootstrap.com/">Bootstrap</a> look-and-feel.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhnv0rv1wX6fwvCUCCP4-XusKUHkvBfrnLqKhuL4C0pTbw0ifqfy3G2ogcTT73hkIrZUcsFGqA2UQP9kUI1Bexg_oU8vm7h8XmQGCZ850AK1jpPCrHo6sVg_QgpFW9eFk6UClCKBsHXbA/s1600/rash_configuration.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhnv0rv1wX6fwvCUCCP4-XusKUHkvBfrnLqKhuL4C0pTbw0ifqfy3G2ogcTT73hkIrZUcsFGqA2UQP9kUI1Bexg_oU8vm7h8XmQGCZ850AK1jpPCrHo6sVg_QgpFW9eFk6UClCKBsHXbA/s1600/rash_configuration.png" height="219" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghpHLRBqtLvZgNOOk3MG6XhnwBRO2r27ozIB_B7IgbcyvFdk0vpI4ItGmeal6TU_pT2hxT6fjnmM3WrUL72mf7wy18dNLD9jQC1E7gWDRk8wB-knbQ037tqbnOrXFc3p53uPO9JRz0-zc/s1600/rash_status.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
The UI includes two base screens - Configuration and Status (below). On the Configuration screen you can select the types and enter the names of your input ports (two sections: Temperature Channels and V & I Channels), specify "Recording Mode". It means mostly how long you would like the Data Logger to run - until data storage fills up or until trigger signal received on GPIO port (currently disabled). The data store overfill is very hard to achieve, we had Data Logger with common 4GB SD card running successfully for several days with the most aggressive settings. Another option is "Sampling Interval" - the number of seconds between data collection events (1...60). The last section is "Start Mode" - you can start the Data Logger immediately after saving the settings, by specifying a date and time, or manually by selecting a menu option on Status screen.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjT8EdZaLx0uGqlRiBZU-xwrIkbyr7HZEAtE4jA8wJlz_C43yWWZphMjuANSivcKP2V82rPHUaaztkSI0MSyIXHSaKp-LHK201Hw26fsYFUuW9iwsda0vuZi3lzAiIEDOCVihNjM7gc63M/s1600/rash_status.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjT8EdZaLx0uGqlRiBZU-xwrIkbyr7HZEAtE4jA8wJlz_C43yWWZphMjuANSivcKP2V82rPHUaaztkSI0MSyIXHSaKp-LHK201Hw26fsYFUuW9iwsda0vuZi3lzAiIEDOCVihNjM7gc63M/s1600/rash_status.png" height="219" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhnv0rv1wX6fwvCUCCP4-XusKUHkvBfrnLqKhuL4C0pTbw0ifqfy3G2ogcTT73hkIrZUcsFGqA2UQP9kUI1Bexg_oU8vm7h8XmQGCZ850AK1jpPCrHo6sVg_QgpFW9eFk6UClCKBsHXbA/s1600/rash_configuration.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
The Status screen is what you will be looking on the most of your time. It displays a mesmerizing live feed of data from sensors with the intervals previously defined on Configuration screen. The graph is based on <a href="http://d3js.org/">d3 framework</a> , with the <a href="http://code.shutterstock.com/rickshaw/">Rickshaw</a> JavaScript toolkit on top - to make a graph coding simpler (for example note the roller on the graph horizontal axis, this is just three lines of code). The table grid is also represent a live data, channels can be filtered out by Search field - thanks to the <a href="http://www.datatables.net/">DataTables</a> JQuery plugin.<br />
<br />
The menu system on Status screen is very simple. The Export option allows you to download the CSV file with the collected data. Note that when a configuration gets updated on Configuration screen, the CSV files are rewritten, so you always get a latest data. If you need to keep the recording results for future - make sure that you export the data before updating configuration.<br />
If on Configuration screen for Starting Mode you selected "Start Menu on Status Screen" (like on the screenshots), you will get one additional menu option named "Action". There you can pick "Start" or "Stop" option to control the logging process manually.<br />
<br />
The last moment worth to mention is that the Data Logger also works as a Wireless Router and Access Point, since the easiest way to get the data from it is to use Web UI over Wi-Fi. For the detailed instructions I'll redirect to the <a href="http://www.daveconroy.com/using-your-raspberry-pi-as-a-wireless-router-and-web-server/">nice article</a> which I followed with a small variations. The Raspberry Pi is a regular Linux box, so basically it should have a DHCP server, Wireless Access Point, Routing and IP-Forwarding services configured. With a proper (Realtek-based) Wi-Fi dongle it work flawlessly in the field, and can provide internet to your tablet, laptop or phone in lab if you plug Ethernet cable from your cable or modem in the Raspberry Pi jack.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com3tag:blogger.com,1999:blog-3682502565034270491.post-63009550093235937742015-03-01T13:36:00.000-06:002015-03-20T14:32:58.360-05:00Multichannel Professional Data Logger on Raspberry Pi - Part 2See the <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger_1.html">Part 1</a> for the overall description and <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger_81.html">Part 3</a> for the software implementation.<br />
<br />
<h3>
Hardware</h3>
<br />
The most interesting part to deal was the hardware. Below you can see an old prototype board, which includes a Raspberry Pi, two custom boards for ADC, sensor connectors and voltage dividers. In the middle - a small board to supply a power. It has real-time input voltage monitoring: on 7.5V - the Data Logger start making warning sounds, on 7.0 V - automatic shutdown. Two power supplies: AC-DC external, and internal pack batteries. Also the embedded <a href="http://www.ti.com/lsds/ti/power-management/oring-controller-products.page?paramCriteria=no">ORing controller</a> makes a smooth transition between battery and external power supply.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjknh5N1XvZAijDJsvgVJm09v8fwPs5n5e11AP1Ue8B9ALe_kVM-2h_xrXKisjqsMEfj7lfZE2wbaaCR59c55GAhMS3P3PMf3aga5hzuBr_Culdt5h8pL7WvQRoQOtf5pY1ayfuXbOAoPU/s1600/protoboard.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjknh5N1XvZAijDJsvgVJm09v8fwPs5n5e11AP1Ue8B9ALe_kVM-2h_xrXKisjqsMEfj7lfZE2wbaaCR59c55GAhMS3P3PMf3aga5hzuBr_Culdt5h8pL7WvQRoQOtf5pY1ayfuXbOAoPU/s1600/protoboard.jpg" height="233" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
The boards has been designed and implemented by Boris Shogan, my old friend and awesome Electronics Engineer. It took a noticeable amount of collaborative work in our spare time and midnight oil burnt to make this happen. But it was a plenty of fun. <br />
<br />
Since we need 16 ports total, the natural choice was to take two <a href="http://www.microchip.com/wwwproducts/Devices.aspx?dDocName=en010530">MCP3008</a> 8-Channel 10-Bit ADC with SPI interface, and have it on two standalone boards (a compact 16-port single board which hosts both ADC is also available). <br />
<br />
Originally we had it working on Raspberry Pi Model B (512 MB RAM),
and later used the Model B+. We did not tried the Raspberry Pi 2, it was
released just a few days ago.<br />
<br />
Below is the IV-board for Currency and Voltage. The <b>low-side</b> external Shunt or Sense resistors are plugged to the blue connectors: <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQpr0xapKA-STqxD9l45j2yzpECiGg1fTo4NXi_PEOAtFfOp9F9N2QXdNvJ-th3bcJfm7VbzpfCVqKXU2aYn1rspWAtBTGxojIEbYNxnARRt6_7efe6Xx7NqmB1OC2cxZdsqqEdfz7iHY/s1600/iv_board.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQpr0xapKA-STqxD9l45j2yzpECiGg1fTo4NXi_PEOAtFfOp9F9N2QXdNvJ-th3bcJfm7VbzpfCVqKXU2aYn1rspWAtBTGxojIEbYNxnARRt6_7efe6Xx7NqmB1OC2cxZdsqqEdfz7iHY/s1600/iv_board.jpg" height="187" width="320" /></a></div>
<br />
Currently this types of Shunt and Sense resistors are supported by default, other types can be easily added:<br />
<ul>
<li>Shunt 500A 50mV</li>
<li>Shunt 300A 50mV</li>
<li>Shunt 100A 100mV</li>
<li>Shunt 100A 50mV</li>
<li>Shunt 10A 100mV</li>
<li>Shunt 2A 100mV</li>
<li>Sense R=0.1 15A</li>
<li>Sense R=0.1 20A</li>
<li>Sense R=0.25 14A </li>
</ul>
Below is T-board for Temperature measurements. The sensor is J-type thermocouples (also not shown, can be plugged to the connectors on the bottom).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF9kR0GLaYUtMjeGetXE_6WiWfTL_PD1DvnHd_XGTjp52pA9Fkre1jIIqGMGbIODHowcXtelO0ZFMOyRutG4ym3v7M69-CtpI-WxOJh99km04Ev_gv053_ZhdZZMjnxEeTFrJncv8qu44/s1600/t_board.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF9kR0GLaYUtMjeGetXE_6WiWfTL_PD1DvnHd_XGTjp52pA9Fkre1jIIqGMGbIODHowcXtelO0ZFMOyRutG4ym3v7M69-CtpI-WxOJh99km04Ev_gv053_ZhdZZMjnxEeTFrJncv8qu44/s1600/t_board.jpg" height="202" width="320" /></a></div>
<br />
<br />
See below the image of the complete product in the box. We have been given the enclosure, and this Customer asked us to make a table-top version, so here it is. Meantime we are working on a smaller "portable" version. Stay tuned. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAOJFyzTYFUcToybREvPWxyTUEQhSt-TvCUGpC2uMrFjvbgFvUzdrpv00sE-eHwykEuq4zmOJjoZ9V8JQIO7q1U4dyu4RJLZ3SgYcKGVZ9WO_xAgpsYqXz9xgv5YI86VKLbYOnp_l3IGs/s1600/enclosure.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAOJFyzTYFUcToybREvPWxyTUEQhSt-TvCUGpC2uMrFjvbgFvUzdrpv00sE-eHwykEuq4zmOJjoZ9V8JQIO7q1U4dyu4RJLZ3SgYcKGVZ9WO_xAgpsYqXz9xgv5YI86VKLbYOnp_l3IGs/s1600/enclosure.JPG" height="320" width="213" /></a></div>
<br />
Feel free to get back to me if you would like to get more details about
the system, get a copy of software, or customized version for your needs. You can also purchase
the assembled ADC boards or DIY kits.<br />
Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com6tag:blogger.com,1999:blog-3682502565034270491.post-83108478287572318212015-03-01T13:34:00.000-06:002015-03-01T13:37:53.144-06:00Multichannel Professional Data Logger on Raspberry Pi - Part 1See the <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger.html">Part 2</a> for the hardware and <a href="http://rokhmanov.blogspot.com/2015/03/multichannel-professional-data-logger_81.html">Part 3</a> for the software implementation. <br />
<br />
In this series of articles I'll explain how to build a multichannel Data Logger, capable of collecting voltage, current and temperature measurements over the specified intervals of time. <br />
<br />
Our alternate goal is to proof that Raspberry Pi is not just a nice tool to promote teaching of basic computer skills in school, but a serious instrument capable of handling professional tasks.<br />
<br />
A "data logger" definition from <a href="https://en.wikipedia.org/wiki/Data_logger">Wikipedia</a>: "A data logger is an electronic device that records data over time, or in relation to location either with a built in instrument or sensor or via external instruments and sensors".<br />
<br />
<h3>
Requirements</h3>
<br />
Let's first clarify our requirements. As a field / service engineer, I want to have:<br />
<ul>
<li>8 input ports which can measure temperature in ranges between 0 and 260 C (about 32...500 F);</li>
<li>8 input ports to measure either voltage (30mV ... 30V, +/- 2mV) or current (10mA ... 500A);</li>
<li>time measurement interval (1 ... 60 sec);</li>
<li>a simple way to configure data logger and store configuration parameters between restarts;</li>
<li>an intuitive way to retrieve collected data or represent it in graph format;</li>
</ul>
Nice-to-haves:<br />
<ul>
<li>lightweight and portable solution: logger should run on batteries; </li>
<li>easy to handle and operate in both field or in lab conditions;</li>
<li>Desktop or laptop computer is not needed to operate and collect data; </li>
</ul>
The requirements above would let you have a tool to test or tune your Segway, car or boat electric circuits, chicken incubator, solar battery or greenhouse.<br />
<br />
<h3>
Architecture and Design</h3>
<br />
<br />
The Raspberry Pi itself is a <a href="https://en.wikipedia.org/wiki/Raspberry_Pi">single-board computer</a>, we will run a <a href="http://www.raspbian.org/">Raspbian</a> Linux distribution on it.We need to build an additional custom board, which will have an Analog-to-digit converter (ADC), connectors for shunts and sensing resistors, temperature sensors, power stabilizer and power outage notifier (we are running on accumulators when we are in the fields).<br />
<br />
From the connectivity standpoint - our Data Logger will be accessible over WiFi. We will put a WiFi dongle in one of the Raspberry Pi USB ports. The WiFi will be configured in HOST mode, so basically in field conditions you can see the Data Logger on your phone or tablet as a "Wireless Network". You can connect to it and access the Data Logger by IP. In case if you work with Data Logger in lab, you can plug your Ethernet cable into the Raspberry Pi and the Data Logger can serve as a free wireless router or WiFi extender.<br />
See hardware and network diagram below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivQ64g-grdlrYCPps06Fxd_VgFYkGG5WN7g2ULMRIrRy_YaTkjHbvesuvzaHc1g7sXU4lapK2CvNh0IJ54mPy6H7Gdcdt_LYi-XcahC4RPFPyEwQNoBVKZh3l0OHsKxCjMZ0XFdiqYw7o/s1600/RaSh_block.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivQ64g-grdlrYCPps06Fxd_VgFYkGG5WN7g2ULMRIrRy_YaTkjHbvesuvzaHc1g7sXU4lapK2CvNh0IJ54mPy6H7Gdcdt_LYi-XcahC4RPFPyEwQNoBVKZh3l0OHsKxCjMZ0XFdiqYw7o/s1600/RaSh_block.jpg" height="156" width="320" /></a></div>
<br />
<br />
Software will be based on assumption that we build a network-connected Data Logger. We will have a web user interface, so all logging configuration can be made from your favorite web browser. The collected data will output on screen in real time and represented as a set of graphs. Alternatively user will have ability to download a whole collected set of data as CSV file and analyze it (for example in MS Excel).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4LWdIp0y9S77FowIR0Tzrsqr5YEeIwjlf6TKeL5ds88pVmlBBjSzIjpDH98pq5uKGUl394C9gt4JdT0Ou8k97cuWBh4ztaH7bLjVnbm0oAeVxZKIVU50Ql4IMaYpMWNB_OzZIaFqrhqo/s1600/Rash_soft.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4LWdIp0y9S77FowIR0Tzrsqr5YEeIwjlf6TKeL5ds88pVmlBBjSzIjpDH98pq5uKGUl394C9gt4JdT0Ou8k97cuWBh4ztaH7bLjVnbm0oAeVxZKIVU50Ql4IMaYpMWNB_OzZIaFqrhqo/s1600/Rash_soft.jpg" height="236" width="320" /></a></div>
<br />
Above is a software design diagram. The ADC connected to Raspberry Pi GPIO ports, we need a way to program ADC and read its values. Also we need a service which will retrieve the ADC data from specified ports over a specified time periods, store and retrieve the configuration settings. We will use REST service to provide a user an easy access to the data and configuration. The Web User Interface consists from two screens: Configuration (so set interval, select ports, trigger conditions) and Status (display retrieved data and draw graphs, export data).<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-51498114187048186422014-12-31T17:13:00.001-06:002015-01-03T18:16:40.620-06:00Representing Your Local Maven Repository Structure in Neo4J Graph DatabaseBelow you will find a way to use <a href="http://neo4j.com/">Neo4J graph database</a> as a storage of metadata from your local <a href="https://maven.apache.org/">Maven</a> repository.<br />
<br />
If you have Maven to manage your project dependencies, then after a while you might get a quite sizable local Maven cache. Maven keeps there a copies of all jars being downloaded from the remote repositories (or being installed by your local builds). Usually it is located in your home folder, under the hidden .m2 directory. For example, my local Maven repository size is ~8GB, and honestly saying I don't remember when the last time I cleaned it up.<br />
<br />
Maven-based projects configured to get the external artifacts (jars, zip, war files, etc.) necessary for a build by looking for a set of <dependency> nodes in the project pom.xml file. Which means you can find out which artifacts in your local Maven repository are still in use and which ones are potential waste by simply examining all your pom.xml files. Note that I am talking here about direct dependencies only, not a dependency-on-dependency chains, which Maven is also capable to handle. If we want to investigate a whole chain, we need to get recursively to each of the pom files and its dependencies. <br />
<br />
We all know that a local storage is not very expensive this days, so we will not gain much benefit by keeping an eye on how Maven uses our local artifacts and periodically wipe out the old ones. It is always cheaper to destroy the local cache completely and let maven to download all it needs automatically during the next build. However, having an automated way of resolving relations between your active projects and your local maven cache might be interesting from several others points of view. For example software audit (if any of your commercial projects use non-licensed code), or some code analysis (what versions of a particular dependencies are used). The Maven own <a href="https://maven.apache.org/plugins/maven-dependency-plugin/">Dependency plugin</a> has some of this functionality, it can build a rudimentary trees and provide the dependency information in one particular project or group of related projects. But I can imagine a situation when you get a vulnerability report about one specific version of a dependency, and you would like to find out quickly where exactly it is used. This can be especially helpful if your organization uses a centralized Maven repository like Artifactory or Sonatype, and you can quickly poke into it's local repository cache.<br />
<br />
There might be another reasons why I came up with this crazy idea to keep Maven repository information in graph database, so to stop speculations I will only add that the Neo4J is a great tool, and the task itself is very familiar to the <a href="http://neo4j.com/use-cases/">"Digital Asst Management"</a> use case, enough said.<br />
<br />
A couple of word about implementation. Let's consider that you have a Neo4J database up and running somewhere, and you know a URL to it's REST endpoint. On my machine it is http://127.0.0.1:7474/db/data/<br />
<br />
GitHub repository with source code: <a href="https://github.com/rokhmanov/repo-graph-maven">https://github.com/rokhmanov/repo-graph-maven</a><br />
<br />
The application itself is a command-line tool, which first scans your local Maven repository for *.pom files, excludes the SNAPSHOTs, and then process the files one-by-one. We are particularly interested only in a subsection of pom file, in particular in <artifactId>, <groupId> and <version> nodes. Another point of interest is a <dependencies> node along with all its child nodes. This chunks of XML gets unmarshalled into java objects, see the corresponding Dependency.java and Project.java source files under com.rokhmanov.graph.sample.entity package. The last step is a graph creation - Neo4J RESTful API gets called by Jersey java client. Nothing really complex. This is a command line syntax:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">usage: repo-graph-maven.xxxx.jar [OPTION]...</span><br />
<span style="font-family: "Courier New",Courier,monospace;">Options:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <directory> - (mandatory parameter) path to the local Maven repository.</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <serverURL> - (mandatory parameter) Neo4j REST server root URL.</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 'clear' - (optional parameter) The existing database will be recreated if specified.</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<span style="font-family: "Courier New",Courier,monospace;">Example: java -jar repo-graph-maven.jar ~/.m2/repository http://192.168.0.1:7474/db/data/ clear</span><br />
<br />
<br />
Structure of Neo4J database: a single Root node "keeps" several links to Project nodes, each of them "has" links to Artifact nodes. So the graph schema is also very simple: "Root", "Project" and "Artifact" are nodes, and "keeps" and "has" are vertices.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[Root]--keeps-->[Project]--has-->[Artifact]</span><br />
<br />
Note that some Projects might use the same Artifacts, for example the same version of jUnit library. The application handles this scenario, and graph includes a cycles, like the one on a screenshot below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOjUEb9OGCNnj4d7LjTHp2gXUVs_wv7bpR7voZZC4X-YRKidoTbOKqBT1B-KIOZdAAXYGmthGiarqLo7UACkvF9juvBD04VNXRCV4hAeQarKiEU8eGJOxqVPdsEPVf1fAUO3pwZwXaYK8/s1600/neo_maven_repository.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOjUEb9OGCNnj4d7LjTHp2gXUVs_wv7bpR7voZZC4X-YRKidoTbOKqBT1B-KIOZdAAXYGmthGiarqLo7UACkvF9juvBD04VNXRCV4hAeQarKiEU8eGJOxqVPdsEPVf1fAUO3pwZwXaYK8/s1600/neo_maven_repository.jpg" height="205" width="320" /></a></div>
<br />
If you have a large repository, the initial run might take a several minutes. Each subsequent run will add only new projects and artifacts so it will be shorter. Keep in mind that if you supply the "clear" parameter to the application, the whole Neo4J graph will be repopulated from scratch.<br />
<br />
After successful execution, we can finally play with the graph. Below are sample Cypher queries along with their results:<br />
<br />
1. What projects use the "commons-io" artifact?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">MATCH p=(b:Component)-->(c:Artifact)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">WHERE c.artifactId='commons-io'</span><br />
<span style="font-family: "Courier New",Courier,monospace;">RETURN b.artifactId, b.version, b.groupId</span><br />
<br />
Result:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2tGyw_n-hO-tKZNpniTrTVvtAMDqO4sMjWXsTC2tMXzpdD4-DYgH5mKzGDQfe312U7sA6KlplY8zBe2CWzCiyjoyyvveGjis_um17oWjCXXahOOmXw_V-UOCTLDaJ0eC1n0Ug8JEgX4g/s1600/query_commons-io.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2tGyw_n-hO-tKZNpniTrTVvtAMDqO4sMjWXsTC2tMXzpdD4-DYgH5mKzGDQfe312U7sA6KlplY8zBe2CWzCiyjoyyvveGjis_um17oWjCXXahOOmXw_V-UOCTLDaJ0eC1n0Ug8JEgX4g/s1600/query_commons-io.jpg" height="280" width="320" /></a></div>
<br />
<a href="https://github.com/rokhmanov/repo-graph-maven"></a>
<a href="https://github.com/rokhmanov/repo-graph-maven"></a>
2. What versions of "commons-io" artifact are used overall?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">MATCH (n:Artifact)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">WHERE n.artifactId =~ '.*commons-io.*'</span><br />
<span style="font-family: "Courier New",Courier,monospace;">RETURN n.artifactId, n.groupId, n.version</span><br />
<br />
Result:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEih37pprmIG96LfisTFH6LtU6vK0Pb24lJrcWBRhGUZGSv5h7d36K4_fFpZ8QFQv05ZlLwR3buJRtWjIeDlRx46tzzWgiYTkUmgxIoe8_lZvnmPJJmzefnVZuVXh_C-qkvZmP5EBxCR4LY/s1600/query_versions_commons-io.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEih37pprmIG96LfisTFH6LtU6vK0Pb24lJrcWBRhGUZGSv5h7d36K4_fFpZ8QFQv05ZlLwR3buJRtWjIeDlRx46tzzWgiYTkUmgxIoe8_lZvnmPJJmzefnVZuVXh_C-qkvZmP5EBxCR4LY/s1600/query_versions_commons-io.jpg" height="280" width="320" /></a></div>
<br />
<br />
<br />
<br />
3. What are the 10 the most used artifacts?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">MATCH (n:Artifact)<-[r]-(x)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">RETURN n.artifactId, n.groupId, n.version, COUNT(r)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">ORDER BY COUNT(r) DESC</span><br />
<span style="font-family: "Courier New",Courier,monospace;">LIMIT 10</span><br />
<br />
Result: <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOv_jHg3do3AY1H8wAX71pVXMvhZGoUa-LX6wPSGISosOFOEKcV1QJtC0z1jXGA3cgMBi0jqfoPKYlkTal6s9oWKi6yx_74Yf9khqAWpar_Pu_fL2lXxdpgoCZg3uklPHjdb5H3U6wgo0/s1600/query_count_artifacts.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOv_jHg3do3AY1H8wAX71pVXMvhZGoUa-LX6wPSGISosOFOEKcV1QJtC0z1jXGA3cgMBi0jqfoPKYlkTal6s9oWKi6yx_74Yf9khqAWpar_Pu_fL2lXxdpgoCZg3uklPHjdb5H3U6wgo0/s1600/query_count_artifacts.jpg" height="272" width="320" /></a></div>
<br />
<br />
Feel free to run all this queries by yourself using Neo4J Browser. Things which I think can be improved or made differently:<br />
<ul>
<li>Implement variable substitution. For example, the ${project.version} can be replaced by the actual value from <parent> section in pom file;</li>
<li>Delete old nodes and edges from graph if the corresponding project is not exist anymore;</li>
<li>Implement batch REST calls to improve performance.</li>
</ul>
Overall this application has a POC quality and serves its needs. The task of course can be achieved by using a regular "relational" approach, the amount of data and the number of joins in the database will not be very large with just two objects and the graph structure like the one above. But nothing prevents us from adding more complexity to the graph in future, like a size of each artifact on filesystem, or information about its internal structure or license used. Keep also in mind that altering the schema (or graph) does not require to bring the database down, like when you alter a schema in a regular relational DB. This might be an important point when considering a solution line that in Production environment.Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com4tag:blogger.com,1999:blog-3682502565034270491.post-33346028398118069422014-12-21T20:15:00.000-06:002014-12-31T11:03:52.385-06:00Realtime Data Percolation with Elasticsearch, Akka and Java 8Finally I've got some time to play with <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html">Elasticsearch Percolator</a> feature. In a couple of words, it is a very efficient way of evaluating your data against a set of rules. Rules are usually defined by some queries. In classic approach, one would save the data in the database, and then run a batch of queries against it to see which corresponding rules are match. The Elasticsearch Percolator approach is opposite - the queries will be placed in database, and data evaluated against them.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhToG7euiotwFOHicyiC339mFjltrsfNrsnfiCF0-G_S4eYnmP_GABbEPz595YfKUzq9oemjGSW-EvoUeiTAPmFUEEmFXvwmG09bDmD8E-NpNFLrP3SeNB_lDmCBhPZHIUeCUPlKq9FRmU/s1600/percolator.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhToG7euiotwFOHicyiC339mFjltrsfNrsnfiCF0-G_S4eYnmP_GABbEPz595YfKUzq9oemjGSW-EvoUeiTAPmFUEEmFXvwmG09bDmD8E-NpNFLrP3SeNB_lDmCBhPZHIUeCUPlKq9FRmU/s1600/percolator.jpg" height="262" width="320" /></a></div>
<br />
This approach can be beneficial when:<br />
<ul>
<li>you have a large amount of queries;</li>
<li>your data does not have a long lifespan (think about application log records for example, they can be safely deleted right after evaluation); </li>
<li>you require fast real-time processing.</li>
</ul>
The data passed to Elasticsearch percolator will be thrown away. The stream of matched queries returned back (almost) immediately.<br />
The example I wrote for my experiment was heavily based on Andrew Easter's sample he made more than a year ago [1], so I had to alter it a bit to use a new Elasticsearch API. My Scala skills are still weak, so I decided to rewrite a whole thing in Java 8, keep Akka actors intact, and drop the Play framework completely, along with the AngularJS UI. The results you can see or clone from my repository in Github [2].<br />
<br />
The design is very simple: a Main class is responsible of starting the embedded Elasticsearch instance, define the Akka actors system, initialize Elasticsearch index with the proper mapping (the latest 1.4.1 version of Elasticsearch requires to have a mapping ready before percolation). The result of initialization Future call is a Stream of tuple objects, each represents a search string and matched data entry. The next step will be a populating Elasticsearch percolator with queries. Nothing prevents us to add or delete this queries in real-time (Elasticsearch has a RESTful API which I used from Jersey client), but for simplicity all the queries will be defined in advance.<br />
<br />
The dummy data supplied by LogEntryProducerActor class. Using built-in Akka scheduler, we can force a periodical log records generated as frequently as we want. The biggest simplification I made is the way how the matched queries are returned back. I've added a BlockingQueue on Worker Actor side, which keeps a matches produced by Percolator. Using Java 8 Streaming API the matched Tuples are directed from Queue back to client and simply printed to stdout.<br />
<br />
Basically, I would like to see how the sample works under load, with the different number of queries defined. On my laptop I tried 1 millisecond interval between logs and got a steady 900...1050 records processed per second, with less than 5% average CPU use. After increasing a number of queries to a several thousands I started to get a Jersey-specific errors, caused by the initialization process. Obviously, if we need to initialize a large amount of documents really fast, sending a huge amount of REST calls will not be a good idea. The Elasticsearch java client probably will be the better option.<br />
<br />
Overall, I am happy with the Percolation feature of Elasticsearch, it works quite fast and efficient. I haven't tried a clustered approach of Elasticsearch or Akka (the Akka's application.conf file is included, feel free to specify your clustering stuff there). The search query optimization was also omitted, for example one can use a "filter" instead of "query_string" Elasticsearch API for a potentially better performance.<br />
<br />
Resources: <br />
[1] <a href="http://www.dreweaster.com/blog/2013/07/08/reactive-real-time-log-search-with-play-akka-angularjs-and-elasticsearch/">Reactive Real-time Log Search With Play, Akka, AngularJS and Elasticsearch</a> (by Andrew Easter);<br />
[2] My<a href="https://github.com/rokhmanov/akka-percolator"> akka-percolator repository</a> in GitHub;<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-29852535048424828582014-07-11T16:43:00.000-05:002015-03-01T13:38:27.265-06:00Monitoring Radon with Arduino<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: justify;">
I live in the Midwest, in Chicago <a href="http://www.naperville.il.us/">suburbs</a>. When I bought my house several years ago I ordered Radon gas check and the levels were in normal range. From what I read about Radon, it's levels can change during the year and season. I always wanted to check its level real-time and find the factors causing its fluctuation if it is possible.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
I bought a Safety Siren Pro Series 3 Radon Gas detector from Amazon (Model No: HS71512) and did some initial measurements. On my living floors the radon levels were OK, but I was mostly interested in basement, which is unfinished non-liveable "crawl space", used mostly for storage. Would be great to have the Radon measurement process automated, without the need to crawl into basement every time to check the readings.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
I found excellent <a href="http://www.howmuchsnow.com/arduino/radon/">article</a> written by Chris Nafis which exactly targeted my task. Recently I've got a spare Arduino Kit from <a href="https://www.sparkfun.com/">SparkFun Electronics</a> and decided to use it for the Radon detection automation. You can get a lots of basics and instructions from Chris's radon work, I will explain here only what I did differently. </div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
First of all, the Safety Siren Radon Gas detector has to be modified, which will void your warranty. Instead of having a set of cables directly attached to a Safety Siren detector board, I installed a 25-pin DB-25 connector on top of Radon Detector, and cut a LPT-port cable in half from my old PC. I soldered the cable wires to my Arduino and put it along with <a href="http://arduino.cc/en/Main/ArduinoEthernetShield">Ethernet Shield</a> into a small plastic box. See the picture below of what I have today running in my basement.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiP7UDzx8CLFQR_vXL1Hjeozmg1LEL-6EICX_oln3VCtyJ4L5UpbzKICg5nQtW7P8i4xPxk2nmqLr6578s6_X5j53h1x-lRPQqImzwaemPXXLbU0xyXz64kpK7lvKC2MDofO-Wlk4sVhc/s1600/assembled.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiP7UDzx8CLFQR_vXL1Hjeozmg1LEL-6EICX_oln3VCtyJ4L5UpbzKICg5nQtW7P8i4xPxk2nmqLr6578s6_X5j53h1x-lRPQqImzwaemPXXLbU0xyXz64kpK7lvKC2MDofO-Wlk4sVhc/s1600/assembled.jpg" height="187" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div style="text-align: justify;">
Below is the picture of LPT cable ends with soldered wire pins I used for experimenting. Later I removed all the pins and soldered cable wires directly to the Arduino board, so the whole construction would fit into the box.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEiLUSESSZWEcBkA9UO8vg82TCAgH8kDUwINlXxvWsNt02BVYhPLTz9xWdlEOl29hvqZaIZYxZ1m4fU40dLx5BtIg8NbN8oArC3jkrqWOY7-Oady6ZhfqFjs_eT2zWWYgXrcUN7k8z-l8/s1600/board_1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEiLUSESSZWEcBkA9UO8vg82TCAgH8kDUwINlXxvWsNt02BVYhPLTz9xWdlEOl29hvqZaIZYxZ1m4fU40dLx5BtIg8NbN8oArC3jkrqWOY7-Oady6ZhfqFjs_eT2zWWYgXrcUN7k8z-l8/s1600/board_1.jpg" height="320" width="240" /></a></div>
<br />
<div style="text-align: justify;">
An interesting challenge was the detection of Long / Short setting of Safety Siren detector (1 week or 1 month of approximate radon data). I noticed that after reset the detector automatically turns on into Long setting, but I wanted to rely on the actual settings and not assume anything. Chris has an Arduino program for older Model HS80002 listed on his website, but I had a newer HS71512, and Long / Short detection logic were missing in the program. First I thought I can read the Long / Short LED, but unfortunately soldering into it was not an easy task without disassembling the Safety Siren board, so I decided to take a programming approach.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I noticed that the Long / Short settings has some timing differences. See two screenshots below, each screenshot has two channels displayed. On a first screenshot the DIGIT_4 strobe (lower channel on screenshot) goes 0 almost immediately with LTL signal (upper channel) when we are in Short setting. In Long setting, this two signals has a timing difference, so the second screenshot illustrates it.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF37QPIn8EqRdSKrLKN-r3YfyoM4uF0w6O1xmCO4bJnUpDYQfia-SVRfYupqXNS3qCzp2fI3s7ukjJ0KVVXoI0QoB-PsB_TVSFZaNnMfy_wXWKqXgl6e8USsTgWUiKdv7-PVetD8t4iFA/s1600/osc_2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjAIjvKCCqmht4Hi9ajYMzXhnJhWrjJdAzvEfAh6XgxYwLGdD-JqTXb8xeEvJL5TfM1JHP0ke75jOjrm9LIapSgayROF0cebr8uXPTjYA_OsOknPGXXdnOAvPX4LX-KGzoMN2KrDtv1hs/s1600/osc_3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjAIjvKCCqmht4Hi9ajYMzXhnJhWrjJdAzvEfAh6XgxYwLGdD-JqTXb8xeEvJL5TfM1JHP0ke75jOjrm9LIapSgayROF0cebr8uXPTjYA_OsOknPGXXdnOAvPX4LX-KGzoMN2KrDtv1hs/s1600/osc_3.jpg" height="253" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF37QPIn8EqRdSKrLKN-r3YfyoM4uF0w6O1xmCO4bJnUpDYQfia-SVRfYupqXNS3qCzp2fI3s7ukjJ0KVVXoI0QoB-PsB_TVSFZaNnMfy_wXWKqXgl6e8USsTgWUiKdv7-PVetD8t4iFA/s1600/osc_2.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjF37QPIn8EqRdSKrLKN-r3YfyoM4uF0w6O1xmCO4bJnUpDYQfia-SVRfYupqXNS3qCzp2fI3s7ukjJ0KVVXoI0QoB-PsB_TVSFZaNnMfy_wXWKqXgl6e8USsTgWUiKdv7-PVetD8t4iFA/s1600/osc_2.jpg" height="268" width="320" /></a></div>
<br />
So to handle this timing difference, I used interrupts functionality of Arduino. The Arduino Uno has int0 on pin2, which I attached to LTL from Safety Siren detector.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">attachInterrupt(0, detectShortHandler, RISING);</span><br />
<br />
The interrupt service routine is very simple - if DIGIT_4 is 0 - we are in Short mode, if opposite - we are in Long.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">void detectShortHandler()<br />{<br /> delayMicroseconds(100);<br /> if (digitalRead(DIGIT_4) == LOW)<br /> state = ST_SHORT;<br /> else<br /> state = ST_LONG;<br />}<b><br /></b></span><br />
Please note a small delay I had to add before processing the DIGIT_4 strobe value. The DIGIT_4 goes to 0 "<b>almost"</b> immediately, see a screenshot below, hence the pause.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigfhYnpNfpgpcLqBp6ZJN34FvGcnHrmQ7j_05X50tAg91Mqbj4lyFBKEtdwpr_dIS_z6rp_JLPlM60OvlK-hZaPS3L0HNJlE9izqZ3hhG1vHXva41dj8V-HAWEJud9YktSdfuHNCLGMIU/s1600/osc_1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigfhYnpNfpgpcLqBp6ZJN34FvGcnHrmQ7j_05X50tAg91Mqbj4lyFBKEtdwpr_dIS_z6rp_JLPlM60OvlK-hZaPS3L0HNJlE9izqZ3hhG1vHXva41dj8V-HAWEJud9YktSdfuHNCLGMIU/s1600/osc_1.jpg" height="267" width="320" /></a></div>
<br />
<div style="text-align: justify;">
Finally, I had the radon values detected, processed and periodically (once per ~30 minutes) published to <a href="https://xively.com/">Xively</a> free service (former Pachube). Below is a graph of the Radon level in my crawl space after several days of monitoring.</div>
<div style="text-align: justify;">
Note that I have only "long" setting presented. After several weeks of collecting data, my Safety Siren detector failed to switch to Short. I think it is a detector issue (probably internal firmware fail), because it still switch into Long and stay there even after I disconnected Arduino and did a full reset of Safety Siren detector. Well, I cannot exchange detector because I opened it and I will not buy another one (~$100 is too steep for a small project), so I'll stuck with Long setting for now.<br />
<b>Update:</b> after about 4 months both the "long" and "short" settings started to operate normally. I think the reason might be a power supply, this is the only thing which I changed with my current setup during that time. </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigEr9yx48qgL3XEQV_j6l4lFDOvFxGo5NSV_dHAgcinvi6xsslHZAejfGCJYHlFKXIW4kC3T04n11BN8x3_Z4Qz_g5t0AVRX-DpfdR5Jn7IJURS4rWISnsG-aQYq8lap6AitDOBQ9vGNY/s1600/xively_long.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigEr9yx48qgL3XEQV_j6l4lFDOvFxGo5NSV_dHAgcinvi6xsslHZAejfGCJYHlFKXIW4kC3T04n11BN8x3_Z4Qz_g5t0AVRX-DpfdR5Jn7IJURS4rWISnsG-aQYq8lap6AitDOBQ9vGNY/s1600/xively_long.jpg" height="214" width="320" /></a></div>
For those who would like to repeat this project - I published my <a href="https://github.com/rokhmanov/arduino-radon">Arduino program on GitHub</a>, feel free to use it and get back to me if any questions.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com4tag:blogger.com,1999:blog-3682502565034270491.post-36432906338337247742013-12-23T20:39:00.000-06:002013-12-23T21:03:50.331-06:00Automate JBoss Teiid Development Environment with Vagrant and VirtualBox<h2>
Why To Automate? </h2>
What benefits has the development environment automation:<br />
<ul>
<li>Ability to run several development versions in parallel. Each running environment can host a separate branch of the same product. Or keep some unique environment options, like different JVMs (Pentaho Mondrian is a good example);</li>
<li>Some complex project which require simultaneous development in several places (happens rare, so it is arguable benefit);</li>
<li>Freedom to experiment with your environment. You can always install something, rollback to a stable step, or start from scratch in minutes;</li>
<li> Your development environment configuration now is a code itself. You can debug it, branch, release, or write tests;</li>
<li>You can migrate your environment to a new machine or upload it to a cloud in several mouse clicks. You just need to find a way to VNC or RDP to it. Finally you can do an Eclipse development from your android phone! (yeah, right...) </li>
<li>If you work for a company, spawn a new environment for new developers would take hours, not days or weeks;</li>
<li>Environment upgrades and updates for everyone in your team are centralized, tested and predictable. No more "it works on my side". </li>
</ul>
Along with benefits above, here are the things to remember:<br />
<ul>
<li>Your Host machine should have enough RAM and CPU. Maven can build in multiple threads, GWT compile permutations over all available cores. So obviously more you have, and more you can slice for your environments - better they work;</li>
<li>Commercial vs Free. If you develop on Windows or Mac - each generated environment should be registered, which means extra $$ and extra complexity;</li>
<li>You need good internet connection to generate the environment (can be several GB to download). On opposite side, most of the time you need to do it only once, the large downloads are cached locally;</li>
<li>Writing environment automation scripts can be a slow job. Depends of your goals the environment generation process might take from several minutes to several hours, and some stupid typo you carelessly made in the last step would require to start over;</li>
<li>You still have to install and get familiar with some (not very complex) tools which help you with the automation;</li>
</ul>
All this pros and cons are individual. If you think that in your case it worth attention, below is my recipe.<br />
<br />
<h2>
Instant Eclipse Development Environment </h2>
My goal is to have a development environment to work on data federation projects with JBoss Teiid. Below is the list of software I put to it, but your task might be different. Due to modularity, you can exclude some items or add new things very quickly.<br />
<ul>
<li>Linux OS. I took Ubuntu 12.04.03 LTS, it is wide-known, considerably stable and supported by large community. No hassle or extra money for licenses.</li>
<li>Oracle Java 7 SDK. Installation requires to accept the license agreement, but it can be automated;</li>
<li>Apache Maven 3.0.4. I need specifically this version, but script can be easily changed to the most recent;</li>
<li>Git as a source control. Just use the one which Ubuntu has in its repository; </li>
<li>Eclipse Kepler SR1 for JEE. I need specifically this version, change several lines in script to put another one;</li>
<li>JBoss Application Server. I put EAP-6.1.0.Alpha, this is the one which works best with the latest Teiid;</li>
<li>Teiid 8.6.0.Final distribution, download and install on JBoss;</li>
<li>Latest Teiid 8.7.0.Alpha1 sources;</li>
</ul>
As a result, you're getting a pretty decent development environment (8 GB RAM, 2 CPU, 32GB storage), which you can immediately fire up and start working on your own project, or extend Teiid itself. <br />
Steps to follow to generate your own environment using the template above:<br />
<ol>
<li>Get the latest version of <a href="https://www.virtualbox.org/">Oracle VM VirtualBox</a>. I used 4.3.6. It is free and easy to use virtualization;</li>
<li>Install latest <a href="http://www.vagrantup.com/">Vagrant</a>. I have version 1.3.5. This is a tool which manages the collection of your virtual environments, takes care about network access and describes how you want to alter the original OS image for your purposes;</li>
<li>Get the latest <a href="http://www.packer.io/">Packer</a>. Version I used is 0.4.1. This is a nice functional addition to Vagrant, which allows you to get and apply the customizations and easy install additional software on generated box;</li>
<li>Get the <a href="https://github.com/rokhmanov/packer-teiid">packer-teiid</a> template I made for the environment described above. It is hosted on GitHub, so clone or simply download it to some folder; </li>
<li>Go to the packer-template folder and run "<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: small;"><b>packer build ubuntu-12.04.3-desktop-amd64.json</b></span></span>". This will start the generation process. After some time (~40 minutes or so), you should see a shiny new "ubuntu-12.04.3-desktop-amd64.box" file in the same folder, around 2GB size;</li>
<li>Now it is a time to add your new box to a Vagrant collection. Type "<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: small;"><b>vagrant box add</b> <b>ubuntu-12.04.3-desktop-amd64 </b><b>ubuntu-12.04.3-desktop-amd64.box</b></span></span>" in the packer-teiid folder, or specify a full path to your *.box file if you do it outside;</li>
<li>You can bring your environment up now. If you want to do it in the same folder, run "<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: small;"><b>vagrant up</b></span></span>". If you want to do it from other place, you should do "<span style="font-family: "Courier New",Courier,monospace;"><b>vagrant init</b></span>" first in this new folder. The generated "Vagrantfile" also included in my project, compare it with the one which appear in your new folder and see the differences ("vb.gui = true" is the most important one, otherwise your environment will start in background).</li>
<li>If everything went well, you should see something line on image below. Feel free to login using vagrant/vagrant credentials, sudo and change the root password, run Eclipse or open JBoss bin folder using a shortcuts on a desktop.</li>
<li>You can get to your running environment from Host using ssh (port 2222 on localhost). To transfer files between Host and Guest, you can use "/vagrant" mapping from inside your Guest. It is mapped to the actual folder on Host machine from where you started your VM box ("packer-teiid" in my case).</li>
<li>To stop your environment, simply run "vagrant halt" or "vagrant suspend". To kill it - "vagrant destroy".</li>
<li>Happy Packing! </li>
</ol>
<span id="goog_747270037"></span><span id="goog_747270038"></span><span id="goog_747270039"></span><span id="goog_747270040"></span><br />
Screenshot of Ubuntu Linux Development Environment running under Windows 7 host machine:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipG43QwJDWZgfow531m9grgrmtGi0tsPN3cmqwaCyFMn0KuIegqQM7sjhxp4l9tpXGjFnnMdfvcF5gNODLOPfdlAXnwjGrFxxSNNfcHt4Y20eJUrdKEH2ksdeCV-NPvnSNIuIKvCKpfA4/s1600/vagrant.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipG43QwJDWZgfow531m9grgrmtGi0tsPN3cmqwaCyFMn0KuIegqQM7sjhxp4l9tpXGjFnnMdfvcF5gNODLOPfdlAXnwjGrFxxSNNfcHt4Y20eJUrdKEH2ksdeCV-NPvnSNIuIKvCKpfA4/s400/vagrant.jpg" width="400" /></a></div>
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-74703084449667646392013-12-06T22:06:00.000-06:002013-12-21T18:31:06.866-06:00Teiid Translator for EBayI wrote a very basic translator for <a href="http://developer.ebay.com/DevZone/Finding">EBay Finding API</a>.
Implemented only findByKeyword API call, but this seems enough to issue a query against
EBay database of active auctions and retrieve result.<br />
<br />
<a href="https://github.com/rokhmanov/teiid-translators/tree/master/teiid-ebay">Link to Teiid EBay translator on GitHub</a>.<br />
<br />
<h3>
Prerequisites</h3>
You should have an active EBay Develpers License, which you can get for free at <a href="http://developer.ebay.com/">http://developer.ebay.com</a>.
Also you have to download an actual EBay FindingKitJava archive from
EBay Developer website, and extract the finding.jar from lib folder.
I used version 1.12.0 of finding API jar (Built-Date: 2011-04-28
14:10:35), let me know if you have problems with other versions.
For convenience you can put this jar into your local maven repository,
see the "version.ebay-finding-java-driver" parameter in parent pom.xml
for the reference.<br />
Put your API key to EbayTest.java (see DEVELOPER_KEY static field)
and run this class as a unit test to make sure your API key is working.<br />
<br />
If everything is fine, you should get a similar result:<br />
<pre><code>Returned results:[
|271336881138|Harry Potter and the Order of the Phoenix (Xbox 360, 2007)|EBAY-US|null|Video Games||http://thumbs3.ebaystatic.com/m/m49aytufjFn4dN70RkI4QMA/140.jpg|http://www.ebay.com/itm/Harry-Potter-and-Order-Phoenix-Xbox-360-2007-/271336881138?pt=Video_Games_Games|null|56274325|PayPal|false|91911|Chula Vista,CA,USA|US||||Active|Auction|false||null||Good|null,
|171183853715|Harry Potter and the Order of the Phoenix Figures Harry, Hermione, Ron and Map|EBAY-US|null|TV, Movie & Video Games||http://thumbs4.ebaystatic.com/m/maptT7UNML82z5uv64Abg2A/140.jpg|http://www.ebay.com/itm/Harry-Potter-and-Order-Phoenix-Figures-Harry-Hermione-Ron-and-Map-/171183853715?pt=US_Action_Figures|null|152040891|PayPal|false|34119|Naples,FL,USA|US|||9.8 USD|Active|Auction|false||null||New|null
]
</code></pre>
<h3>
<a class="anchor" href="https://github.com/rokhmanov/teiid-translators/tree/master/teiid-ebay#implementation" name="implementation"></a>Implementation</h3>
<ul>
<li>translator-ebay - the actual Teiid translator code;</li>
<li>ebay-api- a wrapper for EBayConnection interface;</li>
<li>connector-ebay - a JBoss resource adapter. It supposed to get a EBay
Developer key as a configuration parameter if translator will be
executed in real (non-embedded) Teiid instance.</li>
</ul>
The EBay Teiid translator is a simple stored procedure, which accepts a keywords separated by space as a single input parameter.
The stored procedure input parameter name is obviously "keywords".<br />
<br />
Example:<br />
<pre><code>call ebaydata.findByKeyword('harry potter phoenix');
</code></pre>
Output resultset has several important fields provided by Ebay Finding API:<br />
<ul>
<li>"itemId"</li>
<li>"title"</li>
<li>"globalId"</li>
<li>"subtitle"</li>
<li>"primaryCategory"</li>
<li>"secondaryCategory"</li>
<li>"galleryURL"</li>
<li>"viewItemURL"</li>
<li>"charityId"</li>
<li>"productId"</li>
<li>"paymentMethod"</li>
<li>"autoPay"</li>
<li>"postalCode"</li>
<li>"location"</li>
<li>"country"</li>
<li>"storeInfo"</li>
<li>"sellerInfo"</li>
<li>"shippingInfo"</li>
<li>"sellingStatus"</li>
<li>"listingInfo"</li>
<li>"returnsAccepted"</li>
<li>"galleryPlusPictureURL"</li>
<li>"compatibility"</li>
<li>"distance"</li>
<li>"condition"</li>
<li>"delimiter"</li>
</ul>
The underlying EBay Finding API is a SOAP webservice, see the
FindingService.wsdl from downloaded FindingKitJava.zip for description
of each of the returned fields.<br />
Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-61912990305284024992013-10-04T16:01:00.001-05:002014-02-06T17:45:58.180-06:00Bring HTML Pages Into Relational World Using Web Scraping Teiid Translator You probably heard many times that we live in the era of Semantic Web. Unfortunately not all HTML pages you see were made using RDF. We have to parse them using web browsers, HTTP clients and a variety of custom tools. Many HTML pages are old, unstructured, were built using outdated standards or poorly designed instruments.<br />
Would be really nice to retrieve all data kept in HTML with minimal effort, and be able to access it in relational way. I had a sleepless night last week, and<a href="https://github.com/rokhmanov/teiid-translators/tree/master/translator-scrape"> that's what I came up with</a>. <br />
<br />
In short - this is a poor attempt to wrap a great <a href="http://jsoup.org/">Jsoup java HTML parser </a>in Teiid translator logic. A single example is better than a hundred words. This SQL statement:<br />
<br />
<pre>SELECT text, attributes
FROM (call scrapedata.scrap('<a href="http://www.bing.com/search?q=jboss+teiid%27,%27a%5Bhref%5D%27%29">http://www.bing.com/search?q=jboss+teiid','a[href]')</a>) as S
WHERE upper(text) like '%TEIID%'
</pre>
Returns this: <br />
<pre>Teiid - JBoss Community - Community driven open source … , href="http://www.jboss.org/teiid" h="ID=SERP,5095.1"
Teiid - Downloads - JBoss Community , href="https://www.jboss.org/teiid/downloads" h="ID=SERP,5108.1"
Teiid Download , href="/search?q=Teiid+Download&amp;FORM=QSRE4" h="ID=SERP,5240.1"
Teiid Designer , href="/search?q=Teiid+Designer&amp;FORM=QSRE5" h="ID=SERP,5241.1"
Teiid Forum , href="/search?q=Teiid+Forum&amp;FORM=QSRE6" h="ID=SERP,5242.1"
Teiid - Tools - JBoss Community , href="https://www.jboss.org/teiid/tools" h="ID=SERP,5121.1"
Teiid Installation , Community - JBoss, href="https://community.jboss.org/wiki/TeiidInstallation" h="ID=SERP,5134.1"
Teiid - JBoss Issue Tracker , href="https://issues.jboss.org/browse/TEIID" h="ID=SERP,5147.1"
Teiid 7.0 Installation Guide , href="https://community.jboss.org/wiki/Teiid70InstallationGuide" h="ID=SERP,5160.1",
TEIID on tomcat - Community - JBoss, href="https://community.jboss.org/thread/205308?start=0&amp;tstart=0" h="ID=SERP,5172.1"
</pre>
<br />
Feel free to get a clone a<a href="https://github.com/rokhmanov/teiid-translators/tree/master/translator-scrape"> translator-scrape</a> repository from Github, check the sources, play with ScrapeTest.java - it is a unit test build with Embedded Teiid, should give you an idea of how to use this translator.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com1tag:blogger.com,1999:blog-3682502565034270491.post-8265977715264372382013-04-19T13:27:00.003-05:002013-04-20T20:43:07.436-05:00Running HornetQ Bridge Under JBoss 7Here is the explanation of Bridge from the HornetQ <a href="http://docs.jboss.org/hornetq/2.3.0.CR2/docs/user-manual/html_single/index.html#d0e548">documentation</a>: "Some messaging systems allow isolated clusters or single nodes to be bridged together, typically over unreliable connections like a wide area network (WAN), or the internet. A bridge normally consumes from a queue on one server and forwards messages to another queue on a different server. Bridges cope with unreliable connections, automatically reconnecting when the connections becomes available again".<br />
<br />
My goal is to setup a bridge which will forward messages between two queues residing on a different physical servers over the TCP connection. I plan to use two HornetQ services running as part of the latest JBoss (EAP 6.1.0). The "jms-bridge" example from standalone HornetQ distribution has a sample configuration (standalone-example.xml) for JBoss to setup a bridge on a single JBoss instance with HornetQ service. We will have to modify it to support two servers over the network.<br />
<br />
In my configuration I have two systems set up, let's name them "laptop1" and "remoteHost", each of them has JBoss with HornetQ service installed. Below are the changes to the configuration from "jms-bridge" example:<br />
<ol>
<li>Define new HornetQ connector and acceptor. Note that HornetQ has "netty-connector" defined for JBoss configuration, but seems it does not support "host" and "port" parameters, and rely on mandatory "socket-binding" attribute (see jboss-as-messaging_1_3.xsd from JBoss distribution for more details). That's why I had to redefine both connector and acceptor. For simplicity the acceptor has "0.0.0.0" for host to allow connections from everywhere. Also note that on second server the ports should be listed as opposite - 5457 for acceptor and 5456 for connector.</li>
<blockquote class="tr_bq">
<connector name="netty-bridge"><br />
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class><br />
<param key="host" value="remoteHost" /><br />
<param key="port" value="5457" /><br />
</connector> </blockquote>
<blockquote class="tr_bq">
<acceptor name="netty-bridge"><br />
<factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class><br />
<param key="host" value="0.0.0.0" /><br />
<param key="port" value="5456" /><br />
</acceptor></blockquote>
<li>
Disable the HornetQ security, also for simplicity reasons. The proper approach will be to specify "user" and "password" as parameters for connector above.
</li>
<blockquote class="tr_bq">
<hornetq-server><br />
<security-enabled>false</security-enabled><br />
...</blockquote>
<li>
Define two queues: "source" and "target". If desired, for the purposes of this example would be enough to have only one "source" query on local server and one "target" on remote. </li>
<blockquote class="tr_bq">
<jms-queue name="sourceQueue"><br />
<entry name="queue/sourceQueue"/><br />
<entry name="java:jboss/exported/jms/queues/sourceQueue"/><br />
</jms-queue><br />
<jms-queue name="targetQueue"><br />
<entry name="java:/queue/targetQueue"/><br />
<entry name="java:jboss/exported/jms/queues/targetQueue"/><br />
</jms-queue>
</blockquote>
<li>
Define a new connection factory to use in bridge.
</li>
<blockquote class="tr_bg">
<connection-factory name="RemoteConnectionFactoryBridge"><br />
<connectors><br />
<connector-ref connector-name="netty-bridge"/><br />
</connectors><br />
<entries><br />
<entry name="RemoteConnectionFactoryBridge"/><br />
<entry name="java:jboss/exported/jms/RemoteConnectionFactoryBridge"/><br />
</entries><br />
</connection-factory></blockquote>
<li>
Modify bridge definition added after "hornetq-system" tag in settings.xml.
</li>
<blockquote class="tr_bq">
<jms-bridge name="myBridge"><br />
<source><br />
<connection-factory name="ConnectionFactory" /><br />
<destination name="queue/sourceQueue" /><br />
</source><br />
<target><br />
<connection-factory name="RemoteConnectionFactoryBridge" /><br />
<destination name="queue/targetQueue" /><br />
</target><br />
<quality-of-service>AT_MOST_ONCE</quality-of-service><br />
<failure-retry-interval>1000</failure-retry-interval><br />
<max-retries>7890</max-retries><br />
<max-batch-size>1</max-batch-size><br />
<max-batch-time>1000</max-batch-time><br />
</jms-bridge></blockquote>
</ol>
After starting up both JBoss servers, watch for log message indicating that the bridge is started and connection between two HornetQ instances has been established:<br />
<blockquote class="tr_bq">
18:54:55,621 INFO [org.hornetq.core.server] (MSC service thread 1-1) HQ221024: Started Netty Acceptor version 3.6.2.Final-c0d783c 0.0.0.0:5456 for CORE protocol<br />
...<br />
18:54:57,953 DEBUG [org.hornetq.core.client] (ServerService Thread Pool -- 61) Remote destination: rokan01-VM3762.ca.com/10.130.248.122:5457<br />
...<br />
18:54:58,723 INFO [org.jboss.messaging] (ServerService Thread Pool -- 61) JBAS011610: Started JMS Bridge myBridge</blockquote>
To test the configuration, send a sample message to a "sourceQueue" using for example Teiid messageSender from one of <a href="http://rokhmanov.blogspot.com/2012/09/simple-jms-producer-on-jboss-teiid.html">my previous articles</a>:<br />
<blockquote class="tr_bq">
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">call Times.messageSender('queue/sourceQueue', '<b>My Message1</b>'); </code></pre>
</blockquote>
Now login to JBoss Management Console on your remoteHost (http://remoteHost:9990/console/App.html#jms-metrics) and make sure the messages are gets delivered over the bridge to remote host "targetQueue":<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Rtwl2EIsRs-SoxhZyOaiymlNXlcGgf6NnR1h-L1LBGXHuNusTY-Z1UQDEIUI99kWMIUvC19dW5tjzBrydXTRlyyvLdiNLJM9yaXuDVu1qiV8usgxd6dQ0hiVJtZIDtDJOjC1LDqf0D0/s1600/jbossManagementJMS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Rtwl2EIsRs-SoxhZyOaiymlNXlcGgf6NnR1h-L1LBGXHuNusTY-Z1UQDEIUI99kWMIUvC19dW5tjzBrydXTRlyyvLdiNLJM9yaXuDVu1qiV8usgxd6dQ0hiVJtZIDtDJOjC1LDqf0D0/s1600/jbossManagementJMS.jpg" height="249" width="320" /></a></div>
<br />
There might be a moment when you have only a "connector" side of the bridge up and running. The bridge will automatically reattempt to instantiate the connection every second (configurable by "failure-retry-interval" setting in bridge definition above), until the "acceptor" side of the bridge will be available.Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-30114808270295625132012-12-16T01:30:00.002-06:002013-01-16T09:41:18.758-06:00Caching in TeiidTeiid has several ways to improve performance by <a href="https://docs.jboss.org/author/display/teiid81final/Caching+Guide">caching the data</a>. The simplest approach is to provide a caching hint right in the SQL statement. The query result below will be cached for 1 minute, and stored preferably in memory. The subsequent selects using the same query return the cached results for 1 minute. It is important that query should be "string-equal" to retrieve the cached result. Teiid does not parse the query for performance considerations, and even a single extra space added to the query can force Teiid to get for a fresh result instead of cache. Also the same behavior happens if functions like now() used - the results are not cached.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> /*+ cache(pref_mem ttl:60000) */select id, value from test.table1;
id value
--- -----
1 80
2 20
update test.table1 set value = 10 where id = 2;
/*+ cache(pref_mem ttl:60000) */select id, value from test.table1;
id value
--- -----
1 80
2 20
select id, value from test.table1;
id value
--- -----
1 80
2 10
</code></pre>
<br />
Another approach is to use materialized views, or (simpler approach) a "lookup" function. This function useful to create a system-wide cache on some dictionary and perform a search. The results are cached until Teiid restarted, so it might not be convenient to keep a changing data.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> select lookup('test.table1', 'value', 'id', 1);
expr1
-----
80
update test.table1 set value = 50 where id = 1;
select lookup('test.table1', 'value', 'id', 1);
expr1
-----
80
</code></pre>
<br />
And the most flexible way to operate a cache is to use a custom translator with support of <a href="https://docs.jboss.org/author/display/teiid81final/Caching+API">Caching API</a>.<br />
This is the overridden method from CacheTestExecutionFactory which defines the caching approach for translator:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> @Override
public CacheDirective getCacheDirective(Command command,
ExecutionContext executionContext, RuntimeMetadata metadata)
throws TranslatorException {
CacheDirective cd = new CacheDirective();
cd.setScope(Scope.USER);
return cd;
}
</code></pre>
<br />
For complete source code of CacheTestExecutionFactory teiid translator see <a href="https://github.com/rokhmanov/teiid-test">caching-api</a> project on GitHub ( https://github.com/rokhmanov/teiid-test ). Also the Teiid user forum has a <a href="https://community.jboss.org/thread/215753?tstart=0">related discussion thread</a>.Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-30667297048817717342012-09-30T19:16:00.001-05:002012-10-02T08:18:20.288-05:00Free Private Git HostingToday I have spent some time looking for Git hosting for my new pet project. Last time I did such kind of research about three years ago. I was looking for SVN hosting, and I ended up with <a href="https://www.unfuddle.com/">Unfuddle</a>.<br />
Currently they do not offer a free option any more, and my interests are shifted to Git. My current requirements are very simple:<br />
<ul>
<li>functional free starter option, with no-hassle switch to payed services when needed;</li>
<li>both ssh and https access to repository;</li>
<li>tracking/wiki option (low priority);</li>
<li>private hosting - no requirement to open-source your code like on github;</li>
</ul>
This is my shortlist:<br />
<ul>
<li><a href="http://www.assembla.com/">Assembla</a>;</li>
<li><a href="http://www.cloudforge.com/">CloudForge</a>;</li>
<li><a href="http://www.springloops.com/">Springloops</a>;</li>
<li><a href="http://www.projectlocker.com/">ProjectLocker</a>;</li>
</ul>
I had to drop both ProjectLocker and Springloops because of no https support for Git (only ssh is available). I open accounts on both Assembla and CloudForge. Both of them do not have Trac or any other bugtracking/wiki option for free, but it is not critical on this stage. Registration on Assembla was smooth, in a couple of minutes I committed and pushed a sample project and was able to see it from web ui. Another system (CloudForge) had some glitch during registration: the confirmation email came into my mailbox with about 15-hour delay. During this period the admin module has been annoying me about unfinished registration every time I logged in, asking to provide a confirmation number in a special input field (otherwise my account will be disabled in 24 hours). When email has finally arrived, there was no number, only a registration link to click.<br />
I decided to stay with Assembla for now, but keep an eye on CloudForge in the same time. I like their Publisher option - ability to configure your deployment for Joyent, Amazon WS, Salesforce, Google App Engine, CloudFoundry of custom FTP or SSH option. I had not tried it yet, but from description in sounds useful and time-saving.<br />
<br />
<b>Update</b> (next day): I've tried <a href="http://bitbucket.org/">BitBucket</a> this morning, and I think it fits pretty well into all my requirements: free, private, https, wiki/issues tracking, web access, integration with other systems. It allows to get more than one user account to the repository, which also might be useful. I'll stay with BitBucket for now.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com4tag:blogger.com,1999:blog-3682502565034270491.post-85010797996746879652012-09-12T16:48:00.000-05:002012-09-12T16:51:59.687-05:00Teiid and HornetQ Integration<div style="text-align: justify;">
In my previous article "<a href="http://rokhmanov.blogspot.com/2012/09/simple-jms-producer-on-jboss-teiid.html">Simpe JMS Provider on JBoss Teiid</a>" you can learn about how to create a Teiid translator which will "convert" your SQL commands into messages and put them into queue. This article talks about opposite approach: how to retrieve messages from queue using SQL SELECT statements, so our integration pattern will be complete. The schema below explains the integration logic.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyBtqFxCEdIqTZgDtkkcpEP-Vmqtcf_BD4_UENKxPdWKt_d-oHE9yWOJ2uTYZJlcqFhWPzhqYWEZDQb7m2INJGhWgnL-p_uBCTy4Q_cwHWsX-Iz3pIw45blHOaAQZAHGtQa3dCupVJbLs/s1600/Teiid_JMS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyBtqFxCEdIqTZgDtkkcpEP-Vmqtcf_BD4_UENKxPdWKt_d-oHE9yWOJ2uTYZJlcqFhWPzhqYWEZDQb7m2INJGhWgnL-p_uBCTy4Q_cwHWsX-Iz3pIw45blHOaAQZAHGtQa3dCupVJbLs/s400/Teiid_JMS.jpg" width="400" /></a></div>
<div style="text-align: justify;">
Our goal is to pass a specific command (CALL... or INSERT...) to Teiid with some parameters. This parameters will be converted in Teiid by our custom translator (JMS Producer) into the message, which will be immediately placed into queue. New player is another custom translator (JMS Consumer). Client can issue a Continuous Execution to Teiid, which will automatically repeat itself with some time interval. Each round of execution will retrieve messages from queue and pass back to client as execution results. Client gets notified by Teiid when result is available - client obtains a callback object from Teiid when the execution initiated. The StatementCallback.onRow() method gets executed when data is passed back to client.<br />
The process can be further improved, if instead of retrieving messages periodically we will listen queue and retrieve message back to client right after it will appear. So instead of polling we will implement pushing approach.</div>
<div style="text-align: justify;">
<br /></div>
<h4>
Implementation Details</h4>
<div style="text-align: justify;">
The custom translator code for JMS Consumer should implement MessageListener interface. The provided onMessage(Message msg) method will be called each time the message will appear on subscribed queue. Translator also implements ResultSetExecution (so it will appear as a view in Teiid Virtual Database), and ReusableExecution interfaces. The DataNotAvailableException.NO_POLLING will be thrown from the translator next() method, to put translator "into sleep". This is a non-blocking process and part of Continuous Execution functionality. To bring translator back to life, we will call ExecutionContext.dataAvailable() from onMessage() method.</div>
<div style="text-align: justify;">
The complete code of JMS Consumer Translator (MessageReceiveExecution.java) is attached, and below are the most important methods:</div>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: 350px; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public void execute() throws TranslatorException {
this.callNext = DataNotAvailableException.NO_POLLING;
this.callNext.setStrict(true);
}
public List<?> next() throws TranslatorException, DataNotAvailableException {
if (callNext != null) {
DataNotAvailableException e = callNext;
callNext = null;
throw e;
}
if (msgRow != null) {
List<Object> results = msgRow;
msgRow = null;
return results;
}
return null;
}
public void dispose() {
try {
qreceiver.close();
qsession.close();
qcon.close();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void onMessage(Message msg) {
try {
String messageContent;
if (msg instanceof TextMessage) {
messageContent = ((TextMessage) msg).getText();
} else {
messageContent = msg.toString();
}
System.out.println("!!!!!! MSG:" + messageContent);
setMessage(messageContent);
} catch (JMSException jmse) {
jmse.printStackTrace();
}
this.context.dataAvailable();
}
</code></pre>
<br />
<div style="text-align: justify;">
To make sure it works I used a JBoss with HornetQ configuration from the previous article, and made a sample java servlet which connects to Teiid and executes a SELECT statement continuously. Also I have another SQL client application (SQuirreL or Eclipse Database Explorer) connected to Teiid, which I use to produce messages (CALL JMS Producer, like in my previous article).</div>
Testing approach:<br />
<ul>
<li>start JBoss/HornetQ/Teiid with deployed translator jar module and war file with servlet;</li>
<li>open browser, call servlet to issue a continuous execution;</li>
<li>run call statement from SQL client to send a message;</li>
<li>check servlet logs that message successfully retrieved by translator and passed back to servlet;</li>
</ul>
<a href="https://docs.google.com/open?id=0BxYj8BUOYsxIbHI1QlZkNUhGOVU">MessageReceiveExecution</a> (Google Docs). Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-7254793460104534182012-09-03T22:46:00.003-05:002012-09-03T22:46:41.471-05:00Simple JMS Producer on JBoss TeiidWith it's <a href="https://docs.jboss.org/author/display/teiid81final/Non-blocking+Statement+Execution">Continuous Execution</a> feature, new Teiid 8.1 came closer to functionality which reminds me <a href="http://msdn.microsoft.com/en-us/library/ms345108%28v=sql.90%29.aspx">SQL Server Service Broker.</a> In particular, it might be possible to write Teiid extensions which can serve as integration points between relational data from VDB and asynchronous data available in message-oriented middleware.<br />
For example, Teiid translator can be a JMS client which can send messages into some queue. Another Teiid translator running in Continuous Execution mode can retrieve messages from queue.<br />
Below is results of my experiments with first approach: to have a stored procedure which will act as JMS producer.<br />
<br />
<h4>
Configure JMS in JBoss</h4>
The JBoss 7.1.1 has built-in JMS support with <a href="https://www.jboss.org/hornetq">HornetQ</a>, so it is only a matter of properly editing config files. The HornetQ stuff included in standalone-full.xml profile, and Teiid configuration is in standalone-teiid.xml, so I took simplest approach and merged all JMS-related parts from -full.xml into *-teiid.xml one. Also I defined a test queue "MyQueue":<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> ....
<address-setting match="jms.queue.MyQueue">
<dead-letter-address>jms.queue.MyDLQ</dead-letter-address>
<redelivery-delay>0</redelivery-delay>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
....
<jms-destinations>
<jms-queue name="MyQueue">
<entry name="java:jboss/exported/MyQueue"/>
<durable>false</durable>
</jms-queue>
....
</code></pre>
<br />
<br />
It is worth to write a small producer and consumer java classes to test the queue and make sure all configured correctly. The <a href="http://127.0.0.1:9990/console/App.html#jms-metrics">JBoss Management</a> console and <a href="http://middlewaremagic.com/jboss/?p=2402">this post </a>might be helpful.<br />
<br />
<h4>
Write Teiid Translator</h4>
I made a simple translator which is a stored procedure, implements the ProcedureExecution and acts as a JMS producer. Queue name and message passed as stored procedure IN parameters.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: 300px; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> package com.rokhmanov.test.teiid.translator.async;
import java.util.ArrayList;
import java.util.List;
import javax.jms.JMSException;
import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.teiid.language.Argument;
import org.teiid.language.Call;
import org.teiid.translator.DataNotAvailableException;
import org.teiid.translator.ExecutionContext;
import org.teiid.translator.ProcedureExecution;
import org.teiid.translator.TranslatorException;
public class MessageSendExecution implements ProcedureExecution {
public final static String CNN_FACTORY="/ConnectionFactory";
private String queueName;
private String messageContent;
private Call cmd;
private QueueConnectionFactory qconFactory;
private QueueConnection qcon;
private QueueSession qsession;
private QueueSender qsender;
private Queue queue;
private TextMessage msg;
public MessageSendExecution(Call command,
ExecutionContext executionContext, Object connection) {
this.cmd = command;
final List<Argument> procArguments = new ArrayList<Argument>(this.cmd.getArguments());
this.queueName = procArguments.get(0).getArgumentValue().getValue().toString();
this.messageContent = procArguments.get(1).getArgumentValue().getValue().toString();
}
public List<?> next() throws TranslatorException, DataNotAvailableException {
return null;
}
public void close() {
// TODO Auto-generated method stub
}
public void cancel() throws TranslatorException {
try {
qcon.close();
} catch (JMSException e) {
throw new TranslatorException(e);
}
}
public void execute() throws TranslatorException {
InitialContext ic = getInitialContext();
initJMS(ic, queueName);
sendMsg(messageContent);
}
public List<?> getOutputParameterValues() throws TranslatorException {
return null;
}
private InitialContext getInitialContext() throws TranslatorException
{
try {
return new InitialContext();
} catch (NamingException e) {
throw new TranslatorException(e);
}
}
public void initJMS(Context ctx, String queueName) throws TranslatorException
{
try {
qconFactory = (QueueConnectionFactory) ctx.lookup(CNN_FACTORY);
qcon = qconFactory.createQueueConnection();
qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
queue = (Queue) ctx.lookup(queueName);
qsender = qsession.createSender(queue);
msg = qsession.createTextMessage();
qcon.start();
} catch (NamingException e) {
throw new TranslatorException(e);
} catch (JMSException e) {
throw new TranslatorException(e);
}
}
private void sendMsg(String userMessage) throws TranslatorException
{
try {
msg.setText(userMessage);
qsender.send(msg);
qcon.close();
} catch (JMSException e) {
throw new TranslatorException(e);
}
}
}
</code></pre>
<br />
This is the important parts from the Execution Factory:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: 300px; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> @Override
public ProcedureExecution createProcedureExecution(Call command,
ExecutionContext executionContext, RuntimeMetadata metadata,
Object connection) throws TranslatorException
{
return new MessageSendExecution(command, executionContext, connection);
}
@Override
public void getMetadata(MetadataFactory metadataFactory, Object conn)
throws TranslatorException
{
final Procedure messageSender = metadataFactory.addProcedure("messageSender");
metadataFactory.addProcedureParameter("queue", TypeFacility.RUNTIME_NAMES.STRING, Type.In, messageSender);
metadataFactory.addProcedureParameter("message", TypeFacility.RUNTIME_NAMES.STRING, Type.In, messageSender);
}
</code></pre>
<br />
To post the message, use this SQL statement:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> call Times.messageSender('java:jboss/exported/MyQueue', '<b>My Message</b>');
</code></pre>
<br />
Then monitor JBoss Management console or use your sample java JMS client to retrieve a messsage from queue:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> Sep 03, 2012 10:34:45 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.3.GA
Sep 03, 2012 10:34:45 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.3.GA
Sep 03, 2012 10:34:45 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.3.GA
JMS Ready To Receive Messages (To quit, send a "quit" message from QueueSender.class).
<b>My Message</b>
</code></pre>
<br />
<b>Note:</b> this translator code is not optimized for performance or thread-safety, I wrote it just for illustration purposes. Areas of potential improvement:<br />
<ul>
<li>implement UpdateExecution to make a translator handle INSERT SQL statement (more intuitive approach);</li>
<li>reuse connection to Queue;</li>
</ul>
Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-52719303592978330312012-08-17T18:03:00.001-05:002012-08-17T18:03:04.739-05:00Embedded TeiidWhen developing a new Teiid translators, I found very convenient to use a new 8.1 feature: Embedded Teiid. Basically this is a working Teiid server, which does not require JBoss AS as a host platform. Of course some functionality provided by Application Server has to be added manually (like transactions, clustering or data connection pooling), but in many cases it is unnecessary, especially when you need just a simple federated database or have to quickly test your new translator.<br />
<br />
Basic functionality can be unit-tested with frameworks like Mockito. But only integration test can give you a full picture. For integration tests I have to run a JBoss and Teiid instance as Cargo container with translator jars deployed - basically a mini-scaled copy of our application. This is a fragile and slow way - you have to emulate the environment on which your application is running, provide a proper configs and data sources, make sure the integration tests executed as part of your local build. Better if you have a flexibility of unit tests and completeness of integration tests combined together.<br />
<br />
Below is a sample code how to instantiate an Embedded server:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> EmbeddedConfiguration ec = new EmbeddedConfiguration();
server = new EmbeddedServer();
server.start(ec);
final JdbcDataSource h2ds = new JdbcDataSource();
h2ds.setURL("jdbc:h2:file:target/accounts");
EmbeddedServer.ConnectionFactoryProvider<DataSource> jdbcProvider =
new EmbeddedServer.ConnectionFactoryProvider<DataSource>()
{
public DataSource getConnectionFactory() throws TranslatorException
{
return h2ds;
}
};
server.addConnectionFactoryProvider("source-jdbc", jdbcProvider);
server.addTranslator(new H2ExecutionFactory());
ModelMetaData jdbcModel = new ModelMetaData();
jdbcModel.setName("Accounts");
jdbcModel.setSchemaSourceType("native");
jdbcModel.addSourceMapping("h2-connector", "h2", "source-jdbc");
teiidServer.deployVDB("example", jdbcModel);
</code></pre>
<br />
Create server, define model and data sources, deploy VDB - in a less then twenty lines of code. Now you can connect and run your queries:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> TeiidDriver td = server.getDriver();
Connection c = td.connect("jdbc:teiid:example", null);
final String sql = "select * from Product";
List<String> results = execute(c, sql, true);
</code></pre>
<br />
Really, cannot be simpler. The complete listing of unit-test class <a href="https://docs.google.com/open?id=0BxYj8BUOYsxIWFZEQ3NrMm4yaFU">attached</a> for reference. Also take a look on <a href="http://anonsvn.jboss.org/repos/teiid/branches/8.1.x/build/kits/embedded/examples/">"embedded-portfolio"</a> example from Teiid 8.1 sources, my post was greatly inspired by it.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com4tag:blogger.com,1999:blog-3682502565034270491.post-81205811469678120662012-08-07T12:28:00.000-05:002013-04-20T20:43:37.837-05:00My Experience with OrientDB JDBCI did some graph DB research and POC about six month ago for a "configuration management" task. The short description: the system might have a different set of modules, each consist from a bigger set of different parts. Almost all parts has different types of relations inside one module and across them. The task is to easily store this data, and have some complex search and retrieval logic. A typical graph database task. My weapon of choice at that point was <a href="http://www.orientdb.org/">OrientDB</a>.<br />
My first results were quite good - I was able to quickly import data from our <a href="http://www.jboss.org/teiid">Teiid</a> relational federated database and build a graph with 322591 vertices and 322636 edges. The complete load process took about 40 minutes, mostly because slowness on our side. Our federated database consist from many different parts: fast and small H2 tables, custom data translators to a legacy data, and large and slow DB2 tables with hundreds of thousand records on mainframe.<br />
I used both Native and Tinkerpop API and found it useful and easy to use. In my POC I took <a href="http://jung.sourceforge.net/">Jung</a> for graph visualization, and for a collection of graph algorithms.<br />
Here are just some results for the graph above: <br />
<ul>
<li>"Find Shortest Path" (Dijkstra): 4781.0 ms</li>
<li>"Check Nodes Connected" using Dijkstra algo: 7392.0 ms</li>
<li>"Check Nodes Connected" using SQL TRAVERSE extension from OrientDB: 23829.0</li>
</ul>
Note that results were obtained from in-memory graph, with no optimization (raw quality java code, no indexes defined in OrientDB, unlimited walk depth in traversal). I am confident the results can be improved very easily.<br />
My next step was the integration of OrientDB into Teiid. Since we have a huge federated database, the way to use SQL to send and retrieve data from graphDB considered as convenient but somewhat unnatural approach. I can explain benefits and drawbacks of using this approach separately, it deserves a whole new post. Unfortunately that time (6 months ago) I had to step back: the JDBC implementation of OrientDB was very raw and did not provide enough metadata for Teiid 7.6 to be included in federated database.<br />
I got back to this task recently, trying a new Teiid 8.1 (it has a useful notion of defining metadata in DDL along with NATIVE support like the old Teiid) and with latest OrientDB 1.2.0-SNAPSHOT. I took the latest orientdb JDBC driver sources from git and compiled jar file by myself. Taking the latest JDBC sources was probably my biggest mistake. I've tried to connect to running OrientDB instance using Eclipse Database Explorer and run some queries. Unfortunately even simple and innocent queries against my imported graph or even against built-in "tinkerpop" database provided inconsistent results and even caused Eclipse to freeze sometime:<br />
<br />
<pre>select * from V where o_id like 'demodata.s%';
select from 6:6 where any() traverse(0,10) (o_id = 'demodata.stats');
</pre>
<br />
Something definitely wrong in the latest JDBC driver code. Obviously, the JDBC driver is not a top priority feature for a graph database. I probably can implement the needed functionality by myself - write a custom Teiid translator (might be a considerably big task, depends of the level of SQL support we want to achieve), or try another graphDB implementations. I will be looking also on neo4j - they also have a jdbc driver. The query language they support in JDBC is not SQL, but (I think) it can be worked out.<br />
This task is on hold for now anyway, I'll make an update in two-three months.<br />
<br />Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com4tag:blogger.com,1999:blog-3682502565034270491.post-83467996093498994082012-08-02T14:07:00.000-05:002012-09-03T22:51:50.359-05:00Poll vs Push in Database using JBoss TeiidSometimes you have to monitor a specific table or view in your database, and do some action when INSERT or UPDATE happens. There are two approaches - you can periodically <b>poll</b> the table using Quartz or any other scheduling tool. Or you can be notified by database itself when your table modified - database will <b>push</b> you the new data.<br />
The second approach even more beneficial when your table get updates rarely (so you not waste CPU by returning empty results most of the time), or when you want to be notified immediately when the new data available.<br />
<br />
The good solution to the task above on my mind is database trigger. The commercial database vendors (like Oracle, if I am not mistaken) provide a triggers which (when fired up) are capable to run an arbitrary java code. But what if you don't have Oracle, and the data source of your choice is mySQL, Apache Hive, H2, or even arbitrary CSV file, laying down somewhere on filesystem?<br />
<br />
Here is my approach: I made an UPDATABLE VIEW for my table in H2 database under JBoss Teiid, then created a VIRTUAL FUNCTION (Teiid virtual functions are capable to run specified java class and method). Last step - define two triggers: INSTEAD OF UPDATE and INSTEAD OF INSERT. When my table in H2 got a new record, or existing one updated - my java class gets immediately executed by Teiid.<br />
<br />
Implementation Details:<br />
<ul>
<li>Below is the Teiid 8.1 model definition with VIEW, VIRTUAL FUNCTION and two TRIGGERs defined. Note a property "lib" - it is a name of JBoss 7 Module with your java class in it (yes, you have to create a JBoss module. It is a simple task and well documented).</li>
</ul>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <vdb name="DynamicPortfolio" version="1">
...
<property name="lib" value="com.test.teiid.translator.async" />
<model name="TrigView" type="VIRTUAL">
<metadata type="DDL"><![CDATA[
CREATE VIRTUAL FUNCTION RepHealth(
healthTime varchar,
policyKey varchar,
objKey varchar,
healthState varchar)
RETURNS integer
OPTIONS (JAVA_CLASS 'com.test.teiid.udf.HealthStatesTeiidFunction',
JAVA_METHOD 'reportHealthState');
CREATE VIEW HS_VIEW OPTIONS(UPDATABLE TRUE)
as select * from Accounts.HEALTHSTATE;
CREATE TRIGGER ON HS_VIEW INSTEAD OF INSERT
AS FOR EACH ROW
BEGIN ATOMIC
SELECT RepHealth(
New.HEALTHTIME, New.POLICYKEY, New.OBJKEY, New.HEALTHSTATE)
from HS_VIEW;
END;
CREATE TRIGGER ON HS_VIEW INSTEAD OF UPDATE
AS FOR EACH ROW
BEGIN ATOMIC
SELECT RepHealth(
New.HEALTHTIME, New.POLICYKEY, New.OBJKEY, New.HEALTHSTATE)
from HS_VIEW;
END;
]]></metadata>
</model>
...
</vdb>
</code></pre>
<br />
<ul>
<li>The VDB changes above should work with Teiid's "dynamicvdb-portfolio" example, I installed this example on my Teiid in advance and created schema for additional "Accounts.HEALTHSTATE" table in H2. The DDL is below. You can add a new model right into portfolio-vdb.xml from the example.</li>
</ul>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> CREATE TABLE HEALTHSTATE
(
HS_ID integer,
HEALTHTIME varchar(8),
POLICYKEY varchar(8),
OBJKEY varchar(8),
HEALTHSTATE varchar(8),
CONSTRAINT HEALTHSTATE_PK PRIMARY KEY(HS_ID),
);
</code></pre>
<ul>
<li>Write your java class and package it as JBoss Module. The called java method should be static.</li>
</ul>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> package com.test.teiid.udf;
public class HealthStatesTeiidFunction {
public static int reportHealthState(String healthTime, String policyKey, String objKey, String healthState)
{
String ret = "At:" + healthTime + " for Policy:" + policyKey + " and Object:" + objKey + " the State is:" + healthState;
System.out.println(ret);
return 1;
}
}
</code></pre>
<br />
<ul>
<li>Start JBoss, make sure your module and VDB are deployed and active. Connect to your database and issue INSERT or UPDATE statement against your defined view. </li>
</ul>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> insert into HS_VIEW(HS_ID, HEALTHTIME, POLICYKEY, OBJKEY, HEALTHSTATE)
values(4, '10:28 AM', 'policy1', 'obj1', 'RED');
</code></pre>
<br />
<ul>
<li>See the java output in console:</li>
</ul>
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> 13:51:18,336 INFO [stdout] (Worker0_QueryProcessorQueue0) At:10:28 AM for Policy:policy1 and Object:obj1 the State is:RED
</code></pre>
<br />
Note that since triggers defined as "INSTEAD OF..", the actual insert or update does not happen. Also, in order to be notified by the database changes, your underlying system should issue inserts or updates not to the actual H2 table, but to Teiid UPDATEABLE VIEW we defined (HS_VIEW in example above). Teiid itself is a federated database, available by JDBC, but connecting to it might be an added complexity to your solution.Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com0tag:blogger.com,1999:blog-3682502565034270491.post-17902105298450271602012-08-01T15:47:00.002-05:002012-08-02T12:31:32.280-05:00JBoss 7 and Maven PluginThis days I play a lot with the latest version of JBoss Teiid (8.1.Beta2). As part of my development activity I have JBoss 7.1.1.Final running on my workstation, and I want to shorten my development cycle: modify code -> build war -> deploy. Yes, I know about JRebel, but today I want not to use it.<br />
My project build tool is Maven, so solution is straight: use jboss-as-maven-plugin. I want to generate war file from maven and deploy it to standalone JBoss without restarting it.<br />
<br />
<h3>
Steps to follow</h3>
<br />
<ul>
<li>Define plugin in pom.xml. The definition below attaches to "clean" and "package" maven life cycles.</li>
</ul>
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;"><project...><br /> <build><br /> ...<br /> <plugin><br /> <groupId>org.jboss.as.plugins</groupId><br /> <artifactId>jboss-as-maven-plugin</artifactId><br /> <version>7.1.1.Final</version><br /> <configuration><br /> <filename>${project.build.finalName}.war</filename><br /> </configuration><br /> <executions><br /> <execution><br /> <id>undeploy</id><br /> <phase>clean</phase><br /> <goals><br /> <goal>undeploy</goal><br /> </goals><br /> <configuration> <ignoreMissingDeployment></span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;"> true</span></span><br />
<span style="font-size: small;"><span style="font-family: "Courier New",Courier,monospace;"> </ignoreMissingDeployment><br /> </configuration><br /> </execution><br /> <execution><br /> <phase>package</phase><br /> <goals><br /> <goal>deploy</goal><br /> </goals><br /> </execution><br /> </executions><br /> </plugin><br /> ...<br /> </buiid><br /></project</span><span style="font-family: "Courier New",Courier,monospace;">></span></span><br />
<br />
<ul>
<li>Optionally - create Eclipse launch configuration to execute "mvn package" or "mvn clean package".</li>
<li>Start JBoss, run launch configuration in Eclispe, check messages in your server log that new context is regustered and war file is deployed / replaced. </li>
</ul>
<h3>
</h3>
<h3>
What happens behind the scenes?</h3>
Usually deployed war files placed in <jboss7>/standalone/deployments folder (when standalone configuration used). After JBoss restart the war file gets deployed (new "xxxx.deployed" flag file gets created in the same folder).<br />
In case of "jboss-as-maven-plugin" deployment, the <jboss7>/standalone/configuration/standalone.xml file gets modified: a new "<deployment>...</deployment>" added to it.<br />
The actual deployed war file can be found now under one of the <jboss7>/standalone/data/content/ subfolders.Andriy Rokhmanovhttp://www.blogger.com/profile/15291787076887612433noreply@blogger.com2