9 Months Later...Web Development using RoR.

Its been 9 months since my last post but my adventures in web development never stopped. Well, that's not %100 true. Life/work constantly got in the way but despite the many interruptions and pauses, I've managed to  continue with my web development efforts. However, I have strayed a long way from the application development stack which I explored leading up to my last post.

Initially, I thought that I would only stray temporarily. After the arduous work of developing the  database-backed, J2EE-based, web-service integrated server and deploying the whole stack into the production amazon run-time environment, I hoped that adding more functionality by building on this foundation would be easy. I was wrong. The first challenge I hoped to address was User Interface. GWT proved too unwieldy for making fluid, responsive UIs. So, I searched for better Web UI frameworks. That search led me, inescapably, to the unholy trinity of of Web browser programming technologies:

• HTML
• CSS
• Javascript

Initially, I was highy skeptical of these technologies because, in my mind, they were so 1990's. After all the browser compatibility wars of the 90's and 2,000's, I thought that for sure these technologies had given way to sexy new platform-independent technologies. Well, it turns out that these are the sexy new platform independent technologies. Much has been written about the browser vs native OS vs virtual OS  (like Java's JVM) software development approaches. So, I'm not going to rehash those arguments here. But the bottom-line for me was that as long as browser based web development was going to be a significant part of my software development interest, I had no choice but to dig into the HTML-CSS-Javascript trifecta. Though these technologies are considered client-side technologies...more specifically: browser-based client-side technologies, they have a heavy impact on the server-side technologies used to serve them up. And that's why a plethora of specialized server-side technologies  tailor-made precisely for developing browser applications have cropped up all over the web development scene. These Server-side technologies leverage php, python, ruby, javascript and other interpreted languages coupled with server frameworks like Laravel, Sinatra and Rails, for the sole purpose of making it easier to create dynamic HTML, CSS, Javascript content for browsers to serve-up rich/fluid UI experiences. After several weeks of floundering on my way to  discovering the deceptively simple assertions made thus far in this post, I landed on Ruby on Rails for my web application server-side framework. After many hours reading up on web application frameworks, watching developer conference talks, sheepishly trying php, ruby, javascript, jquery, html, css code snippets, Ruby on Rails stuck with me. One very big reason why RoR stuck was Michael Hartl's most excellent, absolutely awesome Ruby on Rails Tutorial which he has made public at no-charge on the web as a written tutorial. It's also available as an ebook or ebook+video package at very reasonable prices. His tutorial is amazing. The title says "Ruby on Rails Tutorial" but its actually a speedy study of a sizable chunk of modern web application technologies, tools and best practices. As the name implies, this includes RoR but the tutorial covers much more. From generally applicable Client-Server concepts like Model View Controller (MVC) architecture , RESTful application design, Source Control Management using Git and Github, Test Driven Development (TDD) to Web/Browser-specific concepts like HTML, CSS, Javascript, Ajax and much more, all covered in a very pragmatic, hand's-on, engaging way which never gets too heavy despite the sizable depth and breadth of the material...and that is how RoR stuck with me and how I stuck with RoR.

But despite the awesome RoR resources, and huge swathe of web application development capability that I've gotten since finishing Hartl's tutorial (several times) and creating several demo apps of my own, the ability to realize a fully functioning Minimum Viable Product (MVP) continues to remain just outside my grasp. The adventure continues.

Deployed Web App version 0.000001

Since completing coding (several posts ago), my big adventure has been ramping up on the linux system administration skils needed to configure and deploy apps and services into (Amazon AWS/Cloud) Infrastructure. Along the way, I discovered that the AWS Free Usage Tier "Micro" instance server  is not enough to run my application stack on a single server so I had to upgrade to a "Medium" (aka "not-free") server instance. I'm also increasingly under the impression that this piddly app could have been executed (much more effectively) with a direct AJAX (Asynchronous JavaScript and XML) implementation like javascript+php rather than the heavy Google GWT AJAX framework plus the Java Enterprise Edition (J2EE) and SOAP Web Services platfoms that I've employed...but that's another project, possibly the next one, for Adventures in Webdevland. 

Finally, a fully deployed (but very basic & buggy) web application...

WEATHER SCOUT Version 0.000001 

Features:

Front-end/UI:

Display's real time weather for almost any city in the world.

And because it's hosted in the cloud, in the real-live-world-wide-web, its accessible from anywhere in the world!

Also, it has a cross-platform user interface so it works on phones, pc, mac, and ipads ! 

And it maintains consistent look & feel across all devices.

Back-end:

"Administrator" console for configuring application features (available cities can be configured...currently set to USA)

3 Tier Architecture: Web Browser UI, Web Server (Apache Tomcat), Enterprise Server(Oracle Glassfish)

SQL Database for data persistence. Users' available cities' are read from DB (Apache Derby) which is configurable by Application Administrator (via custom Web UI also hosted in the cloud)

Service Oriented Architecture (consumes/exposes SOAP Standard Web Services) to receive and display "live" weather data.

...and more bonus features constantly being discovered (aka bugs)

Release Notes

Corporate firewalls tend to block some amazon services. 

So, if this does not work from your work pc, please try your phone, or try from different network.

Performance is sloooow. Please be patient

Weather Scout is currently configured to work for US cities

User Interface

http://ec2-204-236-197-191.compute-1.amazonaws.com:8081/WeatherScoutUser/

Admin Console

http://ec2-204-236-197-191.compute-1.amazonaws.com:8081/WeatherScoutAdmin/

Amazon Cloud Infrastructure (Continued)

Signing up and instantiating my first ever server in the cloud was impressively easy. Below is what the server instances look like in the admin console. The background window shows my Amazon hosted Virtual Machines or VMs. While the super imposed window is for connecting to the server instances. To get this form, right click on a server instance and select "connect". You have two options to connect SSH standalone client or SSH browser client which requires Java plug-in....and that's where I ran into my first issue (or educational opportunity).

The issue was not with AWS itself. The issue was that I did not have a Java Runtime Environment (JRE) installed nor a Java browser plug-in. The AWS popup screen was even helpful enough the detect and display this fact with links on how to get the JRE and plugin. And that's where it went awry for while. Considering how much GWT Java and J2EE development I've been doing, I was certain that "Of course, I had Java installed". I recall getting/installing both Oracle Java and Open Java dev kits. And after quite a bit of poking around, I realized that what I had were the Java development kits...aka JDKs, used by IDEs and OSs to compile/run Java programs which are quite something different from  the Java run-time environment, JRE which comes in the form of browser plug-ins for  running browser based java....doh. This is the kind of basic stuff that get's you stumbling and drains time but sigh...its how you learn. Remember, when you are not in the world of commercial Windows or Mac OS, this stuff is not automatically, taken care of for you.  So, the next challenge was getting/installing the JDK. Canonical, the makers or Ubuntu linux, have a software repository where they and any linux software supplier can register software so that download/install can be straightforward. Unfortunately, the Java JDK which used to be in this repository is no more. It would have been OK if the JDK was never in Canonical's repository but because it once was, there was all this conflicting information on the web about how to get it from Canonical. Which delayed me from simply getting it from Oracle which I was reluctant to do because my linux/ubuntu skills are still fuzzy, at best. So, I really appreciate a nice script based install which puts things in all the right directories, makes all needed symbolic links and handles shell environment variables etc. I finally stumbled on a credible  looking software repository which hosts the Oracle JDK for easy install. But first, I needed to add the repository to my Ubuntu configuration. Fortunately, I found instructions for that too. Thanks Web Upd8 for the JDK and Instructions.

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

Just a little bit more tinkering and I was ble to get the plug-in installed in my browser(s)....phew. Now, I could use the in-browser SSH client from Amazon Admin console to log into my brand new Amazon (Free Usage Tier) Micro Instance. I did all this, so that I would not have to figure out how to set-up SSH using the native command line application on my system. It turns out that this other approach with native SSH command line was not so hard but also much better experience. Anyway, with the plug-in Installed, everything worked like a charm, following the Amazon documentation. But it did not take long before I figured out how to use direct command line SSH. One more think I had to learn was how to transfer files from my local machine using SCP command and from this point on, accessing my Amazon VM was almost exactly same as accessing my local Ubuntu VM via command line....pretty darn cool.

Getting started with Amazon Cloud Infrastructure

Though slightly intimidated by all the (impressive) infrastructure and software-as-a-service (SAAS) offerings available from Amazon and other IT suppliers hosted on Amazon, I found the intros and documentation quite accessible. I was especially thrilled to discover Amazon's Free Usage Tier.


AWS Free Usage Tier (Per Month):
Elastic Compute Cloud (EC2)
750 hours of Amazon EC2 Linux† Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
750 hours of Amazon EC2 Microsoft Windows Server‡ Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
750 hours of an Elastic Load Balancer plus 15 GB data processing*
30 GB of Amazon Elastic Block Storage, plus 2 million I/Os and 1 GB of snapshot storage*
Simple Storage Service (S3)
5 GB of Amazon S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests*
DynamoDB
100 MB of storage, 5 units of write capacity, and 10 units of read capacity for Amazon DynamoDB.**
Relational Database Service (RDS)
750 hours of Amazon RDS Single-AZ Micro DB Instances, for running MySQL, Oracle BYOL or SQL Server (running SQL Server Express Edition)‡‡ – enough hours to run a DB Instance continuously each month*
20 GB of database storage
10 million I/Os
20 GB of backup storage for your automated database backups and any user-initiated DB Snapshots
SimpleDB
25 Amazon SimpleDB Machine Hours and 1 GB of Storage**
Simple Workflow (SWF)
1,000 Amazon SWF workflow executions can be initiated for free. A total of 10,000 activity tasks, signals, timers and markers, and 30,000 workflow-days can also be used for free**
Simple Queue Service (SQS) and Simple Notification Service (SNS)

100,000 Requests of Amazon Simple Queue Service**

100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service**

more...

The Free Usage Tier was especially welcome after I tried the Simply Monthly Calculator with the reference web application topology selected. With the default load/configurations, the estimated monthly cost came up to the stunningly-out-of-my-budget amount of: $1,660 per month.

Even though this 3-tier web application is the topology I'm going for, I should be able to fit my web app (Weather Scout) into the Free Usage Tier offering (based on expected utillization being close to zero, no load loadbalancing, scaling group, or high availablity clusters).

Finally...a full Web Application Stack!

Yes. Its rudimentary but...boy, did it take a while!

I have been working at this for about 10 hrs per week since my last blog post 9 weeks ago.

APPLICATION DESCRIPTION
The application use case scenario is pretty basic. An administrator uses the "Admin Console" screen on the left to select cities to make available to users. On the right, users select cities to view weather status. As the user types a city that they would like to add, the app helpfully provides suggestions. The suggestions are limited to just the cities permitted by the administrator in the Admin Console.

WHY DID IT TAKE SO LONG
Its hard to remember all the challenges but I'll try to list some. I'll probably add more to this list and get into more details on these in future posts.

Java/POJO (Plain Old Java)
Applied new (to me) concepts to handle Exceptions, Collections, XML parsing, and even variable declarations and scope (a missing use of "final" declaration was the source of 10hrs - 20hrs of tedious debugging)
Java/EJB (Enterprise Java Beans)
Challenges with consuming SOAP Web Service from J2EE/EJB which I was able to do just fine using regular Java
Java/JPA (Java Persistence API)
Challenges with Java Persistence API syntax and usage
Google GWT
Spent many hours learning the idioms of GWT framework particularly around user interactions and Client/Server Remote Procedure calls.
Apache Derby
Challenges with starting, configuring Apache Derby database as standalone server outside of Eclipse
IDE.

NEXT STEP...
Heavy sigh and deep breadth...deploy servers and services to the cloud!

It's kinda like baking a cake...

For this post, I put together a diagram of all the components (aka blood, sweat and tears) that went into the very humble Temperature converter web app in preceding posts. Ok,"blood, sweat and tears" may be a tad bit dramatic but not by much. Certainly, long long long hours went into first learning what constitues an E2E web application stack, then of the many, many, many options out there, selecting a suitable stack. Theeeeen, I had to get them, install them, learn them and use them. An experience which, so far, feels like learning to bake a cake, while learning how to make everything that goes into cake baking. And I mean "everything". The cake and all it's ingredients can be compared to the program itself. The java language abstractions and logical constructs that go into making the program are the ingredients. These are the typical, perhaps obvious, things you think of when approaching the task of cake baking or application development. However, for a complete web application (or almost any non-trivial useful application), you also need tools, appliances, and infrastructure. Even for the "simple" task of cake baking these tools, appliances and infrastructure are quite extensive when you think about it. There are uncountable variations of hand tools, electro-mechanical appliances and fairly expensive infrastructure like furniture, cool/hot storage and of course....an oven!  In cake baking, you also need foundation skills that are easy to take for granted. Skills like reading, writing, vocabulary, math, measuring). Then their are craft skills like kneading, decorating, ingredient sourcing & preparation and others. Sourcing & prep, are not as trivial as one might think. For example, where do you get creme fraiche from and how is it best to stored & prepped for different uses? Finally, and just as importantly, in addition to all these these pre-requisites, you need to have some (good) ideas of satisfactory end results. That is, what does a cake look like, taste like, smell like, how is it served, who uses it, for what (if anything else besides eating), when, how often, what should it cost etc. It begins to be clear that baking the most simple of cakes is far from trivial. 

Ok, so maybe I've succeeded in reminding you about the hidden complexities of cake-baking. Big deal, You are probably thinking "Well, cake baking is one thing. Yes, maybe it is surprisingly more complex than anyone would initially think but software development is totally something else. Of course, that's a complicated thing!." No big surprise there, genius. People, and not just "people", the biggest brained of our kind, spend their lives studying software development aka computer science aka information technology. Clearly, if this is surprising to me, I'm on the other end of the spectrum from exactly those big brained ilk. However, hear me out for a few more seconds. Yes, the best and the brighest spend lifetimes studying software development. But so do the best and the brightest cake makers. Yet, just because the highest forms of software development, like cake baking, takes lifelong dedication and training, this does not mean that regular, modern, affluent, intelligent, capable people should not be able to make delicious cakes or software programs!

Hopefully, prior chronicles provide enough context for the components, tools, appliances and infrastructure outlined in the diagram above. I'll be diving into these more in future adventures which, most immediately, will include adding database/persistence to the humble app. This will also include beefing up the functionality to include features affectly known as user CRUD or record Creation, Reading, Updating, Deletion through the user interface. Also, I'll need to actually deploy the application in to the wild world web...yes, I said "wild" world web...and I meant it.

First E2E Web App Stack (Almost)

Finally E2E Web Application using Google GWT, Apache Tomcat, Glassfish, J2EE and Web services!

In my last attempt, I tried rejiggering the GWT Designer Log-in Manager tutorial to implement my own functionality. Specifically, I planned to keep the UI elements (panels, textboxes, buttons etc) but change the labels and functions to implement a Temperature Converter web application which accepted temperature values and made calls to converter service which actually resided in the server back-end. My desire was for the service to be SOAP/HTTP service which was called by GWT server which in turned serviced the GWT client. My implementation approach was to just deal with GWT client/server aspect first before introducing the SOAP service. This is the Log-in Manager tutorial, I started with:

Tutorial: Creating a Login Manager Application with GWT Designer

And this was the GWT Remote Procedure Call tutorial which I was leveraging in order to rejigger above tutorial to add RPC functionality:

Making Remote Procedure Calls

I had written about completing the RPC tutorial in earlier post (Begining Web/SOA/J2EE Dev). However, this was were I as getting stuck. This RPC tutorial has a rather confusing/clumsy syntax for linnking the web service interface, its implementation, the client proxy and the async callback function. It was not helping that I was trying to learn how to apply the RPC at the same time as I was rejiggering UI tutorial which was also my first exposure to the GWT Designer (Window Builder project). So, I took different approact of first completing start-to-finnish GWT Desinger project and then returing to the RPC aspects. For a fresh start I turned to a different GWT Designer project from the Log-in Manager application. It's another StockWatcher app similar to the one I did in Begining Web/SOA/J2EE Dev but this one is done using GWT Designer/Window Builder:

Tutorial: Creating a Stock Watcher with GWT Designer

I completed this tutorial painlessly and it gave me even more appreciation for GWT for web application UI over straight-up HTML/Javascript which, again, is what GWT ultimately generates/deploys. It just makes the coding/development seemingly easier/better because it allows you to develop in good old Java and uses familiar UI development constructs. So, I'm now a GWT client app developer Wizard (not exactly) but the problem still was that I had no server side implementation or link from client to server. Fortunately, I found a different tutorial which had much more straight forward approach to dealing with this GWT RPC:

GWT RemoteService Wizard

First, I used the GWT RemoteService Wizard to redo the StockWatcher GWT Designer web application to remotely call the stock watcher service (rather than use client side implementation). Worked like a charm. Then I proceeded to create my very own GWT Designer web application from scratch to implement a temperature conversion application. First, I created the client side using same steps as Stock Watcher tutorial with the temperature conversion logic located in the client, then I followed the GWT Remote Service Wizard steps to transfer the conversion logic to the GWT service, finally I used eclipse webservice client generator to create client class from my web service which was a J2EE application (EAR) I had previous created deployed on Glassfish server. that experience will be covered in future blog entry. The GWT pieces are covered in the Google tutorials but the Eclipse web service client generator is not. From the GWT Window Project in Eclipse, navigate to the following:

File -> New -> Other -> Web Services - Web Service Client

Then enter the WSDL location for you web-service. Turn the dial on the client-o-creator to "Develop Client"...you can turn it all the way up to assemble, deploy, or test but then you'd need to add/configure a suitable Web Application Server to your workspace (which can be a very useful thing to test or get a feel for the Web Service operations but not something needed right this second...especially since I develeloped/deployed this particular service).

Hit Finnish...and presto !

Now, you'll see a new package with a bunch Plain Old Java Objects (pojos) in your project explorer new. Below, this new package is appopriately named com.ola.pojo (named by me when I originally created the webservice). The package contains an Interface class called TempConverter.java which shows the methods that the class provides aka as the operations provided by the service. Additionally, the package includes a bunch of supporting classes TempConverterProxy.java, TempConverterPrixy,TempConverterServiceLocator.java,TempConverterSoapBindingStub.java, all created by the Eclipse tooling from the WSDL which I had specified. All you now need to do use this service is to instantiate an obect of TempConverterProxy and you can call any of its methods/properties just like its a pojo object...pretty neat, I think.

And after all that - creating web service using J2EE, deploying that services as a EAR package into Glassfish J2EE container, Creating GWT RPC server to call/consume that web service, and then finally creating GWT client to provide UI and use the web service methods via RPC calls to GWT server - what you have is...

So, at the top of the post, I declared success of full E2E implentation of a web application stack...well,  almost the full stack. This humble app is still missing the database/persistence functionality. Next, I'll try to get a diagram together illustrating the full stack. And I'll continue work on another app to also implement persistence.