JavaScript, Hoisting, Firefox, If/then Statements

Friday, March 21, 2014

Quick intro about JavaScript and Scoping

In most C and Java type languages a block (that is any curly brace) defines the scope of a variable or a function. However in JavaScript, scoping is at the function level. So when I define a variable inside a block (if/then statement for example) it can be used throughout the entire function it is in, including outside its defined block.

JavaScript will also take that function that is defined anywhere in a function and set its definition at the top of the function behind the scenes. This is called hoisting, and basically it allows you to call a function that is defined below the code you are calling it from.

 

So for example:

 

This works in all browsers because JavaScript hoisted the inner function to the top of the outer function behind the scenes.

If/Then statements and Firefox

The problem is in Firefox if you define a function inside an if/then statement the function no longer gets hoisted. This only happens in Firefox.

 

So in the following example, the following code works in IE and Chrome but is broken in Firefox.

 

In Firefox I get an error because it did not hoist the callHoistedFunction therefore it was not defined at the point it was being used.

ReferenceError: callHoistedFunction is not defined
var hello = callHoistedFunction();
 

So the moral of the story is never define a function inside a “if/then” or “try/catch” statement. As a matter of practice, you should probably just define all variables and functions at the top of the outer function as that is where it gets hoisted to anyways.

 

This is from Douglas Crockford’s book the “JavaScript, The Good Parts”. Pg 102.

 

In most languages, it is generally best to declare variables at the site of the first use. That turns out to be a bad practice in JavaScript, because it does not have block scope. It is better to declare all variables at the top of each function.

 

Here is his video on his book in case you were also interested.

http://www.youtube.com/watch?v=hQVTIJBZook

 

Hope this helps,

AngularJS - Using a Service to Communicate Between Two Different Controllers

Monday, January 6, 2014

Introduction

AngularJS, being an MVC type of framework allows you to have a controller that is responsible for marshaling data to and from a view on your web page.

Well, what if you have a multiple controllers on a page that are completely independent from one another, but in certain cases you want those controllers to be able to communicate back and forth. I say completely independent, because you could nest the controllers, making a parent-child relationship, and then you could use the $rootScope object to communicate with each other.

How $rootScope Works

Every AngularJS application has exactly one $rootScope. All other scopes inside the application are children of the $rootScope. So in the case of nested controllers, the child controller's scope object is a descendent of the parent scope which is ultimately a descendent of the application $rootScope. So, for example, if you try to evaluate {{message.text}} on a child scope. and If that property does not exist on the child scope, AngularJS will then evaluate the parent scope for that property, and then ultimately it will evaluate the $rootScope.

So, in the case of controllers that have a parent-child relationship, you can use the existing inheritance between the controller to facilitate the communication.

Using Factories and Services to Facilitate Communication Across Independent Controllers

In the case your controllers are independent from each other, then you will need a service that will link the two controllers. In my example, I am using a factory which essentially a service that returns an object.

My example is just a web page with two text boxes that chat with each other. Each text box is in a different controller, but each controller has the list of message from both text boxes.

Here is the HTML.

    <div ng-controller="Controller1">
      <h1>Controller 1</h1>
      <ul ng-repeat="msg1 in messages1 track by msg1.id">
        <li>{{msg1.text}}</li>
      </ul>
      <p>
        Controller 1 Message: <input type="text" ng-model="text">
        <button ng-click="postMessage()">Post</button>
      </p>
    </div>
    <p></p>
    <div ng-controller="Controller2">
      <h1>Controller 2</h1>
      <ul ng-repeat="msg2 in messages2 track by msg2.id">
        <li>{{msg2.text}}</li>
      </ul>
      <p>
        Controller 1 Message: <input type="text" ng-model="text">
        <button ng-click="postMessage()">Post</button>
      </p>
    </div>

Each ng-controller section above has a copy of the same messages that is sent out by the service that manages the message list.

So to link the two controllers above, I have added a Angular Factory that will be injected into each controller. Here is the code.

app.factory('messageService', function ($rootScope) {
    var messenger = {
        messages: [],
        identity: 0,
        addMessage: function (text, caller) {
 
            this.identity += 1;
            var id = this.identity,
                message = {
                    text: caller + text,
                    id: id
                };
 
            this.messages.push(message);
            $rootScope.$broadcast('messageAdded');
        }
    };
 
    return messenger;
});

In this service, I am injecting the $rootScope, which as available throughout the Angular application. I have a function called addMessage which, when called by a controller, adds a message to the collection of messages and then broadcasts that the message was added using the $broadcast function. Also, keep in mind that a factory in Angular is a singleton object, so there is only one occurrence of its state in the current Angular application, and any other object that access its state will get what ever values it hold at that current time.

Looking at the code for the controllers, you will see that each of the controllers injects the service and calls the addMessage function when a user enters a text message and clicks the Post button.

app.controller('Controller1', function ($scope, messageService) {
    $scope.messages1 = messageService.messages;
    $scope.post = {
        text: ''
    };
 
    $scope.postMessage = function () {
        console.log($scope.post);
        messageService.addMessage($scope.text, "controller 1");
    };
 
    $scope.$on('messageAdded', function () {
        $scope.messages1 = messageService.messages;
    });
});

Each controller also has $on listener which is listening for the messageAdded broadcast. When this broadcast is detected by the controller, it fires the function to update the message collection retrieving the updated message list from the service.

I am finding this approach useful when the application you are working on gets a bit more complex and you have a lot of controllers on a page responsible for different things. In certain cases you may want to update a portion of the page that is managed by a controller, when a user clicks on another part of page managed by a different controller.

Here is the Plunker example

http://plnkr.co/edit/UA28cXfF0I8r3M54DyYa?p=preview

 

Old School MVP for ASP.Net Web Froms

Saturday, November 23, 2013

Introduction

Sometimes you have to work with a legacy site because, let's face it, it works and it's just not worth the effort to rewrite the whole thing via the newer technologies.

But still, if you need to add a web page and you still want to be able to separate the presentation logic from the domain logic, the Model View Presenter Pattern is the way to go.

I am not going to go into much detail about the pattern itself, there is a great article about it up on MSDN, but I would like to show you a great implementation of it that reduces the amount of code you will need to write to get it going.

The Base View

The first thing to do, is to set up the views, which will be interfaces the web page will use to represent the data on the page.

public interface IView
{
     
}

Yeah, not much here, just an empty interface, but this interface will be inherited from all the other views, making them all recognizable.

The Base Presenter

The base presenter class gets the base view injected into it. By doing this, the presenter will have a way to map the model to the web page.

public abstract class Presenter<TView>
{
    public TView View { getset; }
 
    public abstract void OnViewInitialized();
 
    public abstract void OnViewLoaded();
 
}

The Base Page class all Web Pages Will Inherit From

Now that we have the presenter and the view, we can create a base web page class that will inherit from the System.Web.UI.Page class. Using this class as the base, the web page will automatically inject the presenter and the view onto the page (you can also do this for a UserControl base class).

public abstract class MvpBasePage<TPresenterTView> : Page where TPresenter : Presenter<TView>
    {
 
        protected Presenter<TView> Presenter { getset; }
 
        protected MvpBasePage()
        {
            if (!(this is TView))
                throw new InvalidOperationException("Must impliment the generic type TView");
            try
            {
                Presenter = ObjectFactory.GetInstance<TPresenter>();
                Presenter.View = (TView)((object)this);
            }
            catch
            {
                Trace.Write(ObjectFactory.WhatDoIHave());
                throw;
            }
 
            Debug.WriteLine(ObjectFactory.WhatDoIHave());
        }
    }

I am using StructureMap to register the presenter classes, but any Inversion of Controll container should work here as well.

The Default Page Presenter and View

Now that I have all my base classes ready I can then start using them to create web pages that will conform to the Model View Presenter pattern. So for my default page, I am going to impliment a small version of my trusty sample application, which is a project tracker. You can see an almost full version that uses AngularJs on GutHub, but in this case I am just going mock a client list, which is the first page.

So my client list view is just going to have two properties on it: a content title, and a list of clients (Note: you can also have functions and events on the view for more complicated scenarios).

public interface IClientsView : IView
{
    string ContentTitle { getset; }
    List<Client> Clients {  set; }
}

My presenter for the clients just calls a service that is mocking a list of clients.

public class ClientsPresenter : Presenter<IClientsView>
  {
      private readonly IClientsService _clientsService;
 
      public ClientsPresenter(IClientsService clientsService)
      {
          _clientsService = clientsService;
      }
 
      public override void OnViewInitialized()
      {
          View.ContentTitle = "Client List";
          List<Client> clients = _clientsService.GetAll();
          View.Clients = clients;
      }
 
      public override void OnViewLoaded()
      {
      }
  }

Look Ma! Hardly any code behind!

Now with all this in place my Code Behind file only has the responsibility of marshaling data to and from the view, making it very clean.

public partial class _Default : MvpBasePage<ClientsPresenterIClientsView>IClientsView
    {
        private List<Client> _clients;
 
        public string ContentTitle { getset; }
 
        public List<Client> Clients
        {
            set
            {
                clientsRepeater.DataSource = value;
                clientsRepeater.DataBind();
            }
        }
 
        protected void Page_Load(object senderEventArgs e)
        {
            Presenter.OnViewInitialized();
            if (Page.IsCallback == false)
                Presenter.OnViewLoaded();
        }
    }

And lastly, here is the actual web form

<asp:Content runat="server" ID="FeaturedContent" ContentPlaceHolderID="FeaturedContent">
    <section class="featured">
        <div class="content-wrapper">
            <hgroup class="title">
                <h1><%: ContentTitle %></h1>
            </hgroup>
            <section>
                <asp:Repeater runat="server" ID="clientsRepeater">
                   <HeaderTemplate>
                        <table>
                        <thead>
                            <tr>
                                <th>Client Name</th>
                                <th>Contact Name</th>
                            </tr>
                        </thead>
                        <tbody>
                            
                        
                   </HeaderTemplate>
                    <ItemTemplate>
                        <tr>
                            <td><%#Eval("ClientName"%></td>
                            <td><%#Eval("ContactName"%></td>
                        </tr>
                    </ItemTemplate>
                    <FooterTemplate>
                        </tbody>
                    </table>
                    </FooterTemplate>
                </asp:Repeater>
            </section>
        </div>
    </section>
</asp:Content>

That's pretty much it.

The great thing here is if you need to work in legacy applications, you can start this pattern at any point. I have added new pages with this pattern into legacy application that do not follow any pattern without much effort.

I put all the code up on GutHub, in case you would like to take a look.

 

Validation Gotchas to Watch For When Using Angular

Thursday, July 11, 2013

Introduction

Validating a form in Angular is rather straight forward and it is pretty unobtrusive when it comes to the amount of code you have to right, but I thought I would quickly share some gotchas that I experienced when validating a form.

<form name="taskForm" ng-submit="save()" class='form-horizontal' novalidate>

The Form Tag and Controller

First thing to note is that if you want your submit function to fire so you can validate the form in your controller, you need to add the novalidate attribute to your form tag. Otherwise, the browser's built in validating handler will kick and prevent the form submission from firing.

if ($scope.taskForm.$valid === falsereturn;

Notice the form name is automatically attached to scope so it can be validated.  Angular has some built evaluation function you can use to validate the form.

To keep the errors from showing until the user has submitted the form: I created an isSubmitted variable on the $scope object and setting it true when the user clicks the submit button.

$scope.isSubmitted = true;

The Input tag.

Another thing to note is that if you want to use the other custom attributes like email, min, max, etc, then you need to make sure your use the correct input type in your tag. If you don’t, those validation errors will not fire.

Here is a number example:

<div class="control-group" ng-class="{error: isSubmitted && (timesheetForm.rate.$error.required || timesheetForm.rate.$error.number || timesheetForm.rate.$error.min || timesheetForm.rate.$error.max)}">
                <label class="control-label" for="timesheet-rate">Actual Rate:</label>
                <div class="controls">
                    <input type="number" ng-model="timesheet.rate" class="input-mini" 
id="timesheet-rate" name="rate" required min="1" max="999"/>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.rate.$error.required">Actual rate is required.</span>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.rate.$error.number">Actual rate is not a valid number.</span>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.rate.$error.min">Actual rate must be greater than 0.</span>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.rate.$error.max">Actual rate must be less than 999.</span>
                </div>
            </div>

Here is an email example:

            <div class="control-group" ng-class="{error: isSubmitted && (clientForm.email.$error.required || clientForm.email.$error.email)}">
                <label class="control-label" for="email-address">Email Address:</label>
                <div class="controls">
                    <input type="email" ng-model="client.contactEmail" class="input-xlarge" id="email-address" name="email" required/>
                    <span class="help-inline" ng-show="isSubmitted  && clientForm.email.$error.required">Email is required.</span>
                    <span class="help-inline" ng-show="isSubmitted  && clientForm.email.$error.email">Not a valid email address.</span>
                </div>
            </div>

You Need a Name:

You need to include the name attribute in your input tag and the name of the tag cannot have a "-" in the name. Same thing here; the validation will work with the dash.

Styling Using ng-class

For styling errors in Twitter Bootstrap you have to provide an error class in the control-group div tag which is the parent of the input tag that is being validated. To do that with an input element that several validations you can evaluate several conditions at once.

Here I am using the ng-class directive to validate several different scenarios.

<div class="control-group" ng-class="{error: isSubmitted && (timesheetForm.hours.$error.required || timesheetForm.hours.$error.number || timesheetForm.hours.$error.min || timesheetForm.hours.$error.max)}">

Here is the entire form.

        <form name="timesheetForm" ng-submit="save()" class='form-horizontal' novalidate>
            
            <div class="control-group" ng-class="{error: isSubmitted && timesheetForm.timesheetName.$error.required}">
                <label class="control-label" for="timesheet-name">Timesheet Name:</label>
                <div class="controls">
                    <input type="text" ng-model="timesheet.timesheetName" 
class="input-xlarge" id="timesheet-name" name="timesheetName" ng-focus required/>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.timesheetName.$error.required">Timesheet name is required.</span>
                </div>
            </div>
            <div class="control-group">
                <label class="control-label" for="timesheet-description">Description:</label>
                <div class="controls">
                    <textarea ng-model="timesheet.timesheetDescription" 
class="input-xlarge" id="timesheet-description"  cols="20" rows="10"></textarea>
                </div>
            </div>
            <div class="control-group" ng-class="{error: isSubmitted && timesheetForm.startDate.$error.required}">
                <label class="control-label" for="timesheet-start-date">Start Date:</label>
                <div class="controls">
                    <input type="text" jq-datepicker ng-model="$parent.timesheet.startDate" 
class="input-small" id="timesheet-start-date" name="startDate" required/>
                    <span class="help-inline" 
ng-show="isSubmitted && timesheetForm.startDate.$error.required">Start date is required.</span>
                </div>
            </div>
            <div class="control-group" ng-class="{error: isSubmitted && (timesheetForm.hours.$error.required || timesheetForm.hours.$error.number || timesheetForm.hours.$error.min || timesheetForm.hours.$error.max)}">
                <label class="control-label" for="timesheet-hours">Actual Hours:</label>
                <div class="controls">
                    <input type="number" ng-model="timesheet.actualHours" class="input-mini" id="timesheet-hours" name="hours" required min="1" max="999"/>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.hours.$error.required">Actual hours are required.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.hours.$error.number">Actual hours is not a valid number.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.hours.$error.min">Actual hours must be greater than 0.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.hours.$error.max">Actual hours must be less than 999.</span>
                </div>
            </div>
            <div class="control-group" ng-class="{error: isSubmitted && (timesheetForm.rate.$error.required || timesheetForm.rate.$error.number || timesheetForm.rate.$error.min || timesheetForm.rate.$error.max)}">
                <label class="control-label" for="timesheet-rate">Actual Rate:</label>
                <div class="controls">
                    <input type="number" ng-model="timesheet.rate" class="input-mini" id="timesheet-rate" name="rate" required min="1" max="999"/>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.rate.$error.required">Actual rate is required.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.rate.$error.number">Actual rate is not a valid number.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.rate.$error.min">Actual rate must be greater than 0.</span>
                    <span class="help-inline" ng-show="isSubmitted && timesheetForm.rate.$error.max">Actual rate must be less than 999.</span>
                </div>
            </div>
            <div class="control-group">
                <label class="control-label" for="timesheet-revision-number">Revision Number:</label>
                <div class="controls">
                    <input type="text" ng-model="timesheet.revisionNumber" class="input-small" id="timesheet-revision-number"/>
                </div>
            </div>
            <div class="control-group" ng-class="{error: isSubmitted && timesheetForm.status.$error.required}">
                <label class="control-label" for="timesheet-status">Status:</label>
                <select ng-model="timesheet.statusId" 
data-ng-options="t.statusId as t.statusName for t in statuses" id="timesheet-status" name="status" required></select>
                <span class="help-inline" ng-show="isSubmitted && timesheetForm.stats.$error.required">Status selection is required.</span>
            </div>
            <div class="control-group" ng-class="{error: isSubmitted && timesheetForm.user.$error.required}">
                <label class="control-label" for="timesheet-user">User:</label>
                <select ng-model="timesheet.userId" data-ng-options="m.userId as m.userName for m in members" id="timesheet-user" name="user" required></select>
                <span class="help-inline" ng-show="isSubmitted && timesheetForm.user.$error.required">Member selection is required.</span>
            </div>
            <div class="form-actions">
                <button class="btn btn-primary" >Save</button>
                <button class="btn" ng-click="cancel()" >Remove</button>
            </div>
        </form>

Hope that helps.

Multiple Column Sorting of a Table Using Angular

Thursday, June 20, 2013

This year I have been focusing a lot on Single Page Application (SPA) and the various tools that can be used to build these applications efficiently.
 
So far, I have checked out knockout.js, backbone.js and angular.js, but have been mainly focused on getting my head around angular.js. I have been hearing great things about ember.js too, so perhaps I'll tackle that next. For now, though, I am focusing on angular.js. 
 
So in this post, I thought I would share how to take advantage of angular's filtering functionality. Specifically: how to have a table with a sort button on each of the table columns. It's actually pretty easy.

Binding

One of the core features of angular.js is its ability to do "two-way binging". That is, once the page loads, Angular, binds any models on the its scope to the designated fields on the page. These fields can be a single record or a collection of records. This means properties I change on the scope will immediately change on the screen. Angular handles all that magic.

Filter

Angular has a filter function that allows you to do things like query your collection filtering the result on page. The filter function can also sort your collection instantaneously on the page. In my example of sorting, I can also change the sort the sort column dynamically and I can also reverse the sort dynamically by storing the sort values in the $scope. Note that the $scope is sort of like a ViewBag if you think of it in terms of ASP.Net ASP. It is holds properties that the page has access to and that you can change at anytime.

Template

Here is what my template looks like
<table class="table table-striped table-condensed">
        <thead>
            <tr>
                <th></th>
                <th>Client Name <button class="btn btn-mini btn-link" ng-click="sort('clientName')"><i  class="icon-sort"></i></button> </th>
                <th>Contact Name<button class="btn btn-mini btn-link" ng-click="sort('contactName')"><i  class="icon-sort"></i></button></th>
                <th>Email Address<button class="btn btn-mini btn-link" ng-click="sort('contactEmail')"><i  class="icon-sort"></i></button></th>
                <th></th>
                <th></th>
            </tr>
        </thead>
        <tbody>
            <tr ng-repeat="client in clients | orderBy:orderProp:direction">
                <td><a ng-href="/#/clients/edit/{{client.clientId}}"><i class="icon-edit"></i></a></td>
                <td><a ng-href="/#/clients/view/{{client.clientId}}"> {{client.clientName}}</a></td>
                <td>{{client.contactName}}</td>
                <td>{{client.contactEmail}}</td>
                <td></td>
                <td><remove-button text="Remove Client" action="removeClient()"></remove-button></td>
            </tr>
        </tbody>
    </table>

I tell Angular to loop through the collection by using the ng-repeat command and then I can set the sorting options by using the filter command orderBy. I am then telling it what property in the scope to use for the sorting (orderProp) and which property will be used for ascending and descending direction (direction).

Also notice my heading column links. They have a ng-click command to call the function sort. The sort function takes a parameter for which column to sort and will either change the column sorting or the direction depending on what column was clicked.

Then in the controller class, I initialize those properties and create the sort function which checks to see if it was the same column that is set in the scope and if it is it simply changes the direction of the sort. If the column is different than what is in scope, then it resort based on the new column and resets the direction to ascending.

                    $scope.direction = true;
		    $scope.orderProp = "clientName";
		    $scope.sort = function(column) {
		        if ($scope.orderProp === column) {
		            $scope.direction = !$scope.direction;
		        } else {
		            $scope.orderProp = column;
		            $scope.direction = true;
		        }
		    };

Here is my Plunker Example.

 

My Daughter's Flute Evaluation

Saturday, May 4, 2013

My oldest daughter had her flute evaluation today. I can't recall doing anything that stressful when I was her age. Any how, as a proud pop, I couldn't resist and posted this on my website. She did really great.

They're many schools represented at this event, and as a matter of fact, my daughter's particular judge had traveled 5 hours to be at the competition. When we arrived at the particular school holding the competion, we settled in the cafeteria where kids from all over the region were practicing before their evaluation.  All I can say is, I am glad my daughter plays the flute and not trumpet like I did growing up, as trumpets can be quite loud when a bunch of them are together in one room plaing different music at the same time.

Free DotNetCurry Subscription

Thursday, May 2, 2013

If you are looking for some good programming related content on the latest on the Microsoft development stack, DotNetCurry has a free digital magazine that you can subscribe to.

http://www.dotnetcurry.com/magazine/

One of the things I like about this mazazine is that it has a lot of walk through examples where you can go through step by step and recreated what the article recreates. I find that I grasp new concepts a lot better when I can go through and code things up myself. For me it sinks in better that way.

Any how, check it out.

Building a Blog Redux - Creating a Lucene.Net Index (Part 8)

Thursday, October 18, 2012

This is the eighth post in a series of posts about how I went about building my blogging application.

  1. Building a Blog Redux - Why Torture Myself (Part 1)
  2. Building a Blog Redux - The tools for the trade (Part 2)
  3. Building a Blog Redux - Entity Framework Code First (Part 3)
  4. Building a Blog Redux - Web fonts with @font-face and CSS3 (Part 4)
  5. Building a Blog Redux - Goodreads feed using Backbone.js (Part 5)
  6. Building a Blog Redux - Mapping View Models to Entities Using AutoMapper (Part 6)
  7. Building a Blog Redux - Setting Up Lucene.Net For Search (Part 7)

In the previous post in the series I talked about how I went about Lucene.Net in my blogging application using StructureMap. In this post, I am going continue with the searching feature and write about how the indexing is set up.

What Initiates Writing an Index

Researching other implementations of creating an index, I saw several different options of when an index was created and refreshed. I saw examples of applications reindexing when the application starts up, when the content is updated or on demand on an admin screen. For my first implementation, I chose to keep it simple, so I just create a button on my admin site that kicks off the reindexing when the button is clicked.

Here's the view. Pretty simple.

<h2>Manage Search Index</h2>
<div id="manage-index-panel">
    
    <input id="reindex" type="button" value="Re-index Posts" />
    <div id="index-waiting-panel" style="displaynone;">
        <img src="/Content/images/sb_wait.gif" alt="Waiting"/>
    </div>
    <div id="reindex-status-panel">
        <ul></ul>
    </div>
</div>

The event kicked off via a AJAX call. I was thinking the reindexing would be taking a long time, but actually it is pretty fast so I am not so sure I needed to do this but hey what the heck.

var indexAdmin = function ($) {
 
    var $waitPanel = $('#index-waiting-panel'),
        $reindex = $('#reindex'),
        $status = $('#reindex-status-panel ul'),
        postUrl = '/manage/index/reindex',
        displayIndexCompletionStatus = function (items) {
            console.log('data returned', items);
            $waitPanel.css({ display: 'none' });
            $.each(items, function () {
                $status.append('<li>' + this + '</li>');
            });
        },
        init = function () {
 
            $reindex.on('click'function () {
                $waitPanel.css({ display: 'inline' });
                $.ajax({
                    url: postUrl,
                    cache: false,
                    contentType: 'application/json; charset=utf-8',
                    dataType: "json",
                    type: "POST",
                    success: function (data) {
                        displayIndexCompletionStatus(data);
                    },
                    error: function (jqXHR, textStatus, errorThrown) {
                        console.log(jqXHR);
                        console.log(textStatus);
                        console.log(errorThrown);
                        displayIndexCompletionStatus(errorThrown);
                    }
                });
            });
        };
 
 
    return { init: init };
 
}(jQuery);
 
jQuery(document).ready(function() {
    indexAdmin.init();
});

Here is the MVC action that receives the API call.

public JsonResult Reindex()
        {
            var errors = _searchIndexService.RebuildIndex();
            if (errors != null && errors.Count > 0)
            {
                var errorList = errors.Select(error => error.Exception.Message).ToList();
                return Json(new { Status = errorList.ToArray() });
            }
            var status = new { Status = new[] { "Success" } };
            return Json(status);
        }

Pretty clean action. Just calling my Reindexing Service and if there were any errors then return a list of the errors. I am then sending a status message back in the response.

Creating the Index Document

Before I talk about the code that actually writes to the index let me first talk about the main Lucene.Net classes that are going to be involved in process.

  • The Directory is a flat list of files that may only be writing to once. What this means is once a file is created. It cannot be modified; rather, it can only be read from or deleted.
  • Documents are the primary units which stored within a directory. A document will have a collection of fields with will specify what content to index, store and compare with other content.
  • The IndexWriter is the class that actually writes out the index. It also has the ability to remove items from the index as well.
  • The Analyzer are the objects that tokenizes the content so it can be searched on later. There are various derivations of the analyzers that very in there complexity in searching. I am using the SnowballAnalyzer which does things like matches on root word on longer words. So for example is someone searching on "programming" would get match on the word "program". (Note: The SnowballAnalyzer is in a different Nuget package called Lucene.Contrib). 

The IndexWriter is the first thing that I setup. I really like how elegantly this object was set up in SubText so I am pretty much using the same implementation for my IndexWriter. You can see their version of the implementation by downloading their source code from their Google Code Repository. Essentially every time you need to call a IndexWriter function, you need to ensure the current thread is the only instance that is using object during the transaction. So rather then repeating the same code for every transaction, SubText creates a function that locks the object first and passes back a delegate where you can then call an action to executed.

Here is the locking function:

        private void EnsureWriterInstance()
        {
            if (writer != nullreturn;
 
            if (IndexWriter.IsLocked(_directory)) IndexWriter.Unlock(_directory);
 
            writer = new IndexWriter(_directory, _analyzer, IndexWriter.MaxFieldLength.UNLIMITED);
            writer.SetMergePolicy(new LogDocMergePolicy(writer));
            writer.SetMergeFactor(5);
        }

Then every time an action needs to be excuted the DoWriterAction is called.

        private void DoWriterAction(Action<IndexWriter> action)
        {
            lock (WriterLock)
            {
                EnsureWriterInstance();
            }
            action(writer);
        }

Be sure the writer is static:

private static IndexWriter writer;

The code is returning the current IndexWriter if one already exists and if it does not exists, it makes sure the document object and setting the merging policy. The merge factor is a setting that determines how often the segment indices are merged. According to the specs, "With smaller values, less RAM is used while indexing, and searches on unoptimzed values are faster, but indexing speed is slower. With larger, values, searches on unoptimized indices are slower, indexing is faster."

For my blog site, I would like the user to be able to search on the title, content, tags, and meta-description, but I would like to store other things in the document that I can use in the displaying the search result like the publishing date, and the slug for the link. Conversely, some field I just want to index, but I do not want to store like the post content.

//create the fields
            var postId = new Field(
                PostId,
                NumericUtils.IntToPrefixCoded(post.Id),
                Field.Store.YES,
                Field.Index.NOT_ANALYZED,
                Field.TermVector.NO);
 
            var title = new Field(
                Title,
                post.Title,
                Field.Store.YES,
                Field.Index.ANALYZED,
                Field.TermVector.YES);
 
            var body = new Field(
                Body,
                post.PostContent,
                Field.Store.NO,
                Field.Index.ANALYZED,
                Field.TermVector.YES);
 
            var tags = new Field(
                Tags,
                tagDelimited.ToString(),
                Field.Store.NO,
                Field.Index.ANALYZED,
                Field.TermVector.YES);
 
            var metaDescirption = new Field(
                MetaDescription,
                post.Description ?? string.Empty,
                Field.Store.NO,
                Field.Index.ANALYZED,
                Field.TermVector.YES
                );
 
            var published = new Field(
                IsPublished,
                post.IsPublished.ToString(),
                Field.Store.NO,
                Field.Index.NOT_ANALYZED,
                Field.TermVector.NO);
 
            var slug = new Field(
                Slug,
                post.Slug,
                Field.Store.YES,
                Field.Index.NO,
                Field.TermVector.NO);
 
            var pubdate = new Field(
                DatePublished,
                pubDate,
                Field.Store.YES,
                Field.Index.NOT_ANALYZED,
                Field.TermVector.NO);

So each document has a collection of fields and each field I specify the following:

  • The key (name) of the field.
  • The value of the field.
  • Whether or not I want to store the raw value in the field for later use,
  • Whether or not I want to index the value of this field to be used in searching.
  • Whether or not and/or how I want the field to use term vectors.

I can also weight some fields so that I get a higher match on a word if for example it is located in the title verses being located in the body. I can do this by calling the SetBoost function and giving it a weighting factor.

So then once my fields are created and weighted, I can then add them to a document.

            //boost some of the entries
            title.SetBoost(4);
            tags.SetBoost(2);
            body.SetBoost(1);
            slug.SetBoost(2);
            metaDescirption.SetBoost(1);
 
            //add the fields to the docuement
            doc.Add(postId);
            doc.Add(title);
            doc.Add(body);
            doc.Add(tags);
            doc.Add(published);
            doc.Add(pubdate);
            doc.Add(slug);
            doc.Add(metaDescirption);

Once the document is prepared then you add this current document to the index writer where the collection of documents are contained.

 DoWriterAction(indexWriter => indexWriter.AddDocument(CreateDocument(currentPost)));

Once all the documents are added then the commit function where all the pending changes.

DoWriterAction(
                indexWriter =>
                    {
                        indexWriter.Commit();
                        if (optimize) indexWriter.Optimize();
                    });

The Optimize function on the indexWriter reorganizes the indexes so that they can be queried faster and provide better performance.

Conclusion:

So now I have functionality that when I click a button, the current index is refreshed with the latest blog content from the database. In my next post in this series I will show you how I can now query this index.

As always you can check out the code at my GitHub repository.

Resources:

Building a Blog Redux - Setting Up Lucene.Net For Search (Part 7)

Thursday, September 27, 2012

This is the seventh post in a series of posts about how I went about building my blogging application.

  1. Building a Blog Redux - Why Torture Myself (Part 1)
  2. Building a Blog Redux - The tools for the trade (Part 2)
  3. Building a Blog Redux - Entity Framework Code First (Part 3)
  4. Building a Blog Redux - Web fonts with @font-face and CSS3 (Part 4)
  5. Building a Blog Redux - Goodreads feed using Backbone.js (Part 5)
  6. Building a Blog Redux - Mapping View Models to Entities Using AutoMapper (Part 6)

Search is a segnificant part of website usability. When someone comes to my website looking for something, I want to be able to provide a good user experience where if they look down at the tags and don't see a specific tag they are looking for, then give them the ability to search my blogs for word matches. I would rather not have the user leave my site to go to Google and search for term that will lead a user to another website. Users should be able to enter a term or phrase on my website, and get the most relevant responses back for them to review.

That being said, the latest functionality added to my blog site is a searching capability. I chose Lucene.net mainly because it is the big open source player on the capability for ASP.Net applications. The common features that come with this framework that I am using is the indexing of content from by blogs and the querying of the indexes to return relevant information in a quick response. It is a direct function to function port over from the original Java version Lucene.

Lucene.Net is an indexing and search framework that can run on many different platforms, but most commonly it is used for searching capabilities on websites. A few years ago, it was essentially a dead project, however some dedicated programmers revived the projected and entered it into the Apache Foundation where was in an incubator status until this past August 15th, where it was then voted out of the incubator status.

Be aware that now that the project has graduated from the incubator status, there project site has moved and it took me some time to actually find it. At least at the time of this writing, it didn't even come up on a Google search. It was on Twitter where I found current project link.

What I would like to do in this post, is show you how I have implement Lucene.net in a MVC ASP.Net web application. This article is mainly going to focus on setting up the components in a MVC web application using StructureMap for Inversion of Control. Later on, I will have a post on setting up the index and another post on setting up querying the index.

Getting Started

First off, I should say that documentation on the project site for Lucene.net is a bit lacking, but you can get a pretty good introduction from the CodeClimber web site starting with this article. Furthermore, since this article talks about how Lucene.net was implemented in the SubText blog engine project, I went to their project site and downloaded the Subtext source code. I tried not to copy any code directly from the Subtext source code, but if you compare my code to the code that in the Subtext source code you will see a lot of similarities.  I think they way the Subtext code is implement on that site is pretty good, and pretty much the most common way to have the framework implemented. For the most part, all the samples I have seen either refer to the same articles I am referring to, or give examples very similar to the code I have setup as well. The point being is that I am standing on other developers shoulders with this code mainly taking there ideas and modifying them to fit my needs.

Set Up

Lucent.Net is up on Nuget so its easy enough to do download install as a package. Just download the package via Nuget, and you are ready to start writing code.

Since I am using StructureMap to manage all of my dependencies, I need to explicitly add the Lucene.Net components to the registry because I want to specify a singleton instance of the components so only one of each of the components run while the application is running. Typically, I have StructureMap scan all my objects and then use StructureMap's "Convention of Configuration" pattern to register them all; however, this unique situation where this pattern does not work. I need to explicitly tell StructureMap that components must run as Singletons; that is, there can only be one instance of the objects running at a single given time.

Regarding how I have set up StructureMap in general, I mentioned this in an earlier post, but if you want to see a good example, of how I have StructureMap setup see Elijah Manner's post.

Here is my StructureMap registry code:

    public class AviBlogRegistry : Registry
    {
        #region Constructors and Destructors
 
        public AviBlogRegistry()
        {
            Scan(
                x =>
                    {
                        x.TheCallingAssembly();
                        x.Assembly("AviBlog.Core");
                        x.WithDefaultConventions();
                    });
 
            //Register Search and Indexing Services
            For<ISearchEngineService>().Singleton().Use(
                () => new SearchEngineService(SingletonDirectory.Instance, SingletonAnalyzer.Instance));
            For<ISearchIndexService>().Singleton().Use<SearchIndexService>();
        }

As I stated, I have the scan feature, which registering all my objects, but right below that I am registering to Lucene.net objects explicitly that will be using one single instance of the SearchEngine object and a single instance of the SearchIndexService object. These objects are my custom objects that wrap the Lucene.net objects which also need to be and are set up as Singletons. I am also telling StructureMap, that when the SearchEngineService is instantiated, they should use the singleton versions of the Directory and Analyzer Lucene.Net objects.

I am using the "Multithreaded Singleton Pattern" to instantiate these objects as Singletons. The Singleton pattern is used when you want to accomplish the following:

  • You want an object to have only one instance
  • You want an object to one global entry point, in my case I am loading the objects up on the Application Start Event.
  • You want the single instance object to "thread-safe". That is, in a multi threaded in environment, the object should be safely created without the chance that another. thread could accidently create a multible instance at the same time.

I would have liked to have StructureMap create the singletons for the Directory and Analyzer, but I think because I am still in the AviBlogRegistry class, the objects are not registered at this point and thus the instantiated objects cannot be used. I am getting null objects from StructureMap at this point. Perhaps if I go back and create a separate StructureMap registry class for these two objects before this registry is executed, I can once again have every object in the application initiated by StructureMap. I might try that later and update this post.

Therefore to create a Singleton instance of the Directory and Analyzer Lucenet.Net objects, I have created some static classes. Here is the Lucene.Net Directory Singleton instance:

public sealed class SingletonDirectory
    {
        private static volatile Directory instance;
 
        private static readonly object syncRoot = new Object();
 
        private SingletonDirectory()
        {
        }
 
        public static Directory Instance
        {
            get
            {
                if (instance == null)
                {
                    lock (syncRoot)
                    {
                        if (instance == null && HttpContext.Current != null)
                            instance =
                                FSDirectory.Open(
                                    new DirectoryInfo(HttpContext.Current.Server.MapPath("~/folder/subfolder/")));
                    }
                }
                return instance;
            }
        }
    }

The Directory object creates the index that the application will use to read search queries from. In this case I am implementing the concrete FSDirectory version of Directory which is the version of the object that writes the index out to a file. There is also MMapDirectory mmap for reading, and RAMDirectory which reads and writes to memory, which from what I can see is mainly used for unit testing. The FSDirectory takes a DirectoryInfo object which specifies where you want the index files to be saved to.

Conclusion

I'll stop here for now. In my next post in the series I'll talk how to read all the blog records from the database and create an index for searching. As always you can check out the code on my GitHub account.

Resources

Using Node.js and Socket.IO to Stream Tweets From the Twitter API

Wednesday, June 20, 2012

I've discussed in a previous blog post about the Twitter API. Specifically in that post I talked about making a specific call, and then getting a response back in a traditional request/response manner. However, Twitter also has a streaming service that you can connect to that will push to messages real time in a persistent, open HTTP connection.

In general, this type of service can be very powerful, because it makes it very easy to stream information to clients with out the client having request the updates. In the past, streaming data to a browser was not been easy. In actuality, websites were really just making the browser appear to be streaming data but they were really just polling for data with multiple requests. However, today's browsers now support the WebSockets protocol which makes this type of streaming rather easy to implement. It essentially gives the browser the ability to open a connection to a host service and retrieve data through the open connection. Here is what is said about this functionality on Wikipedia.

"The WebSocket protocol makes possible more interaction between a browser and a web site, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open. In this way a two-way (bi-direction) ongoing conversation can "take place between a browser and the server."

Enter Node.js

Since this type of service is asynchronous, its the type of functionality that really fits into the Node.js sweet spot, because JavaScript is by its very nature asynchronous.

To set this Node.js application up, I am going to need a few packages. Like Ruby on Rails' Gems package manager and Microsoft.Net's Nuget package manager; Node.js has a package manager called NPM. I'm not looking to reinvent the wheel here. I just want an easy way to get access to the Twitter stream. So to accomplish, this I am using the package nTwitter. There are tons of Twitter packages for Node.js, but I found this one the easiest to get up and running.

I am also going to set up the Express NPM package which gives Node.js the ability to have a more MVC approach to building a website.

So the first thing to do is to reference all the needed packages and also reference my custom nTwitter module.

var express = require('express')
  , routes = require('./routes')
  , socketIo = require('socket.io')
  , twitter = require('ntwitter')
  , util = require('util')
  , twitterModule = require('./modules/twitterModule');

 

My twitter module reference looks like this. Here I am initializing my nTwitter reference and passing all the needed Twitter credentials to make the streaming call. As aside, you need to go the Twitter Development Site and set up an application to get all your needed credentials.

var twitter = require('ntwitter'),
	util = require('util');
 
// Twitter Setup
module.exports.twit = new twitter({
  consumer_key: 'consumer key',
  consumer_secret: 'consumer secret',
  access_token_key: 'access token',
  access_token_secret: 'access token secret'
});

 

Enter Socket.IO

In my example, I am going to stream tweets that are in my general location. To do this, the "statuses/filter" streaming API needs to be referenced. This is the streaming API provided by Twitter.

To stream the API from my server to the client I am going to use Socket.IO. Socket.IO was built mainly for Node.js, although it can be used by other languages, to be the broker that sits on both the server and the client and handles all the heavy lifting of passing the data back in forth. The nice thing is, Socket.IO has the ability to check the browser for compatibility, so if the current client browser does not support WebSockets, then Socket.IO will revert to something like polling instead.

Thus the only server code that is needed to stream the tweets to the client is this:

io.sockets.on('connection'function(socket) {
  var twit = twitterModule.twit;
  twit.stream('statuses/filter', {'locations':'-80.10,26.10,-80.05,26.15'},
    function(stream) {
      stream.on('data',function(data){
        socket.emit('twitter',data);
      });
    });
});

Socket.IO has an "on" function, where a callback function is passed in. Inside this callback function the Twitter API stream is called, and as the stream "emits" new tweets, those entries get sent to the client. The locations parameters specifies the general longitude and latitude I want the tweets to originate from.

Socket.IO and JSRender on the Client

On the client there is the corresponding Socket.IO code that is configured to listen to the URL that was set up on the server for emitting the tweets. When each tweet is emitted to the client, an event is fired to render a new item to the list.

To render the tweets on the browser, I am using jsRender. If you have ever used jQuery Templates, jsRender is very similar and is probably going to end up replacing jQuery Templates going forward; although, at the time of this writing jsRender is still in Beta.

(function($) {
	$(document).ready(function() {
		var $container = $('ul.tweets'),
			socket = io.connect('http://localhost:3000'),
			template = $('#tweetTemplate');
			
 
	    socket.on('twitter'function(data) {
	        $container.append(template.render(data));
	    });
	});
})(jQuery);

For my Node.js views, I am using Jade. Very similar to HAML in Ruby on Rails, it is very terse and clean way creating markup. No angle brackets to deal with which is nice. Jade works by determining the indentation of the each line to base how things are nested. By the way, you can get this syntactical goodness on ASP.Net MVC as well by using The Spark View Engine.

This view also contains the jsRender template that is appended for each tweet.

h1= title
p #{title}
#local-tweet-container
	ul.tweets
 
script(id="tweetTemplate", type="text/x-jsrender")
	{{:#index+1}}: <li><img src='{{:user.profile_image_url}}' /> {{:text}}</li>

Running the App

When start the application by running node app.js and then browse to http://localhost:3000, the Socket.IO code makes the following request.

 

GET http://localhost:3000/socket.io/1/websocket/21369566461136251045 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:3000
Origin: http://localhost:3000
Sec-WebSocket-Key: Y65qAJoayH795Vbhn3Bj4w==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: x-webkit-deflate-frame
 
The response that is sent back is:
 
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: vCWC+o5mWP5mnD2enVI3/PJOhMk=
 
The 101 status means that the requestor has asked the host to change its protocal and in this case the change was to WebSockets.
 
If I look at my browser I see the tweets getting added realtime.
Socket.Io Tweets
It's kind of hard to demonstrate what is happenning in screenshot but the tweets are being added to the page and the actual page is not doing anything. Looking on Fiddler, no post or get requests are being made from my localhost domain.
 
Pretty cool stuff!
 

References