Build Parsers using PEG.js

Writing parser from scratch is a tedious task. Recently I came across a simple parser generator for JavaScript called PEG.js

PEG.js is a simple parser generator for JavaScript that produces fast parsers with excellent error reporting. You can use it to process complex data or computer languages and build transformers, interpreters, compilers and other tools easily.

In order to create a parser, provide a grammar as the input and PEG.js will build a parser for that grammar. Then that generated parser can be used to parse the content.

PEG.js grammar has a particular syntax, you can learn more about it from http://pegjs.org/documentation#grammar-syntax-and-semantics. If you are familiar with regular expression it will easy understand.

Take a simple example for writing grammar which will parse texts like “$100”, “$150” etc…

Example text follow a pattern where currency symbol i.e. “$” is followed by a number. Translating this into pEG.js grammar looks like this

1
Text
 = "$" Number

Number
 = Digit+

Digit
 = [0-9]

Every grammar has set of rules, in this case we have three rules. Every rule has a name that identify the rule and a parsing expression. Rule name is followed by “=” then parsing expression.

  1. Rule Text
    Parsing expression for this rule is “$” Number. This parsing expression says “Match text that has literal “$” followed by a Number“. Note Number is name of a rule.
  2. Rule Number
    Parsing expression “Digit+” means - match text that has one or more Digit.
  3. Rule Digit
    Last rule says match any character that is between 0 to 9

Hope you got basic idea how a grammar is created. PEG.js provides playground where you can try your grammar and see how it is working. To try this go to http://pegjs.org/online and copy paste the grammar we have created on text box 1 and enter “$100” on text box 2. Parsed result will be shown on the output section and it looks like this

1
2
3
4
5
6
7
8
[
"$",
[
"1",
"0",
"0"
]
]

PEG.js was able to parse successfully, if you provide an invalid input like “$100$” it will show a parser error.

Note if you want to just get the value from that text (“100” from “$100”), the output that we got is not that useful. In order extract only the required values, parsing expression allows to write custom JavaScript expressions inside the curly braces (“{“ and “}”). Updating rule Text like below will return just the value “100”

1
Text
 = "$" n:Number
 { return n.join(""); }

Two changes we have made to the previous rule

  1. n:number - added a prefix n to rule Number
    Here we are assigning the match result of rule Number to a variable called n
  2. { return n.join(""); }
    JavaScript statement inside the curly brace call join method on the variable n. In this case variable n contains an array of characters, so calling join on that will concatenate and produce a string which returned as the result of rule Text

Parsing Grocery List

Take some more complex example where you want to parse grocery list and return the result in a JSON format like below where quantity should be in grams

1
2
3
4
[
{ "name": "Onion", "quantity": 1000 },
{ "name": "Tomatoes", "quantity": 500 }
]

Grocery list text will be in the following format - where the quantity can be in kg or gm

1
Onion 1kg
Tomatoes 500gm
Beans 1kg

Please try for yourself and share your grammar.

Here is the grammar I have created to do this; Below JsFiddle is interactive you can change grammar and grocery list, clicking on Parse button will show the result.

Hope this article was helpful. Please share your thoughts and comments.

A Different Way to Organize JavaScripts in ASP.Net MVC

New JavaScript SPA(Single Page Application) frameworks are getting written day by day, generally one core feature these frameworks solves is how to organizing the code so that maintaining it in the future won’t be a night mare. When it comes to organizing JavaScripts in a large scale server side application, resources available is very less SPA frameworks.

Organizing JavaScripts in a server side application is entirely different from SPA, most of the articles that I came across is suggesting to include page level script inside the server side page itself - e.g. in ASP.Net MVC you include common JavaScript files inside the layout file and views will include the scripts it required

Layout(_Layout.cshtml)

1
2
3
4
5
6
7
8
9
10
11
<!DOCTYPE html>
<html>
<head>
.....
</head>
<body>
......
@Scripts.Render("~/bundles/commonjs")
@RenderSection("scripts", required: false)
</body>
</html>

View(Index.cshml)

1
2
3
4
.....
@section Scripts {
<script src="~/....."></script>
}

For large applications a page will have different components which requires it’s own set of JavaScript files in order work. Including and organizing these component level scripts and it’s dependencies can cause lot of issues.

So here I want explain a different approach which I am not going to claim is best optimal solution for all applications but I think it does work for majority of the applications.

In this article I am using TypeScript instead of JavaScript. If you don’t know about TypeScript it’s a typed superset of JavaScript, learn more about it from http://www.typescriptlang.org

Application Class

Application is a TypeScript class which is the main entry point to client side script, this class is responsible for initializing page component. This is done by reading data-js-component attribute on body, then import the component dynamically and initialize.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import { Component } from "Component"
import * as Utils from "Utils"

export class Application {

public initialize(): void {

// Get page component name
var componentName = document.body.dataset["jsComponent"];
if (!componentName) {
// No page level component found
return;
}

System.import("Pages/" + componentName)
.then((mod) => {
if (typeof mod[componentName] === "function") {
var pageComponent = Utils.createInstance<Component>(mod[componentName], document.body);
pageComponent.initialize();
}
});
}
}

In order to load the component dynamically I am using a module loader called SystemJS, line 15 shows dynamically importing a page component. Once the component is imported it’s initialize() method is called after creating an instance of that component. One of the main advantage of using a module loader here is it will manage the dependencies which means any dependent modules will be imported automatically.

Every page will have a corresponding page component, name of this page level component is set as a data-js-component attribute on the body tag. Page component name is generated based on a convention - it will be same as the controller name. e.g. component name for CustomerController will be Customer. All page level components resides in a separate directory called Pages.

I have created a Html helper extension method which will return controller name, it also provide flexibility where an action method could change the page component name using a view data jsComponent.

1
<body data-js-component="@Html.JsPage()">
    ......
</body>

Helper extension

1
2
3
4
5
6
7
8
9
public static MvcHtmlString JsPage(this HtmlHelper helper)
{

var pageNameViewData = helper.ViewContext.ViewData["JsComponent"];
var pageName = pageNameViewData != null
? pageNameViewData.ToString()
: helper.ViewContext.RouteData.GetRequiredString("controller");

return new MvcHtmlString(pageName);
}

Component

All page level components derive from a base class Component, this base class implements the core initialization logic. Component can have child components as well, so initializing parent component will initialize child components as well. When Application class initializes page component, then any child components will also get initialized. OnReady method on the component is called after initialization is completed.

Component is associated with an HTML element on which the component is going to act up on. This container element is passed as a constructor argument. Page level component is initialized with body as associated element.

Component Class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
export class Component {

....

constructor(private element: HTMLElement) {
}

public get container() {
return this.element;
}

public get componentName() {
return this.container.dataset["jsComponent"];
}

public initialize(): Promise<any> {
......
}

public onReady() {
}

....
}

Home page component

1
2
3
4
5
import { Component } from "Component";

export class Home extends Component {

}

Child Components

A page will have many child components as well. To create these child component element associated with that component should have a data-js-component attribute.

Below code shows Popover component which initializes the bootstrap popover

1
2
3
4
5
6
7
8
9
10
11
import { Component } from "Component"

export class Popover extends Component {

public onReady() {
$(this.container).popover({
content: "Popover Component",
placement: "right"
});
}
}

In order apply this in the razor view, I will be setting data-js-component attribute like below

1
2
3
4
5
<a  href="#" 
class="btn btn-primary btn-lg js-component"
data-js-component="Popover">

Learn more &raquo;
</a>

Connecting the dots

Finally to wire up everything - Application class needs to initialized. For that import Application module in _Layout.cshtml and initialize it.

1
2
3
4
5
6
7
8
9
10
11
12
13
<script src="~/Scripts/system.js"></script>
<script>
System.config({
baseURL: '/Scripts/App',
defaultJSExtensions: true
});

System.import('Application')
.then(function (m) {
var app = new m.Application();
app.initialize();
});
</script>

Source code used in this article can be found here https://github.com/cvrajeesh/myblogsamples/tree/master/MVC/PageScripts

How to Fill Remaining Height with a Div

Sometimes things are too obvious but don’t work the way we think.

This is the HTML I have; my requirement was child1 height will be auto(based on the content) and child2 will occupy the remaining height of the main-container.

1
2
3
4
<div class="main-container">
<div class="child1">Child1 contents</div>
<div class="child2">Child2 contents</div>
</div>

First I thought this is easy and created the following styles

1
2
.main-container { height: 500px;  width: 100%; }
.child2 { height: 100%; }

This produces a wrong result - if child1 takes 100px I expected child2 to be 400px, but the height was 500px.

Child2 takes full height

There are many ways to fix this. One easy way is using display:table-row. Here is fixed CSS

1
2
3
4
5
6
7
.main-container {
height: 500px;
width: 100%;
display: table;
}

.child1 { display: table-row; }
.child2 { display: table-row; height: 100%; }

see this in action below

Run Gulp Tasks in a Predefined Order

Gulp is build automation tool which runs on top of Node. In my current project I had to execute set of tasks in a predefined order, in which some of the tasks can be run in parallel to improve the efficiency.

By default Gulp runs all the tasks at the same time, so after googling I saw this official article. In this approach every task will takes a callback function so that Gulp knows when that task is completed. Then other task add this as a dependency

E.g. from the official article

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
var gulp = require('gulp');

// takes in a callback so the engine knows when it'll be done
gulp.task('one', function(cb) {
// do stuff -- async or otherwise
// if err is not null and not undefined, the orchestration will stop, and 'two' will not run
cb(err);
});

// identifies a dependent task must be complete before this one begins
gulp.task('two', ['one'], function() {
// task 'one' is done now
});

gulp.task('default', ['one', 'two']);
// alternatively: gulp.task('default', ['two']);

Here task two is depends on task one, so Gulp ensures that task one is completed before running task two.One drawback with this approach is tasks are tightly coupled to each other which reduces the flexibility if you want to change the order.

So I came across a plugin called run-sequence, this plugin allows you to order the tasks much easy. We can re-write the first example like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var gulp = require('gulp'),
runSequence = require('run-sequence');

gulp.task('one', function() {
// Returning the stream is improtant
return gulp.src(....)
.pipe(gulp.dest(....);
});

gulp.task('two',function() {
// Returning the stream is improtant
return gulp.src(....)
.pipe(gulp.dest(....);
});

gulp.task('default', function(){
runsSequence('one', 'two');
});

Caution
Make sure each of the tasks returns a stream

Running tasks in parallel

run-sequence can do much more, like running in parallel

1
2
3
4
5
6
7
8
gulp.task('default', function(){
runsSequence(
'task-1',
'task-2',
['task-3', 'task-4', 'task-5'],
['task-7', 'task-8']
);
});

The above one ensures that task-1 and task-2 will be running in series, after that task-3, task-4, task-5 will run in parallel, then task-7 and task-8 are run in parallel as well.

Main advantage I see when compared to official documentation is - tasks are independent and orchestration is done by someone else.

EmberJS Error Logging in Production

In order to log all errors in an EmberJS application, EmberJS provides a global hook called onerror

Here is a link to the official documentation - http://emberjs.com/guides/understanding-ember/debugging/#toc_implement-an-ember-onerror-hook-to-log-all-errors-in-production

Code sample from the documentation (v 1.5.1) look like this

1
2
3
4
5
6
7
8
9
Ember.onerror = function(error) {
Em.$.ajax('/error-notification', {
type: 'POST',
data: {
stack: error.stack,
otherInformation: 'exception message'
}
});
}

What this piece of code does is, whenever an error occurs that stack trace is send to the server using an AJAX request. This will works great for most of the scenarios but….

Consider a situation where an error occurs inside a mouse move event handler. Then you will be trying to sending hundreds of requests to server in short interval. This can bring down your server if not handled properly, it’s like DDOS-ing your own server.

Good approach will be to control the rate at which errors are send to the server for logging. This can be done using a concept called Throttling - which ensures that the target method is never called more frequently than the specified period of time.

Google for JavaScript Throttling you can find lot different implementations. EmberJS also provide it’s own throttling function Ember.run.throttle

Here is the example from the official EmberJS website

1
2
3
4
5
var myFunc = function() { console.log(this.name + ' ran.'); };
var myContext = {name: 'throttle'};

Ember.run.throttle(myContext, myFunc, 150);
// myFunc is invoked with context myContext

This throttling function can be used for sending errors to the server. Here is a working demo which shows the throttled error logging -
http://cvrajeesh.github.io/emberjs/errorhandling.html

EmberJS - How to Detect User Inactivity

Scenario:

I am working on a web application that has a Single Page Application module which is built using EmberJS framework.

This EmberJS application is a page designer where user can drag and drop items to the design surface and change properties of these items. Finally when the page is ready, he can save the changes back to server.

Problem:

Whenever a user works on the page designer for long period of time without saving the changes in between - finally when he try to save all of his changes in one shot, server side session was already timed out.

Solution:

Implemented an API on the server side which can refresh the session if this API is invoked.

So on the EmberJS application side we had to calculate how long a user not active. Then we can check for this inactivity time periodically and decide whether to refresh session or not.

An easy way to do it in EmberJS is by hooking to the mousemove, keydown events of the ApplicationView. Details can be found in the EmberJS documentation http://emberjs.com/api/classes/Ember.View.html#toc_responding-to-browser-events

Here is code sample which does this

First I am creating the application object with an additional property for tracking the last active time.

1
2
3
4
5
6
var App = Ember.Application.create({
lastActiveTime: null,
ready: function(){
App.lastActiveTime = new Date();
}
});

After that, hook the events in which you are interested in ApplicationView, which will update the last active time defined in the application object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
App.ApplicationView = Ember.View.extend({
templateName: 'application',

mouseMove: function() {
App.lastActiveTime = new Date();
},

touchStart: function() {
App.lastActiveTime = new Date();
},

keyDown: function() {
App.lastActiveTime = new Date();
}
});

In our case, I am interested on mouseMove, touchStart (for touch devices) and keyDown events. When they are getting fired I will update the last active time with current time.

Now you can use the last active time property to find out how long the user is active.

Demo: http://cvrajeesh.github.io/emberjs/inactivity.html

This demo uses moment.js for calculating time difference as well as for formatting the time.

Visual Studio : Debugging Silverlight Application with Chrome and Firefox

Here is a tip if you want to debug Silverlight applications from Visual Studio 2012 if it is running in Google Chrome or Firefox

Google Chrome :

  1. Run the application and select Debug –> Attach to Process from the menu
  2. From the Available Processes list, look for process named “chrome.exe” with type Silverlight
  3. Select that process and click the Attach button

Here is screenshot from my machine
Visual Studio attach to Silverlight in Chrome

Mozilla Firefox :

Firefox has a slightly different approach

  1. Run the application and select Debug –> Attach to Process from the menu
  2. From the Available Processes list, look for process named plugin-container.exe
  3. Select that process and click the Attach button

Here is the screenshot
Visual Studio attach to Silverlight in Firefox

Voila!!! now you are able to get the breakpoints in Visual Studio

One of the Top E-Commerce Website in India Stores Passwords Incorrectly

This incident made me feel very bad – couple of weeks before I ordered a product from naaptol a well-known E-Commerce website in India. It was the first time I was ordering something from them. So I had to register first, after the registration was completed got a confirm email from them

Registration confirmation email from naaptol

Oops!!! My password is in clear text.

I thought they might be generating the email during the registration process before it is stored in their database. Without bothering much, I ordered the item. Thought about investigating about the “clear text password” little bit, but somehow forgot it.

After few days when I got the product which I have ordered, it reminded me about this password incident. So in order see whether they are hashing my password or not, I used their Forgot my password option. It asked for my email address. After the submission got a message saying

We’ have sent you an e-mail at the submitted ID including instructions. You’ll be back to your shopping place in no matter of time.

As expected, got an email from them

Forgot my password email from naaptol

It reads Here is your new Login and Password and surprisingly password they gave is same as my old password even though in the email it is mentioned that they are sending a new password. It confirmed that they are not hashing the passwords.

Will they be storing it in plain text? Who knows…

Did someone tell you internet is not a good place to store your secrets?

I tried to play a nice role, so sent them an email telling about about the password hashing problem. You know what happened… no reply from them till now.

If you are in the naaptol technical team, convince your boss about the importance of securing the password and push that functionality in the next release.

If you are a non-technical manager in naaptol, tell your developers to read this article from Jeff - You’re Probably Storing Passwords Incorrectly

Automating WYSIWYG Editors using Selenium Web Driver

Automating web application using selenium for acceptance/integration tests is very easy. Recently I have faced few issues when automating a page with a JavaScript based WYSIWYG editors.

The reason why it is hard because most of these editors create a new editable HTML document inside an iframe. There are two ways I aware that you can set text on these editors

  1. Executing a JavaScript code in the current page
  2. Sending keys on the editable iframe

Here is how I automated two WYSIWYG editors using selenium web driver.

bootstrap-wysihtml5

Website: http://jhollingworth.github.io/bootstrap-wysihtml5/

1
2
3
4
5
private void SetBootstrapWysihtml5Content(IWebElement textArea, string content)
{

string script = string.Format(@"$('#{0}').data('wysihtml5').editor.setValue('{1}');", textArea.GetAttribute("id"), content);
((IJavaScriptExecutor)this.WebDriver).ExecuteScript(script);
}

What this method does is, it accepts the TextArea web element which you are converting to an editor and the actual content to be set. Then it executes a piece of JavaScript on the current WebDriver context.

For e.g. if you have a TextArea element with id summary then executing below JavaScript statement will insert a h1 tag with “Test” to the editor

$('#summary').data('wysihtml5').editor.setValue('<h1>Test</h1>');

TinyMCE

Website: http://www.tinymce.com/

1
2
3
4
5
6
7
8
9
private void SetTinyMCEContent(IWebElement textArea, string content)
{

string textAreaId = textArea.GetAttribute("id");
string frameId = string.Format("{0}_ifr", textAreaId);
var frame = this.WebDriver.FindElement(By.Id(frameId));
this.WebDriver.SwitchTo().Frame(frame);
((IJavaScriptExecutor)this.WebDriver).ExecuteScript("document.body.innerHTML = '" + content + "'");
this.WebDriver.SwitchTo().DefaultContent();
}

This method also accepts the TextArea web element which you are converting to an editor and the actual content to be set. Then it finds the corresponding iframe and set the innerHTML by executing a JavaScript statement.

Introducing SourceCodeReader

Have you ever tried to understand a project just by going through the source code?

That’s how I do most of the time, till now I haven’t seen any up to date documentation which gives the clear understanding of how the code is implemented. In reality after a certain period during the development, documentation goes out of sync with the source code. So some one joins the project lately has only one way to understand the project Go through the source code

Sometimes I go through the open source projects hosted on either Github, Codeplex or Google code. All the hosting site provide you with a user interface for browsing through the source code with syntax highlighting support, which is nice. The problem I had with this is – if I see a object creation statement, method call there is no way for me to find out how it is implemented other than manually going through the files and finding it out.

Most of the full blown editors provide a feature where you can navigate to the implementation directly from where it is used for e.g. Microsoft Visual Studio provides Go To Definition feature. I find this feature very much useful for understanding the project. Only draw back with this approach is you have to completely get the latest version of source code to your local machine.

SourceCodeReader is trying solve this navigation problem on the web. With this application you can open a project and browse through the files with code navigation support.

Source code for this project can be found at https://github.com/cvrajeesh/SourceCodeReader

Limitations

  1. Now supports only Github code repository
  2. Only supports .Net C# projects, other type of projects also works without code navigation support

How to use this application

Go to SourceCodeReader and enter the URL of a C# project on Github

SourceCodeReader home page

Once you have entered a Github project link and open the project, it get the source code from the Github and present you with file browser.

SourceCodeReader project directory

When you navigate to a C# file you will be able to see clickable links for identifiers which takes you the file location where that is declared.

SourceCodeReader Go To Definition feature

Sample projects :

Hope you liked this idea, please provide me with your valuable feedback.