Saturday, November 10, 2012

Why I migrated from Pair NIC to GoDaddy

I was using Pair NIC for my domains. However, I had to migrate to another and finally chose GoDaddy due to the following reasons.

The foremost reason is that I am using Google Sites to host the websites and Google has started requiring "domain verification". That is, to verify that you do own that domain in order for the site to accept HTTP requests for that domain name. To do that, I have to add an TXT resource record to the zone. This can be done with Pair NIC but then it won't allow email forwarding anymore (technically there is no reason why both can't be done at the same time but it is the way the Pair NIC admin tool works). It means that I can no longer have email addresses like kent@mydomain.com forwarding emails to my Gmail account. Therefore, I have to choose between website branding (www.mydomain.com) or email branding (kent@mydomain.com).

An added benefit is that GoDaddy's email forwarding won't cause any delay like Pair NIC. Pair NIC artificially returns an error to an "unfamiliar" source SMTP server to make sure it is not sending spam. That will cause a delay of 10-20 minutes for that server to retry. GoDaddy doesn't have such a stupid feature. After all, Gmail is doing spam filtering, please just forward the damned mails. Worse yet, that "feature" of Pair NIC is not found anywhere and not mentioned before users making purchases.

In fact, I also found that GoDaddy's admin tool and management functionality is more powerful than that of Pair NIC. It also enjoys better support from Google Sites in terms of auto domain verification. GoDaddy is also cheaper, even though I am happy to pay more as Pair NIC is environmentally friendly and bearing the 100% renewable energy logo.

Tuesday, January 3, 2012

Implementing an async servlet in Scala with less than 10 lines of code

What is async servlet and why use it? An async servlet can tell the container that it will handle a request later, so that the current thread is not tied up and can be used for another request. Why is it useful? For example, if the request is going to do something relatively time consuming (e.g., generating a report from a DB), instead of tying up the thread doing nothing waiting for the data, you can free up the thread for other requests and wait for the data in a background thread. When the data is received, you can complete the response.


It turns out to be very easy to implement an async servlet (new in servlet 3.0). In Scala, it took less than 10 lines of code (not counting brace brackets and imports):



package com.ttdev.webqos
import javax.servlet.http._
import java.util.concurrent._

class MyServlet extends HttpServlet {
private val pool = Executors.newFixedThreadPool(1) // create a thread pool

override def doGet(req: HttpServletRequest, res: HttpServletResponse) {
req.startAsync //tel the container the response will be generated later
pool.execute(new Runnable() { // add the runnable to the queue
def run { // when a thread in pool picks up the runntable...
res.getWriter.append("hello world!") // write the response
req.getAsyncContext.complete // tell the container that the response is done
}
})
// return without a response yet
}

}

Friday, December 30, 2011

What is software architecture

Practical definition


Many people have different interpretations of what software architecture actually means. Some people take it as the high level decomposition of the system (subsystems or physical distribution). However, a much better interpretation is simply "the most important design decisions". What are important design decisions? If they are going to impact what you (or your customer) care the most, then it is important. Typically, if the design decision is going to impact the availability, performance, scalability, security, usability, manageability, maintainability, cost, then it is part of the architecture.

Examples


Below are some examples of architectural decisions:

  • Availability: how to build in redundancy in all the layers (storage, middle tier) and how to fail-over (e.g., client initiated vs virtual IP)? How to handle excessive load (e.g., QoS)? Selecting well proven products with strong support?

  • Scalability: how to scale (scale out or scale up) and load balance?

  • Performance: multi-threading, algorithm, processing in batches?

  • Security: encrypting data against exposure, using HMAC to check data integrity, secure communication, auditing?

  • Manageability: monitoring and operational control (e.g., logging, JMX)? Live upgrading (OK to upgrade just one node at a time in a cluster)?

  • Maintainability: automated tests, using a language, tools and frameworks familiar to the team, proper layering of the code (UI, service, data access layers)?

  • Cost: language, tools and frameworks familiar to the team, open source vs commercial products?


Specifying and maintaining the architecture


I find the best way to specify the architecture is to explain it in use cases (e.g., how to perform fail-over when a certain component fails or how to ensure security in a typical use case).
Also, specifying the architecture is just the first step. Some team members may make sub-optimal architectural decisions in programming without knowing it, or you may realize that your initial decisions can be further improved when you see the code. Therefore, as the architect, your work is far from over and you must keep the architecture (as implemented in the code) fit as the project goes.

Sunday, August 28, 2011

Reaction to "Clojure: Towards The Essence Of Programming" from a Scalaperspective

How Scala addresses those problems


Just watched Howard Lewis Ship's Clojure: Towards The Essence Of Programming, which is an excellent introduction to Clojure and how it addresses the problems in Java. As a Scala fan, I thought it would be a good idea to see how one could use Scala to do the same.

Ceremony vs Essence


Howard uses the example below to show how much ceremony (code with little real info) is required, hiding the essence (the real intention) when using Java to sort a list of Stock objects (a portfolio) either using different properties such as open values or last trade values:
 public static void sortByOpen(List<Stock> portfolio) {
    //ceremony: a lot of code but very little real info
    Comparator<Stock> c = new Comparator<Stock>() {
       public int compare(Stock o1, Stock o2) {
          return o1.getOpen() - o2.getOpen(); //essence: the beef is here
       }
    };
    Collections.sort(portfolio, c);
}

The Clojure version is:
//a Stock object is just a map with each field as a key. ":open" is the key to get the open value.
(sort-by :open portfolio)

In Scala, it can be as simple as the Clojure version:
//full version: using an anonymous function instead of a comparator object
portfolio.sortBy((s: Stock) => s.open)

//simpified version: sortBy is expecting the argument to be a function
// (Stock) => SOME-RESULT-TYPE, so we don't need to declare the 
//type for "s".
portfolio.sortBy((s) => s.open)

//final version: as "s" only appears once in the function body, just use
//an underscore and get rid of the parameter declaration.
portfolio.sortBy(_.open)

The Clojure version is like the final Scala version in the first attempt because it doesn't need typing information in the code. In Scala, the compiler infer it from the code.

In addition, in Clojure the object is just a map so people can refer to a property directly with its name without referring to the object. In Scala, we must use something to denote the object in the form of obj.property. However, the underscore syntax eliminates the need to declare that object, so the code is still very succinct.
The benefits of the Scala approach are:

  • Compile time checking: if the property name mismatches the Stock class, in the Scala the compiler will tell us immediately.

  • Runtime performance: it is a lot faster to access the property by offset than by looking up a key.

The cost we pay is mainly the learning curve: we must learn how Scala infers the typing information so that we know what to type and what to NOT type and learn the meaning of the underscore in an anonymous function.

Form is structure


I may not be fully understanding this example. Howard uses the Fahrenheit conversion, f =9*c/5+32, to show that in Clojure the form (the look on the surface) is the same as the structure (the actual relationship):
//the look is the same as the actual syntax tree
(+
  (/
    (* 9 c)
    5)
  32)

I definitely think the Java or Scala version is much simpler and more concise:
9*c/5+32

This is, after all, what we call DSL. Arithmetics has its rules of precedence and associativity, allowing us to write in a very succinct form to express more complex structure.

Read Eval Print Loop


Scala has it too. I like it very much too.

Functional programming


Howard that moves onto functional programming with examples using filter, reduce, map, for comprehension, stream. Obviously, Scala can do all these in a similar fashion. For example, to get the last trade total value of the portfolio, we can map each stock to its last trade value and then use reduce to sum them all up. In Clojure:
//#() means anonymous function. % refers to the one and only parameter (the stock).
(reduce + (map #(* (% :last-trade) (% :shares)) portfolio))

In Scala:
//+ in Scala is not a function, but a method. So we can't use it directly; 
//instead, use an anonymous function with underscores again.
portfolio.map((s)=>s.lastTrade*s.shares).reduceLeft(_+_)

There is not much difference. However, I think the significance is much deeper. The question is not whether functional programming is supported or not, but how much it is intended to be used in place of procedural programming. Scala allows both styles of programming, which may or may not be a good thing, depending on how you view it. If you're committed to the functional programming style for concurrency, testability, predictability, then using a language that is less capable of procedural programming is actually a good thing. If you're more comfortable with procedural programming and just using functional programming in a smaller scale, then a hybrid language would be better.

Extending the language with DSL


Then Howard moves onto extending Clojure with DSL. One example he uses is building a DOM tree with a DSL.
(defview root-index
  [env]
  :html [
    :head [
      :title [ "Cascade Blog" ]
    ]
    :body [
      :h1 [ "Cascade Blog" ]
      :ul { :class "recent-postings" } [
        (template-for [posting (recent-postings env)]
          :li [
             (render-link env show-posting (posting :id) (posting :title))
           ])
      ]
    ]
  ])

As an exercise I implemented a similar DOM DSL in Scala:
//the DSL is implemented in the Html object
import Html._

//using the DSL to define a DOM tree
p @@ ("style" -> "color: red") containing (
        "This is an ",
        b containing ("interesting"),
        "DOM tree"
        )

They aren't that different except that in Clojure each list item is separate from its neighbors by being enclosed in its own () but in Scala we must use a comma to separate the list items. So, choose your own poison: either tolerate a million ()'s or the ugly commas in the DSL.
Another difference is that in Clojure text can be freely mixed in the DOM tree as there is no typing restriction, while in Scala text is magically converted into a TextNode object. Again, the latter provides structure, compile-time protection and runtime performance, but we must learn the implicit conversion mechanism to understand what is going on.
Just to be complete, the DOM DSL is implemented in Scala as below:
//a DOM node
class Node

//a text node
case class TextNode(val text: String) extends Node

//an element node
case class Element(val name: String,
                   val attrs: Map[String, String],
                   val children: Seq[Node]) extends Node {
  def this(name: String) = this (name, Map(), List())

  //we can use symbols as method names, but @ is a reserved keyword in Scala,
  //so I used @@ as the name of the method that adds attributes to the element.
  def @@(attrPairs: (String, String)*) =
    //just add each pair to the map
    Element(name, attrPairs.foldLeft[Map[String, String]](Map())(_ + _), children)

  //add the child nodes (elements or text nodes) to this element
  def containing(children: Node*) = Element(name, attrs, children)
}

object Html {
  //convert strings into text nodes automatically
  implicit def string2TextNode(s: String) = TextNode(s)
  def p = new Element("p")
  def b = new Element("b")
}

Howard also shows the need for lazy evaluation so that some code is only executed on demand. For example, the DSL code to generate each <li> is repeatedly called in a loop. In Clojure lazy evaluation is implemented with macros which generate well-form code instead of arbitrary text. This seems to be more powerful (able to do more) than the call-by-name mechanism Scala, although I don't know if/when this extra power is really needed.


Sunday, August 21, 2011

Revealing the Scala magician’s code: method vs function

How's a method different from a function in Scala?


A method can appear in an expression as an internal value (to be called with arguments) but it can't be the final value, while a function can:

//a simple method
scala> def m(x: Int) = 2*x
m: (x: Int)Int

//a simple function
scala> val f = (x: Int) => 2*x
f: (Int) => Int = <function1>

//a method can't be the final value
scala> m
<console>:6: error: missing arguments for method m in object $iw;
follow this method with `_' if you want to treat it as a partially applied function
m
^

//a function can be the final value
scala> f
res11: (Int) => Int = <function1>

Parameter list is optional for methods but mandatory for functions


A method can have no parameter list or have one (empty or not), but a function must have one (empty or not):

//a method can have no parameter list
scala> def m1 = 100
m1: Int

//a method can have an empty parameter list
scala> def m2() = 100
m2: ()Int

//a function must have a parameter list
scala> val f1 = => 100
<console>:1: error: illegal start of simple expression
val f1 = => 100
^
//a function's parameter list could be empty
scala> val f2 = () => 100
f2: () => Int = <function0>

Why a method can have no parameter list? See below.

Method name means invocation while function name means the function itself


Because methods can't be the final value of an expression, so if you write a method name and if it doesn't take any argument (no argument list or an empty argument list), the expression is meant to call that method to get the final value. Because functions can be the final value, if you just write the function name, no invocation will occur and you will get the function as the final value. To force the invocation, you must write ():

//it doesn't have a parameter list
scala> m1
res25: Int = 100

//it has an empty parameter list
scala> m2
res26: Int = 100

//get the function itself as the value. No invocation.
scala> f2
res27: () => Int = <function0>

//invoke the function
scala> f2()
res28: Int = 100

Why we can provide a method when a function is expected?


Many Scala methods such as map() and filter() take functions arguments, but why can we provide methods to them like:

scala> val myList = List(3, 56, 1, 4, 72)
myList: List[Int] = List(3, 56, 1, 4, 72)

//the argument is a function
scala> myList.map((x)=>2*x)
res29: List[Int] = List(6, 112, 2, 8, 144)

//try to pass a method as the argument instead
scala> def m3(x: Int) = 3*x
m3: (x: Int)Int

//still works
scala> myList.map(m3)
res30: List[Int] = List(9, 168, 3, 12, 216)

This is because when a function is expected but a method is provided, it will be automatically converted into a function. This is called the ETA expansion. This makes it a lot easier to use the methods we created. You can verify this behavior with the tests below:

//expecting a function
scala> val f3: (Int)=>Int = m3
f3: (Int) => Int = <function1>

//not expecting a function, so the method won't be converted.
scala> val v3 = m3
<console>:5: error: missing arguments for method m3 in object $iw;
follow this method with `_' if you want to treat it as a partially applied function
val v3 = m3
^

With this automatic conversion, we can write concise code like:

//10.< is interpreted as obj.method so is still a method. Then it is converted to a function.
scala> myList.filter(10.<)
res31: List[Int] = List(56, 72)

Because in Scala operators are interpreted as methods:

  • prefix: op obj is interpreted as obj.op.

  • infix: obj1 op obj2 is interpreted as obj1.op(obj2).

  • postfix: obj op is interpreted as obj.op.


You could write 10< instead of 10.<:

scala> myList.filter(10<)
res33: List[Int] = List(56, 72)

How to force a method to become a function?


When a function is not expected, you can still explicitly convert a method into a function (ETA expansion) by writing an underscore after the method name:

scala> def m4(x: Int) = 4*x
m4: (x: Int)Int

//explicitly convert the method into a function
scala> val f4 = m4 _
f4: (Int) => Int = <function1>

scala> f4(2)
res34: Int = 8

A call by name parameter is just a method


A call by name parameter is just a method without a parameter list. That's why you can invoke it by writing its name without using ():

//use "x" twice, meaning that the method is invoked twice.
scala> def m1(x: => Int) = List(x, x)
m1: (x: => Int)List[Int]

scala> import util.Random
import util.Random

scala> val r = new Random()
r: scala.util.Random = scala.util.Random@ad662c

//as the method is invoked twice, the two values are different.
scala> m1(r.nextInt)
res37: List[Int] = List(1317293255, 1268355315)

If you "cache" the method in the body, you'll cache the value:

//cache the method into y
scala> def m1(x: => Int) = { val y=x; List(y, y) }
m1: (x: => Int)List[Int]

//get the same values
scala> m1(r.nextInt)
res38: List[Int] = List(-527844076, -527844076)

Is it possible to maintain the dynamic nature of x in the body? You could cache it as a function by explicitly converting it:

//explicit conversion, but then you must invoke the function with ().
scala> def m1(x: => Int) = { val y=x _; List(y(), y()) }
m1: (x: => Int)List[Int]

scala> m1(r.nextInt)
res39: List[Int] = List(1413818885, 958861293)

Saturday, August 6, 2011

equals() and Scala

Problem with equals()


The problem with equals() is that it is difficult to get it right. For example, for a simple class Foo, you may write the equals() method as:

class Foo(val i: Int) {
override def equals(that: Any) = {
that match {
case f: Foo => f.i == i
case _ => false
}
}
}

The problem is that what happens if the other object ("that") belongs to a subclass of Foo, which may have its own fields? As the equals() method is only comparing the "i" field, it will ignore the other fields and return true prematurely. Below is such a subclass Bar:

class Bar(i: Int, val j: Int) extends Foo(i) {
override def equals(that: Any) = {
that match {
case b: Bar => super.equals(b) && b.j == j
case _ => false
}
}
}

With the erroneous equals() method in Foo, we could get incorrect results:

scala> val f1 = new Foo(2)
f1: Foo = Foo@14c0275

scala> val b1 = new Bar(2, 3)
b1: Bar = Bar@171bc3f

scala> f1.equals(b1)
res0: Boolean = true

scala> b1.equals(f1)
res1: Boolean = false

A solution to the problem


The problem is that the equals() method in Foo is treating the Bar object exactly as a base Foo object, but the equality contract in Bar has changed from that in Foo. Of course, not every subclass of Foo will use a different equality contract; some do and some don't (by default, we should assume that they don't). Therefore, the equals() method in Foo should make sure that the "that" object uses the same equality contract as "this":

object FooEqualityContract {
}

class Foo(val i: Int) {
//by default all Foo objects and subclass objects use this equality contract
val equalityContract: Any = FooEqualityContract

override def equals(that: Any) = {
that match {
//make sure the two objects are using the same equality contract
case f: Foo => f.equalityContract == this.equalityContract && f.i == i
case _ => false
}
}
}

Now, as Bar is using its own equality contract, it should say so:

class Bar(i: Int, val j: Int) extends Foo(i) {
//tell others that we're using our own equality contract
override val equalityContract: Any = BarEqualityContract

override def equals(that: Any) = {
that match {
case b: Bar => super.equals(b) && b.j == j
case _ => false
}
}
}

Now, the equals() method in Foo will rightly determine that a Bar object is using a different equality contract and thus will never be equal to a bare Foo object:

scala> val f1 = new Foo(2)
f1: Foo = Foo@34b350

scala> val b1 = new Bar(2, 3)
b1: Bar = Bar@7c28c

scala> f1.equals(b1)
res2: Boolean = false

scala> b1.equals(f1)
res3: Boolean = false

scala> val b2 = new Bar(2, 3)
b2: Bar = Bar@5dd915

scala> b1.equals(b2)
res6: Boolean = true

Of course, it should also work for subclasses that use the same equality contract:

scala> val f2 = new Foo(2) { }
f2: Foo = $anon$1@a594e1

scala> f1.equals(f2)
res9: Boolean = true

scala> f2.equals(f1)
res10: Boolean = true

Friday, July 15, 2011

A simple but highly useful feature request for DNS

Most people believe that by having two Windows domain controllers can provide transparent fail over, i.e., if one DC fails, the clients will automatically use the other. However, this is not true. The client will simply use the first DC returned by the DNS. Similarly, if you use DNS to load-balance between multiple web servers, when one of them fails, some clients will still be directed to it.
To fix the problem, there is a very simple solution: enhance the DNS server to perform a health check against the resulting host of the resource record. For example, the administrator could specify the TCP port to connect to as in the imaginary syntax below:
  www        A      1.1.1.1    80     ; return this record only if we can connect to its TCP port 80
www A 1.1.1.2 80
www A 1.1.1.3 80

Of course, the health check could be more general, then you could use a script:
  www        A      1.1.1.1    web-check.sh  ; return this record only if the script returns true

where the IP would be passed to that script as an argument for checking.
It works for domain controllers too:
  _ldap._tcp.dc._msdcs.foo.com.   SRV  1.1.1.1  dc-check.sh
_ldap._tcp.dc._msdcs.foo.com. SRV 1.1.1.2 dc-check.sh

Finally, one might ask why implement this checking in the DNS server instead of the clients? The idea is that problems should be detected as early as possible to avoid bad effects downstream. In concrete terms, if a server is down but the DNS server (broker) still refers the clients to it, many clients will need to perform this health check themselves. But if the DNS server performs this health check, the checking is only done once, saving a lot of trouble downstream.