Module loading in dojo (1.9)

How Dojo locates a module

The official Dojo tutorials unfortunately contain many errors or omissions. So, I had to write down my findings from my experiment. When we request the Dojo loader to load a module with id, say, foo/bar, it will basically start the search from the "base URL". The base URL by default is the folder containing dojo.js. Then it will drill down the path using the module id. For example, if dojo.js is loaded like:

then the base URL will be http://www.abc.com/js/dojo-dist/dojo and therefore, in principle, the Dojo loader will drill down and try to load http://www.abc.com/js/dojo-dist/foo/bar.js file. However, because Dojo is distributed with the following folder structure, it will mean that you need to put your foo folder into the dojo folder (which contains the core files of Dojo):

 js
  |
  +-- dojo-dist
       |
       +-- dojo  <--- 1: the base URL by default
       |     |
       |     +-- dojo.js
       |     |
       |     +-- foo  <--- 2: drill into here
       |          |
       |          +-- bar.js
       |
       +-- dijit
       |
       +-- dojox
       |
       +-- util

This doesn't make sense. Therefore, Dojo by default assumes that your files are put into a folder that is a sibling of the folder pointed to by the base URL. That is, after locating the base URL, it will go back one level before drilling down using the module id:

 js
  |
  +-- dojo-dist  <--- 2: go back to here
       |
       +-- dojo  <--- 1: the base URL by default
       |      |
       |      +-- dojo.js
       +-- dijit
       |
       +-- dojox
       |
       +-- util
       |
       +-- foo  <--- 3: drill into here
            |
            +-- bar.js

To observe the effect of this "going back one level", you can set the tlmSiblingOfDojo ("top level module sibling of Dojo") option between true (default, go back) and false (don't go back):

Putting the foo folder as a sibling of the dojo folder is better (at least the foo folder is not mixed with the core Dojo files). You might interpret the dojo-dist folder containing all Dojo related files, i.e., files from the Dojo Foundation and files created by you that use Dojo.

Putting your own files outside the Dojo folder structure

However, you may not like the above folder structure and decide to put your own files outside the dojo-dist folder, like:

 js
  |
  +-- dojo-dist
  |    |
  |    +-- dojo
  |    |     |
  |    |     +-- dojo.js
  |    +-- dijit
  |    |
  |    +-- dojox
  |    |
  |    +-- util
  |
  +-- foo
       |
       +-- bar.js

Then you must set the base URL yourself. You may set it to the dojo-dist folder and continue to rely on the default tlmSiblingOfDojo to go back to the js folder, before it drills down to the foo folder:

 js  <--- 2: let it go back to here
  |
  +-- dojo-dist  <--- 1: set the base URL to here
  |    |
  |    +-- dojo
  |    |     |
  |    |     +-- dojo.js
  |    +-- dijit
  |    |
  |    +-- dojox
  |    |
  |    +-- util
  |
  +-- foo  <--- 3: drill into here
       |
       +-- bar.js

Obviously, when setting the base URL yourself, it no longer has the meaning of "the folder containing dojo.js" and thus the tlmSiblingOfDojo option is really meaningless; it only means whether it should go back one level. The code is like:

As going back one level in this case is really confusing, it is better to just turn that off and set the base URL to the js folder:

 js  <--- 1: set the base URL to here and do NOT go back one level
  |
  +-- dojo-dist
  |    |
  |    +-- dojo
  |    |     |
  |    |     +-- dojo.js
  |    +-- dijit
  |    |
  |    +-- dojox
  |    |
  |    +-- util
  |
  +-- foo  <--- 2: drill into here
       |
       +-- bar.js

The code is like:

Instead of a full URL, the base URL can also be set as an absolute path (on the same host with the same protocol used to access the enclosing HTML file) or a relative path (to the enclosing HTML file). For example, If the enclosing HTML file is a sibling of the js folder, then it is better to use a relative path:

A more flexible location mechanism

Putting the modules as direct children of the base URL is fine, but may be too limiting. If you would like to have complete control over where to put each module, you can make use of the concept of a "package". A package can be considered a location with a name. For example, you can define a package named "foo" whose location is http://www.abc.com/js/myapp, then when you're trying to load the foo/bar/baz module, Dojo will note that the module id (foo/bar/baz) starts with foo, so it is in the foo package and thus it will go to http://www.abc.com/js/myapp first and then drill down to the bar folder and load baz.js:

Instead of using a full URL as the package location, it is possible to use an absolute path (use the same protocol and host as the base URL) or a relative path (to the base URL). Note that when locating a module through through the package method, which is much more flexible than the "assuming everything is a sibling of dojo", the tlmSiblingOfDojo is NOT used at all. That is, Dojo will locate the base URL, never go back one level, combine it with the package location (absolute or relative path) to determine the full URL for the package. Here is an example of a relative path:

Here is an example of an absolute path:

Of course, it is possible to define multiple packages:

Beauty of Scala: construction of hierarchical objects

In Java, one would construct a GUI (e.g., Swing) like:

JMenuBar menuBar = new JMenuBar();
JMenu file = new JMenu("File");
menuBar.add(file);
JMenuItem open = new JMenuItem(new AbstractAction("Open Word Bank") {
    public void actionPerformed(ActionEvent e) {
        openWordBank();
    }
};
file.add(open);

But in Scala, it is done like:

menuBar = new MenuBar {
  contents += new Menu("File") {
    contents += new MenuItem(new Action("Open Word Bank") {
      def apply {
        openWordBank
      }
    })
  }
}

That's the beauty of Scala! The code can be written like that because we can write initialization code (usually inside a constructor in Java) directly inside a Scala class.

Network access related protocols: PPP, EAP, RADIUS, 802.1x, L2TP, GRE

PPP

• PPP is protocol to negotiate network layer protocol parameters (e.g., IP) and a encapsulation to carry IP packets for transmission over a point-to-point link (a layer 2 functionality). The former allows the server to assign IP, netmask and DNS server etc. to the client. The latter allows the actual transmission of IP packets.
• When a packet is sent to the PPP's IP interface (ppp0 on Linux), PPP will put the IP packet into a PPP packet and deliver it to the point-to-point link. The other side will extract the IP packet from the PPP packet. This is very much link IP over Ethernet.
• Before allowing the client to connect, the client needs to authenticate with the server. PPP has built-in authentication protocols like PAP and CHAP.

EAP

• EAP was devised to run on top of PPP (without IP). Instead of requiring a particular authentication mechanism like PAP and CHAP, it allows different authentication mechanisms to be plugged into it.
• As EAP is not run on top of IP or TCP, it handles re-transmission (for packet loss) by itself. However, it does assume that packet ordering is preserved by the transport (typically ensured by point-to-point link).
• EAP is a peer to peer protocol. Either side can authenticate the other side. The side being authenticated (e.g., a remote access client) is called the supplicant or the peer, the other side is called the authenticator (e.g., a network access server).
• The authenticator can authenticate the peer itself or can pass through to another server (authentication server). The latter approach is useful to centralized the user accounts. Typically, it is a RADIUS server.
• EAP-TLS is an authentication mechanism of EAP which uses TLS for authentication. That is, using the TLS handshake protocol (using the certificate of the peer) for authentication. The last step of the TLS handshake protocol is to verify that both sides have the master secret which was established by random numbers generated by each side and sent to the peer encrypted by the public key of the peer.
• PEAP (protected EAP). There are some weaknesses in EAP with some mechanisms such as sending the identity of the peer in clear. To avoid the problem, PEAP works by establishing a TLS session with the authenticator (or authentication server in pass-through mode) first and then perform EAP (so, the actual mechanism is needed such as EAP-CHAP). This way, everything is encrypted. PEAP is not an open standard but created by Cisco and MS.

RADIUS

• RADIUS provides its own authentication (quite weak) using a central user database. It can also return additional information through the attributes to the authenticator (e.g., the privilege level of the user).
• Even though RADIUS has its own authentication protocol, it can support CHAP and EAP using its attributes.
• A RADIUS attribute has a type (an integer) and a value. It is called attribute-value pair (AVP). There are some standard RADIUS types and vendor proprietary attribute types.

802.1x

• 802.1x (EAP over LAN) is basically just EAP over 802 (Ethernet or 802.11 wireless LAN) so that a switch or an AP can authenticate a device being connected to it. This is like the case in PPP for allowing the client to connect or not, except that PPP deals with a remote client while 802.1x deals with a LAN client. As on a LAN there is no need for another layer 2 protocol, only the authentication part (EAP) of PPP is needed.
• As the authentication is done in layer 2, there is no IP yet. So, just like normal EAP, EAP packets are encapsulated in layer 2 frames (here, in 802 frames). 
• The switch or AP is called the authenticator as it needs to authenticate the peer.
• In pass through mode, the authenticator will use RADIUS protocol with EAP support to talk to the authentication server (for central management).
• A RADIUS server can return RADIUS attributes telling the switch which VLAN to put the client/port into.

L2TP

• L2TP is used to simulate a point-to-point link with an IP network such as the Internet. Therefore, PPP can be run over L2TP. This way, a VPN tunnel can be built across the Internet. Essentially L2TP serves as a layer 2 protocol that runs on top of a layer 3 protocol (IP). When people say it is an L2TP VPN, actually it is L2TP and PPP VPN as the address assignment and etc. are done with PPP. The authentication is also the same (e.g., EAP).
• L2TP itself provides no encryption, so typically it is run over IPSec to protect the traffic. So, it is PPP over L2TP over IPSec.

GRE

• GRE is very much like L2TP except that it can tunnel any protocol over any protocol (i.e., the transport is not necessarily IP).
• When using for VPN, PPP is also run over GRE. In addition, for security, the PPP payload is typically encrypted and this modified version of PPP is PPTP (proprietary by MS).

Concepts of Nagios

• Purpose. Basically Nagios is used to monitor if your servers and network devices are up and working properly. If something is not working, it can notify you (the administrator) so that you can fix the problems ASAP.
• How it works (basically). Nagios runs on a central server so that you can view the status of everything in one web interface. It will check each host (server or network device) by pinging its hostname or IP, say, every 5 minutes ("check interval"). If a host is down, it will notify the administrator ("contact") or the administrators (the "contact group").
• Notify on state change. Once a host is detected as down, the checking will continue as usual to detect if it is up again.
• If it is still down, usually no further notification is sent (to avoid bombarding the admins). However, to avoid the problem from being forgotten, you can configure it to send a notification at a certain interval ("notification interval").
• If it becomes up, another notification will be sent so that the admins will know that it has recovered.
• Therefore, notification is sent when the state is changed, not when a host is down.
• Soft and hard state. Sometimes a host is not down, but just very busy or the network is very slow so some pings may timeout. Therefore, if a host is considered down by ping, the state is changed to a "soft" down, meaning that it is possibly down. No notification is sent yet. Nagios will check it ("retry") a few times more (the "max check attempts"). If it is still down for these checks, the state will be changed to a "hard" down and then a notification is sent. Similarly, when a host recovers from a hard down to up, it is just a soft up. Only when it is still up in the subsequent checks, it will change to a hard up and then a notification will be sent. In summary, notification is sent only when the state is changed into a different hard state.
• Unreachable state. In addition to up or down, there is a third possible state: unreachable. It can happen if the host is running fine, but a router between them is down. But how can Nagios distinguish which case it is when ping fails? You can create a host record for the router in Nagios  and tell it that the router is the "parent" of that server host (on the path from the Nagios server to that server host). This way, when ping to that server host fails, Nagios will check if the router is up or down. If it is down, it will mark the server host as unreachable instead of down. Again, a one time result only leads to a change to a soft state.
• Will the admins still get a notification if it is changed to a hard unreachable state? By default yes but you can configure it (below).
• Command. In Nagios you can configure the actual command used for ping or for sending notifications. To allow you to refer to the command by a simple name without worrying about the actual path, each command is assigned a name for later use. Many such commands will also retrieve information from environment variables such as HOSTADDRESS ("macro" and set up by Nagios). Below is the command typically used for ping:  

define command {
  command_name check-host-alive
  command_line /var/lib/nagios/plugin/check_ping -H $HOSTADDRESS$
}

Below is a simplified example of a command used for sending email notification (again, Nagios will provide the information such as email, host name, host state through the macros):


define command {
  command_name notify-by-email
  command_line /bin/echo $HOSTNAME$ in $HOSTSTATE$ | /usr/bin/mail $CONTACTEMAIL$
}
• Contact. A contact is simply a configuration item for sending a text message (a command line) to a person. It specifies the email address, pager/mobile number and etc. so that the command know where to send the message to.
• Notification options. It can also specify what kinds of notifications it will accept: it can choose to accept down (from up to down), recovery (from down to up), unreachable (from up or down to unreachable), etc.
• Notification period. It can also specify what time of the day on which days notifications can be sent. Usually it should be 24x7 for critical hosts, but if the host is unimportant and this is contact sending SMS, you may configure it to only send in business hours. The notification outside of the period is not lost though: Nagios will schedule it to be sent at the start of the next time slot (e.g., next morning).
• Notification settings for a host. The notification options and notification period can also be specified for a host. It means some hosts may be more important and some admins may be available only for certain issues in certain time periods.
• Below is an example of a contact (using the notify-by-email command above):

define contact {
  contact_name kent
  email kent@foo.com
  host_notification_options d,r,u
  host_notification_period 24x7
  host_notification_command notify-by-email
}

• Contact group. A contact group is just a set of contacts (e.g., for all the administrators). You define the individual contacts first and then add them to a group. This way, you can configure the host checking to notify a contact group instead of individuals. Below is an example:

define contactgroup {
  contactgroup_name admins
  members kent, paul
}

• Check period. Instead of performing checking at all time, Nagios requires that for each host you specify a period in which checking is performed. You can set it to 24x7 or your business hours depending on the supposed up-time window of the host.

• Host configuration example. The configuration of host in Nagios includes the address (hostname or IP) and the various checking and notification settings mentioned above. Below is an example (using the check-host-alive command above):

define host {
  host_name host1
  address 192.168.1.10
  parents router1
  check_interval 5
  check_period 24x7
  check_command check-host-alive
  max_check_attempts 4
  contact_groups admins
}

define host {
  host_name router1
  ...
}

• Time period. The time period used above such as 24x7 is defined like below:

define timeperiod {
  timeperiod_name 24x7
  sunday 00:00-24:00
  monday 00:00-24:00
  tuesday 00:00-24:00
  ...
}

• Object and object inheritance. Each item above such as host, contact, command is called an "object" in Nagios. To save on specifying all those options, Nagios allows you to define an object template that can be inherited by objects. For example, there is a template coming with Nagios called generic-host defined as below to set sensible values for many settings. The name is the name of the template so that you can refer to it by name and "register 0" tells Nagios not to treat it as a real host:

define host {
  name generic-host
  register 0
  max_check_attempts 10
  check_command check-host-alive
  contact_groups admins
  ...
}

• To use ("inherit") the template, just do it like this:

define host {
  host_name host1
  address 192.168.1.10
  parents router1
  use generic-host
  check_period 24x7
}

• Service monitoring. Just monitoring if a host is up or down isn't enough. It is needed to monitor the services such as website or email services are running properly. This is configured with a service object. A service object specifies the command used for checking (so the actual meaning of "up" is up to you to decide) and one or more hosts to provide the IP addresses. It means Nagios will perform the checking on each such host independently and pass the IP to the command via the $HOSTADDRESS$ macro. This is useful if, say, you have Apache running dozens of hosts, as you only need one service object. So, the service object is not really a single service, but just a way to apply a type of service checking across multiple hosts. Other than this, a service checking is very much similar to host checking and support similar settings such as max check attempts, check interval, contact, notification options, etc. Below is an sample service object (the check-http command will simply check if it can successfully retrieve a web page from the IP):

define service {
  service_description website
  host_name host1, host2, host3
  max_check_attempts 4
  check_interval 3
  check_period 24x7
  check_command check-http
  notification_options c,w,r
  notification_period 24x7
  contact_groups admins
  ...
}

• Service state. Instead of up, down, unreachable, a service (running on a particular host) can be up, critical (obvious error such as no response or connection refused or 404 error), warning (e.g., slow response. The actual meaning is defined by the command )  or unknown (usually meaning that the check command itself has failed). The corresponding notification options are r (recover), c (critical), w (warning).
• Service notification settings in a contact. Probably because the notification options for service checking are different from those for host checking, Nagios requires that in a contact object you  specify the notification settings separately for host notifications and for service notifications:

define contact {
  contact_name kent
  email kent@foo.com
  host_notification_options d,r,u
  host_notification_period 24x7
  host_notification_command notify-by-email
  service_notification_options c,w,r
  service_notification_period 24x7
  service_notification_command notify-by-email
}

• Passing explicit arguments to a command. Just checking if a web page can be retrieved may not be good enough. A better check is to check if the web page contains the right content (e.g., containing a particular string). To do that, you can define your own command and pass the expected string (e.g., abc) to it. This is done using an exclamation mark and it is retrieved by $ARG1$ in the command definition:

define service {
  service_description website
  host_name host1, host2, host3
  max_check_attempts 4
  check_interval 3
  check_period 24x7
  check_command check-http-string!abc
  ...
}

define command {
  command_name check-http-string
  command_line /usr/lib/nagios/plugins/check_http -H $HOSTNAME$ -I $HOSTADDRESS$ -s $ARG1$
}

• Finding out the plugins. There are many commands (both the defined command objects and the actual programs) coming with Nagios. They are called plugins and many 3rd party ones can be installed. To find out what is available and how to use each one, just do it like:

# find the command object definitions by the plugins
$ ls /etc/nagios-plugins/config
  ...
# learn about the options supported by each program
$ /usr/lib/nagios/plugins/check_http --help
  ...

• Inheritance for service. Just like a host object, you can use object inheritance for a service object too. Nagios comes with a service template called "generic-service" defining many defaults for you to use:

define service {
  service_description website
  use generic-service
  check_command check-http-string!abc
  ...
}

• Host group. Instead of listing individual hosts in a service object, it may be a better way to define a "host group" named web-servers containing such hosts and just let the service object refer to the host group. This is particularly useful if you need to perform two kinds of service checking on the same group (e.g., checking web and SSH).

define service {
  service_description website
  hostgroup_name web-servers
  ...
}

define hostgroup {
  hostgroup_name web-servers
  members host1, host2, host3
}

• Service dependency. If the DNS or LDAP is down, then many other services will also not function. So, you will receive lots of notifications, burying the real issue. If you don't want to receive such "bogus" notifications, you declare that such services are dependent on the DNS or LDAP services. This way, when an upper level service fails, Nagios will check if the lower level service also fails. Given a specific option, you can tell Nagios to not send notification for the upper level service when the lower level service is in a certain "failure" states (e.g., critical or warning). As this dependency is about a service on a host depending on another service on another host, you must specify the hosts in addition to the services:

define servicedependency {
  dependent_service_description website
  dependent_host_name host1
  service_description dns
  host_name host2
  notification_failure_criteria c,w
}

• Service-host dependency. In principle, the same dependency should be specified for service and host as if a host is down, it is impossible for the service to continue to run properly. However, at present Nagios doesn't support this concept. So, if a host is down, you will still receive notifications for the services running on it.

Concepts of SNMP (including v3)

SNMP

• Purpose. SNMP is a protocol for getting the status (e.g., CPU load, free memory, network load) of computing devices such as routers, switches and even servers.
• Object descriptor, managed object. The client can provide a globally unique names such as cpmCPUTotal5secRev (the average CPU load of a Cisco device for the past 5 seconds) to indicate the information that it wants, then the server should return such information. Such a textual name is called the "object descriptor". The word "object" or "managed object" refers to the concept of CPU load. The actual CPU load in the device is called the "object instance".
• Object identifier (OID). To make sure that each object descriptor is unique, actually it is defined using a list of integers such as 1.3.6.1.4.1.9.9.109.1.1.1.1.6. Each integer is like a package in Java. For example, the integers in 1.3.6.1.4.1.9 represents iso (1), org (3), dod, i.e., department of defense (6), internet (1), private (4), enterprises (1), cisco (9) respectively. This allows the Internet authority to delegate the management of the namespace hierarchically: to private enterprises and then to Cisco, which can further delegate to its various divisions or product categories. Such a list of integers is called an "object identifier". This is the ultimate identification for the managed object.
• Even though the object descriptor should be unique, it is useful to see the hierarchy. Therefore, usually the full list of object descriptors is displayed such as iso.org.dod.internet.private.enterprises.cisco...cpmCPUTotal5secRev.
• Why use integers instead of symbolic names? Probably to allow the network devices (with little RAM or CPU power) implementing SNMP to save space in processing. Symbolic names such as object descriptor can be used by human in commands, but in the protocol's operation it is done using object identifier.
• In principle, the object a.b.c.d and the object a.b.c.d.e on a device have NO containment relationship. That is, they are NOT like a Java object containing a child object. In fact, the value of each object in SNMP is basically a simple value (scalar) such as an integer or a string. The only relationship between them is their names.
• Identifying an instance. Now comes the most complicated concept in SNMP. Consider the concept of the number of bytes that have been received by a network interface on a router. This concept is an object. As a router should have multiple interfaces, there must be multiple instances of that object. Then, how can an SNMP client indicate to the SNMP server which instance it is interested in? The solution is more or less a kludge: to allow the instance of, say, a.b.c.d, to represent a table (a compound, structural value), which contains rows (also compound, structural value) represented by a.b.c.d.e. Each row contains child object instances (with scalar values only). Each child object is called a "columnar object". For example, each row may contain three object instances: a.b.c.d.e.f, a.b.c.d.e.g, and a.b.c.d.e.idx. If you'd like to refer to the a.b.c.d.e.f instance in a particular row, you will write a.b.c.d.e.f.<index>. The meaning of the index is defined by a.b.c.d.e (the row). For example, it may be defined as finding the row in the table which contains a columnar object a.b.c.d.e.idx whose value equals to <index>, then return the columnar object a.b.c.d.e.f as the result.
• Note that this is the only situation where the value of an object can be a structure and that there is object containment relationship in SNMP.
• What is confusing is that a.b.c.d.e.f is used both as an object identifier and the lookup key to find the child instance in the row. Unlike other object identifiers, the identifier now represents an object containment relationship so it must have a.b.c.d.e as the prefix, otherwise the server won't know which table to look into and what is the definition for the index.
• The complete identifier a.b.c.d.e.f.<index> is called an instance identifier.
• Here is a concrete example: Consider iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets.1.  The definition of iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry says that to find the row in the table, it should search for a row which contains a child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex with the value of 1 (the index specified), then it will return the value of child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets in the row. Of course, for this to work, the  iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex child object in each row must have been assigned with sequential values 1, 2, ..., etc. (which is indeed the case). 
• Finally, a simple case is that there is no table at all. For example, to find the up time of the device, use the object identifier iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime and append a .0 as the index, so the instance identifier is iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0. BTW, "hr" stands for "host resources".
• MIB (management information base). A MIB is just a a collection of managed objects (not instances). There are standard MIBs so that device manufacturers can implement and users can find the right object identifiers to use. There are also proprietary MIBs such as those designed by Cisco to provide information only available on its devices.
• Finding the supported OIDs. How can you find out the MIBs or the object identifiers supported by a device? It is easier to just "walk the MIB tree": to show all the instances in the tree or in a subtree. On Linux, this is done as below. You specify the IP or hostname of the server and optionally specify a node so that only that subtree is displayed (the object identifier starts with a dot, otherwise it will be assumed it is relative to iso.org.dod.internet.mgmt.mib-2):

# snmpwalk <some args> localhost
# snmpwalk <some args> localhost .iso.org.dod.internet.mgmt.mib-2.system
# snmpwalk <some args> localhost system

• Getting an instance. Just specify the instance identifier:

# snmpget <some args> localhost .iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0
# snmpget <some args> localhost host.hrSystem.hrSystemUptime.0

• SNMP enttiy, engine and applications. SNMP is a peer to peer protocol. There is no concept of a server and a client (these terms are used here for simplicity). Instead, both peers have the same core capabilities such as sending or receiving SNMP messages, performing security processing (see below), integrating v1, v2c and v3 processing (dispatching) and etc. This part is called the SNMP engine. On top of the engine, there are different "applications": one application may only respond to requests for object instances (called SNMP agent in v1 and v2), another application may probe others (called SNMP manager in v1 and v2), yet another may forward SNMP messages (SNMP proxy). The whole server or client is called the "SNMP entity".
• Context. On some devices there are multiple copies of a complete MIB subtree. For example, a physical router may support the concept of virtual routers. Each such virtual router will have a complete MIB subtree of instances. In that case, each virtual router may be indicated as a "context" in the SNMP server. A context is identified by name. For example, a virtual router could be identified as the "vr1" context. There is a default context with empty string ("") as its name. When a client sends a query, it can specify a context name. If not, it will query the default context.
• Notification (trap). An SNMP server may actively send a notification to the client when some condition occurs. This is very much like a response without a request. Otherwise, everything is similar. The condition (e..g., the changes of the value of an instance or its going out of a range), the destination, the credentials used (see the security section below) and etc. are configured on the server.
• Transport binding. Typically SNMP runs on UDP port 161.

SNMP security

• SNMP v1 and v2c security. In SNMP v1 and v2c (v2 was not widely adopted), there is little security. The only security is the "community string". That is, the server is configured to be in a community identify by a string such as "foo", "public" (commonly used and the default for many devices to mean no protection) or "private". If the client can quote the community strnig, then it is allowed access. As the community string is included as plain text in SNMP packets, it practically provides no security. Therefore, in v1 and v2c, to access an SNMP server, you will do something like:

# snmpwalk -v 2c -c public localhost
# snmpget -v 2c -c public localhost <INSTANCE ID>

• SNMP v3 security. In SNMP v3, there is user-based security. That is, the client may be required to authenticate the messages to the server as originating from a user using a password (authentication password). In addition, the client may be furthered required to encrypt the messages using another password (privacy password). This security requirement is called the "security level" (no authentication needed, authentication but no privacy, authentication with privacy). Therefore, in v3, you will access the server like:

# snmpwalk -v 3 -l noAuthNoPriv localhost
# snmpwalk -v 3 -l authNoPriv -u kent -A "my auth passwd" localhost
# snmpwalk -v 3 -l authPriv -u kent -A "my auth passwd" -X "my priv passwd" localhost 

• Client configuration file. To save typing all those every time, you can store these parameters into the snmp.conf file as defaults.
• Security limitation. It is a bad idea to specify the password on the command line as it can be revealed by local users using "ps". Storing it into the configuration file is better. However, the file only allows a single authentication password and a single privacy password, not enough to handle the case of using different passwords for different servers.
• Security name. A security name is just a user name. No more, no less. That's the term used in the RFC (maybe in the future it could be something else?)
• Algorithm. Further, there are different algorithms for authentication (HMAC using MD5 or SHA) and for privacy (encryption using DES or AES). So, you need to specify the algorithms to use:

# snmpwalk -v 3 -l noAuthNoPriv localhost
# snmpwalk -v 3 -l authNoPriv -u kent -a MD5 -A "my auth passwd" localhost
# snmpwalk -v 3 -l authPriv -u kent -a MD5 -A "my auth passwd" -x DES -X "my priv passwd" localhost 

• Ensure the algorithms match. As SNMP uses UDP and each query and response may use just a single UDP packet, there is no negotiation of algorithm at "connection phase" at all. In fact, presumably for simplicity in implementation, the algorithms used are not even indicated in the message, so the client must use the agreed-on algorithms as configured in the user account on the server, otherwise the server will simply fail to authenticate or decrypt the message.

• Localized keys. The authentication password and privacy password of a user account are not used directly. The idea is, most likely you will use the same password for all the user account on all devices on site. If it is directly used, then a hacker controlling one device will be able to find the password and use it to access all the other devices. Therefore, when creating a user account, you specify the password, but the Linux SNMP server will combine it with a unique ID (called the "engine ID") generated for the device (such as the MAC or IP and/or a random number generated and stored on installation), hash it and use the result as the password (the "localized key"). This way, even if the hacker can find this localized key, he will still be unable to find the original password.
• But how can a client generate the same key? It has to retrieve the engine ID first and then perform the same hashing. This is supported by the SNMP protocol.
• User accounts creation. Due to the need to generate localized keys, the way to create user accounts on Linux is quite weird. You stop the server, specify the user account's name and password in a file, then start the server. It will read the password, convert it to a localized key and overwrite the file. This file is /var/lib/snmp/snmpd.conf on Linux:

createUser kent MD5 "my auth password" DES "my privacy password"
createUser paul SHA "my auth password, no encryption needed"

• Access control. The access control can specify the user account, the lowest security level required, which part of the MIB tree is accessed (may use an OID to identify a subtree), the type of access (read or write), in order to grant the access. Here are some example settings on Linux (although human user names are used, but in practice they should be representing devices):

rouser john noauth .iso.org.dod.internet.mgmt.mib-2.system
rouser kent priv .iso.org.dod.internet.mgmt.mib-2.system
rouser kent auth .iso.org.dod.internet.mgmt.mib-2
rwuser paul priv 

• View. How to specify several subtrees in an access control rule? You can define a view. A view has a name and is defined as including some subtrees and excluding some subtree. Then you can refer to it by name in access control:

view myview included .iso.org.dod.internet.mgmt.mib-2.system
view myview included .iso.org.dod.internet.mgmt.mib-2.host
view myview excluded .iso.org.dod.internet.mgmt.mib-2.host.hrStorage

rwuser paul priv -V myview 

• Access control for v1 and v2c. For v1 and v2c, access control can specify the community string, the IP range of the client (the "source"), the subtree (OID) or the view:

rocommunity public 192.168.1.0/24 .iso.org.dod.internet.mgmt.mib-2.system
rwcommunity private localhost -V myview

• Most flexible access control model. The above access control model is called the "traditional model". The new, most flexible access control model is called the "view-based access control model (VACM)", even though the former can also use views. It may be more suitable to called it group-based access control as it uses user groups in the rules (NOT the precise syntax yet!):

group g1 kent
group g1 paul
#access <group> <context> <min sec level> <exact context?> <view for read> <view for write> <view for notify>
access g1 "" auth exact myview1 myview2 myview3

• Mapping community string to user name. When using the VACM, instead of granting access to community strings, you need to merge v1 and v2c into the user-based access control processing. To do that, a community string along with the source can be mapped to a user name (the user name mapped to do NOT have to be existing):

com2sec user1 192.168.1.0/24 public
# "default" source means any
com2sec user2 default private

• Security model. Even though the different types of identity in the different SNMP versions are represented uniformly as a user name, their trustworthiness is still significantly different. So, in specifying group memberships and access control rules, you are required to specify the "security model" (v1, v2c or the user security model as in v3) and that's the correct syntax:

group g1 usm kent
group g1 usm paul
group g2 v2c user1
group g2 v1 user1
group g2 v1 user2
access g1 "" usm auth exact myview1 myview2 myview3
access g2 "" any noauth exact myview4 myview5 myview6

Basic concepts of OpenLDAP

LDAP

• LDAP is just a type of database specialized for look ups. Reads are fast while writes are supported but relatively slow. That's why LDAP is commonly used to store centralized user accounts.
• In LDAP, data is stored nodes (much like a record in a relational DB). Each node has some named attributes. For example, a node representing a user may have the following attributes (cn stands for "common name", while sn stands for "surname"):

cn: Kent Tong
sn: Tong
givenName: Kent
email: kent@ttdev.com

• Just like a record in a DB or an object in Java, the attributes that a node can have is fixed. This is specified by an attribute named objectClass. For example, there is an objectClass named "person" that says that it can have cn, sn and email, then a node belonging that this "person" class can have such attributes:

cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• It is possible for a node to belong to multiple object classes. For example, there may be an objectClass named "securityObject" that says that it can have a password attribute, then if you need to put a password into a node, you can do it like:

cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person
objectClass: securityObject
password: abc123

• How to use a node to present a department or a company? This is done with the "organizationalUnit" objectClass ("ou" stands for "organizational unit"):

ou: Foo Ltd.
objectClass: organizationalUnit

ou: Finance dept
objectClass: organizationalUnit

• If the person Kent Tong is in that Financial department, how to represent that relationship? Unlike a relative DB where such relationship is represented a foreign keys, in LDAP one node can be put under the other as a child node. For example, let's put the person node as a child node of the financial dept node which is put as a child node of the company node:

dn: ou=Foo Ltd.
ou: Foo Ltd.
objectClass: organizationalUnit

dn: ou=Finance dept,ou=Foo Ltd.
ou: Finance dept
objectClass: organizationalUnit

dn: cn=Kent Tong,ou=Finance dept,ou=Foo Ltd.
cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• This parent-child relation is represented by an attribute called "dn" (distinguished name, just like an full path to a file). That is, you can determine the dn of the parent node from the dn of the node.
• Usually you will add the cn to the dn of the parent to get the dn of the node itself. However, you don't have to use cn. You are free to any attribute to serve that purpose (called "relative dn") as long as it is unique among the siblings. For example, cn is good if there are no two people with the same cn values  the dept. This is so that given a dn, it is very fast to locate the node.
• When you create the top node of the directory, you need to specify its dn. Usually, you will use the DNS domain name of your company (let's say it is foo.com):

dn: dc=foo,dc=com
ou: Foo Ltd.
objectClass: organizationalUnit

dn: ou=Finance dept,dc=foo,dc=com
ou: Finance dept
objectClass: organizationalUnit

dn: cn=Kent Tong,ou=Finance dept,dc=foo,dc=com
cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• in which dc means "domain component", just a label in a domain name.
• Some object classes such as person or organizationalUnit are designed to be used as the "main" object class for the nodes, while some other object classes such as securityObject (officially, it is called "simpleSecurityObject") are designed to be "auxiliary" and to be added to the nodes. The former is called a "structural object class" while the latter an "auxiliary object class".
• For each node, it must have exactly one structural object class. It can have zero or multiple auxiliary object classes.

Schema

• The attributes and object classes are defined instead of hard-coded. For OpenLDAP on Debian, you can find the schemas in /etc/ldap/schema. For example, you can find the following object class in core.schema:

objectclass ( 2.5.6.6 NAME 'person'
        DESC 'RFC2256: a person'
        SUP top STRUCTURAL
        MUST ( sn $ cn )
        MAY ( userPassword $ telephoneNumber $ seeAlso $ description ) )

• It says that it must have the "sn" and "cn" attributes and may have a "userPassword" attribute and etc. It says that its super class is "top" which is a built-in object class representing the topmost level object class. It also says that the "person" object class is a structural class.
• You can find the most commonly used object classes and attributes here.

Authenticating with LDAP

• A straightforward way is to retrieve the node according to the username, get the password and compare it to the password the user entered.
• However, this allows retrieving sensitive information (password) of any user over the network by services. Therefore, there is another way: you send dn and the password to the LDAP service and let it perform the checking. This is called "binding" to that dn. Essentially you are logging in as a particular node.
• Both methods above will transfer the password in the clear. So, you should probably use TLS when accessing the LDAP service (using ldaps://)
• The above methods are most commonly used by other services such as mail, Samba or Apache for checking the passwords of the users.
• When performing administration with OpenLDAP, you may use the "bind" approach above to log in as the node representing the administrator. This is called "simple authentication":

# ldapsearch -x -W -D cn=admin,dc=foo,dc=com ...

• However, sometimes it is more convenient to authenticate not as a node in the directory but just as an administrator. For example, if your site is using Kerberos, you can perform single sign-on with OpenLDAP. This is done with its SASL support. SASL in turn supports many mechanisms such as PLAIN (plain text), GSSAPI (Kerberos), EXTERNAL (using the transport such as the client cert with TLS). When SASL to authenticate, OpenLDAP will make up a dn to represent your identity. For example, for user Kent in the Kerberos domain foo.com, the made-up dn will be uid=kent,cn=foo.com,cn=gssapi,cn=auth. This is needed for granting permissions.

Setting up and running OpenLDAP on Debian

• When setting up OpenLDAP on Debian, you need to reconfigure the package so that it creates the initial directory for you as well as the node representing the administrator:

# dpkg-reconfigure -p low openldap

• Then, to check if it is working, you can use SASL EXTERNAL mechanism and use Unix domain socket (ldapi:///) as root to connect to it:

# ldapsearch -b cn=config -H ldapi:/// -Y external

Query

• Just like a relation DB, you can perform queries to retrieve the nodes and their attributes. For example, you can try to find all nodes with cn=Kent Tong (the "filter") under the node whose dn is dc=foo,dc=com (the "base" of the search):

# ldapsearch -b dc=foo,dc=com -H ldapi:/// -Y external "cn=Kent Tong"

• You may try to find all nodes whose objectClass is "person" and whose cn contains the word "Kent":

# ldapsearch -b dc=foo,dc=com ... "(&(objectClass=person)(cn=*Kent*))"

• You may tell it to retrieve only a certain attributes:

# ldapsearch -b dc=foo,dc=com ... "(&(objectClass=person)(cn=*Kent*))" dn sn

Updating the data

• To add a new node, make sure the parent already exists, then specify the dn and other attributes in a text file (called an LDIF file) and mark it as an "add" request:

dn: cn=Paul Chan,ou=Finance dept,dc=foo,dc=com
changetype: add
cn: Paul Chan
sn: Chan
email: paul@ttdev.com
objectClass: person

• Then feed the LDIF file to ldapmodify:

# ldapmodify < myfile.ldif

• To modify (add, change or delete) the attributes of an existing node, specify the node first and then list each attribute change and separate them by a line containing a single hyphen. Finally, end the node with an empty line:

dn: cn=Kent Tong,ou=Finance dept,dc=foo,dc=com
changetype: modify
add: givenName
givenName: Kent
-
replace: email
email: kent@foo.com
-
delete: phoneNumber

• To delete a node:

dn: cn=Paul Chan,ou=Finance dept,dc=foo,dc=com
changetype: delete

Basic IPv6 concepts

• Basic concept. IPv6 is a completely separate protocol from IPv4! It means that the configuration and operation of IPv6 have nothing to do with those of IPv4 at all. They will run independently just like IPv4 and IPX or AppleTalk.
• Format of IPv6 address
• An IPv6 address contains 128 bits and is written in 8 groups of 16 bits like abcd:0000:1111:2222:1357:2468:5555:6789. Each group contains 4 hexadecimal digits (so, 16 bits).
• Many IPv6 addresses contain many consecutive groups of 0000 such as abcd:0000:0000:0000:0000:0000:0000:1234. To save typing, such consecutive groups of zero can be written as ::, so the address above can be written as abcd::1234. Or working from the reverse direction, because there are eight groups in an IPv6 address but only two groups were given, so :: is representing the six groups that are all zero.
• Different types of IPv6 addresses
• Just like IPv4, there are global IPv6 addresses that can be obtained from ISP. They begin with 2???: or 3???:.
• Just like private IPv4 addresses like 192.168.?.?, in IPv6 we use fec?:, fed?:, fee?:, fef?:. These are called site local addresses as they can only be used privately on a site. However, IPv6 provides a better alternative called "unique local address" in the range of fd??. To use it you will first generate a random number and append it to fd to form a random prefix, so that it will not collide (in practice) with any other range used by other private networks.
• Just like 127.0.0.1 in IPv4, there is ::1 (i.e., 0000:0000:0000:0000:0000:0000:0000:0001) in IPv6.
• Just like 169.254.?.? in IPv4, there are fe8?:, fe9?:, fea?:, feb?: address ranges in IPv6. They are called link local addresses as they can only be used privately on a link (a broadcast domain such as a switch or a set of interconnected switches). They are mainly used for network discovery (see below).
• Netmask and subnetting. The host bits in IPv6 take up at least the lower 64 bits (and maybe more), so the netmask is always <= 64 bits. The only exception is the loopback address ::1 which has a netmask of /128. For a LAN, usually just use /64 as the netmask. The ISP may assign a /64, /56 or /48 address block to a site.
• Assignment.
• Stateless auto configuration. In IPv6 it is fine to let a host configure its own address by getting the prefix (the network ID) from the router and then using its own MAC address as the host ID (with some encoding). Therefore, you only need to configure the router with static IPv6 address (so it knows the prefix). How can the host find the router? The host can use multicast to discover the router (part of the IPv6 Network Discovery Protocol) so you don't need to manually configure the default gateway of the host. The idea is that it can detect the running router automatically and perform fail-over as needed.
• DHCPv6. It is very much like DHCP for IPv4. Use it if you need centralized control.
• DNS.
• Domain name to address. You use an AAAA record to map a domain name to an IPv6 address. You can use either IPv4 or IPv6 to query the server and it can return A or AAAA records in either case (depending what record type you are searching for).
• NS, MX, etc. You can just put a domain name there and the name server will try to find its A and AAAA records and return them in the additional section.
• PTR. The reverse domain name is in the form of a.b.c.d.?.?.?.?.?.?.?.IPv6.arpa where a, b, c and d represent a hexadecimal digit (the address is like ??:??:??:...:dc:ba).
• Binding with application protocols. Most higher level protocols already support IPv4 and IPv6 simultaneously (e.g., ping/ping6, OpenSSH, Chrome, Firefox, Apache). To use an IPv6 address directly, you may need to put it in square brackets, e.g., http://[2001::1] in a browser otherwise the colon may look like starting a port. If you use a domain name and already has an AAAA record, the OS will return both the IPv4 and IPv6 to the application. According to the standard, actually global IPv6 are listed before  (preferred) public IPv4.
• In Linux, you can configure a static IPv6 address just like IPv4 in the interfaces file:

iface eth0 inet6 static
address 2001::1
netmask 64

• Checking the IPv6 addresses can be done with:

# ip addr show

Concepts of Squid

Basic proxy operation

• The normal mode of operation is for squid to serve as a proxy for internal users accessing the Internet. The user's browser is configured to use Squid as the proxy. Then, when the user tries to access a URL, the browser will send the request to Squid instead of the server in the URL (the "origin server").
• The request sent to Squid (a "proxy request") is slightly different than normal HTTP request as the former contains the full URL in the, say, GET command, while the latter only contains a path. This allows the proxy to contact the origin server as needed.
• When squid receives a request (proxy request), by default it will try to find the resource (the web page) in the following order:

1. Its cache.
2. The siblings and parents. Siblings should be deployed on the same site for load-balancing. Parents should be deployed in a larger ISP or regional headquarter. The idea is that the on-site proxies will in time contain mostly resources commonly accessed by the users on that site, while the ISP proxy will contain mostly resources commonly needed by the users in that region.
1. Squid will query all its siblings and parents (all are called "peers"). If there is a HIT response, it will send the request to the first such peer. If there is none (only MISS response or no response), the first parent returning MISS will get the request. It is supposed to get the resource from the origin server or its own parent if it is not in its cache. If there is no reply from the parents at all (e.g., all are busy or down), Squid will use the parent that is marked as the default.
2. The chain of request handling should normally stop here, unless, say, all the parents are down.
3. The origin server. The server hosting the resource (identified in the full URL in the GET command).

• After getting the resource, it will save it into its cache to speed up future accesses. The path to the cache and maximum disk space can be configured.
• Squid uses the ICP protocol to query its siblings and parents (the peers) to check if they have the resource in their caches. If yes, it will use HTTP to retrieve it. ICP is usually based on UDP and uses UDP port 3130. Squid usually listens for HTTP requests on TCP port 3128.
• A sibling or parent (peer) is defined like this:

cache_peer <hostname1> parent 3128 3130
cache_peer <hostname2> sibling 3128 3130

• The <hostname> above is used to resolve the IP.

Access security

• Usually you should only allow your internal users to use your Squid server (by restricting the client IPs or user authentication). To do that, you can  configure ACLs so that only authorized internal users can connect to it (the http_access configuration).

http_access allow myACL1

• An ACL is a named Boolean expression that can refer to various properties of the request (e.g., URL, domain name in the URL, HTTP method, source IP, authenticated user name):

acl myACL1 url_regex ^http://...
acl myACL2 dstdomain .foo.com

• When an ACL is evaluated and it refers to authenticated user name and etc. but that information is not yet in the request, Squid will perform proxy authentication with the client first.
• You can specify multiple ACLs to the same http_access and they will AND'ed together:

http_access allow myACL1 myACL2

Controlling routing of request

• You can control the routing between peers by applying ACLs to the peers. When Squid is selecting the peers, it will filter out those whose ACLs aren't satisfied. For example, to force Squid to use a particular parent for accesses to www.foo.com:

acl myACL1 dstdomain www.foo.com
cache_peer mycache1.local parent 3128 3130
cache_peer mycache2.local parent 3128 3130
cache_peer_access mycache1.local allow myACL1
cache_peer_access mycache2.local deny myACL1

SSL processing

• Usually the client will send a CONNECT command to Squid to establish a TCP proxy connection to the origin server. Squid will simply forward the raw TCP data (encrypted HTTP commands) back and forth.
• In newer versions of Squid, it can fake the origin server's certificate on the fly (so it needs the CA's cert and private key) or just a cert that doesn't really match the origin server. In either case, such cert faking and interception (called "ssl bumping") is configured in the http_port configuration.

Reverse proxy

• If you need it to serve as a reverse proxy for Internet users to access your DMZ web servers, most likely you won't need to use http_access to restrict the access. On the other hand, you can't just not configure siblings and parent in the hope that Squid will contact the origin server (using the server URL in the request) because that URL will resolve to the IP of Squid itself! So, the way to do it is to configure origin server as a parent. However, usually Squid sends proxy requests to the parent, but some HTTP servers can only handle normal requests. So, you need to mark the parent as the origin server so that Squid sends it a normal request. You also need to tell Squid not to query it (e.g., ICP) for any resource (because it is not really a cache!):

cache_peer <hostname-of-origin> parent 80 0 originserver no-query

• To serve as a reverse proxy, Squid must be prepared to receive normal request instead of proxy request. It must also find out the site name from the Host header to construct the URL. These are done in the http_port with the accel option. If no Host header is included in the request, the defaultsite option provides a default site to use. In an older version, Squid by default will rely on the defaultsite option (probably at that time HTTP 1.0, which didn't have the Host header, was more common). To tell it to use the Host header, you need to add the vhost option:

http_port 80 accel defaultsite=www.foo.com
http_port 80 accel defaultsite=www.foo.com vhost

• Usually your public website is on port 80, so you need to configure Squid to listen on port 80:

http_port 80

• If the clients are supposed to connect to Squid using SSL, you need to enable SSL processing and the port configure using the http_ports configuration. It is only meaningful for reverse proxy mode of operation (for normal mode, Squid usualserves as a TCP proxy).

http_ports 443 cert=<path to cert file> key=<path to key file>

• You should allow the public to access Squid:

 http_access allow all 

Transparent proxy

• Transparent proxy means that the client believes that it is accessing the HTTP server but in fact it is just talking to the proxy. One way to do it is to let the router perform DNAT so that packets to port 80 will be sent to Squid. Just like the case with reverse proxy, Squid needs to accept normal HTTP request instead of proxy request. By DNAT, the reply packets are translated back and thus the client won't notice any problem.
• To tell Squid to run in transparent mode:

http_port 80 transparent

• Why not just use the accel mode? There are some differences between them:
• In transparent mode, Squid will not perform any authentication because if both it and the real server requires authentication, the client will be confused.
• In transparent mode, if there is no Host header in the HTTP request, Squid will try to read the NAT connection state (only works if it is running on the router) to find out the original destination IP and port use those to construct the URL.
• If Squid is running on the router, just use a simplified form of DNAT:

iptables -t nat -A PREROUTING -s <client subnet> --dport 80 -j REDIRECT --to-ports 80

• If Squid is running on the another host, just use DNAT:

iptables -t nat -A PREROUTING -s <client subnet> --dport 80 -j DNAT --to <ip of squid>

Juniper (ScreenOS) concepts

Juniper (ScreenOS) concepts

Basic concepts

• Zone. Juniper controls traffic between "zones" using policies. Each interface of the Juniper firewall device belongs to a certain zone. For example, the interface to the internal network is marked as connecting to a zone named "trust", the interface to the Internet is marked as connecting to a zone named "untrust". The idea is that you can multiple interfaces connecting to the same zone (e.g., to the "trust" zone), they can be grouped together so that you don't need to configure policies for each interface. Instead, you only configure policies for traffic between the "trust" and "untrust" zones.
• Obviously, the interfaces in the same zone should have the same security protection. Therefore, traffic between different zones must be checked and traffic between different interfaces in the same zone is NOT checked (by default).
• Virtual router. There are multiple "virtual routers" in a Juniper device. Each virtual router contains its own routes so that, for example, routes for internal network are separate from other routes. The idea is to avoid exposing the routes to the outside, e.g., through RIP or OSPF. Virtual routers are just like physical routers: if you need to forward a packet from one virtual router (VR) to another, just define a route and use the latter as the gateway.
• Each zone is associated with a virtual router (also called a "routing domain"). As each interface belongs to a certain zone, it also belongs to a certain virtual router, so packets arrived at it are processed by that virtual router.
• There are two pre-defined virtual routers: trust-vr (for all routes) and untrust-vr (for routes to be exposed to the Internet).
• Policy. Stateful TCP and UDP inspection is performed by a policy so that reply packets are automatically included. That is, to allow HTTP access from untrust zone to a host in the DMZ zone, it is enough for the policy to allow the first packet (untrust => DMZ) and there is no need to explicitly allow the reply packet (DMZ => untrust).
• For NAT and VPN, policy checking is done before translation or encapsulation (initiating from internal to the Internet) and after translation or decapsulation (reply from Internet to internal), so you will use the "original" addresses in the policy. For example, to allow public access to a web server (using a private IP) in the DMZ through a public IP, the destination address in the policy should be the private IP instead of the public IP.

NAT

• Source NAT. To access the Internet with source NAT, just put the internal interface (usually in the "trust" or DMZ zone) into "NAT mode". When a packet is passing through it to the "untrust" zone, the device will perform source NAT on the packet. The reply packet is also handled automatically.
• The external interface (usually in the "untrust" zone and connecting to the Internet) should be in the "routing mode".
• If a packet is passing between two internal interfaces in NAT mode, no NAT is performed. The idea is that NAT is only used for accessing the Internet, not between different departments in the company.
• Source NAT can also be done with a policy (instead of in the incoming interface). In that case ("policy-based source NAT"), the mode of the incoming interface isn't important anymore and the NAT policy will take priority. Policy-based source NAT can perform NAT more selectively, but then you can not apply further policies to restrict the traffic (but it is not required really because if NAT is not done, the packet can't go out).
• Destination NAT. This can be done with MIP (mapping a public IP to a private IP), VIP (mapping a public IP+port to a private IP+port) or a policy.
• MIP is usually defined in the public interface (more precisely, the "hosting" interface should have be on the same subnet as that MIP). It will accept incoming packets and the reply packets automatically. In fact, it will perform source NAT when the internal host initiates traffic to the Internet.
• VIP is similar to MIP except that you can map different ports of the same public IP to different internal hosts. Also, NAT will not be performed if the internal host tries to initiate traffic to the Internet. To allow that, normal source NAT is needed. A VIP can be the same IP as that of the public interface, while this is not allowed with MIP.
• All MIP and VIP "interfaces" belong to the "global zone". They are special because they are just "proxies" for the private IPs. 

VPN

• There are two ways to implement a VPN in Juniper: policy-based (encapsulate the packet when it matches a policy) or route-based (encapsulate the packet when it matches a route and is processed by a tunnel interface).
• Route-based VPN. It is implemented using a tunnel interface on the two sides. The tunnel interface in the Juniper device is configured to be in a certain zone. It can be different from the ultimate outgoing physical interface (e.g., the tunnel interface may be in "branch-office" zone while the physical interface in "untrust" zone).
• Route-based VPN works by routing packets to the tunnel interface, which is bound to a VPN tunnel (or called the "VPN gateway"). All the VPN information such as pre-shared key,  algorithms to use and the peer IP is stored in the VPN gateway. The tunnel interface doesn't need an IP.
• Dial up peer. If the peer is using a dynamic IP, there is no way to set the peer IP in the VPN gateway object. In that case, just choose "dynamic" so that it will just wait for connections from the clients and use their IPs. To compensate the security (to authenticate user instead of device), you should specify a user or user group in the VPN gateway object and bind that user to an "IKE user" who can be authenticated with PKI or a pre-shared key.
• Proxy ID. In ISAKMP in phase 2 (establishing phase 2 SA), it is possible for the gateways to establish different tunnels (SA's) for different "entities" (e.g., different internal servers, different internal networks). Each such entity is identified by an ID. Juniper calls it the "proxy ID". Usually, it can be just a string representing the internal network such as 192.168.1.0/24. The local Juniper gateway will send its local proxy ID (specified in the VPN gateway object) to the Juniper gateway at the other side, which will try to match it against the remote proxy ID (specified in the VPN gateway object). So, make sure that they match.
• Why not merge the concept of VPN gateway and the tunnel interface into one? Two reasons:
• A VPN gateway can be referred to by a policy to set up a policy-based VPN.
• A tunnel interface can be bound to multiple VPN gateways (but why is this needed? It seems to be an "artificial" feature).
• Null route. Juniper can monitor a VPN tunnel (SA) to see if it is still working by pinging it from time to time ("VPN monitoring").  If using a route-based VPN and the tunnel is detected as down, the interface will be marked as down and then the route will be skipped. Then the traffic may be sent to the default route (the Internet)! Either don't use VPN monitoring or add a route that sends the traffic to the Null interface (which just drops the packet). The route should be similar to the route performing encapsulation but with a higher metric so that it is normally not used.
• Policy-based VPN. The source IP used in the policy is the IP assigned to the tunnel interface on the client.
• Route-based VPN is more versatile than a policy-based VPN as:
• You can apply multiple policies for a single SA. This is because filtering (policy) and encapsulation are done independently.
• You can use a routing protocol to propagate the route to the remote site throughout the headquarter internal networks. A policy is not a route and can't be propagated.
• The tunnel interface can be put into a desired zone (e.g., "branch") instead of the "untrust" zone.
• L2TP VPN. It is possible to use IPSec tunnel mode to build a VPN, but IPSec can't assign IP to the client. To do that, you should use an L2TP VPN. L2TP can be considered just like PPP (IP assignment and encapsulation) except that PPP requires a point-to-point. So, by configuring the peer IP, the L2TP protocol software can emulate a point-to-point link to PPP, then it can use the peer IP to deliver the PPP packet (PPP inside a UDP inside an IP). L2TP doesn't provide encryption, so it is used with IPSec to build VPN. L2TP, just like PPP, can use PAP and CHAP for user authentication.
• To configure for establishment of L2TP tunnel:
• Set the L2TP configuration (e.g., IP pool to use, authentication method) as global default or in the user object.
• Create an L2TP tunnel object,  specify the user group that is allowed to connect and create a policy to send the packets to be encapsulated to that tunnel. The source address can be set to the "Dial-Up VPN" address book entry (What does it mean? All the IPs connected with L2TP and IPSec?).
• L2TP over IPSec. To apply IPSec to protect the L2TP packets, in the same policy enabling the L2TP tunnel, enable the IPSec tunnel by setting the IPSec VPN gateway. Each user are both an IKE user (for IPSec authentication) and L2TP user (for L2TP authentication).