Concepts of SNMP (including v3)

SNMP

• Purpose. SNMP is a protocol for getting the status (e.g., CPU load, free memory, network load) of computing devices such as routers, switches and even servers.
• Object descriptor, managed object. The client can provide a globally unique names such as cpmCPUTotal5secRev (the average CPU load of a Cisco device for the past 5 seconds) to indicate the information that it wants, then the server should return such information. Such a textual name is called the "object descriptor". The word "object" or "managed object" refers to the concept of CPU load. The actual CPU load in the device is called the "object instance".
• Object identifier (OID). To make sure that each object descriptor is unique, actually it is defined using a list of integers such as 1.3.6.1.4.1.9.9.109.1.1.1.1.6. Each integer is like a package in Java. For example, the integers in 1.3.6.1.4.1.9 represents iso (1), org (3), dod, i.e., department of defense (6), internet (1), private (4), enterprises (1), cisco (9) respectively. This allows the Internet authority to delegate the management of the namespace hierarchically: to private enterprises and then to Cisco, which can further delegate to its various divisions or product categories. Such a list of integers is called an "object identifier". This is the ultimate identification for the managed object.
• Even though the object descriptor should be unique, it is useful to see the hierarchy. Therefore, usually the full list of object descriptors is displayed such as iso.org.dod.internet.private.enterprises.cisco...cpmCPUTotal5secRev.
• Why use integers instead of symbolic names? Probably to allow the network devices (with little RAM or CPU power) implementing SNMP to save space in processing. Symbolic names such as object descriptor can be used by human in commands, but in the protocol's operation it is done using object identifier.
• In principle, the object a.b.c.d and the object a.b.c.d.e on a device have NO containment relationship. That is, they are NOT like a Java object containing a child object. In fact, the value of each object in SNMP is basically a simple value (scalar) such as an integer or a string. The only relationship between them is their names.
• Identifying an instance. Now comes the most complicated concept in SNMP. Consider the concept of the number of bytes that have been received by a network interface on a router. This concept is an object. As a router should have multiple interfaces, there must be multiple instances of that object. Then, how can an SNMP client indicate to the SNMP server which instance it is interested in? The solution is more or less a kludge: to allow the instance of, say, a.b.c.d, to represent a table (a compound, structural value), which contains rows (also compound, structural value) represented by a.b.c.d.e. Each row contains child object instances (with scalar values only). Each child object is called a "columnar object". For example, each row may contain three object instances: a.b.c.d.e.f, a.b.c.d.e.g, and a.b.c.d.e.idx. If you'd like to refer to the a.b.c.d.e.f instance in a particular row, you will write a.b.c.d.e.f.<index>. The meaning of the index is defined by a.b.c.d.e (the row). For example, it may be defined as finding the row in the table which contains a columnar object a.b.c.d.e.idx whose value equals to <index>, then return the columnar object a.b.c.d.e.f as the result.
• Note that this is the only situation where the value of an object can be a structure and that there is object containment relationship in SNMP.
• What is confusing is that a.b.c.d.e.f is used both as an object identifier and the lookup key to find the child instance in the row. Unlike other object identifiers, the identifier now represents an object containment relationship so it must have a.b.c.d.e as the prefix, otherwise the server won't know which table to look into and what is the definition for the index.
• The complete identifier a.b.c.d.e.f.<index> is called an instance identifier.
• Here is a concrete example: Consider iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets.1.  The definition of iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry says that to find the row in the table, it should search for a row which contains a child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex with the value of 1 (the index specified), then it will return the value of child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets in the row. Of course, for this to work, the  iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex child object in each row must have been assigned with sequential values 1, 2, ..., etc. (which is indeed the case). 
• Finally, a simple case is that there is no table at all. For example, to find the up time of the device, use the object identifier iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime and append a .0 as the index, so the instance identifier is iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0. BTW, "hr" stands for "host resources".
• MIB (management information base). A MIB is just a a collection of managed objects (not instances). There are standard MIBs so that device manufacturers can implement and users can find the right object identifiers to use. There are also proprietary MIBs such as those designed by Cisco to provide information only available on its devices.
• Finding the supported OIDs. How can you find out the MIBs or the object identifiers supported by a device? It is easier to just "walk the MIB tree": to show all the instances in the tree or in a subtree. On Linux, this is done as below. You specify the IP or hostname of the server and optionally specify a node so that only that subtree is displayed (the object identifier starts with a dot, otherwise it will be assumed it is relative to iso.org.dod.internet.mgmt.mib-2):

# snmpwalk <some args> localhost
# snmpwalk <some args> localhost .iso.org.dod.internet.mgmt.mib-2.system
# snmpwalk <some args> localhost system

• Getting an instance. Just specify the instance identifier:

# snmpget <some args> localhost .iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0
# snmpget <some args> localhost host.hrSystem.hrSystemUptime.0

• SNMP enttiy, engine and applications. SNMP is a peer to peer protocol. There is no concept of a server and a client (these terms are used here for simplicity). Instead, both peers have the same core capabilities such as sending or receiving SNMP messages, performing security processing (see below), integrating v1, v2c and v3 processing (dispatching) and etc. This part is called the SNMP engine. On top of the engine, there are different "applications": one application may only respond to requests for object instances (called SNMP agent in v1 and v2), another application may probe others (called SNMP manager in v1 and v2), yet another may forward SNMP messages (SNMP proxy). The whole server or client is called the "SNMP entity".
• Context. On some devices there are multiple copies of a complete MIB subtree. For example, a physical router may support the concept of virtual routers. Each such virtual router will have a complete MIB subtree of instances. In that case, each virtual router may be indicated as a "context" in the SNMP server. A context is identified by name. For example, a virtual router could be identified as the "vr1" context. There is a default context with empty string ("") as its name. When a client sends a query, it can specify a context name. If not, it will query the default context.
• Notification (trap). An SNMP server may actively send a notification to the client when some condition occurs. This is very much like a response without a request. Otherwise, everything is similar. The condition (e..g., the changes of the value of an instance or its going out of a range), the destination, the credentials used (see the security section below) and etc. are configured on the server.
• Transport binding. Typically SNMP runs on UDP port 161.

SNMP security

• SNMP v1 and v2c security. In SNMP v1 and v2c (v2 was not widely adopted), there is little security. The only security is the "community string". That is, the server is configured to be in a community identify by a string such as "foo", "public" (commonly used and the default for many devices to mean no protection) or "private". If the client can quote the community strnig, then it is allowed access. As the community string is included as plain text in SNMP packets, it practically provides no security. Therefore, in v1 and v2c, to access an SNMP server, you will do something like:

# snmpwalk -v 2c -c public localhost
# snmpget -v 2c -c public localhost <INSTANCE ID>

• SNMP v3 security. In SNMP v3, there is user-based security. That is, the client may be required to authenticate the messages to the server as originating from a user using a password (authentication password). In addition, the client may be furthered required to encrypt the messages using another password (privacy password). This security requirement is called the "security level" (no authentication needed, authentication but no privacy, authentication with privacy). Therefore, in v3, you will access the server like:

# snmpwalk -v 3 -l noAuthNoPriv localhost
# snmpwalk -v 3 -l authNoPriv -u kent -A "my auth passwd" localhost
# snmpwalk -v 3 -l authPriv -u kent -A "my auth passwd" -X "my priv passwd" localhost 

• Client configuration file. To save typing all those every time, you can store these parameters into the snmp.conf file as defaults.
• Security limitation. It is a bad idea to specify the password on the command line as it can be revealed by local users using "ps". Storing it into the configuration file is better. However, the file only allows a single authentication password and a single privacy password, not enough to handle the case of using different passwords for different servers.
• Security name. A security name is just a user name. No more, no less. That's the term used in the RFC (maybe in the future it could be something else?)
• Algorithm. Further, there are different algorithms for authentication (HMAC using MD5 or SHA) and for privacy (encryption using DES or AES). So, you need to specify the algorithms to use:

# snmpwalk -v 3 -l noAuthNoPriv localhost
# snmpwalk -v 3 -l authNoPriv -u kent -a MD5 -A "my auth passwd" localhost
# snmpwalk -v 3 -l authPriv -u kent -a MD5 -A "my auth passwd" -x DES -X "my priv passwd" localhost 

• Ensure the algorithms match. As SNMP uses UDP and each query and response may use just a single UDP packet, there is no negotiation of algorithm at "connection phase" at all. In fact, presumably for simplicity in implementation, the algorithms used are not even indicated in the message, so the client must use the agreed-on algorithms as configured in the user account on the server, otherwise the server will simply fail to authenticate or decrypt the message.

• Localized keys. The authentication password and privacy password of a user account are not used directly. The idea is, most likely you will use the same password for all the user account on all devices on site. If it is directly used, then a hacker controlling one device will be able to find the password and use it to access all the other devices. Therefore, when creating a user account, you specify the password, but the Linux SNMP server will combine it with a unique ID (called the "engine ID") generated for the device (such as the MAC or IP and/or a random number generated and stored on installation), hash it and use the result as the password (the "localized key"). This way, even if the hacker can find this localized key, he will still be unable to find the original password.
• But how can a client generate the same key? It has to retrieve the engine ID first and then perform the same hashing. This is supported by the SNMP protocol.
• User accounts creation. Due to the need to generate localized keys, the way to create user accounts on Linux is quite weird. You stop the server, specify the user account's name and password in a file, then start the server. It will read the password, convert it to a localized key and overwrite the file. This file is /var/lib/snmp/snmpd.conf on Linux:

createUser kent MD5 "my auth password" DES "my privacy password"
createUser paul SHA "my auth password, no encryption needed"

• Access control. The access control can specify the user account, the lowest security level required, which part of the MIB tree is accessed (may use an OID to identify a subtree), the type of access (read or write), in order to grant the access. Here are some example settings on Linux (although human user names are used, but in practice they should be representing devices):

rouser john noauth .iso.org.dod.internet.mgmt.mib-2.system
rouser kent priv .iso.org.dod.internet.mgmt.mib-2.system
rouser kent auth .iso.org.dod.internet.mgmt.mib-2
rwuser paul priv 

• View. How to specify several subtrees in an access control rule? You can define a view. A view has a name and is defined as including some subtrees and excluding some subtree. Then you can refer to it by name in access control:

view myview included .iso.org.dod.internet.mgmt.mib-2.system
view myview included .iso.org.dod.internet.mgmt.mib-2.host
view myview excluded .iso.org.dod.internet.mgmt.mib-2.host.hrStorage

rwuser paul priv -V myview 

• Access control for v1 and v2c. For v1 and v2c, access control can specify the community string, the IP range of the client (the "source"), the subtree (OID) or the view:

rocommunity public 192.168.1.0/24 .iso.org.dod.internet.mgmt.mib-2.system
rwcommunity private localhost -V myview

• Most flexible access control model. The above access control model is called the "traditional model". The new, most flexible access control model is called the "view-based access control model (VACM)", even though the former can also use views. It may be more suitable to called it group-based access control as it uses user groups in the rules (NOT the precise syntax yet!):

group g1 kent
group g1 paul
#access <group> <context> <min sec level> <exact context?> <view for read> <view for write> <view for notify>
access g1 "" auth exact myview1 myview2 myview3

• Mapping community string to user name. When using the VACM, instead of granting access to community strings, you need to merge v1 and v2c into the user-based access control processing. To do that, a community string along with the source can be mapped to a user name (the user name mapped to do NOT have to be existing):

com2sec user1 192.168.1.0/24 public
# "default" source means any
com2sec user2 default private

• Security model. Even though the different types of identity in the different SNMP versions are represented uniformly as a user name, their trustworthiness is still significantly different. So, in specifying group memberships and access control rules, you are required to specify the "security model" (v1, v2c or the user security model as in v3) and that's the correct syntax:

group g1 usm kent
group g1 usm paul
group g2 v2c user1
group g2 v1 user1
group g2 v1 user2
access g1 "" usm auth exact myview1 myview2 myview3
access g2 "" any noauth exact myview4 myview5 myview6

Basic concepts of OpenLDAP

LDAP

• LDAP is just a type of database specialized for look ups. Reads are fast while writes are supported but relatively slow. That's why LDAP is commonly used to store centralized user accounts.
• In LDAP, data is stored nodes (much like a record in a relational DB). Each node has some named attributes. For example, a node representing a user may have the following attributes (cn stands for "common name", while sn stands for "surname"):

cn: Kent Tong
sn: Tong
givenName: Kent
email: kent@ttdev.com

• Just like a record in a DB or an object in Java, the attributes that a node can have is fixed. This is specified by an attribute named objectClass. For example, there is an objectClass named "person" that says that it can have cn, sn and email, then a node belonging that this "person" class can have such attributes:

cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• It is possible for a node to belong to multiple object classes. For example, there may be an objectClass named "securityObject" that says that it can have a password attribute, then if you need to put a password into a node, you can do it like:

cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person
objectClass: securityObject
password: abc123

• How to use a node to present a department or a company? This is done with the "organizationalUnit" objectClass ("ou" stands for "organizational unit"):

ou: Foo Ltd.
objectClass: organizationalUnit

ou: Finance dept
objectClass: organizationalUnit

• If the person Kent Tong is in that Financial department, how to represent that relationship? Unlike a relative DB where such relationship is represented a foreign keys, in LDAP one node can be put under the other as a child node. For example, let's put the person node as a child node of the financial dept node which is put as a child node of the company node:

dn: ou=Foo Ltd.
ou: Foo Ltd.
objectClass: organizationalUnit

dn: ou=Finance dept,ou=Foo Ltd.
ou: Finance dept
objectClass: organizationalUnit

dn: cn=Kent Tong,ou=Finance dept,ou=Foo Ltd.
cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• This parent-child relation is represented by an attribute called "dn" (distinguished name, just like an full path to a file). That is, you can determine the dn of the parent node from the dn of the node.
• Usually you will add the cn to the dn of the parent to get the dn of the node itself. However, you don't have to use cn. You are free to any attribute to serve that purpose (called "relative dn") as long as it is unique among the siblings. For example, cn is good if there are no two people with the same cn values  the dept. This is so that given a dn, it is very fast to locate the node.
• When you create the top node of the directory, you need to specify its dn. Usually, you will use the DNS domain name of your company (let's say it is foo.com):

dn: dc=foo,dc=com
ou: Foo Ltd.
objectClass: organizationalUnit

dn: ou=Finance dept,dc=foo,dc=com
ou: Finance dept
objectClass: organizationalUnit

dn: cn=Kent Tong,ou=Finance dept,dc=foo,dc=com
cn: Kent Tong
sn: Tong
email: kent@ttdev.com
objectClass: person

• in which dc means "domain component", just a label in a domain name.
• Some object classes such as person or organizationalUnit are designed to be used as the "main" object class for the nodes, while some other object classes such as securityObject (officially, it is called "simpleSecurityObject") are designed to be "auxiliary" and to be added to the nodes. The former is called a "structural object class" while the latter an "auxiliary object class".
• For each node, it must have exactly one structural object class. It can have zero or multiple auxiliary object classes.

Schema

• The attributes and object classes are defined instead of hard-coded. For OpenLDAP on Debian, you can find the schemas in /etc/ldap/schema. For example, you can find the following object class in core.schema:

objectclass ( 2.5.6.6 NAME 'person'
        DESC 'RFC2256: a person'
        SUP top STRUCTURAL
        MUST ( sn $ cn )
        MAY ( userPassword $ telephoneNumber $ seeAlso $ description ) )

• It says that it must have the "sn" and "cn" attributes and may have a "userPassword" attribute and etc. It says that its super class is "top" which is a built-in object class representing the topmost level object class. It also says that the "person" object class is a structural class.
• You can find the most commonly used object classes and attributes here.

Authenticating with LDAP

• A straightforward way is to retrieve the node according to the username, get the password and compare it to the password the user entered.
• However, this allows retrieving sensitive information (password) of any user over the network by services. Therefore, there is another way: you send dn and the password to the LDAP service and let it perform the checking. This is called "binding" to that dn. Essentially you are logging in as a particular node.
• Both methods above will transfer the password in the clear. So, you should probably use TLS when accessing the LDAP service (using ldaps://)
• The above methods are most commonly used by other services such as mail, Samba or Apache for checking the passwords of the users.
• When performing administration with OpenLDAP, you may use the "bind" approach above to log in as the node representing the administrator. This is called "simple authentication":

# ldapsearch -x -W -D cn=admin,dc=foo,dc=com ...

• However, sometimes it is more convenient to authenticate not as a node in the directory but just as an administrator. For example, if your site is using Kerberos, you can perform single sign-on with OpenLDAP. This is done with its SASL support. SASL in turn supports many mechanisms such as PLAIN (plain text), GSSAPI (Kerberos), EXTERNAL (using the transport such as the client cert with TLS). When SASL to authenticate, OpenLDAP will make up a dn to represent your identity. For example, for user Kent in the Kerberos domain foo.com, the made-up dn will be uid=kent,cn=foo.com,cn=gssapi,cn=auth. This is needed for granting permissions.

Setting up and running OpenLDAP on Debian

• When setting up OpenLDAP on Debian, you need to reconfigure the package so that it creates the initial directory for you as well as the node representing the administrator:

# dpkg-reconfigure -p low openldap

• Then, to check if it is working, you can use SASL EXTERNAL mechanism and use Unix domain socket (ldapi:///) as root to connect to it:

# ldapsearch -b cn=config -H ldapi:/// -Y external

Query

• Just like a relation DB, you can perform queries to retrieve the nodes and their attributes. For example, you can try to find all nodes with cn=Kent Tong (the "filter") under the node whose dn is dc=foo,dc=com (the "base" of the search):

# ldapsearch -b dc=foo,dc=com -H ldapi:/// -Y external "cn=Kent Tong"

• You may try to find all nodes whose objectClass is "person" and whose cn contains the word "Kent":

# ldapsearch -b dc=foo,dc=com ... "(&(objectClass=person)(cn=*Kent*))"

• You may tell it to retrieve only a certain attributes:

# ldapsearch -b dc=foo,dc=com ... "(&(objectClass=person)(cn=*Kent*))" dn sn

Updating the data

• To add a new node, make sure the parent already exists, then specify the dn and other attributes in a text file (called an LDIF file) and mark it as an "add" request:

dn: cn=Paul Chan,ou=Finance dept,dc=foo,dc=com
changetype: add
cn: Paul Chan
sn: Chan
email: paul@ttdev.com
objectClass: person

• Then feed the LDIF file to ldapmodify:

# ldapmodify < myfile.ldif

• To modify (add, change or delete) the attributes of an existing node, specify the node first and then list each attribute change and separate them by a line containing a single hyphen. Finally, end the node with an empty line:

dn: cn=Kent Tong,ou=Finance dept,dc=foo,dc=com
changetype: modify
add: givenName
givenName: Kent
-
replace: email
email: kent@foo.com
-
delete: phoneNumber

• To delete a node:

dn: cn=Paul Chan,ou=Finance dept,dc=foo,dc=com
changetype: delete

Basic IPv6 concepts

• Basic concept. IPv6 is a completely separate protocol from IPv4! It means that the configuration and operation of IPv6 have nothing to do with those of IPv4 at all. They will run independently just like IPv4 and IPX or AppleTalk.
• Format of IPv6 address
• An IPv6 address contains 128 bits and is written in 8 groups of 16 bits like abcd:0000:1111:2222:1357:2468:5555:6789. Each group contains 4 hexadecimal digits (so, 16 bits).
• Many IPv6 addresses contain many consecutive groups of 0000 such as abcd:0000:0000:0000:0000:0000:0000:1234. To save typing, such consecutive groups of zero can be written as ::, so the address above can be written as abcd::1234. Or working from the reverse direction, because there are eight groups in an IPv6 address but only two groups were given, so :: is representing the six groups that are all zero.
• Different types of IPv6 addresses
• Just like IPv4, there are global IPv6 addresses that can be obtained from ISP. They begin with 2???: or 3???:.
• Just like private IPv4 addresses like 192.168.?.?, in IPv6 we use fec?:, fed?:, fee?:, fef?:. These are called site local addresses as they can only be used privately on a site. However, IPv6 provides a better alternative called "unique local address" in the range of fd??. To use it you will first generate a random number and append it to fd to form a random prefix, so that it will not collide (in practice) with any other range used by other private networks.
• Just like 127.0.0.1 in IPv4, there is ::1 (i.e., 0000:0000:0000:0000:0000:0000:0000:0001) in IPv6.
• Just like 169.254.?.? in IPv4, there are fe8?:, fe9?:, fea?:, feb?: address ranges in IPv6. They are called link local addresses as they can only be used privately on a link (a broadcast domain such as a switch or a set of interconnected switches). They are mainly used for network discovery (see below).
• Netmask and subnetting. The host bits in IPv6 take up at least the lower 64 bits (and maybe more), so the netmask is always <= 64 bits. The only exception is the loopback address ::1 which has a netmask of /128. For a LAN, usually just use /64 as the netmask. The ISP may assign a /64, /56 or /48 address block to a site.
• Assignment.
• Stateless auto configuration. In IPv6 it is fine to let a host configure its own address by getting the prefix (the network ID) from the router and then using its own MAC address as the host ID (with some encoding). Therefore, you only need to configure the router with static IPv6 address (so it knows the prefix). How can the host find the router? The host can use multicast to discover the router (part of the IPv6 Network Discovery Protocol) so you don't need to manually configure the default gateway of the host. The idea is that it can detect the running router automatically and perform fail-over as needed.
• DHCPv6. It is very much like DHCP for IPv4. Use it if you need centralized control.
• DNS.
• Domain name to address. You use an AAAA record to map a domain name to an IPv6 address. You can use either IPv4 or IPv6 to query the server and it can return A or AAAA records in either case (depending what record type you are searching for).
• NS, MX, etc. You can just put a domain name there and the name server will try to find its A and AAAA records and return them in the additional section.
• PTR. The reverse domain name is in the form of a.b.c.d.?.?.?.?.?.?.?.IPv6.arpa where a, b, c and d represent a hexadecimal digit (the address is like ??:??:??:...:dc:ba).
• Binding with application protocols. Most higher level protocols already support IPv4 and IPv6 simultaneously (e.g., ping/ping6, OpenSSH, Chrome, Firefox, Apache). To use an IPv6 address directly, you may need to put it in square brackets, e.g., http://[2001::1] in a browser otherwise the colon may look like starting a port. If you use a domain name and already has an AAAA record, the OS will return both the IPv4 and IPv6 to the application. According to the standard, actually global IPv6 are listed before  (preferred) public IPv4.
• In Linux, you can configure a static IPv6 address just like IPv4 in the interfaces file:

iface eth0 inet6 static
address 2001::1
netmask 64

• Checking the IPv6 addresses can be done with:

# ip addr show

Concepts of Squid

Basic proxy operation

• The normal mode of operation is for squid to serve as a proxy for internal users accessing the Internet. The user's browser is configured to use Squid as the proxy. Then, when the user tries to access a URL, the browser will send the request to Squid instead of the server in the URL (the "origin server").
• The request sent to Squid (a "proxy request") is slightly different than normal HTTP request as the former contains the full URL in the, say, GET command, while the latter only contains a path. This allows the proxy to contact the origin server as needed.
• When squid receives a request (proxy request), by default it will try to find the resource (the web page) in the following order:

1. Its cache.
2. The siblings and parents. Siblings should be deployed on the same site for load-balancing. Parents should be deployed in a larger ISP or regional headquarter. The idea is that the on-site proxies will in time contain mostly resources commonly accessed by the users on that site, while the ISP proxy will contain mostly resources commonly needed by the users in that region.
1. Squid will query all its siblings and parents (all are called "peers"). If there is a HIT response, it will send the request to the first such peer. If there is none (only MISS response or no response), the first parent returning MISS will get the request. It is supposed to get the resource from the origin server or its own parent if it is not in its cache. If there is no reply from the parents at all (e.g., all are busy or down), Squid will use the parent that is marked as the default.
2. The chain of request handling should normally stop here, unless, say, all the parents are down.
3. The origin server. The server hosting the resource (identified in the full URL in the GET command).

• After getting the resource, it will save it into its cache to speed up future accesses. The path to the cache and maximum disk space can be configured.
• Squid uses the ICP protocol to query its siblings and parents (the peers) to check if they have the resource in their caches. If yes, it will use HTTP to retrieve it. ICP is usually based on UDP and uses UDP port 3130. Squid usually listens for HTTP requests on TCP port 3128.
• A sibling or parent (peer) is defined like this:

cache_peer <hostname1> parent 3128 3130
cache_peer <hostname2> sibling 3128 3130

• The <hostname> above is used to resolve the IP.

Access security

• Usually you should only allow your internal users to use your Squid server (by restricting the client IPs or user authentication). To do that, you can  configure ACLs so that only authorized internal users can connect to it (the http_access configuration).

http_access allow myACL1

• An ACL is a named Boolean expression that can refer to various properties of the request (e.g., URL, domain name in the URL, HTTP method, source IP, authenticated user name):

acl myACL1 url_regex ^http://...
acl myACL2 dstdomain .foo.com

• When an ACL is evaluated and it refers to authenticated user name and etc. but that information is not yet in the request, Squid will perform proxy authentication with the client first.
• You can specify multiple ACLs to the same http_access and they will AND'ed together:

http_access allow myACL1 myACL2

Controlling routing of request

• You can control the routing between peers by applying ACLs to the peers. When Squid is selecting the peers, it will filter out those whose ACLs aren't satisfied. For example, to force Squid to use a particular parent for accesses to www.foo.com:

acl myACL1 dstdomain www.foo.com
cache_peer mycache1.local parent 3128 3130
cache_peer mycache2.local parent 3128 3130
cache_peer_access mycache1.local allow myACL1
cache_peer_access mycache2.local deny myACL1

SSL processing

• Usually the client will send a CONNECT command to Squid to establish a TCP proxy connection to the origin server. Squid will simply forward the raw TCP data (encrypted HTTP commands) back and forth.
• In newer versions of Squid, it can fake the origin server's certificate on the fly (so it needs the CA's cert and private key) or just a cert that doesn't really match the origin server. In either case, such cert faking and interception (called "ssl bumping") is configured in the http_port configuration.

Reverse proxy

• If you need it to serve as a reverse proxy for Internet users to access your DMZ web servers, most likely you won't need to use http_access to restrict the access. On the other hand, you can't just not configure siblings and parent in the hope that Squid will contact the origin server (using the server URL in the request) because that URL will resolve to the IP of Squid itself! So, the way to do it is to configure origin server as a parent. However, usually Squid sends proxy requests to the parent, but some HTTP servers can only handle normal requests. So, you need to mark the parent as the origin server so that Squid sends it a normal request. You also need to tell Squid not to query it (e.g., ICP) for any resource (because it is not really a cache!):

cache_peer <hostname-of-origin> parent 80 0 originserver no-query

• To serve as a reverse proxy, Squid must be prepared to receive normal request instead of proxy request. It must also find out the site name from the Host header to construct the URL. These are done in the http_port with the accel option. If no Host header is included in the request, the defaultsite option provides a default site to use. In an older version, Squid by default will rely on the defaultsite option (probably at that time HTTP 1.0, which didn't have the Host header, was more common). To tell it to use the Host header, you need to add the vhost option:

http_port 80 accel defaultsite=www.foo.com
http_port 80 accel defaultsite=www.foo.com vhost

• Usually your public website is on port 80, so you need to configure Squid to listen on port 80:

http_port 80

• If the clients are supposed to connect to Squid using SSL, you need to enable SSL processing and the port configure using the http_ports configuration. It is only meaningful for reverse proxy mode of operation (for normal mode, Squid usualserves as a TCP proxy).

http_ports 443 cert=<path to cert file> key=<path to key file>

• You should allow the public to access Squid:

 http_access allow all 

Transparent proxy

• Transparent proxy means that the client believes that it is accessing the HTTP server but in fact it is just talking to the proxy. One way to do it is to let the router perform DNAT so that packets to port 80 will be sent to Squid. Just like the case with reverse proxy, Squid needs to accept normal HTTP request instead of proxy request. By DNAT, the reply packets are translated back and thus the client won't notice any problem.
• To tell Squid to run in transparent mode:

http_port 80 transparent

• Why not just use the accel mode? There are some differences between them:
• In transparent mode, Squid will not perform any authentication because if both it and the real server requires authentication, the client will be confused.
• In transparent mode, if there is no Host header in the HTTP request, Squid will try to read the NAT connection state (only works if it is running on the router) to find out the original destination IP and port use those to construct the URL.
• If Squid is running on the router, just use a simplified form of DNAT:

iptables -t nat -A PREROUTING -s <client subnet> --dport 80 -j REDIRECT --to-ports 80

• If Squid is running on the another host, just use DNAT:

iptables -t nat -A PREROUTING -s <client subnet> --dport 80 -j DNAT --to <ip of squid>

Juniper (ScreenOS) concepts

Juniper (ScreenOS) concepts

Basic concepts

• Zone. Juniper controls traffic between "zones" using policies. Each interface of the Juniper firewall device belongs to a certain zone. For example, the interface to the internal network is marked as connecting to a zone named "trust", the interface to the Internet is marked as connecting to a zone named "untrust". The idea is that you can multiple interfaces connecting to the same zone (e.g., to the "trust" zone), they can be grouped together so that you don't need to configure policies for each interface. Instead, you only configure policies for traffic between the "trust" and "untrust" zones.
• Obviously, the interfaces in the same zone should have the same security protection. Therefore, traffic between different zones must be checked and traffic between different interfaces in the same zone is NOT checked (by default).
• Virtual router. There are multiple "virtual routers" in a Juniper device. Each virtual router contains its own routes so that, for example, routes for internal network are separate from other routes. The idea is to avoid exposing the routes to the outside, e.g., through RIP or OSPF. Virtual routers are just like physical routers: if you need to forward a packet from one virtual router (VR) to another, just define a route and use the latter as the gateway.
• Each zone is associated with a virtual router (also called a "routing domain"). As each interface belongs to a certain zone, it also belongs to a certain virtual router, so packets arrived at it are processed by that virtual router.
• There are two pre-defined virtual routers: trust-vr (for all routes) and untrust-vr (for routes to be exposed to the Internet).
• Policy. Stateful TCP and UDP inspection is performed by a policy so that reply packets are automatically included. That is, to allow HTTP access from untrust zone to a host in the DMZ zone, it is enough for the policy to allow the first packet (untrust => DMZ) and there is no need to explicitly allow the reply packet (DMZ => untrust).
• For NAT and VPN, policy checking is done before translation or encapsulation (initiating from internal to the Internet) and after translation or decapsulation (reply from Internet to internal), so you will use the "original" addresses in the policy. For example, to allow public access to a web server (using a private IP) in the DMZ through a public IP, the destination address in the policy should be the private IP instead of the public IP.

NAT

• Source NAT. To access the Internet with source NAT, just put the internal interface (usually in the "trust" or DMZ zone) into "NAT mode". When a packet is passing through it to the "untrust" zone, the device will perform source NAT on the packet. The reply packet is also handled automatically.
• The external interface (usually in the "untrust" zone and connecting to the Internet) should be in the "routing mode".
• If a packet is passing between two internal interfaces in NAT mode, no NAT is performed. The idea is that NAT is only used for accessing the Internet, not between different departments in the company.
• Source NAT can also be done with a policy (instead of in the incoming interface). In that case ("policy-based source NAT"), the mode of the incoming interface isn't important anymore and the NAT policy will take priority. Policy-based source NAT can perform NAT more selectively, but then you can not apply further policies to restrict the traffic (but it is not required really because if NAT is not done, the packet can't go out).
• Destination NAT. This can be done with MIP (mapping a public IP to a private IP), VIP (mapping a public IP+port to a private IP+port) or a policy.
• MIP is usually defined in the public interface (more precisely, the "hosting" interface should have be on the same subnet as that MIP). It will accept incoming packets and the reply packets automatically. In fact, it will perform source NAT when the internal host initiates traffic to the Internet.
• VIP is similar to MIP except that you can map different ports of the same public IP to different internal hosts. Also, NAT will not be performed if the internal host tries to initiate traffic to the Internet. To allow that, normal source NAT is needed. A VIP can be the same IP as that of the public interface, while this is not allowed with MIP.
• All MIP and VIP "interfaces" belong to the "global zone". They are special because they are just "proxies" for the private IPs. 

VPN

• There are two ways to implement a VPN in Juniper: policy-based (encapsulate the packet when it matches a policy) or route-based (encapsulate the packet when it matches a route and is processed by a tunnel interface).
• Route-based VPN. It is implemented using a tunnel interface on the two sides. The tunnel interface in the Juniper device is configured to be in a certain zone. It can be different from the ultimate outgoing physical interface (e.g., the tunnel interface may be in "branch-office" zone while the physical interface in "untrust" zone).
• Route-based VPN works by routing packets to the tunnel interface, which is bound to a VPN tunnel (or called the "VPN gateway"). All the VPN information such as pre-shared key,  algorithms to use and the peer IP is stored in the VPN gateway. The tunnel interface doesn't need an IP.
• Dial up peer. If the peer is using a dynamic IP, there is no way to set the peer IP in the VPN gateway object. In that case, just choose "dynamic" so that it will just wait for connections from the clients and use their IPs. To compensate the security (to authenticate user instead of device), you should specify a user or user group in the VPN gateway object and bind that user to an "IKE user" who can be authenticated with PKI or a pre-shared key.
• Proxy ID. In ISAKMP in phase 2 (establishing phase 2 SA), it is possible for the gateways to establish different tunnels (SA's) for different "entities" (e.g., different internal servers, different internal networks). Each such entity is identified by an ID. Juniper calls it the "proxy ID". Usually, it can be just a string representing the internal network such as 192.168.1.0/24. The local Juniper gateway will send its local proxy ID (specified in the VPN gateway object) to the Juniper gateway at the other side, which will try to match it against the remote proxy ID (specified in the VPN gateway object). So, make sure that they match.
• Why not merge the concept of VPN gateway and the tunnel interface into one? Two reasons:
• A VPN gateway can be referred to by a policy to set up a policy-based VPN.
• A tunnel interface can be bound to multiple VPN gateways (but why is this needed? It seems to be an "artificial" feature).
• Null route. Juniper can monitor a VPN tunnel (SA) to see if it is still working by pinging it from time to time ("VPN monitoring").  If using a route-based VPN and the tunnel is detected as down, the interface will be marked as down and then the route will be skipped. Then the traffic may be sent to the default route (the Internet)! Either don't use VPN monitoring or add a route that sends the traffic to the Null interface (which just drops the packet). The route should be similar to the route performing encapsulation but with a higher metric so that it is normally not used.
• Policy-based VPN. The source IP used in the policy is the IP assigned to the tunnel interface on the client.
• Route-based VPN is more versatile than a policy-based VPN as:
• You can apply multiple policies for a single SA. This is because filtering (policy) and encapsulation are done independently.
• You can use a routing protocol to propagate the route to the remote site throughout the headquarter internal networks. A policy is not a route and can't be propagated.
• The tunnel interface can be put into a desired zone (e.g., "branch") instead of the "untrust" zone.
• L2TP VPN. It is possible to use IPSec tunnel mode to build a VPN, but IPSec can't assign IP to the client. To do that, you should use an L2TP VPN. L2TP can be considered just like PPP (IP assignment and encapsulation) except that PPP requires a point-to-point. So, by configuring the peer IP, the L2TP protocol software can emulate a point-to-point link to PPP, then it can use the peer IP to deliver the PPP packet (PPP inside a UDP inside an IP). L2TP doesn't provide encryption, so it is used with IPSec to build VPN. L2TP, just like PPP, can use PAP and CHAP for user authentication.
• To configure for establishment of L2TP tunnel:
• Set the L2TP configuration (e.g., IP pool to use, authentication method) as global default or in the user object.
• Create an L2TP tunnel object,  specify the user group that is allowed to connect and create a policy to send the packets to be encapsulated to that tunnel. The source address can be set to the "Dial-Up VPN" address book entry (What does it mean? All the IPs connected with L2TP and IPSec?).
• L2TP over IPSec. To apply IPSec to protect the L2TP packets, in the same policy enabling the L2TP tunnel, enable the IPSec tunnel by setting the IPSec VPN gateway. Each user are both an IKE user (for IPSec authentication) and L2TP user (for L2TP authentication).