Study notes on Cobit: Managing projects

Scope, cost and schedule

Suppose that you're the software development manager of a publishing company. To create a new market, the company is considering developing a self-publishing platform for authors to publish their works (initiating a project).To help the CEO to decide to go ahead or not, you have gathered the rough requirements (deliverables) and estimated a rough cost (e.g., MOP1,000,000) and schedule (1 year). So, the CEO decided to go ahead and you have been assigned as the project manager for the project. Your mission is to ensure that the deliverables (scope) are delivered within the allocated cost and schedule (the three most important constraints for a project). This is called project management.

Good planning

Is this mission possible? You must make sure that for the given scope, the cost and schedule are reasonable. At the initiation phase, your estimate may be quite rough because there were no detailed requirements yet (so, you should add a large buffer to it, whose size depends on the level of details available). Now, you need to identify all the requirements in details and get a better estimate on the cost and schedule.
How? You could adopt a divide and conquer approach. First, you can break down the project into:

1. A short upfront analysis to identify the features (which you will use to estimate the cost & duration): 3 days (according to previous experience)


2. Development: ???


3. System testing: ???

For each part, you can further divide it. For example, for the development part, you can break it down into the deliverables (features):

1. Analysis


2. Development
   
   
   
   1. Author management
      
      
      
      1. Register account: 2 days (given the previous experience of the same development team)
      
      
      2. Edit profile: 1 day
      
      
      3. ...
      
      
      
      
   
   
   2. Publication management
      
      
      
      1. Create a publication: ...
      
      
      2. Push a publication to Amazon: ...
      
      
      3. ...
      
      
      
      
   
   
   
   


3. System testing: ??? (you can estimate it by the number of features, the number of tests, etc.)

You can the cost of each small item similarly. The total cost is just the sum of all item costs. For the schedule, basically you can assume that the analysis, development and system testing will be in sequence, so you can determine their individual schedules (not really 100% true, as system testing could be done before everything is implemented). For the features in development, basically they can be implemented in any order you want. As not all programmers will be working on the same feature, they won't be done in sequence, but it is fare to plan what features will be done in a particular month or week.
Note that with this approach you can plan not only the cost and the schedule, but also other resources needed (e.g., programmers, testers, business analyst, 3rd party software components and tools, hardware) and when.

Uncertainties, changes, monitoring and control

After creating the project plan above, can you just assign people to the tasks and come back again in one year to receive the working system? No. There are way too many uncertainties and changes in a typical project that will throw your project off the track. For example:

• What if the progress of the development team is slower than planned? (planning risk)


• What if a programmer quits in the middle of the project? (human risk)


• What if the framework adopted for the project doesn't work as well as expected? (technology risk)


• What if the new programmer is not getting along with another programmer? (human risk)


• What if the testers aren't working well with the programmers? (human risk)


• What if the testing hardware is delayed by the supplier? (supplier risk)


• What if the CEO wants to add some new features? (scope change)


• What if the CEO wants to release the system two months earlier? (schedule change)

Therefore, you (the project manager) must monitor for any potential and actual problems and any deviations from the plan and take actions to correct the problems or deal with changes (control the project). Therefore, in accordance with the project management framework or methodology adopted by your company, you should have phases in the project as inspection and approval points. The phases can be time-based such as monthly or weekly meeting (as in XP or SCRUM) or based on deliverables (e.g., when the author management sub-system is implemented).

Quality as the 4th major constraint

Suppose that the progress of the development team is slower than planned or the CEO wants to add some new features, what would you do? Unfortunately, instead of really fixing the problems by speeding up the team or removing unnecessary features, many project managers will choose to cut the quality by overloading the programmers and decreasing the testing (unit testing embedded in the development, the system testing, etc.). The result is that, the features seem to work, but only in some occasions and will behave incorrectly in others.
Why is quality often the first to get cut? It is because quality is not as visible as the other constraints: scope, cost and schedule. Any deviation in these three can be found easily, but it takes very closed interactions to discover quality deviations. In addition, the longer the problems are covered (e.g., after being deployed), the more money it will cost (e.g., service interruptions, loss of businesses, lots of service calls, training for workarounds, redeployment for the fixes).
Therefore, quality is not an option. It should be considered the 4th top constraints in project management. As mentioned earlier, you should plan, monitor and control for quality in your project (that's why you have all kinds of testing and validation in the project).

Keeping the sponsor happy to fuel the project

What if you only report to the project sponsor (here, the CEO) and other stakeholders in a year? They will lose enthusiasm by then. So, to keep their enthusiasm and more important their commitment and support for the project (that's how to ensure the resources for your project), you should report the project status, deliverables so far and etc. to them regularly. This is like fueling your car regularly on a journey.
Also, after the project is ended (closing), you should provide a final report to the project sponsor and other stakeholders mainly to show that you have delivered the required deliverables in high quality, within cost and schedule. Again, this is to make the project sponsor happy about the project and you for his future support.

Getting long lasting benefits out of the project

Basically, after the project is closed, all you enjoy is the deliverables. However, there is more you can get out of the project. For example, if you have found a best practice with working with that framework or working with a particular stakeholder, you should document it for future use.

Coordination between projects

In addition to the self-publishing platform, your company has other projects supporting the same strategy of enabling authors to self-publish. For example, it has a market research project to identify the needs, a promotion project to promote the service, and etc. Obviously, the projects must be coordinated. For example, it is impossible for you to get the detailed requirements before the market research project identifies them.
Therefore, there needs to be a program manager (e.g., a vice president) to do that (managing the program which consists of the interrelated projects). In addition, the company should adopt a program management framework to ensure proper organization (e.g., a program management committee consisting of the program manager and all the project managers) and processes are in place to support the prioritization and coordination of the projects.

Adopting and tailoring a project management framework

You've seen the key control objectives above for project management. Cobit recommends that your company adopt a project management framework (e.g., based on PMP body of knowledge or PRINCE2) covering the above, with customizations based on project size, risk, complexity and etc.

Study notes on Cobit: Assessing and managing IT risks

Life could be at stake

Suppose that you were the CIO of a hospital. To allow the people to quickly locate expensive medical equipment, you had implemented a WiFi tag tracking system and it had been working fine. Until one day, a patient was rushed to the ICU for urgent operation and the surgery doctor was looking for a certain piece of critical but rarely used equipment but it just couldn't be found. As a result, the patient died and the hospital was sued and ordered to pay a million dollars for the damage. The incident was also widely reported in the media. Investigation showed that the WiFi tag on that piece of equipment simply ran out of battery, but the hospital didn't have a procedure for checking for low-battery WiFi tags, so it was considered a negligence.
The above example shows a risk (medical equipment not found when it is urgently needed) that has become true. It had a very serious impact (loss of human life, public image of the hospital, financial damage). So, the question is, how to mitigate the risks?

Identifying and assessing the risks

The most important thing is to identify (find out) the risks first, otherwise you won't even consider them at all. How to identify the risks? As risks are not just about IT, your organization should appoint someone higher up to take overall accountability for risk management in the company. This role could be the chief risk officer. He should form an enterprise risk management committee and the members should include himself, the CIO (or someone in IT responsible for IT risks), compliance officer (for legal risks), and probably other business executives. Then, the members will work together to identify the risks. This should take the form of a brainstorm session or some other forms.
As there are so many possible risks to consider (the Earth hit by a meteor from the outer space?), it is ineffective to list them all. Instead, you should focus on those applicable to your organization's activities and environments (treating patients, laws for hospitals, local labor laws, in earthquake zone? etc.). This is called establishing the risk contexts. Then, you should focus on those risks with a high impact and/or a high probability (so, you're assessing the impacts and probabilities of the risks at the same time). For example:

1. High impact and high probability such as a baby is stolen (assuming you have very poor security) or a patient dies due to wrong dose of medication (assuming your staff are very lousy). Usually this kind of risks should not exist for very long as the company would have gone out of business.


2. High impact and medium probability such as the WiFi tag case above.


3. High impact and low probability such as avian flu pandemic.


4. Medium impact and a high probability such as hardware failure of a server.

Note that there is no strict definition of what is a high impact (MOP10,000 high impact or just medium impact?) and what is high probability (once a month or once a year?); it all depends on the rough concepts across the risk management committee members.
In addition, it is usually very difficult to quantify the impact to a dollar amount (how much a human life is worth in dollars?) or to quantify the probability to a number (the chances of having avian flu pandemic in this year is 0.1%?). So, in most cases you will just roughly classify them as high, medium or low.

Mitigating the risks

Once you have the list of risks (called the risk register), the risk management committee can prioritize them (by their impact and probability) and consider how to deal with each of them. How? It is simple: either reduce the probability or reduce the impact. For example:

• For the risk of babies being stolen due to poor security, just enhance the security to reduce the probability of losing a baby. Put security guards at various spots in the hospital (in particular, the baby room). Attach WiFi or RFID tags to the babies for checking whenever they are transported.


• For the risk of WiFi tags running out of battery, locate all the the equipment everyday to detect low-battery and replace the battery as required. This reduces the probability of running out of battery when the equipment is needed.


• For the risk of avian flu pandemic, while you can't reduce the probability, you can reduce the impact. For example, set up remote interaction system between the hospital and n quarantined facility for avian flu patients that supports remote diagnosis and automated delivery of medication, so that the medical staff staff members don't need to physically enter that facility and won't get infected.


• For the risk of a server hardware failure, you can reduce the impact by having a stand-by server, a 2-hour emergency repair service contract, a pool of spare parts and etc.

As you can see above, to mitigate the risks you may need to change your procedures (e.g., locating all the equipment everyday), IT infrastructure (stand-by server), contracts (emergency repair), to integrate risk management into all elements of your enterprise.

Transferring the risks

Sometimes it is just impossible or too difficult to reduce the probability or the impact of a risk in isolation. For example, a patient dies due to a mistake by the doctor. In that case, you can buy insurance. So, you will pay a certain small amount (a small impact but 100% probability). Essentially, you are trading a risk with a high impact (could be millions) and a low probability for a risk with a small impact and a high probability with the insurance company. For a smaller organization with limited cash such as your hospital, the latter is easier to absorb. For a larger organization with a lot of cash such as the insurance company, the former is more profitable.
In addition to insuring, outsourcing is another way to trade the risk: you just pay someone else to do the risky task for you. You pay more but the outsourcer takes the risk.
In either case, the event now has a 100% probability so in common sense is no longer a risk as it will definitely happen. Therefore, you may view it as transferring the risk to the insurance company or the outsourcer by paying them.

Monitoring and reviewing the risks

So, you have taken some measures to mitigate the risks, but do they work? Therefore, you should review regularly to see if the measures are indeed effective enough to reduce the probability or the impact enough. If yes, fine. If no, you need to come up with more mitigation measures. To allow for such reviews, you should record the corresponding mitigation measures for each risk in the risk register.
Will a certain existing or new risk become more prominent? For example, are there indications that an avian flu pandemic is coming? A new law is coming into effect? To deal with these cases, the risk management committee should review and update the risk registry regularly accordingly.

Study notes on Cobit: Managing quality

What is quality?

Are your users happy with, say, your email service? This is what quality is all about. If the email system is so slow or fails to catch spam effectively, the users may be unhappy about it. If you (the CIO) don't know about it and thus aren't doing anything to address the problem, they (probably including the CEO) will lose trust in you and you will have difficulty getting budget for existing or new initiatives. So, getting the users happy about your services (quality) is critical for your career as the CIO.

How to ensure quality?

Take the email service as an example, it is basically an automated service so the quality mainly depends on the email software and the infrastructure (server and network) instead of human being. How to ensure the quality of, e.g., the email software? You may follow such a process:

1. The CIO, the business analyst and the solution architect discuss with the representative users and other stakeholders on the requirements. For example, you may determine that it needs to support shared and personal address book, anti-spam, anti-virus, web access from anywhere, digital signature, company wide archiving (for legal purpose). There are also non-functional requirements such as: it shouldn't take more than 1 minute to process and deliver an incoming mail (performance), all client communication must be secure (security), the service should be available 99.9% of the time (availability), etc.


2. The architect evaluates the email packages on the market to identity those that support the above requirements.


3. The architect selects the one that meets the requirements, at a lower cost with a lower risk.


4. The architect requests the representative users, in your actual environment with a heavy work load.


5. The CIO finalizes on the decision with all the stakeholders.


6. Deploy the solution.


7. Get user feedback.


8. Get user feedback regularly to see if the service gets deteriorated (e.g., is it getting slow due to increased volume of emails?) or there are new requirements (e.g., shared calendar?).

If you check carefully, you will note that this process is actually applicable to most types of solution acquisitions and has little with the specifics of email service. The process is basically:

1. Get requirements (control point for quality).


2. Evaluate & test possible solutions (control point for quality).


3. Select a solution.


4. Validate the solution (control point for quality).


5. Finalize & deploy on the solution.


6. Validate the solution again in production (control point for quality).


7. Monitor the quality regularly and improve as needed (control point for quality).

That is, to ensure quality, all solution acquisitions should adopt this process. Adopting such a process with quality control points is the basis of quality management. Of course, you also need people capable of doing this type of work (in this case, the business analyst and the architect).

Ensuring quality throughout the whole enterprise

The above process deals with the quality of acquired solutions, but what about others such as development of applications, support services and maybe other non-IT products or services? To ensure that everything is covered, there should be someone higher up such as the quality director who is in overall charge of quality (in ISO 9000, this is called the management representative). He should somehow ensure that the processes for the company to develop its products or to deliver its services (including those for internal use which is typically the case for IT) have incorporated control points for quality.
As this is a lot of work and much of the work requires professional knowledge (e.g., it's difficult for a non-IT person to ensure quality in software development), the quality management representative should ignite the care for quality in the minds of all staff members, so that they drive the development of the processes to ensure quality instead of them being driven by the processes.

Summary

As you can see, to ensure quality, your company needs the organization (quality management representative and all other people actively working to ensure quality and their relationships) and the processes in place designed to ensure quality. These two are the main elements of a quality management system (QMS).

Study notes on Cobit: Managing IT human resources

It is the people who will make or break the IT capability

Suppose that you have in place the best IT processes in the world and all the resources you want, but what if:

• Your programmers are always blamed by their manager for any faults found in the software while he takes all the credit (motivation issue)?


• Your system administrators don't care about security and use simple passwords (attitude issue)?


• Your business analysts can't communicate with the users smoothly and pushing the users away (skill issue)?


• Your support personnel are not familiar with the services being provided (skill issue)?


• Your architect was actually hired by a competitor to steal your business secrets (law and ethic issue)?

Then there is no way for you to deliver business values through IT. Therefore, after all it is the people who will make or break your IT capability. Therefore, it is critical that you ensure your people:

• tend to comply by the laws and ethics.


• have the required skills.


• have the required attitudes.


• are motivated.

How? These items are of different natures so you need to treat them differently.

People tending to comply by laws and ethics

Most people do try their best to comply by laws and ethics. To ensure that it is indeed the case (in particular, if you're in a sensitive business such as department of defense or drug research company), you can perform a background check, like:

• check if the candidate has a criminal record.


• check with his previous company to see why he left or why he was fired.


• check his public records such as blogs, Facebook, forum postings for any tendency for illegal activities.

People with the right skills

There are many skills in IT that can come and go easily (e.g., Netware administration). So, instead of hiring a person who has such a particular skill, you may look for someone with the ability to learn, by self-study or training, then create a training plan for him and other employees on an annual basis to meet the then current demands.
However, there are some fundamental skills that is very hard to acquire. For example, the skill to write readable code, the skill to diagnose a problem, the skill to search for answers, the skill to understand others, the skill to think strategically, the skill to think out of the box and etc. If such skills are important to you, you should hire the people with such skills.
How to test if the candidate possesses such skills? Usually I will just give him some practical tests. For example, to see if he can search for answers, ask him some questions and tell to him search for answers and see how good he is.

People with the right attitudes

Attitude is very much like the fundamental skills above. It is difficult to change a person's attitudes:

• Pessimistic vs optimistic


• Risk-aversive vs risk taking


• Blaming others vs responsibility taking


• Improvising vs planned


• Self-centered vs empathic


• Following vs leading


• Negative vs positive


• ...

Therefore, you should hire the people with the desired attitudes. How to test if the candidate possesses such attitudes? Usually I will ask him for his past actions. For example, to see if he has the leading attitude, ask him if he has taken any initiative to lead people to solve a problem.

Motivating people

Motivation is not something you do when hiring. It is what you do everyday. This is like fueling your engine constantly. How to motivate people?
First of all, money or other extrinsic rewards don't really motivate people. Why not? For example, let's say you got a pay raise of MOP2,000, you would be motivated and work harder, right? What if you found that all the other colleagues got a pay raise of MOP4,000? Then you would be demotivated even though you had more money. It means what you really care is the recognition of your value, not the money you receive.
From my experiences the following simple actions do motivate people:

• Trust (autonomy). Trust your staff members and let them decide how to achieve the targets you set. Do not micromanage.


• Praise (recognize). Praise them for the good things they have done.


• Help. Help them remove the obstacles they face. Help them become better to avoid any similar bad things from happening.

Study notes on Cobit: Communicating management aims and direction

Defining or maintaining IT policies, standards and procedures

Suppose that there has been a user in your company who lost many important files because the hard disk of his computer died. Obviously, he didn't save his files into the network drive. To prevent similar incidents again in the future, the chief information security officer proposed to make an addition (bold below) to the IT standard, which belongs to a certain IT policy and is shown below.
IT policy 10: All employees should take care of the information they created against possible failures.
IT standard 123: File handling

1. All user files created must be saved to the network drive instead of the local hard disk.


2. If some files (at the confidential level or above) must be put onto a portable device such as a USB disk (to acquire a USB disk, refer to procedure 456), the file must be password protected (refer to the technical procedure 789).


3. ...

Procedure 456: Applying for USB disks

1. Fill in the form xyz.


2. Get the approval from the department manager.


3. Get the USB disk from the service desk.


4. Return the USB disk by the deadline stated on the form.


5. ...

Technical procedure 789: How to encrypt files

1. ...


2. ...


3. ...

After defining or amending the policies, standards and procedures, depending on the authority of the chief information security officer, he may seek approval for this change (from the CIO or the chief risk officer of the company). But it is far from done yet, as the people will not automatically follow them.

Rolling out the IT policies, standards and procedures

So, he will need to inform all the people relevant (all users in this case) of the changes and the rationale behind it (if they don't understand the rationale, they will have no motivation to adhere to it). How?

• Emailing them is a quick way but many people will simply ignore it. You may insist to get a reply and chase those who don't reply. But later people may simply reply to you without reading the mail.


• You may conduct a briefing but it is very time consuming, in particular, for such a small change or if there are multiple locations in your company.


• You may record a video to talk about the change and email them the link.


• You may seek the help of the department managers to disperse this information.


• Any other ways that you think is the most effective.

The above is for existing employees. What about new employees? Make sure the policies, standards and procedures are included in the orientation.

Compliance

As you can imagine, it is difficult to get all people to understand the change, not to mention to really act according to it. So, it is almost always necessary to check if they're really doing it. That is, go to some users' PCs to check if there are any files saved locally (to save time, check against other terms in the IT standard or other standards). Of course, to perform such audits, you must get management approval first and inform the people in advance that audits will be conducted.

Study notes on Cobit: Managing IT investment

Forecasting (planning) the business values and costs

Suppose that you're the CIO of a movie rental store chain with 10 stores at different locations. The CEO has selected your proposed IT strategy of implementing an enterprise wide collaboration system (e.g., Sharepoint or Alfresco) to enable document sharing and project management, using a SaaS solution. As discussed before, you (the CIO) have forecast (planned) the business values and costs as below.
Values:

• Allow all the users to access the same documents even though they are on different sites. The information to be shared includes updated price list, current promotion programs, company policies, procedures and guidelines, document templates, best practices, staff phone number extension list.


• Allow different project team members to see and update project tasks. Allow project managers to see the overall status of the projects.

Costs:

• Subscription fee to the SaaS solution: MOP100 per user per year. There are totally 30 users, so the annual cost is MOP3,000.


• Internet access: the company already has Internet access for all the sites. It is estimated that it will not require upgrading the bandwidth. If the Internet cost is significant, you may estimate the percentage that will be used for this service so that you can justify the Internet access. In this example, let's ignore it.


• Service desk support: your service desk will provide support to the users. However, it is difficult to estimate how much of the service desk will be used for this service. But if you'd like to justify the value of existence of your service desk, you may need to estimate or measure it, to link the cost to the business value. In this example, let's ignore it.


• PC clients: the existing PC clients can handle it easily as almost everything happens in the browser.


• Testing by IT: it is estimated that an IT architect and the QA manager will spend 1 week to test the solution. Assuming that their hourly rate is MOP200, this will cost 2*5*8*200=MOP16,000.


• PC client configuration: you will need to add a bookmark to the browser and a shared folder (to hold the document templates) for each user. This is not a good way, but let's suppose that you lack an automated solution. Let's say it will take 2 hours to visit and configure the PCs on each site, so it will take 2*10=20 hours. Assuming that the hourly rate of a technician is MOP100, this will cost MOP2,000.


• User training: you will conduct two user training sessions at the headquarter. It takes 1 day (8 hours) for preparation and for each session 2 hours for delivery. It will also take each user 1 hour to travel to the headquarter and 2 hours to attend the session. Assuming that the hourly rate of the trainer is MOP200 and that of the user is MOP50, this will cost (8+2*2)*200+30*(1+2)*50=MOP6,900.

So, the CEO has accepted this IT strategy and approved the budget (once-off: MOP24,900. MOP3,000 annual).

Tracking the costs

So, some time later you start to implement the project. In the process, you should keep track of the costs to make sure they don't exceed the budget. For example, if the testing of the architect and the QA manager shows that the existing Internet bandwidth must be upgraded from the existing 256Mbps to 512Mbps for headquarter (due to the need to perform lots of uploads), this will cost an extra MOP300 per month.
When seeing this deviation, what should you do? There are a couple of options like:

• Try to reduce the cost. For example, check if the bandwidth of the headquarter is being used wisely (e.g., many staff members watching unrelated videos on Youtube?). If so, cut those wastes.


• Report to the CEO and other stakeholders about this bad news. They may decide to go ahead, to abort the project or to even suggest some other solutions for you.


• Keep it secret. This is not really an option and you will pay dearly later! So, don't do that.

Tracking the business values

In addition to tracking the costs, you should also keep track of the business values to show to the CEO and other stakeholders. For example, after implementing the project, the users should start using it. But how to show or measure the business values? Usually it is very difficult to quantify the business values into money amounts. In fact, it is also very difficult to measure, qualify or describe business values. Here are some possible ways to "measure" the business values for this example:

• show that the updated price list has indeed been put into the system and how frequently it has been accessed.


• show the comments from the users regarding how much better they feel when getting the updated price list this way instead of the old way (e.g., fax).


• show the comments from project managers regarding how much better they feel when managing projects with this system.


• so on.

What happens when there are deviations? For example, if after implementing the system, nobody accesses the updated price list in the system? Then obviously you're in big trouble! Try to find out why they aren't using it. For example, if the guy in the headquarter is uploading the price list in a timely fashion, talk to him to find out why and try to resolve the problem.

Quantifying business values

In the official Cobit document, it is recommended that we quantify business values into amount of dollars and then prioritize or select IT strategies based on monetary comparison (e.g., ROI, net present value, payback period). While this may sound scientific, I seriously doubt its practical usefulness. For example, how to measure the value of easier project management by having a global and updated view? It really depends on the nature of the projects being managed, the people using the system and even the current business environment, etc.

Study notes on Cobit: Defining the IT organization and relationships

The CIO job position and relationships

Recall the IT strategic planning process involving the CEO, CIO and etc.? What if the company has no CIO (or any job position assuming such responsibilities), then all deals are off. It means that CIO is a important job position for the success of IT governance.
Consider the CIO position, in the process we implicitly assumed that the CIO reported to the CEO. But what if he reports just to the CFO (IT considered part of finance)? Then, IT may not play as a strategic role as when the CIO is reporting directly to the CEO. That is, in addition to the responsibilities of the position, the relationship (here, reporting relationship) between this position and others will have high impact on the significance of IT.
In addition to the reporting relationship, there are some other important relationships. For example, as described in the process for strategic IT planning, the executive-level collaborators (CEO, CIO, executives) also form a working relationship to steer IT. To highlight and formalize this relationship, we can give it a name like "IT steering committee" or "IT strategy committee". In Cobit, the former refers to the case we described before (up to the CEO level), and the latter refers to the case when the board of directors are involved (typically for major IT strategies involving a large investment).

Other strategic positions in IT and relationships

If we assume that CIO will take care of the business value alignment, then we will next see who will take care of quality and risk (other main elements of IT governance):

• For quality, we should have a QA manager. If the CIO really values IT quality, he should have the QA manager reporting to him directly. In addition, the QA manager should have a working relationship with the enterprise wide quality director (or a similar position).


• For risk, it can be divided into information security and legal compliance. For information security, we should have a chief information security officer (CISO). He may report to the CIO or even the CEO (if information security is considered very important by the enterprise).


• For legal compliance, we should have a position in IT who reports to the CIO and has a working relationship with the chief compliance office in the enterprise.

Protecting from the loss or malfunctioning of key individuals

What if the CIO quits to work for another company? What if the CIO is the only one making purchase decisions but he is bribed? In order to limit the damages from the loss or malfunctioning of a key individual, you should design the IT organization so that there are backup positions (e.g., assistant CIO or a senior IT manager acting in that role) and important decisions are approved or reviewed by more than one person (duty segregation) (e.g., bid evaluation committee).

Study notes on Cobit: Defining IT processes

What is a process?

In my previous notes, you have learned that you, the CIO, must suggest IT strategies to support the business objectives or business strategies. How to do that? You might do it this way:

1. The CEO conducts the annual business strategic planning process with other executives (COO, CFO, vice presidents, senior managers and etc.).


2. Assuming that the CIO (you) is involved in the process, so, he  learns about the business objectives and strategies for the coming financial year and proposes rough ideas about the potential IT strategies to support the business objectives and strategies. He should provide very rough estimates on the business values, costs, resources and risks of such IT strategies.


3. The CEO, with input from the CIO and other executives, selects a short list of IT strategies for further studying.


4. For each selected IT strategy (which supports a business objective or strategy) you should be able to identify the executive who will be benefited the most. That executive should be enthusiastic enough to act as the champion.


5. The CIO works with the architects to give better estimates on the costs, resources and risks for such IT strategies.


6. The architects may conduct researches and if required, feasibility studies.


7. The CIO obtains the support from the champion.


8. The CIO obtains the support from the CEO and other participants.

The work flow above is called a process. Why is it useful to clearly specify (define) the process?

Seeking support

In the above the process, it is clear that the CIO, CEO, champions and architects need to collaborate to execute the process (these are called "roles" in the process). Each role has to carry out a certain responsibilities (e.g., an architect has the responsibility of estimating the cost, resource and risk of the IT strategies). Therefore, to establish the process, you must first obtain their consent on those responsibilities. In fact, as they're the collaborators, it is best to seek their suggestions to make the process better in order to obtain their strong support for the process.
Therefore, the process definition above can serve as a tool to get their agreements and support on the process including their respective responsibilities. This is the first important purpose of defining the process.
BTW, if you're careful, you may note that while roles like CIO, CEO and architect are probably job titles, but champion is probably not as it varies by case (e.g., the chief marketing officer may be the champion for an IT strategy involving facebook marketing). It means that you need to include all such possible people to act as the champion in this agreement seeking exercise.

Continuous care and delegation

Is it done after defining the process? Once you have a process, you need to continuously check if it is really working well and improve it. For example:

• Is it effectiveness (does it work at all)? For example, has it generated IT strategies which turned out to be not supporting any business objectives or strategies?


• Is it producing high quality output? For example, has it generated just mediocre IT strategies or excellent ones?


• Is it efficient? For example, does it turn out that it takes many times of back-and-forth to convince the CEO?


• Is it embedded into the DNA of the organization (maturity)?


• Is it dealing with risks properly? For example, has it generated IT strategies which turned out to overlook a major risk?

Who will do this work on a continuous basis? As it is a lot of work, having a single person to do it for all the processes (or just all the IT processes) in the organization makes no sense. So, for each process, a process owner must be assigned. For this particular process, the process owner should probably be the CIO as this process has the most relevance for him. For a process regarding, say, user acceptance test, it should probably be the QA manager.

Defining IT processes

As it is extremely important to have IT processes properly defined, monitored and improved, this is itself an important process in Cobit.

Study notes on Cobit: Determining technological direction

The need for enterprise wide standardized technologies

Suppose that you're the CIO of a company and have decided to acquire an ERP system and a CRM system to support the business strategies of optimizing the supply chain and enhancing the relationship with the customers. You've assigned two architects to work on those two projects respectively. If they don't communicate and you don't coordinate their decisions, it is possible that they will adopt completely different technologies that they consider the most suitable, like:

• ERP: Linux server, Windows XP clients, DB2 database, web-based, Java.


• CRM: Windows server, Windows 7 clients, Oracle database, client-server, .NET.

If they are allowed to go ahead like that, you'll be in big trouble:

• Your infrastructure team will have to manage both Windows and Linux servers and thus have to learn both and deal with the quirks of both.


• Your DBA will have to manage both Oracle and DB2 and thus have to learn both and deal with the quirks of both.


• Your PCs just won't be able to access the two systems at the same time! Some PCs will be used to access the ERP system, while some others will the CRM system.


• It is difficult for the two systems to share information (as mentioned in "defining information architecture") as they don't support the same database management system.


• If, say, you're already using Oracle, introducing DB2 into the environment may create uncertainty and instabilities.


• If you need to customize the two systems, you will have to find both Java programmers and .NET programmers.

Therefore, you (the CIO) should have a process to coordinate the technologies used by different applications as proposed by the architects. For example, you might standardize all new projects on:

• Linux server


• Oracle database


• Java


• web-based


• Windows 7

Then you'll be able to leverage the same people (system administrators, DBAs, programmers) and the same resources (Linux server, Windows 7 clients) for both applications, enjoying economy of scale and better interoperability (e.g., now possible to share information).

What about legacy technologies?

For example, you may have an existing accounting system written in Cobol running in AS/400 accessed through a terminal. This is certainly holding you back in economy of scale and interoperability. Therefore, you should consider working out a plan to migrate the legacy technologies to your selected standardized technologies, after considering the benefits, costs and risks.

Looking forward and updating your standardized technologies

Of course, you can't stick to your standardized technologies for very long; Windows 8 may be released in a couple of years, cloud computing may take off, there might be legal regulations for cloud providers to ensure the security for customers, credit card security standard might change (PCI DSS), mobile phones may become the primary computing client, SOX might be updated and etc. So, you need to keep monitoring for new technological trends, legal, social or economical trends that may have technological impacts, and then create a future set of standardized technologies for your company (thus establishing a technological direction) and a plan for migration.

Study notes on Cobit: Defining information Architecture

Creating, maintaining and using information for the business

Suppose that you're the owner of a retail store. You'd like to use IT to enhance your business (so, you're also acting as the CIO in addition to the CEO). You figure that if some products are missing on shelves, you'll lose business. So, you have to replenish the items on shelve from the storage area promptly and reorder the items if they're running out in the storage area in a just-in-time manner. To do that, you can use a POS system to keep track of the items sold, the items on shelve and in the storage area. That is, you need to create, maintain and use information that is modeling the real world. That's probably the most common purpose of business IT systems.
However, there comes a cost of storing information: you have to take good care of (manage) the information properly otherwise bad things can happen. This is like owing a car or a dog: once you own it, you have to keep it clean, keep it healthy and etc.

Keeping information correct

One problem with storing information is that, if unmanaged, the information can become erroneous. For example,the data in the POS may not match the real world as time goes by, so you need to perform inventory from time to time to reconcile them (a process to keep information correct).

Keeping information consistent

Another problem is that information, if gets duplicate, will become inconsistent. For example, while the POS system contains the information about your suppliers, you may also need to input such information into your accounting system for the accounts payable. Why is this a problem? Let's say if a supplier changes his address and informs you, you will have to update it in both the POS system and the accounting system. If you forget to do either, the information will become inconsistent.
How to solve this problem? The ideal solution is to not duplicate the data. For example, if both the POS system and the accounting system support it, they could be put into the same database but in different schemas (one schema for each application), while storing the supplier information into a third schema to be shared. If this is not supported (some applications may not allow you to specify the schema), you may have to synchronize the supplier information somehow between two databases. Both approaches can be considered enterprise-wide data coordination between different applications, departments or processes.

Keeping information secure

In addition, once keeping the information, you must keep it secure. For the information regarding the product items, it is not that confidential (confidentiality is one aspect of information security), but you must prevent ordinary users or other people from changing their retail prices (integrity is the second aspect of information security). For the financial information (accessed through the accounting system), reports like profit and loss should be confidential and accessible to the management of the company only.

Making sure the information & the processing system are available

As the operation of your business now relies on the POS system to read and update the product item information, if the database (information) or the POS system (processing system) breaks, your business will be severely hindered. Therefore, you must ensure the availability of the database and the POS system (availability is the third aspect of information security).
How to do that? You should conduct backup of the database and the POS system and practice restoration regularly. You could also consider redundant hardware (e.g., RAID, fault-tolerant server, dual power supply), redundant power (UPS or power generator), redundant systems (e.g., database replication, server cluster) and disaster recovery planning.

Archiving or removing the information

Finally, for performance of the system or to save space, you may want to archive the information to a long term storage (e.g., DVD) as historical records.
In some companies, if the law requires that the information be kept for a certain years, they will remove that information after meeting that requirement in case that the information could be used against them in court (just think Edison Chen who failed to really delete his photos in his laptop!).

Delegating the management responsibility of information

As there is a lot of different information in the company, it is difficult to depend on you, an individual, to decide how much care should be put on that information (How strong should the access control be? How long to keep the information? How much availability?). Therefore, for each type of information (e.g., financial information, product information), you (acting as the compliance officer or chief security officer) should ensure that there is a process to assign someone as the owner of that information, then let him decide how much management care is required. Usually, this can be done by classifying the data (e.g., in security: top secret, confidential, internal use, public. In availability: highly available, important, normal, rarely needed).

Information architecture

In summary, to support the business, you plan what information to create, maintain and use, how to keep it correct, consistent (sharing or synchronizing), secure (confidential, untempered, available), when and how to archive or remove it. The processes, methods and documents you create to achieve the above purposes (properly managing the information) are called the "integration architecture" for the enterprise.

Study notes on Cobit: Overview and strategic IT planning

What is strategic IT planning?

Simply put, strategic IT planning is to suggest various IT solutions to support business objectives or strategies. For example, suppose that you are the CIO of a school and that the school has an objective of improving education quality. It may have defined business strategies to achieve that objective like:

• Making learning fun


• Engaging parents


• ...

As the CIO of the school, you could suggest various IT solutions to support one of those strategies. For example, for the first strategy ("Making learning fun"), you may suggest:

• Provide educational games to the teachers.


• Provide training to enable teachers to develop interactive courseware.


• Locate and provide the best student-engaging lecture videos to the teachers.


• ...

Note that each IT strategy must have a clear linkage to the business strategy so that it creates business value ("Value" is one of the three major concerns of Cobit). For example, it is assumed that educational games will make learning fun because students like playing games.

Cost and risk

After coming up with these IT strategies, you will explain them to the CEO (the principal) and business executives (the vice principals) to let them choose which ones to proceed. However, without the information regarding the cost ("Cost" is the second major concern of Cobit) and other needed resources for each IT strategy, it is just impossible to choose. So, you have to provide such information. For example, for the IT strategy of providing educational games, you may need:

• MOP1,000 (cost) for a site license for each game (software).


• Teachers (people) to attend a training session.


• A server (hardware) to host the games.


• A certain internal network bandwidth (network).


• ...

These needed resources (people, software, hardware, network) are called "Enterprise architecture for IT" in Cobit.
In addition, what if some students become addicted to the games? What if some parents don't want their kids to play games? These risks must be communicated to the CEO and business executives to help them choose which IT strategies to pursue ("Risk" is the third major concern of Cobit).

Control objective

So, you can perform strategic IT planning as described above. But how to ensure the desired output (IT strategies as selected by the CEO will have good value, low cost and low risk) will be produced? For example, there are some success factors:

• You (the CIO) knows the business objectives and strategies well.


• The CEO is willing to make clear decisions.


• You, the CEO and business executives can discuss and negotiate well.


• You can plan (estimate) the costs, risks and other needed resources well.

Each process has its own such success factors and they're key to ensure that the process is effective (producing the desired output) and performance (producing the output efficiently). Such success factors are called "control objectives" in Cobit and is a key concept in Cobit (the term "Cobit" simply stands for control objectives for IT).

Maturity level

However, even if you're doing the strategic IT planning process very well, it doesn't mean that the process is embedded in the DNA of the organization. For example, if you leave the organization and your successor doesn't do it or does it poorly, then it is not in the DNA of the organization. This is reflected in the "maturity level" of the process:

• Level 0 (No process): The organization has no process at all.


• Level 1 (Ad-hoc): Different people in the organization have different processes to do the same thing.


• Level 2 (Repeatable): Different people in the organization follow the same process, but the process is informal (not written down), so it is not formally reviewed, approved and used for training.


• Level 3 (Defined): The process is formally defined.


• Level 4 (Managed): The process is continuously improved.


• Level 5 (Optimized): The process is considered among the best in that  industry.

Scala exercise 6: Tackling the Wicket hierarchy mismatch problem

Introduction

This is the sixth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from the previous exercises. Below is exercise 6: Tackling the Wicket hierarchy mismatch problem.
Some people have alleged that a weakness in Wicket is that you have to keep the hierarchy of the component tree in sync with that of the HTML elements. For example, for the HTML code:

<html>
    <form wicket:id="f">
        <input type="text" wicket:id="num">
        <input type="submit" value="OK">
    </form>
    <span wicket:id="result">10</span>
</html>

You would construct a component tree (in Scala) like:

class MyPage extends WebPage {
    val f = new Form[MyPage]("f") {
      override def onSubmit() {
        ...
      }
    }
    add(f)  //add the form to the page
    val numField = new TextField[Int]("num")
    f.add(numField)  //add the text field to the form
    val r = new Label("result")
    add(r) //add the label to the page
}

The problem is that, if, say, you'd like to move the result span into the form, you must change the code accordingly:

class MyPage extends WebPage {
    ...
    val r = new Label("result")
    f.add(r) //you must add the label to the form, not to the page!
}

and a common problem is that we may forget to do so. While there is no easy way to solve this problem, you could make the Scala code reflect the HTML structure visually (and look more like a declarative UI):

class MyPage extends WebPage {
  //the "ctx" represents a surrounding container context for the 
  //construction of child components
  this containing { ctx =>
    val f = ...
    ctx.add(f)  //add the form to the context (the page)
    f containing { ctx =>
      val numField = ...
      ctx.add(numField) //add the text field to the context (the form)
    }
    val r = new Label("result")
    ctx.add(r)  //add the label to the context (the page)
  }
}

Note that the Scala code reflects the structure of the HTML code so you can compare them visually. In addition, if you need to move the label into the form, you can simply cut and paste the construction code of the label into that context of the form:

class MyPage extends WebPage {
  //the "ctx" represents a surrounding container context for the 
  this containing { ctx =>
    val f = ...
    ctx.add(f)
    f containing { ctx =>
      val numField = ...
      ctx.add(numField)
      //the following lines were simply cut and pasted into here
      val r = new Label("result")
      ctx.add(r)
    }
  }
}

Now, your task is to implement this solution by creating the necessary code. Try to do it now.
See the answer here.

package com.foo.myapp

import org.apache.wicket.{MarkupContainer, Component}

object MarkupContainerContext {
  //automatically convert a MarkUpContainer into a container context
  implicit def wrapContainer(c: MarkupContainer) = new MarkupContainerContext(c)
}


class MarkupContainerContext(c: MarkupContainer) {
  //create a container context to wrap around the children construction code
  def containing(childrenConstructionCode: (MarkupContainerContext) => Unit) {
    childrenConstructionCode(this)
  }

  //add one or more child component
  def add(children: Component*) {
    //this special syntax will expand the List into individual arguments
    //and pass all the child components one by one
    c.add(children :_*)
  }
}

package com.foo.myapp

import org.apache.wicket.markup.html.WebPage
import org.apache.wicket.markup.html.form.{TextField, Form}
import reflect.BeanProperty
import com.foo.myapp.MarkupContainerContext._
import org.apache.wicket.markup.html.basic.Label
import org.apache.wicket.model.{IModel, CompoundPropertyModel, PropertyModel}

class MyPage extends WebPage {
  @BeanProperty
  var num: Int = 0
  @BeanProperty
  var result: Int = 0

  setDefaultModel(new CompoundPropertyModel[MyPage](MyPage.this))
  this containing { ctx =>
    val f = new Form[MyPage]("f") {
      override def onSubmit() {
        result = 2*num
      }
    }
    ctx.add(f)
    f containing { ctx =>
      val numField = new TextField[Int]("num")
      ctx.add(numField)
    }
    val r = new Label("result")
    ctx.add(r)
  }

}

Scala exercise 5: monitoring pattern

Introduction

This is the fifth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from the previous exercises. Below is exercise 5: monitoring pattern.
Suppose that you have created a web application displaying product information to and taking order from customers. From your testing, let's say you have determined that it can at most display the product page 100 times per minute, otherwise it will get painfully slow or even crash. Now, you'd like to monitor the application in production to make sure its loading is not approaching that limit. If yes, you'll promptly allocate more computing resources to it.
In order to do that, it would be great if the application would support something like SNMP and provide a variable named "display counter" for you to access. Then you could use a monitoring system like zabbix or nagios to monitor it. The good news is, there is something similar for Java: it is called JMX (Java Management eXtension).
Next, you'll do it step by step. Your job is to fill in the missing code.
First, create a Maven project. Use the following pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>scala-jmx</groupId>
    <artifactId>scala-jmx</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>
    <properties>
        <spring.version>3.0.5.RELEASE</spring.version>
    </properties>
    <repositories>
        <repository>
            <id>java.net2</id>
            <name>Repository hosting the jee6 artifacts</name>
            <url>http://download.java.net/maven/2</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-context</artifactId>
                <version>${spring.version}</version>
            </dependency>
            <dependency>
                <groupId>org.springframework</groupId>
                <artifactId>spring-core</artifactId>
                <version>${spring.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-web-api</artifactId>
            <version>6.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.8.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.0.2</version>
                </plugin>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.9.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.1-beta-1</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

There is nothing special except that you'll use Spring. Next, create a Scala trait PerfCountersMBean in the com.foo package:

trait PerfCountersMBean {
  def getDisplayCounter(): Int
}

It says that you'll have an "MBean" that has a property named displayCounter. MBean stands for managed bean, which is just a Java bean that can be inspected or called from a remote management console. Next, create a Scala class to implement the MBean in the com.foo package (fill in the code later):

...
class PerfCounters extends PerfCountersMBean {
  ...
}

In order to initialize the MBean, you'll turn it into a Spring bean (you'll do it later by annotating the PerfCounters class). Then, to register it with the JVM, you'll create a Spring bean that performs the registration. So, create a beans.xml file in the src/main/resources folder:

<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context-3.0.xsd">
    <context:component-scan base-package="com.foo"/>
    <!-- init this exporter eagerly as nobody will ask it to export the MBeans -->
    <bean id="mbeansExporter" class="org.springframework.jmx.export.MBeanExporter" lazy-init="false">
        <property name="beans">
            <map>
                <!--
                   the key is the name of the MBean, which starts with the domain
                   name (foo.com) and then some name=value pairs.

                   the value is the Spring bean acting as the MBean
                -->
                <entry key="com.foo:name=performance-counters" value-ref="perfCounters"/>
            </map>
        </property>
    </bean>
</beans>

Of course, you need to increment the counter whenever the product page is displayed. So, create a servlet which should display something as the product information and more importantly, get access to your MBean (a Spring bean) and increment the counter (you'll fill in the code later):

class DisplayServlet extends HttpServlet {
  ...
}

Finally, create the web.xml file the src/main/webapp/WEB-INF folder:

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd">
    <servlet>
        <servlet-name>s1</servlet-name>
        <servlet-class>com.foo.DisplayServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>s1</servlet-name>
        <url-pattern>/product</url-pattern>
    </servlet-mapping>
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:/beans.xml</param-value>
    </context-param>
    <!-- initialize Spring -->
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
</web-app>

Run "mvn package" to build the war file. Then deploy it into Tomcat. To enable JMX in the JVM on which Tomcat runs, you need to pass some JVM arguments by setting the CATALINA_OPTS environment variable before running startup.sh (or startup.bat). On Linux:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

On Windows:

set CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Now, fill in all the missing code above.
To verify that it is working, run Tomcat with startup.sh (or startup.bat). Use netstat to check if it is listening on port 1234 (the port you specified above). Finally, run jconsole and connect to localhost:1234. Choose the "MBeans" tab and you should see your MBean in the com.foo folder. Check the value of the DisplayCounter. Access the product page at http://localhost:8080/scala-jmx/product. Then check the value again and it should have been incremented.
See the answer here.

package com.foo

import org.springframework.stereotype.Component
import reflect.BeanProperty
import actors.Actor


//make it a Spring component
@Component
//it will be accessed by concurrent threads, so it should be an actor
class PerfCounters extends PerfCountersMBean with Actor {
  //this will create the Java beans getter and setter
  @BeanProperty
  var displayCounter = 0

  start //start myself just like a thread
  
  def act() {
    while (true) {
      receive {
        //note that there is no ++ in Scala
        case IncDisplayCounterMsg => displayCounter += 1
      }
    }
  }
}

//only one instance is needed, so use a case object as the message
case object IncDisplayCounterMsg

package com.foo

import javax.servlet.http.{HttpServletResponse, HttpServletRequest, HttpServlet}
import org.springframework.web.context.support.WebApplicationContextUtils
import org.springframework.web.context.WebApplicationContext

class DisplayServlet extends HttpServlet {
  //look up the Spring application context created by the Spring context listener.
  //You can't do that in the constructor as the servlet context hasn't been set
  //yet. So, use the "lazy" keyword to do it on-demand.
  lazy val context: WebApplicationContext = WebApplicationContextUtils.getWebApplicationContext(getServletContext())
                                                          
  override def doGet(req: HttpServletRequest, resp: HttpServletResponse) = {
    val writer = resp.getWriter
    writer.print("This is the product page!")
    //get access to the MBean
    val counters = context.getBean("perfCounters").asInstanceOf[PerfCounters]
    //increment the display counter by sending it a message
    counters ! IncDisplayCounterMsg
  }
}

Scala exercise 4: circuit breaker pattern

Introduction

This is the fourth exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from exercise 1: template method, exercise 2: observer design pattern and exercise 3: decorator and composite design patterns. Below is exercise 4: circuit breaker pattern.
Problem: Create a console program that reads commands from the user. If the command is "q", it will query a hard-coded URL (e.g., http://localhost:1234) and print the result to the console. If the command is "x", it will exit. Below is a sample session:

Enter: q
Hello <-- this is the content at the URL
Enter: q
Hello <-- ditto
Enter: x
Exiting

In practice, if the remote server is overloaded, out of order or the firewall is mis-configured, every time you try to access the URL, it may take up to several minutes for your client to timeout or receive an error. It would also place unnecessary load on that server which possibly is already overloaded. Therefore, a better way to do it is, if the server returns an error (or does that for a consecutive number of times), your client will not try to access it anymore (treat it as an error immediately), until the administrator resets it. This is called the "circuit breaker" pattern.
Now, your task is to implement this behavior by filling in the code below. There is also a new command "r" to reset the circuit breaker. For simplicity, you'll stop calling the server as long as one exception is caught.

//it extends the CircuitBreaker trait to obtain the functionality. Unit is the return
//type of the code block to be protected by the circuit breaker.
object HttpClient extends CircuitBreaker[Unit] {
  def main(args: Array[String]) {
    while (true) {
      print("Enter: ")
      try {
        readLine match {
          case "q" => queryHttpServer
          case "r" => reset
          case "x" => {
            println("Exiting")
            return
          }
          case _ => println("Unknown command")
        }
      } catch {
        case e: Exception => e.printStackTrace
      }
    }
  }

  def queryHttpServer {
    //run the code block in the protect() method
    protect {
      ... //get the content at http://localhost:1234
    }
  }
}

trait CircuitBreaker[T] {
  var isOpen = false

  //run the code block only if the circuit breaker is not open. If it is,
  //thrown an CircuitOpenException immediately.
  def protect(codeBlock: ...): T = {
    ...
  }

  def reset {
    println("Resetting the circuit")
    isOpen = false
  }
}

class CircuitOpenException extends Exception

In addition, for your own testing, you can create a simple HTTP server with netcat. Just run:

  echo -e "HTTP/1.0 200 OK\n\nHello\n" | nc -l 1234

It will quit after serving one request. So, you need to run it again to serve multiple requests.
Try to do it now! Then, click here to see the answer.

import java.net.URL
import io.Source

object HttpClient extends CircuitBreaker[Unit] {
  def main(args: Array[String]) {
    while (true) {
      print("Enter: ")
      try {
        readLine match {
          case "q" => queryHttpServer
          case "r" => reset
          case "x" => {
            println("Exiting")
            return
          }
          case _ => println("Unknown command")
        }
      } catch {
        case e: Exception => e.printStackTrace
      }
    }
  }

  def queryHttpServer {
    protect {
      //a Source object is a like a sequence of char
      val s = Source.fromURL(new URL("http://localhost:1234"))
      s.foreach(ch => print(ch))
      s.close
    }
  }
}

trait CircuitBreaker[T] {
  var isOpen = false

  //the => in the type for the codeBlock argument means that the codeBlock parameter
  //will be passed by name. That is, it will not be evaluated when passed. Instead,
  //whenever you refer to codeBlock in the method body, it will be evaluated on demand.
  def protect(codeBlock: => T): T = {
    if (isOpen) {
      println("Circuit is open, not calling the real thing")
      throw new CircuitOpenException
    } else {
      try {
        codeBlock //evaluate the code block on demand
      } catch {
        case e: Exception => {
          println("Got an exception, opening the circuit")
          isOpen = true
          throw e
        }
      }
    }
  }

  def reset {
    println("Resetting the circuit")
    isOpen = false
  }
}

class CircuitOpenException extends Exception

Scala exercise 3: decorator and composite design patterns

Introduction

This is the third  exercise in my Scala exercises series. If you haven’t seen it before, you may want to start from exercise 1: template method and exercise 2: observer design pattern. Below is exercise 3: decorator and composite design patterns.
Problem: In most UI frameworks including JSF, Wicket or Swing, you will need to provide a callback/listener object to handle requests from the user. Typically in such a callback, if there are some errors, you'd like to display a specific error message instead of propagating it to the framework, otherwise the framework would simply display a generic error to the user.
To hand code such a callback, you may do it like:

new Callback() {
  def onCallback(ev: Any) {
    try {
      //perform the business logic here
    } catch {
      case e: LoginException => {
        //assuming that error() will display the error
        error("failed to login") 
      }
      case e: SQLException => {
        error("error accessing the database")
      }
    }
  }
}

The problem with this approach is that there is a lot of boilerplate code there, while most usually we only want to say for exception class E1, display some error message M1:

new ErrorHandlingCallback(
  //classOf[Foo] is the same as Foo.class in Java
  classOf[LoginException], "failed to login",
  classOf[SQLException], "...") {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

But what if you'd like to extract some information from the exception and include it into the error message or would like to do something special? Then, ideally, you should be able to specify a function as the error handler:

new ErrorHandlingCallback(
  classOf[LoginException], "failed to login",
  classOf[SQLException], "...",
  (e: Exception) => doSomething(e)) {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

Finally, you should be able to pre-define an object to handle the commonly seen exceptions:

//ideally you should be able to "add" the error 
//handlers together to get a compound error
//handler
val defaultErrorHandler =
  (classOf[IOException], "I/O error") + 
  (classOf[Exception], "Unknown catch all error")

new ErrorHandlingCallback(
  classOf[LoginException], "failed to login",
  ...,
  defaultErrorHandler) {
  def performBusinessLogic(ev: Any) {
      //perform the business logic here
  }
}

Your task is to complete the code below and create the other necessary classes as needed:

trait Callback {
  def onCallback(ev: Any)
}

abstract class ErrorHandlingCallback(errorHandler: ErrorHandler) extends Callback {
  //overload the constructor to take multiple error handlers (the star does that)
  def this(errorHandlers: ErrorHandler*) = ...

  def performBusinessLogic(ev: Any)

  def onCallback(ev: Any) {
    ...
  }
}

object ErrorHandlerUtil {
  //allow you to use a function as an error handler
  implicit def fromFunc(f: Exception => Boolean): ErrorHandler = ...
  //allow you to use a pair (error class, error message) as an error handler
  implicit def fromPair(p: (Class[_ <: Exception], String)): ErrorHandler = ...
}

Then, the following code should compile and run:

object ErrorHandlerTest {
  //in order to use the implicit conversion methods, you
  //must import these objects so that those methods can 
  //be invoked without a prefix.
  import ErrorHandlerUtil._

  def main(args: Array[String]) {
    //assume that this is the default error handler in this context
    //classOf[Foo] is the same as Foo.class in Java
    val defaultErrorHandler = (classOf[IOException], "I/O error") + (classOf[Exception], "Unknown catch all error")
    //create a decorator to handle additional errors
    val decorator = new ErrorHandlingCallback(
      //convert a pair to an error handler
      (classOf[IndexOutOfBoundsException], "index out of bound"),
      //ditto
      (classOf[NullPointerException], "hit a null pointer"),
      //you can define a custom error handler using a function to, say,
      //access the info in the exception (not just its class).
      (e: Exception) => if (e.getMessage.contains("xyz")) {
        println(e.getMessage)
        true //indicate that it has been handled
      } else false,
      //specify the default error handler here
      defaultErrorHandler) {
      def performBusinessLogic(ev: Any) {
        println("called")
        ev match {
          //do nothing. No error.
          case "foo" =>
          //try to access the 100th element of an array which has only 3 elements
          case "bar" => Array[Int](1, 2, 3).apply(100)
          //Try to call a method on null
          case "baz" => null.equals("oops!")
          //throw a custom exception
          case "baz2" => throw new RuntimeException("I am xyz!")
          //divided by zero (something unexpected to test the ultimate fallback)
          case "baz3" => 100 / 0
        }
      }
    }
    decorator.onCallback("foo")
    decorator.onCallback("bar")
    decorator.onCallback("baz")
    decorator.onCallback("baz2")
    decorator.onCallback("baz3")
  }
}

Try to do it now! Then, click here to see the answer.

trait Callback {
  def onCallback(ev: Any)
}

trait ErrorHandler {
  def handle(e: Exception): Boolean

  //in Scala you can use symbols as method names
  def +(fallback: ErrorHandler): ErrorHandler =
    new ErrorHandler() {
      def handle(e: Exception) =
        //try to handle it myself first, if not handled, try the fallback
        ErrorHandler.this.handle(e) || fallback.handle(e)
    }
}

//implicit conversion methods must always be inside an object
//but not a class, so that the client can static-import the
//object's methods.
object ErrorHandlerUtil {
  //allow you to use a function as an error handler
  implicit def fromFunc(f: Exception => Boolean): ErrorHandler =
    new ErrorHandler() {
      def handle(e: Exception) = f(e)
    }
  //allow you to use a pair (error class, error message) as an error reporter
  implicit def fromPair(p: (Class[_ <: Exception], String)): ErrorHandler =
    //use _1 and _2 to access the elements of a pair (tuple)
    new ClassBasedErrorReporter(p._1, p._2)
}

//print an error message if the exception belongs to the specified class.
class ClassBasedErrorReporter(
        //the type means Class[E] where E is a subclass of Exception
        errorClass: Class[_ <: Exception],
        msg: String) extends ErrorHandler {
  def handle(e: Exception) = {
    if (errorClass.isInstance(e)) {
      println(msg)
      true
    } else {
      false
    }
  }
}

abstract class ErrorHandlingCallback(errorHandler: ErrorHandler) extends Callback {
  //overload the constructor to take multiple error handlers (the star does that)
  def this(errorHandlers: ErrorHandler*) =
    //the anonymous function _+_ is the same as (x, y) => x+y as each argument
    //only appears once in the body. Note that you have defined a method
    //named + above to combine two error handlers together.
    this (errorHandlers.reduceLeft(_ + _))

  def performBusinessLogic(ev: Any)

  def onCallback(ev: Any) {
    try {
      performBusinessLogic(ev)
    } catch {
      case e: Exception => {
        //handle the error
        if (!errorHandler.handle(e)) {
          throw e //if unhandled, re-throw it.
        }
      }
    }
  }
}

Scala exercise 2: observer design pattern

Introduction

This is the second exercise in my Scala exercises series. If you haven't seen it before, you may want to start from exercise 1: template method. Below is exercise 2: observer design pattern.

Problem: Complete a Scala trait Observed (shown below) to represent the subject being observed and the Scala trait Observer to represent an observer. The Observed object allows one or more Observers to register with it. Later, it can fire an event and notify all such Observers. The code is like (where E is the type of the event to be fired):

trait Observed[E] {
  def addObserver(o: Observer[E]) ...
  def notifyObservers(ev: E) ...
}

trait Observer[E] {
  def eventOccurred(ev: E)
}

Then use these traits to implement Java bean "bounded properties", e.g., to allow others to get notified when properties of a Book instance is changed:

//Let others observe changes to its properties
case class Book(var title: String, var price: Double) extends Observed[PropertyChangeEvent] {
  def setTitle(title: String) {
    val oldTitle = this.title
    this.title = title
    //Notify the observers
    ...
  }
  def setPrice(price: Double) {
    val oldPrice = this.price
    this.price = price
    //Notify the observers
    ...
  }
}

//A sample observer class
class Foo extends Observer[PropertyChangeEvent] {
  //Just print some info after a property has been changed
  def eventOccurred(ev: PropertyChangeEvent) = {
    printf("Foo: %s of %s has changed from %s to %s\n", ev.getPropertyName, ev.getSource, ev.getOldValue, ev.getNewValue)
  }
}

object BeanTest {
  def main(args: Array[String]) {
    val b1 = new Book("Scala programming", 35.95)
    val foo = new Foo
    b1.addObserver(foo) //Register the observer
    b1.setTitle("Thinking in Scala")  //foo should get an event
    b1.setPrice(39.95) //ditto
    b1.setTitle("Effective Scala") //ditto
  }
}

The above code should print:

Foo: title of Book(Thinking in Scala,35.95) has changed from Scala programming to Thinking in Scala
Foo: price of Book(Thinking in Scala,39.95) has changed from 35.95 to 39.95
Foo: title of Book(Effective Scala,39.95) has changed from Thinking in Scala to Effective Scala

Try to do it now! Then, click here to see the answer.

trait Observer[E] {
  def eventOccurred(ev: E)
}

trait Observed[E] {
  var observers: List[Observer[E]] = Nil

  def addObserver(o: Observer[E]) {
    observers = o :: observers
  }

  def notifyObservers(ev: E) {
    //for an anonymous function x => ...x..., if x only appears once in the function body,
    //we can turn it into an underscore and get rid of the formal parameter list, ie,
    //just write it as ..._...
    observers.foreach(_.eventOccurred(ev))
  }
}

case class Book(var title: String, var price: Double) extends Observed[PropertyChangeEvent] {
  def setTitle(title: String) {
    val oldTitle = this.title
    this.title = title
    notifyObservers(new PropertyChangeEvent(this, "title", oldTitle, title))
  }
  def setPrice(price: Double) {
    val oldPrice = this.price
    this.price = price
    notifyObservers(new PropertyChangeEvent(this, "price", oldPrice, price))
  }
}

Scala exercise 1: template method design pattern

Introduction

As part of my studying with Scala, I have tried to find some exercises to do but most are either too simple or too "academic". So, I decided to create a series of Scala exercises to implement the well known design patterns in an industrial context. Here is the first one: template method.

Problem: Create a Scala class JdbcTemplate that has two template methods: execute() and load(). They will execute a SQL statement and load the query results using JDBC respectively. The methods should open and close the connection, create and close the statement and etc. so that the caller doesn't need to worry about those. For example, the client could use your code like this:

object Test {
  def main(args: Array[String]) {
    Class.forName("org.h2.Driver")
    val t = new JdbcTemplate
    val ds = () => DriverManager.getConnection("jdbc:h2:~/test")
    t.execute(ds, "drop table products if exists")
    t.execute(ds, "create table products (id long primary key, name varchar(128))")
    val insertSql = "insert into products values(?, ?)"
    t.execute(ds, insertSql, List(100, "p1")) //the list is the parameters
    t.execute(ds, insertSql, List(101, "p2")) //ditto
    val ps = t.load[Product](ds, "select * from products", Nil,
      { //this function maps a record into an object
        (rs) => new Product(rs.getLong("id"), rs.getString("name"))
      })
    println(ps)
  }
}

//make it a case class so that it can be printed to the console nicely
case class Product(val id: Long, val name: String) {
}

To make it run, you need to include the h2 database into your classpath. If you use Maven, just add the following to pom.xml:

       <dependency>
             <groupId>com.h2database</groupId>
             <artifactId>h2</artifactId>
             <version>1.2.143</version>
       </dependency>

Note that I am not suggesting that you do this in production code; you'd probably want to use an ORM framework.

Click here to see the answer.

import java.sql.{DriverManager, ResultSet, PreparedStatement, Connection}

class JdbcTemplate {
  def execute(ds: () => Connection, sql: String) {
    execute(ds, sql, Nil)
  }
  //Use List[Any] instead of List[Object] so that elements can be primitives
  def execute(ds: () => Connection, sql: String, params: List[Any]) {
    execute(ds, sql, params, (st: PreparedStatement) => st.executeUpdate)
  }

  def execute(ds: () => Connection, sql: String, params: List[Any], executor: PreparedStatement => Unit) {
    val con = ds()
    try {
      var st = con.prepareStatement(sql)
      try {
        //zip (a, b, c) with (0, 1, 2) to get ((a, 0), (b, 1), (c, 2))
        params.zipWithIndex.foreach {
          //case p1 => e1 can be seen as an anonymous function: x => x matches { case p1 => e1 }
          case (value, idx) => st.setObject(idx+1, value)
        }
        executor(st)
      } finally {
        st.close
      }
    } finally {
      con.close
    }

  }

  def load[A](ds: () => Connection, sql: String, params: List[Any], objMapper: ResultSet => A): List[A] = {
    var result = List[A]()
    execute(ds, sql, params, (st: PreparedStatement) => {
      val rs = st.executeQuery
      try {
        while (rs.next) {
          result = objMapper(rs) :: result
        }
      } finally {
        rs.close
      }
    })
    result.reverse
  }

}