How the Models are Connected to the Web

This page describes the Web programming side of the Virtual Economy model, and the techniques that we used to handle the interaction. Unlike the rest of the Virtual Economy site, it is mainly concerned with computing. We hope it will interest those who want to know how we connected the models to the Web, or who want to do something similar for themselves. Much of this page will be intelligible to the non-specialist, but in a few places, we have gone into some detail about the software we've used and the bugs we've found, for the benefit of the more advanced implementor.

• The Virtual Economy model's history and predecessors
  • Be Your Own Chancellor and Budget95
  • Tow - Teaching economics On the Web
• The Virtual Economy model's Web technology
  • Java and servlets
  • How the Virtual Economy model uses servlets
  • Graph plotting
  • Improving plotting speed
  • Servers and servlet adaptors
  • Program testing
  • Connecting the microeconomic model to the Web
  • Connecting the macroeconomic model to the Web
  • Spreadsheets and debugging output
  • The File Vulture

The Virtual Economy model's history and predecessors

Be Your Own Chancellor and Budget95

The Virtual Economy evolved from earlier work on connecting economic models to the Web, done at the Institute for Fiscal Studies. The IFS was set up to promote public understanding of fiscal policy, and for many years has published printed research papers. By 1994, it was obvious that the Web would be an important alternative for making information available to the general public, and the IFS set up its own Web site, holding on-line copies of many research papers.

In 1995, the IFS took the next step. Shortly before the November 1995 Budget, we were experimenting with Web browsers as an in-house front-end to the IFS tax-and-benefit model Taxben, to replace an old DOS menu interface. The idea was that users would submit data to it by filling in HTML forms, and it would send back results as dynamically-generated Web pages. This seemed a wonderful idea: we would not have to write a special interface for each type of machine; we could use hyperlinks to point at help text; and the Web browser would handle the complicated windowing stuff, so we would have much less programming to do.

We then realised there was no need to keep this in-house, and the idea surfaced of a Web-based Budget analyser. Anyone with a Web browser would be able to submit details of income and expenditure via an HTML form. The model, primed with the pre- and post-Budget tax systems, would return a breakdown showing how the Budget affected their Income Tax and National Insurance (based on income, broken down into earnings, pension, etc) and their VAT and other indirect taxes (based on expenditure on food, alcohol, and other consumables), The post-Budget parameters would be programmed in just after the Chancellor's speech, and the model would go live at 6pm the same evening. In the event, we went live early, and our server performed impeccably, handling a maximum of 450 model runs per hour without crashing or slowing down appreciably. We update Budget95 each year, and it is due to become Budget99 on March 9th this year (1999). The BBC asked us to implement it on their News Online site for 1998, and we shall be doing the same again this year.

To prove our techniques, we implemented and released another model a few weeks earlier, thus giving us some breathing space to detect and fix bugs. This was our first on-line model, "Be Your Own Chancellor" (BYOC), where the user submits a "Budget" much as for the Virtual Economy model, but without the macroeconomic variables. Although connecting programs to the Web is old hat now, it was something really exciting then, and BYOC received some very favourable comments, including a nice review in the November 1995 Computers in Higher Education Economics Review.

Tow - Teaching economics On the Web

Budget95 and BYOC are intended for the general user. After finishing them, we spent a year looking at how the Web might be used for teaching economics - in particular, labour-supply economics - to school and university students. The result was Tow (for "Teaching economics On the Web"). One of the main differences between it and the earlier models was that it was session-based. This means that users could continue their interaction over a series of transactions, saving their input data between each, in the same way that (for example), on-line shopping services work, where the state of each user's order is saved on the server between transactions. Tow also allowed users to store their Budgets and sample families in a database on our server, and to reuse them days or even weeks later. A paper on Tow, showing some screen dumps, is available as TOW: a system for teaching economics on the Web in zipped Word format.

One innovation was Jocelyn Paine's Web-O-Matic (WOM) authoring tool for interactive Web pages, described in Web-O-Matic/Rexx: a tool for building complex interactive Web applications by compiling HTML to Object Rexx. We have been using a Java version of this to build the Virtual Economy model. WOM enabled session-based applications to be written easily - again, this is old hat now, but was novel then.

Tow was a prototype and was never made public. This was partly because of efficiency problems with the Object Rexx language that we programmed the Web side in. Basically, the language was much worse supported by its implementors, IBM, than we were lead to believe it would be. Some of our problems are described near the end of the Web-O-Matic/Rexx paper, and make a dire warning on the perils of using badly-supported non-portable programming languages. (This is not to slight Object Rexx itself: we believe there is now an efficient version available for Linux.)

The Virtual Economy model's Web technology

The principles behind the Virtual Economy model are similar to those behind BYOC. When the user submits the input form, the data is sent to the server, which passes it to both modelling programs. These write their results into temporary files: the server reads these, reformats into HTML where necessary, pastes the chunks of HTML together into a complete page, and sends this back to the browser. We say more about the problems of connecting the micro and macro models in later sections.

Java and servlets

We implemented BYOC using CGI scripting. This is a technique which causes the Web server to run a "script" (a program written either as a batch file, or in a special-purpose scripting language such as Perl) when a certain URL is requested. This script can do anything a batch file can do, including running a program (such as a model) and sending back its output. For a non-technical account of how this works, see Jocelyn Paine's How to connect existing educational programs to the Web: a simple guide.

Scripting languages like Perl have lots of facilities for string- and file-handling, and can be very powerful. However, they also have disadvantages. For example, the Perl compiler does not do type-checking, and Perl does not require variable declarations. This means that errors such as adding a number to a string, or using a misspelt variable name can pass the compiler by, only to cause havoc at run time. For these reasons, and also because we had the Java version of WOM available, we decided to use Java servlets instead.

For those unfamiliar with Java, it is a general purpose object-oriented programming language. It came to prominence in 1995, when its designers, Sun Microsystems, promoted it for writing applets - programs that can be downloaded from a Web server and run on a Web browser. This is "client-side programming" - providing the user with downloadable Java code that will run on their browser. However, the Virtual Economy model uses Java not for client-side, but for server-side programming. Here, as with CGI, the Java program runs on the Web server, receiving data from the user and generating HTML pages, graphs, and other content, which it sends back for the browser to display. The Java programs that do this are termed "servlets", by analogy with applets.

Java is less error-prone than CGI. Another advantage is that servlets can call Java code from anywhere else. There are many excellent Java libraries available - some commercial, some free- or share-ware - and using them can make one's servlets much more powerful. We bought one such library, Visual Engineering's JavaChart, for plotting our graphs, and recommend it, and the support Visual Engineering offer.

Another advantage is portability. Java has been designed to run almost all programs unchanged on a wide range of machines, and achieves this goal much better than did previous languages such as C++ and Fortran. And, although there are now some excellent Perls for most common systems, such as ActivePerl for Windows (available via http://www.perl.com/) Java is still more portable. So this will make it easier for us to move our code from one server to another.

How the Virtual Economy model uses servlets

In the Virtual Economy model, we use two servlets. One - ModelOutput - receives the string of data sent from the input form and returns the "Model Output" page which you see on running the Virtual Economy model. To do so, it translates the user's menu selections as described in the section below on WOM and menus, and writes them into a temporary file. It then calls both the macro and micro models, picks up the results, converts into HTML where necessary, pastes these pieces of HTML into a complete output page, and returns that to the server.

Graph plotting

The other servlet handles graph-plotting. When ModelOutput needs to place a graph in the output page, it cannot just include the binary data making up the image, because HTML does not work that way. Instead, it needs to place an HTML IMG tag, which contains another URL pointing at the graph. Normally, this would point at a temporary file holding the graph: for example,

<IMG SRC="/scratch/graph_02758.gif">

However, the URL need not point at a file. Instead, it can refer to a CGI script or servlet which takes the graph's data points and generates the image on demand. This is what we do. ModelOutput generates a URL containing the data points, axis annotations, and other graphical parameters. The URL points at our plotting servlet. When the server receives this URL, it passes the data in it to the servlet, which decodes them and calls JavaChart to generate the graph as an image file, which is then sent back to the browser. A typical URL would look like this:

<IMG SRC="/servlet/Plotter/xs=1999;10&ys= 1,1.2;2,2.4;3,3.75&unit=Percent">

Here, xs gives the starting year and increment, and ys the before- and after data points. unit is the y-axis annotation. This technique was suggested to us by Damion Yates of the BBC.

Improving plotting speed

One problem we faced with the plotting servlets was lack of speed. It took about two seconds to generate each graph (each 300 by 300 pixels), leading to at least 16 seconds for the output page, not counting other delays. We tried an alternative plotting library, KL Group's JClass Chart. Like JavaChart, this library has lots of features, and we would recommend it for many purposes (KLG also sells excellent tabulation and windowing libraries). However, for our graphs, JClass Chart turned out to be no faster than JavaChart. Both libraries create the graphs by drawing them into an instance of the class java.awt.Image and then converting that to a GIF file; discussions with Visual Engineering indicated that this conversion is what produces the delay.

We therefore followed Jan Chipchase's advice, and headed the output page with eight thumbnail graphs, each 40 by 40 pixels. These plotted more rapidly, and can be loaded with little delay, enabling the student to see some output even though the rest of the page may still be loading. Each graph is an image link to its big brother. Jan has pointed out that the thumbnails also let students see immediately the general trend resulting from their changes, without being overloaded with details.

Just before releasing the Virtual Economy model, we moved from Java 1.1 to 1.2, having heard that it might be faster. This in itself did not help. However, Dave McMurdie from Visual Engineering has been trying some experiments, and found that Java 1.2 comes with a JPEG encoder that translates instances of java.awt.BufferedImage directly into a JPEG image. On his machine, this was able to create 500 by 300 pixel images in about 0.1 second. By writing our graphs into a BufferedImage rather than an Image and using the JPEG encoder, we greatly improved our speed, and we're very pleased with Visual Engineering's advice. It appears that Sun's JPEG encoder is not available on all Java platforms (it does exist for NT and Solaris); what would be really nice is a portable GIF encoder.

Servers and servlet adaptors

All Web servers implement CGI, but only a few, such as Sun's JavaWebServer can run servlets directly. Some servers that can't can still be equipped with "servlet adaptors", which enable them to call servlets. We run the Virtual Economy model on Windows NT 4, using Netscape's Enterprise Server, bought from Accord IT, coupled with New Atlanta Communication's ServletExec servlet adaptor.

Enterprise is fast, and Accord have been able to provide us with local support. We considered several different servlet adaptors - New Atlanta's has all the features we needed, and the company, though small, replies quickly to technical queries, and is extremely helpful. (For this and the graphics libraries, we have had to make several phone calls to the US. It may be helpful to know that the "phone-on-account" companies such as First Telecom make this possible for about 10p per minute.)

We considered other servers. Since Sun invented Java and servlets, their server ought perhaps to be the best. However, when we tried the evaluation download, it was slightly more sluggish than Netscape's (this may not matter in practice), and more expensive. Sun also failed to reply to any of our technical evaluation emails, whereas New Atlanta answered the first one in under an hour, and have been consistently prompt ever since. We believe Sun may now have changed their pricing, making the server free to educational organisations, and we do have a high regard for the quality of most of their software - our main quibble is with their poor response to support queries.

We also considered the freeware Apache server. At the time however, Apache did not support servlets. It does now, as part of the Apache JServ Project, and we would certainly consider it if starting again.

Finally, it is worth saying that although fast, Enterprise can be temperamental. When developing servlets, you need to stop and restart your server before it will pick up new versions of your classes: this is because it caches them once loaded. Unfortunately, Enterprise sometimes hangs when you try to restart it, making it necessary to reboot the machine. There are other oddities too, including a bug in its CGI-handling, which makes it necessary to add and then delete a dummy directory before it will agree to process CGI requests.

WOM and menus

We use WOM to make writing our interactive pages easier. To describe an interactive page to WOM, you write in an extended version of HTML, augmented with Java code and special constructs that describe how the page should react to input from the user. WOM compiles this description into Java code that runs on the server.

In this version of the Virtual Economy model, our main use for WOM is in writing menus. All data entry on the input form is by menus rather than text fields, because this means we do not need to send back error messages (particularly important for students, some of whom will delight in trying to break programs by typing in negative numbers and other junk data). To provide sufficient range in the variables they denote, the menus need a large number of closely-spaced options. For example, our Income Tax Rate menus vary from rates of 0% to 50%. WOM provides shorthands for specifying sets of options as loops, allowing the author to write simply

<IntegerMenu Id="TaxRates[1] Options={/ 0 to 50 /} Data="20">

rather than fifty-one HTML statements.

In some menus, text strings such as Cut by 1% (for the Government spending menu) are more helpful then numeric codes. WOM allows the page-designer to write these in the options list, together with a numeric translation:

<IntegerMenu Id = "GovCur"
Options = {/ "Cut by 1%" : -1,
"Leave unchanged" : 0,
"Raise by 1%" : 1
/}
Data = "Leave unchanged"
>

The user will see the text strings in the input form, but WOM will generate code which translates these to the numbers when they reach the server.

Like most of today's authoring tools, WOM also provides facilities for writing session-based applications. We used these in our prototype version of the Virtual Economy model, which allowed students to define their own tax systems and save them in our database. However, they aren't present in the final version - we wanted to find out how students used the simple interface that we've implemented, before proceeding to anything more complicated.

Program testing

The Virtual Economy model has a very large number of combinations of inputs. There are 54 menus, and if we assume about 30 options per menu, this makes 30⁵⁴ - about 5 × 10⁷⁹ - possible combinations. It is obviously impossible to test these all! However, we do need to test as well as we can.

WOM makes this easier. It translates menus into Java data structures, which can then be accessed by any Java program. We have written a little testing program which examines each menu and gathers the two extreme values, the default, and the values either side of the default. Having done this, it uses them to exercise the model, running through each menu, keeping all other menus at their default value, but varying the one being tested through these five possibilities. While not exhaustive, this method of testing does test extreme inputs, which will often flush out errors.

The testing program has acess to our servlets, and can grab their HTML output and encapsulate it as a series of Web pages, each describing one test, the test's input, and the results. This makes it easy to view test results and to pass them to our collaborators. We could automate further by writing code that scanned the generated pages for strings such as "Exception" (which the server will send back if a run-time error occurs). Graham Stark has also written a check, built into the output of the macro model, that all output variables are within reasonable range. If this fails, it writes a warning message into the model output page. This can then be detected by our tester.

We also need to run load tests, to ensure that our server can cope with many simultaneous users. Such tests are also needed to pick up gradual memory-leak errors. These might not crash the server immediately, but could slowly degrade its performance over time. We are looking at various commercial Web testing programs for this - our favourite so far is OCLC's WebArt.

Connecting the microeconomic model to the Web

Taxben, the IFS microeconomic model, is written in Modula-2, using the XDS Modula compiler - a high-quality compiler generating efficient code. One can be glad that the compiler is reliable - it was developed for USSR defence applications such as missile control. With the break-up of communism, the implementors formed a company to sell it, and the IFS has been using it for a number of years.

To communicate with Taxben, the ModelOutput servlet writes the user's data into a scratch file. The servlet then constructs a DOS command line to run Taxben, passing the name of this file as a command-line argument. It uses the exec method to start a process which will run this command. If you look at the model output page, you will see that there are two pieces of output from Taxben - the summary and the family details - separated by some other stuff. Taxben writes each of these out as HTML, into its own scratch file. The servlet then picks up these files and copies them into the correct place in the output page.

This is all simple enough. However, we are beginning to suffer from problems caused by the number of different page designs that Taxben must handle. Taxben is used not only in the Virtual Economy model, but on the BBC News Online site, and also on the IFS' own site. Each organisation has its own ideas about how the output pages should look. Some of the designs, such as the Virtual Economy's, are also quite complex, with heavy use of stylesheets, and fine-tuning for effects like cell spacing in tables. Trying to accomodate all these designs in one program is becoming overwhelming. In the next version of the Virtual Economy model, we shall probably rewrite Taxben so that it just emits a set of output values, and use a separate piece of software to substitute these into HTML templates for the output pages.

Connecting the macroeconomic model to the Web

Strictly speaking, the macroeconomic model is not the program, but the equations that predict the economy's future state from its past history and the user's changes. These equations can be run by any program capable of solving them. We tried three: WinSolve, Time Series Processor, and the Fair-Parkes modelling program.

Our first attempt used Richard Pierse's WinSolve, However, we found that although it performed well when called from the command-line or from a stand-alone Java program, it crashed when called from a Java program running under our Web server. The reasons for this remain obscure. However, discussions with New Atlanta suggest that it may be related to the way that Enterprise calls ServletExec. This is done using an "NSAPI" interface, in which ServletExec and any programs it is running live in the same process space as the server. There is an alternative interface, "WAIS", which runs them as separate processes, and which should therefore reduce interference. New Atlanta have offered us a special version of ServletExec which runs this way (something not normally available under NT), but we have not yet tried it.

We next tried the Time Series Processor. Here again, the program worked perfectly when called from a stand-alone Java program, but crashed when run from one running under our Webserver. TSP is written in Fortran, and uses DOS memory-extender calls to extend its memory beyond the 64K allowed by DOS. These may be interfering with the server's memory allocation, but again, we have not followed the matter up.

Finally, we tried the FairModel modelling program This is written in standard Fortran, and does not use any special operating system calls or other features, so we hoped that we could get running under our server without problems. Indeed we were able to - but only under Microsoft's Fortran compiler. When we tried compiling it under the freeware G77 compiler, it crashed. This is probably because it does disgusting things with types, for example declaring a routine that expects integer arrays as arguments, and then passing it real arrays. Our G77 compiler settings were probably violating some obscure assumption about Fortran 66 storage conventions upon which FairModel depends.

Presumably because Fortran lacks dynamic storage, FairModel restricts file names to 16 characters. This was an irritation. We need to generate scratch files to hold model input (FairModel has its own command language, and we need to generate a driver file in this language for each run) and output: to avoid files for one transaction getting mixed up with those for another, the names must be unique. One can generate unique names from the date and time, but only if two transactions can never occur within the same clock interval. On our machine, this is 10 milliseconds, which is not quite short enough. Instead, we use unique "session identifiers" generated by the Web server (available by the HttpSession.getId() method). However, these are long, and overflowed FairModel's filename limit.

We circumvented this by generating a scratch directory with the same name as the session identifier, and changing the current directory so as to run FairModel from under that. This meant that it could refer to its files via relative names, which need not contain either session identifiers nor directory names, and so could be very short. Another minor inconvenience was that Java does not provice any means to change current directories (i.e. no equivalent of the cd command). We circumvented that by invoking a Perl script from Java. Perl does know how to change current directories, so the script did that and then ran FairModel.

One final difficulty was that FairModel has its own output format which we were unable to modify. It returns results as blocks of space-separated numbers, giving the values of the macroeconomic variables over time. The servlet has to read these and remove headings and other extraneous text, before outputting them as HTML tables and graphs. Reading this stuff is a non-trivial task, and we used the freeware JavaCC parser-generator to make it simpler. JavaCC, freely available from http://www.metamata.com/javacc/index.html, is an extremely useful program that reads a grammar and generates Java code for parsing according to it. In our case, the grammar is that of FairModel's output file.

Spreadsheets and debugging output

The bottom of the the Virtual Economy model output page contains three rather inconspicuous links in a section headed "Downloadable results". These are the outputs from each model, and the user's inputs, written out as lists of numbers separated by commas. This so-called comma-separated value format can be loaded into most spreadsheets, including Lotus 1-2-3 and Excel, and gives a way for users to use the data in other calculations. These three pages are just held under another scratch directory created under our document root. As with the FairModel files, there is one unique directory for each run, its name derived from the session identifier.

the Virtual Economy model also writes out a fourth file, containing debugging output. This is just an internal trace of the programs, showing such things as the user's data, the menu-option translations, and the command file generated for FairModel. One would often write such stuff to the standard error stream, from which it would emerge into the Web server's error log file. However, putting it out into a new page means it can be examined from any browser, enabling us easily to check that our code has correctly generated the FairModel command file parameters, for example.

The File Vulture

With all the temporary files generated by a run, both the pages mentioned in the last section and the scratch input and output files, it is essential to do a regular clearup. We have a separate Perl program, the "File Vulture", for this. It awakens every five minutes and examines all the files in the scratch directories, deleting any which are older than a certain age, currently six hours. This delay gives ample time for users to examine and collect their results. We weren't able to write the File Vulture in Java, because there is no way to determine the age of a file.