Adding Interactive Services To Your Web Server

Look around the World Wide Web, and most of what you'll find today are online equivalents of printed propaganda. There may also be some downloadable files or imagesæas well as the seemingly requisite overabundance of hypertext links to every imaginable corner of the Internet. But, for the most part, Web servers are about as interactive as their paper-based counterparts.

While getting to this level of highly-functional-yet-somewhat-interoperable online publishing can seem miraculous to many, it is far from where these systems need to be if they are ever to be something more than an outlet for marketing's creativity. In order for the Web to become the next-generation platform for distributed computing, it must go beyond read-only into the world of interactive. Users should be able to add, edit and delete data in real time, as security and application locking allow.

We'll outline some of the concepts and technologies to help you get there today. We'll also look at some basic examples, and provide pointers to other materials where appropriate and available. By no means can we cover everything. We will attempt to give you enough information so that you can get started in the right direction.

"What kinds of applications can be called from the server, and where can they be found?" These are the two most common questions asked at this point. Before we can answer these questions, we must first look at how the server and applications communicate.

The Common Gateway Interface (CGI)

Most, but not all, Web servers have two basic methods of handling back-end data. They either read a file or communicate with other programs through the Common Gateway Interface (CGI). It's up to the application to figure out what to do with that data, and to return HTML to the server. This is an extremely simplistic overview of the process, and there are many possible exceptions and extensions.

Examples of CGI can be seen on many servers today. Clickable image maps such as those found on Network Computing's home page use CGI. The client sends an HTTP request containing the URL and XY coordinates to the server, which passes the coordinates to a local application. The application executes an event that has been associated with those coordinates and returns HTML to the server for delivery to the client. In the case of image maps, most of the time this HTML data consists of an HTTP redirect command that instructs the client to request another document from the servers.

Another common use of CGI is for searching through files or databases (Yahoo and WebCrawler are good examples). A user constructs a query that is sent to the server, which then spawns an application that conducts the actual search and returns the "hits" in HTML form.

So, to answer the question about the kinds of applications that can be called by the server: They include any application that supports CGI on the front end and can generate HTML responses. In fact, you can call almost any program you want, as long as it supports command-line operation.

However, the unfortunate truth is that no two Web servers are alike. They have all been developed with specific audiences in mind and "enhanced" to suit those markets. This means that just because something works with server brand X does not mean that it will work with server brand Y. For example, there are considerable differences between the Netscape server's CGI interface and the O'Reilly and Associates WebSite server's CGI interface, even though they both use Windows NT's CMD.EXE command interpreter. The differences between the various Unix shells, Digital's DCL, DOS' COMMAND.COM, and all other platforms, can also impact what you want to do.

CGI Detailed: GET Vs. POST

One of the most common methods of implementing CGI is through the use of GETs and POSTs. In the "Clickable Image Map" figure, the client submitted a URL with a question mark and XY coordinates appended to the end. This method of CGI call is known as a GET. Not only are GETs the oldest, but they are also the default and easiest to implement. Another form of call, known as POST has become more popular, and is generally considered to be much more powerful. The differences between them are subtle, but significant.

The GET method is named for what the client does. It asks the server to GET the HTML document at HTTP://URL?Param. The server sees that the URL points to an executable, calls it and passes it the parameters.

In contrast, the POST method is used when the client submits a block of data to the server. The client is not necessarily assuming that anything will be returned to it, although the connection is still open. In most cases, the server's Webmaster has configured things so that the client is given a thank you message, or response data is sent to the client.

CGI Examples

Let's walk through an example using O'Reilly and Associates WebSite server, since its use of Windows NT's CMD.EXE makes it easy to write simplistic demonstration scripts. WebSite can also run on Windows 95, so it's easy to use on a personal system for learning purposes. Besides, you can evaluate it free for 60 days. It's available at http://website.ora.com.

Once the WebSite Manager is downloaded, use the administrative tool to locate the directory for DOS executables (C:\WEBSITE\CGI-DOS is the default). Using notepad or some other text editor, create a file called DIRSEARCH.CMD in that directory and carefully copy the contents of script contents shown at the bottom of this article and save the file. Using a Web client, connect to your server and point to http://<servername>/DOS-CGI/DIRSEARCH.CMD . Assuming that your server is configured to allow your client to access DOS-based CGI scripts, you should get an HTML page stating No Files Specified. Don't worry. This is what we want! Connect again and this time add ?command.com to the end. You should see a standard directory listing of all the files named command.com. DIRSEARCH.CMD is a simple NT batch file that scans your server's C: drive for any occurrence of a specified file and returns the directory list to the user. If a file is not specified, then an error message is displayed. We'll extend this program later, but first we need to examine it in detail.

Always Test the Data First

The first line in the box below is a parameter test to make sure that a user has supplied a filename. If "%1" is blank (that is, does not exist), then the batch file jumps to the :errlabel label, which returns an error message to the user. Any seasoned programmer will tell you that it's important to test your program inputs before acting on them, and it is even more crucial here for several reasons.

Most important is an open issue regarding system security. There are known weaknesses with many operating systems that allow shell metacharacters to break your script or a called application, thereby exposing your system. It sounds far-fetched, but it happens all the time. By adding keystroke traps to your script, however, you can go a long way toward preventing this from happening. Refer to the CERT archives for more information on this subject. This weakness is not prevalent on all systems, but it is on many of them.

Another good reason to test the parameters is for program control. As we'll show below, you can use this function to minimize your development. Finally, there's a programming reason to test the input: to make sure that the data is valid. If instead of conducting a directory search, you queried a database, you would want to make sure that all of the necessary information had been provided. Testing the data in a script is more efficient than letting the called application fail. You would still have to test for errors, so you might as well do it up front.

Define Your MIME Type

The use of MIME types to coordinate data exchange is now part of the HTTP specification, as of version 1.0. As you can see, both the :goodlabel and :errlabel sections begin with MIME type declarations echoed to STDOUT.

For our purposes, we are simply declaring that the following data is HTML. The Web client interprets this and knows to read and apply whatever HTML formatting tags it sees in the data. If the MIME type had not been declared, then your browser may or may not interpret the HTML tags.

As you may have guessed, you can also send binaries or other nontext data back to the client. For example, if you wanted to return a GIF image, you would define the MIME content-type as Image/GIF. The client would then receive and display the image appropriately. Likewise, you could send application binaries by using the Application/Octet-stream MIME type, and so on. Note the echo statement that follows the MIME declaration. You must send a blank line after you send the content-type: statement.

The Rest of It

The remaining code is fairly self explanatory. There are, however, some things worth noting. Notice that the < and > characters are proceeded by ^. This is to prevent CMD.EXE from interpreting them as redirection commands. Also notice that DIR C:\"%1"/s is the actual command that we are calling. The remaining lines of code are used to prepare the environment and command line, as well as generate HTML on its behalf.

Image map servers and dynamic directory builders are two of the more common CGI-aware applications that you'll find. Additionally, there are a variety of third-party tools, such as database requesters and indexed searching agents, available on the Internet. If you wish to write your own, you can apply the same concepts we've shown you to a compiled language.

Adding Support for Forms

In its current form, DIRSEARCH.CMD must be given a parameter on the URL's command line. This is far from intuitive, and if you made your users work like this, you'd have very little traffic on your site. Most users prefer to enter information on a form, and so do administrators, since the level of control increases tremendously.

The easiest way to build a form is through the use of the <ISINDEX> HTML tag. This simple tag has the delightful affect of putting an edit box widget on your page. When a user enters text into the edit box and presses the enter key, the text is appended to the URL (with the question mark), and the page is reloaded. At this point, a valid search string exists, and the script will process accordingly.

For our example, change the line in DIRSEACH.CMD that reads ^<H1^> No Files Specified^</H1^> to Enter a filename to search for: ^<ISINDEX^>. Now point your Web browser to /BIN/CGI-DOS/DIR SEARCH, and you will be greeted with an edit box. Type in COMMAND.COM and press enter, and the same script will now branch through the :goodlabel section, and return a directory listing.

Don't be confused by the name ISINDEX, it's an unfortunate moniker for such a flexible gizmo. This tag does not automatically make your documents searchable; some other tool (such as DIR, in our case) must do the searching if that's what you want. The <ISINDEX> tag simply provides an edit box whose contents are appended to the current URL.

Just as the <ISINDEX> tag is poorly named, so is it poorly treated by most browsers. Many of them prepend the edit box with an inane comment about the document being searchable, which is just plain wrong. In order for us to have an attractive form that doesn't confuse people, we'll have to use real forms.

Creating Real HTML Forms

Form creation with HTML is a fairly simple task, and as we've shown, passing data with CGI is also straightforward. When you combine these two components' simplicity and strength, tremendous results can develop.

In order to illustrate this, we need to modify our sample code. Edit DIRSEARCH.CMD once again, and replace the <ISINDEX> tag with the code shown in the "ISINDEX Replacement" at the bottom of this page.

Save these changes, and reload our now familiar URL. Now, instead of seeing an <ISINDEX> input field, you should see a real HTML form. When you enter a file name into the edit box and click on the "submit" button, the script will process the directory search and return the list of matching files.

Let's examine this syntax in detail. The line that contains the HTML tag does what you might guess: It notifies the client that the following text contains form elements, until a closing tag is encountered.

There are additional keywords that can be used with the tag, which provide greater amounts of control. One of these is the Action= statement, which tells the browser where to send the form contents. By default, the destination is a URL but it could be any CGI application available to the user. For companies that have several distributed Web servers, but only one or two CGI servers, this is a great way to split the load. Also of importance is the Method= keyword, which allows you to specify that the query data is to be submitted using the GET or POST methods as described earlier.

Widget Lingo

Another keyword, the input tag, has many options, but it basically applies to almost every type of form widget available. This includes edit boxes, radio buttons, check boxes and command buttons. The only types of widgets not controlled by <input> are list boxes (either drop-down or scrollable) and large, multi-line text boxes, both of which have their own specific tags.

In our example, the first occurrence reads <Input Name="FILENAME">. This syntax will create an edit box with the resource name of Filename. The Name= command associates a name with the control data. After successfully running a query, look at the client's URL command line. Notice that where it used to have ?command.com, it now has ?filename=command.com. One final thing to remember about the Name= option is that it does not apply to command buttons (you'll see why in a moment).

Notice that we did not explicitly declare the edit box as such. That's because the default of <Input> widget is an edit box. If we chose to do so, we would declare it using the Type=Text attribute. Other valid Type= declarations include Password fields (like edit boxes, only keystrokes are echoed as asterisks), CheckBox, Radio, Submit and Reset.

Additionally, we could preload a string into the edit box by using the Value= statement. This can also be used with password fields, as well as checkboxes, radio buttons and command buttons, although its usage varies widely among them.

The default width (in theory, at least) of an edit box is 20 characters, and the height is one. You can define different sizes using the Size= directive. When setting the width, use Size=W. When setting the height, use Size=W,H. Note that setting the width does not restrict input to that amount of characters, but instead defines the width of the box itself.

One problem with the GET method is that text is appended to the URL as entered. If you let the user enter as much as they want, you run the risk of script or called application failure due to overflown buffers. The Maxlength= attribute is useful for limiting the maximum number of characters that an edit box will accept, thereby limiting your system's exposure.

Altogether, our simplistic <Input Name="Filename"> example could (and probably should) read as <Input Type="Text" Name="Filename" Width= 20 maxlength=20>. Note the use of quote marks around the textual values; this is an HTML requirement that needs to be followed for maximum compatibility.

Command Buttons

Command buttons are not nearly as complex as the edit boxes; there are only two possible types. Our example uses the Type="Submit" variety, which builds the parameter list and appends it to the URL specified by the <Form Action=....> tag. The other type of command button is Type="Reset", which clears the form and resets all of the widgets to their original state.

The only optional attribute that can be set for command buttons is Value=. In this case, the Value= attribute defines the text that appears on the button's face. It does not change the button's behavior. For our example, we could change <Input Type="Submit"> to <Input Type="Submit" Value= "Search">, and add another button that cleared the editbox by using <Input Type="Reset" Value="Clear">.

And Then Some

You should experiment with the different types of widgets and their optional parameters. Together they provide a fairly comprehensive set of tools for getting user input into your CGI scripts and applications. A good starting place is http://www.ncsa.uiuc.edu/ SDG/Software/Mosaic/Docs/fill-out-forms/ overview.html, and by searching Yahoo's (http://www.yahoo.com) database for "HTML and Forms."

Anyone familiar with other types of graphical forms-based tools will tell you that these gizmos, while handy, are hardly sufficient. New extensions have been added to the HTML drafts that address some concerns, but not all. Vendors are stepping up with products that address these concerns, however, so you should be able to find something that suits your need.

On the back end, Lotus Development and Netscape have both promised to ship HTML filters for their respective groupware products, allowing you to leverage in-form value testing, mathematical functions and the like. How these extensions will be handled by generic clients is anybody's guess. But it is a step in the right direction.

As for the clients, Netscape Navigator 2.0 and Oracle's PowerBrowser clients both promise to offer event-driven scripting language extensions that will allow Webmasters to embed extended controls into the forms directly. Although these clients will perform the functions as directed, it means more work for the administrators, since you may have to support multiple proprietary extensions. Again, it's a step that will drive further standards-based development.

GET vs. POST Revisited

We have been using the GET method to pass data down to our script. The POST method, which offers more capabilities, deserves some attention as well. To see a quick demonstration, change the <Form...> line in DIRSEARCH.CMD to read <Form Method="POST" Action= "/BIN/CGI-DOS/DIRSEARCH">, and reload the form again. This won't change the behavior of your script, but you may receive a security notice from your client.

Remember, we've been using O'Reilly's WebSite. It's CGI interface to CMD.EXE is relatively consistent between POST and GET. However, there is one significant difference. WebSite manager generates a unique temporary file that contains the data provided during the post operation. You can sniff through this output file on a programmatic basis if needed, allowing your script to branch according to content. This is extremely handy if you have to allow users to submit large strings of text or binary objects.

Beyond Simple CGI

GET and POST both offer value, depending on your objectives. However, in this implementation theyare also both limited in that they rely on NT's CMD.EXE, a character-based command interpreter, as well as require that scripts and applications support the use of STDIN and STDOUT. This precludes the use of Windows-based applications as CGI-callable services.

A handful of vendors are working together to develop WIN-CGI to circumvent this problem. Implemented as a CGI-aware library, you can extend any application for which you have the source code, recompiling it to read and write CGI instead of the more traditional I/O interfaces. Such efforts aren't new. Many sources are available for CGI libraries for various servers and operating systems. Contact your Web server vendor for leads.

Now that you understand some of the concepts behind CGI programming, you should be able to develop applications that communicate between the Web and any data system. Remember to follow the specifications, and do a lot of testing, and you'll come out ahead.

Sample File: DIRsearch.CMD

  IF "%1=="" goto errlabel
  :goodlabel
  echo Content-type: text/html
  echo.
  echo ^<html^>^<head^>
  echo ^<title^>All occurances of %1^</title^>^</head^>
  echo ^<body^>
  echo ^<h1^>All occurances of %1^</h1^>
  echo ^<pre^>
  dir c:\"%1" /s
  echo ^</pre^>
  echo ^</body^>
  echo ^</html^>
  goto exitlabel
  :errlabel
  echo Content-type: text/html
  echo.
  echo ^<html^>^<head^>
  echo ^<title^>No files specified!^</tile^>^</head^>
  echo ^<body^>
  echo ^<h1^>No files specified^</h1^>
  echo ^</body^>
  echo ^</html^>
  goto exitlabel
  :exitlabel

ISINDEX Replacement

  echo ^<Form^>
  echo Enter A File Name to search for:
  echo ^<Input Name="FILENAME"^>
  echo ^<Input Type="SUBMIT"^>
  echo ^</Form^>

Written by Eric A. Hall.
Copyright © 1996 CMP Media, Inc. Used with permission.