DandyWebDesign

Webserver Notes

Website Upload Management Utility

One thing I really hate is repeating myself, or performing the same task twice, especially when it comes to file management. Maintaining separate development and live copies of a website on your PC and on a remote host server respectively requires a fair degree of organisation, there are a lot of steps, a high level of complexity and many pitfalls.
Another thing I hate is clutter, by the time you have installed Apache, Perl, MySQL and God only knows what else your desktop is awash with icons you will rarely use. And getting a web development session up and running is fraught with complications, you have to be sure that you have started Apache and the database listener before you try to open the browsers and so on.

All of these things things are irritating and over time they niggle away and cry out for a simple solution...
...which is of course to automate the process. This is not a simple task in itself but once implemented it will very quickly pay you back, in time saved during uploads, and time not wasted fixing problems that would never have occurred if you had an automated web management system! This can be achieved with something so low-level it is often ignored yet works on all PCs running MS-Windows; A DOS batch file or 'shell-script'! And of course if you're running Linux anyway then shell scripting should be de rigeur...

With a single DOS shell-script I can control the Apache server, the MySQL database and, most importantly keep track of all upload operations and ensure that only the files that have changed ever get uploaded. With the assistance of a little Perl, I also ensure that all uploaded files have the correct shebang lines, and that all basic safeguards and precautions are applied, up to and including maintaining a back-up copy of the uploaded files. All of the desktop clutter can be poked away into a sub-directory somewhere in the start-menu and replaced with a short-cut to the new DOS shell-script; 'WebDevUtil.bat'

The Utility: an Overview...

The utility is fairly straight forward, a single MS-DOS shell-script which when executed generates a text menu with a dozen or so options in, each one assigned to a particular letter. Hit the appropriate key and that option will be invoked. On completion the utility will report what needs to be said and then return to the default menu display where it will await your next instruction. Much of the functionality is particular to my setup, however the general methods are easily modified to suit anyone else's, including your's.

The functions provided by the utility fall into two categories. The first allows various programs to be launched or stopped and so on. This is a handy but by no means essential aspect of this utility. if it is not of use to you then simply leave these options out.

The second category is far more significant though and takes care of so many issues that I consider it to be an invaluable routine. These are the upload routines, which ensure that only those files which have actually changed get uploaded and that a copy of the uploaded files is preserved for reference. Using this, user-interaction and confusion are kept to a minimum, I no longer need to remember which files I have changed, easy enough when there are only 20 files in your site but utterly out of the question when you're dealing with several thousand files. Upload time itself is kept to a minimum, no file is uploaded if it is the same as an existing live copy, a waste of time otherwise!

The routine also takes care of a number of specific 'gotchas'. Although the live and development copies of each site are the same there are usually a few files which are most definitely not the same and would break the live site if the development copy was ever uploaded by accident. Generally these are items such as password files, .htaccess files and any configuration files with path specific settings. The upload routine is set to watch out for certain files or occurrences and make them 'safe'.

So at any point all I have to do is call a few options from this utility and all of the uploading operation is carried out for me. My interaction is kept to a minimum, all I have to do is run my eye over what is going up and hit 'OK'... ...and then confirm the clear-up at the end. A done job! Very quick, easy, and reasonably fool-proof. Most of the pitfalls of not having such a routine were learnt the hard way in the early days, of all my 'tools' this is one of the greatest time-savers and has proved its worth many times over.

Third Party Applications:

The whole point of the utility is to tie a number of other applications together. Obviously my setup is specific to the software that I use, however the method can easily be modified for whatever equivalent applications you may be using. My setup includes:

• Windows: This method will work in all versions of Windows from 3.11 right up to and including XP. You may encounter some minor differences in the DOS implementation under NT, however these should not cause any problems.
• DOS: Ships with Windows!
• Windows Explorer: The file manager that ships with Windows, Firefox is easy to find on the 'net.! I find it useful to launch it as part of my environment setup, you may choose not to bother.
• Web Browser, Firefox or Interet Explorer 5.0+: IE ships with Windows! This are the browsers I use for testing, however any browser can be used here, it does not matter which one as long as it is DOM compliant. (Up to date!)
• WpWiz.exe: 'Web Publishing Wizard'. This is a simple FTP application that ships with Windows. There is no need to go out and download (or even buy!) any of the many GUI based FTP applications out there just to perform your uploads, the functionality already exists on your PC.
• NoteTab Lite: This is my text editor which I cannot rate highly enough and which I would not be without. This is designed for working with HTML and allows all sorts of very useful features such as a 'clips' library, macros a reasonably sophisticated search and replace and much more.
  Alternatively you could just use Windows NotePad in which case; good luck with that!
• Apache: From its unix heritage Apache can be started, stopped, reconfigured, interrogated and tested from the command prompt. This makes it ideally suited to running from a simple shell-script utility like this one which provides full control over the Apache server.
• MySQL: Database applications. Ditto Apache comments above...
• Perl: CGI applications. Ditto Apache comments above...
• ...and a handful of Perl scripts: These perform certain detailed file modification tasks and changes to the live and development databases. These will be discussed individually in due course.

The Menu: Its basic functions explained...

The menu display is shown here. All fairly simple stuff, each letter corresponds to an option and a brief precis, hit the key and that option is executed.

On completion the display will confirm the previous operation (as appropriate) and then return to this menu after a specified time interval or when the user presses a key.

So, lets take a brief stroll through these options, not necessarily in order!

• Menu control options.
• Q Quit : Exit the menu and return to the DOS command prompt.
• X Exit : Terminates the DOS session.
• Apache and MySQL database control options.
• A Apache : This starts the Apache server. Note that an error will be returned if you attempt to start a second instance of Apache.
• B Bounce : This stops and then restarts the Apache server, necessary if you have made any config changes. Note that an error will be returned if Apache is not running.
• K Kill : It is necessary to cleanly shut-down Apache before shutting down your PC, this option takes care of that. It also shuts down the MySQL database listener. If either service is not running an error will be returned, however these can be ignored as the end result will be correct insofaras neither Apache nor MySQL will be running!
• M MySQL : This launches the MySQL database listener.
• T Test : This invokes the Apache self-test option which will check the httpd.conf file and either report 'OK' or details of the invalid configuration.
• Website files and FTP upload control options.
• C Changed : This scans the files under the document root and compares their timestamps to those under /live. Those files which have changed since the last upload are identified and copied to /ftp ready for upload.
• D Changed : This is exactly the same as option 'C'. I run several websites, 'C' maintains the upload lists for wcms.org.uk, this option performs the same tasks but for the dandylife.org site (this one!).
• F Flush : This clears the /ftp directory. This is invoked by the upload routine on completion to clear up, however it is also handy to have this directly available.
• L List : This lists the contents of /ftp, this information can be displayed before upload.
• U Upload : This calls the WPWiz program which is pre-configured to upload whatever it finds within /ftp. This takes care of the upload operation and on completion it reports back and then (if confirmed by the user) clears down /ftp.
• Working environment and website maintenance options.
• E Environment : This launches a working session for me, this starts Apache and MySQL, opens the Explorer to the required directory, launches my text editor and opens a variety of browser windows for the various sites. All personal to me, however this method can be used in so many ways, you can launch almost any program like this. With one option you can go from a freshly started PC with nothing running to having all of your regular working programs up and running. Very simple and much quicker than laboriously launching each program in turn.
• I Import : This opens a particular CGI page on my utilities server; utilities which help to maintain the websites. This one invokes a few routines on the live sites which cause them to output the database contents in a particular way. When this has completed all of the various data files are downloaded from the live sites back to my PC where they can subsequently be used to update the development sites to keep them in synch with the live sites. (See the Synch option below...) Again this is highly specific to my usage but the idea can be adapted for your own purposes.
• S Synchronise : This is similar to the Import option above. This particular routine takes the data which has been downloaded and loads it back into the development sites. It then provides another CGI utility which allows me to 'push' the contents of the live databases onto the development ones to keep them in synch.

Efficient Uploading: The DOS file sortation method...

For a website with a few dozen pages there is no real penalty in uploading the entire site from the document root, it's quick and simple. However as the site gets larger this becomes less and less desirable and eventually a point is reached at which the number of files makes this impractical.

In order to keep FTP connection time to a minimum (significant for anyone who is limited to a dial-up connection) it is important to only upload those files that actually need it, in other words those that have changed since the last upload.

In order to keep your live site consistent it is of course imperative to upload all of your changes, which means keeping track of which files have changed since the last upload and making sure that they are included in the next one.

Neither of these tasks are particularly difficult, however for a large site they may prove to be complex tasks. But with a little thought they can be relegated to background admin tasks, this is the primary function of this utility.

The utility uses the 'archive' attribute of each file, some strategically chosen directory locations and the DOS command XCOPY to achieve the desired results quickly and easily as follows...

A copy of the site is kept, this provides a reference to what is currently on the 'live' server. At any instant this should be exactly the same as the development copy of the site under the document root except for any changes that you have made since the last upload.

The ftp operation is carried out from a separate directory. This directory is populated by the 'changed files' routine, uploaded from and then cleared down afterwards to leave this directory empty once again.
Thus we have the following directories: (and their references in this example...)
c:\website - [/docroot]
c:\live - [/live]
c:\ftp - [/ftp]
The routine works as follows...

1. The archive flags are set to up (on) for all files under /docroot
2. The archive flags are set to down (off) for all files under /live
3. XCOPY /A /D is used to copy all files from /docroot to /live which either have more recent timestamps or which do not already exist under /live.
   /A copy files with archive flag up, copies flag as well...
   /D source timestamp newer than destination...
4. XCOPY /A is then used to copy files from /live to /ftp.
5. A complete 'tree' is built under /ftp of all the files which are destined for the next upload. All you have to do now is select the next option from the menu: Upload!
6. On completion /ftp is emptied ready for the next cycle.

The Perl Programs:

The real power of any shell environment is not so much what the shell can do as what it can invoke; the shell-script or batch-file is merely the 'binder' which pulls applications together.

There are a number of tasks in the upload operation which require detailed file interaction or logic, DOS is suited to neither and so instead we call a few simple homespun Perl routines to do the nitty-gritty work.

Setuploadshebang.pl

The first line of any shell-script usually takes the form of an instruction as to which interpreter should be used to execute the script. This is a common unix method and so CGI scripts generally start with: #!/usr/bin/perl or whatever the equivalent path to perl is on your live hosting system.

But if you are developing under Windows you will be running win32 Perl which requires a different shebang line; #!perl, consequently you will have to change this before you upload the file.

It is possible to configure CGI programs to run without the shebang line under windows by instructing Apache to use the file extension and a registry look-up to decide what action to take with a file. However, this is not recommended.

This script is invoked before an upload commences and checks the cgi-bin directories for each of the sites under the /ftp upload area looking for CGI programs. If it finds any it checks the code and if it contains a win32 shebang line replaces it with the required unix one.

Setuploadtimestamp.pl

There are two methods by which a webpage can be made to output its timestamp, one by using a serverside Include #EXEC command and the other using the JavScript document.lastModified method. Unfortunately neither method can be guaranteed to work as some host servers are configured not to give out such information from the filesystem.

The next best thing is to set an upload timestamp for the entire site and have it refreshed everytime you run the upload utility.

The actual nature of how the timestamp is passed to the website can vary, you can either write it into an .ssi file for direct inclusion or as I have done and create a JavaScript page which contains the timestamp as a variable which can be referenced by the various JavaScript programs that I use to create the repetitive parts of my pages.

This program simply writes the file directly to the required location under the /docroot for the site. There are actually two separate versions of this routine, one for each of the two sites that I maintain in this way.

Securitysafetycheck.pl

Although the development site should be identical to the live one there will always be a few configuration related files which differ in some way between the two sites. Often the differences are such that an accidental upload of the development copy to the live site would be disastrous. This is especially true of files such as password and group files, .htaccess directive files and any configuration options that require a system specific path, the effect if these are accidentally uploaded are usually fairly terminal!

This program provides a safeguard against such issues by checking the /ftp directory for such oversights. It's invoked just before an upload takes place and scans the /ftp directory looking for certain pre-specified filenames. If any are found then one of two actions will take place depending on the particular file and its requirements.
If the file is really incompatible with the live one it is deleted from the /ftp directories. If it is merely 'sensitive' then it is renamed with an additional suffix that will prevent the live file from being overwritten.

In the unusual event of actually wanting to upload one of these files then the desired file will need to be manually put into the /ftp directories or have the suffix removed from the filename after this filtering has taken place and before the FTP operation commences.

The DOS Code: A walk through the code...

The DOS code that makes up this utility is displayed below in convenient blocks, each is section discussed in detail below...

The first key line is to turn ECHO OFF to stop the output being polluted with each command as it is executed...
The label :BEGIN marks the start of the program and is the point to which control will be redirected at the end of each operation.

There are a number of applications and settings required, these are all set here rather than hard-coding the values into the program. You should be able to adapt this program to your system from here without having to go through the code looking for paths and other specific values.

Note! In its default mode the DOS prompt only allows a small environment space, by the time you have stored a few variables you run out of memory and it all goes mysteriously wrong!

The solution is to invoke the script using
c:\windows\command.com /E:32767 /K webdevutil
Create a desktop icon for this utility, right-click it and select Properties. Select the Program tab and enter the command line shown here into the field marked: 'Cmd Line'. This will cause the DOS shell to launch with 32Kb of memrory, more than ample for your needs, the /K flag will keep the shell open even if the utility is itself is exited.

The final command sets the current drive as required, (on my home sytem I have several partitioned drives, very few of these utilities are actually on my C drive, this is just an example!)

As soon as the variables have been set the screen is cleared and the menu is drawn, this is exactly the same as can be seen in the diagram above.

The CHOICE command waits for a key input. If it does not get one it defaults to 'O', an unadvertised option that simply returns it to the start. Thus the menu will wait indefinitely in the background.
Once a key is pressed an ERRORLEVEL value is returned, 1 for the first letter in the list: 'A', 2 for 'B' and so on up to 17 for 'Z', another unadvertised feature which reinvokes the program. (Handy during development when the code is changing to force a re-read)
the ERRORLEVEL vaue is then tested and directed to the appropriate section by the GOTO commands.

Note! the test 'IF ERRORLEVEL 1' should be treated with caution, its actual meaning is: 'If errorlevel is GREATER THAN or equal to 1'. Some thought about the implications of this will explain why the options in the program code are listed in descending order of errorlevels!

The final GOTO BEGIN is a catchall to prevent execution crashing through into the code below in the event of an error in the IF statement block.

The first two fairly self-explanatory blocks here, the first starts Apache, the second stops and restarts Apache. Both use the variable APACHEXE that was previously defined for the path to the Apache executable.

This is by far the most critical of all of the functions of this utility, the file sortation prior to an upload. All of the values are set as variables at the start of this program. There are two blocks like this in my version of the utility, one for each of the two sites that I am maintaining; this is the WCMS one.

%PERLEXE% %WCMSUTPL% this invokes the 'setuploadtimestamp' perl program that does just that!
ATTRIB -A ... takes the archive flags down under /live and ATTRIB +A ... puts them all up under /docroot

XCOPY ... /D does a timestamp sensitive copy from /docroot to /live,
XCOPY ... does copy from /live to /ftp
The CHOICE gives the user 5 seconds to note what has happened before defaulting back to the menu to await the next command.

Exactly the same as the block above, this one is for the DandyLife.org website, the other for WCMS...

This is a sub-routine that I find very useful, you might not, so leave it out if you wish. when I fire up my PC all I have to do is launch this utility, hit this option and sit back for a few seconds whilst everything that I want opens itself ready for me to work.
It opens the Explorer to my web folders, opens my text-editor, fires up Apache and the MySQl database and then opens three browser windows into my three principal areas of interest in a regular working session.

Note! Any program for which you can set an icon or short-cut to in Windows can be called this way. right-click the icon, select properties and copy whatever is given in the 'target' field after the START /MAX comamnd.
Type START/? at the DOS prompt for more details on syntax and options for this very useful DOS command.

This block takes care of flushing the /ftp directory either as a sub-routine of the upload operation or as a routine in its own right called by the user.

It checks for the existence of the FTP list file and removes it if found. Once the deletion has finished the DandyLife top level directory is recreated as /ftp/dandylife.org as it has a slightly different root to the WCMS site.
On completion the routine reports back to the user and then returns to the main menu after 5 seconds or when the user presses a key.

This is one of the simpler sub-routines, it simply invokes a specific page on the home system utilities server. This page contains links to a CGI program that allows me to import data from the live sites back to the development ones using additional CGI programs built into the site database code.

This shuts down the Apache webserver and then the MySQL listener. Both are fairly simple calls to these programs using their command line options.

Note! If either service is not already up when this is invoked an error will be returned, however since the end result will still be that neither service will be running these errors can be ignored.

This generates and then displays a list of all files under /ftp The user gets 50 seconds to look at the file list before it defaults back to the main menu.

Another self-explanatory call to start the MySQL database service and listener.

This also handles the synchronisation routines, see IMPSYNCH for more details

Another command line option for Apache, this runs the self test option.

Blah blah blah...

Using the Utility