Workable Solutions

Workable Solutions is a growing occupational therapy and rehabilitation clinic in Mountlake Terrace, Washington which provides physical and occupational therapy services and a respected work injury rehabilitation program focused on workers in the North Puget Sound area. The clinic was established in 2000 and has been expanding ever since, with the company currently making plans to open another clinic site somewhere in the area within the next twelve to twenty-four months.

IMS was brought in to the clinic to deal with a number of issues that had appeared in the course of normal use of the system since its installation by a third party. Laptops were having wireless connectivity problems and were often unable to logon to the network in the morning. Some files were locking up when accessed on the server from some desktops. Finally, another of the desktops was having consistent problems accessing certain websites outside of the local network. As is common, we found that these were only symptoms of a number of more subtle infrastructure problems.

The company IT challenges revolve around storing patient documentation and financial and billing information, as well as providing application support for some line of business clinical applications and the usual electronic communications protocols such as e-mail and web services. The clinic has several desktop systems and provides eight laptops which are portable within the clinic in order to provide direct data entry and specialized program access for employees directly out on the working floor of the business. Connectivity for the laptops was provided via a secure WLAN, which was patched in to the existing wired LAN. A single Windows Small Business Server 2003 provides file storage, electronic mail, and database hosting. A mixture of Windows XP and Windows 2000 were installed on the workstation machines, along with a number of different versions of MS Office.

As there was very little documentation available from the original installation, we proposed to review and document the entire system from the ground up rather than only addressing the identified problems. Even when this process does not directly attempt to address the exposed issues, it produces knowledge of the system that can be invaluable in identifying and correcting future issues proactively and shortening troubleshooting time on existing problems. In the course of the documentation process, we identified a number of other problems. Some of these were underlying technical issues that users would not typically encounter directly, but some were chronic annoyances that had not been reported to us because staff had reported them to prior consultants with no result and therefore had no expectation that they could ever be solved. This situation is also common and can lead to an insidious loop between staff and system which often results in bizarre and inefficient practices.

Among the problems that were identified in our 4 hour analysis:

• All devices on the network had been assigned static IP addresses. This resulted in additional consulting overhead any time a new device was to be added, particularly since there was no existing documentation as to which device had what address; it also introduced additional potential for human error to cause network problems, as was in fact the case on the PC with website access issues--a typo in the manually entered DNS configuration was preventing it finding many sites on the Internet. The server itself also had DNS mis-configured, a potentially major problem for a Windows Active Directory based network.


• A number of basic security issues were found--it was discovered in the course of correcting the above issue that an old Internet connection was still plugged into the network, which interfered with DHCP (probably the reason that static addresses were being used in the first place) but worse, bypassed the firewall entirely and allowed direct outside access to every PC on the network. This was compounded by the fact that all users had been made members of the powerful Domain Administrators security group, and that was compounded by the fact that strong passwords were not enforced. On top of all of that, neither the server nor workstations were being patched with any regularity, leaving them vulnerable to any number of completely correctable security issues. Different versions of Windows and Office were installed through the clinic, making patch distribution and verification complex. Finally, the firewall that was in place was effectively disabled as it was set to forward automatically all traffic to the server, dramatically increasing the attack surface instead of filtering it.


• Two wireless access points were installed to provide coverage throughout the clinic--it developed that these were operating on the same frequency and conflicting with one another, effectively reducing wireless network range and power when the addition of the second access point had been intended to correct those very issues. A combination of this issue and the following one were found to be the cause of the laptop logon problems reported.


• Server processor utilization was extremely high, making it slow or unresponsive, a combination of an out of control database application process and the installation of a number of real-time anti-spyware applications (which are of dubious value on a server, as they are intended to safeguard client systems with regular desktop users).


• The server backup was backing up recursively to the same drive the data was being stored on. No backups were being taken off-site, or even to a separate computer on-site--meaning that had the server failed (which is the circumstance in which you most need the backup files) they would have been lost with the originals.

The system was never a candidate for a complete, ground-up rebuild due to budgetary constraints and the lack of criticality of any of the issues to operations. Additionally, as is frequently the case, previous consultants had attempted to address many of the issues by recommending and installing new hardware, and after several unproductive cycles of this, management was not inclined to attempt it again, nor did IMS recommend it. Instead, we documented the problems and added specific recommendations to solve each of them, making best possible use of previously purchased hardware and software. The largest expenditure of the project, ultimately, was the purchase of sufficient new Windows XP licenses to upgrade all machines to a stable and consistent operating system version. Also keeping in mind the customer's bottom line, IMS arranged to sub-contract much of the straightforward (but time-consuming) upgrade and installation work to a less experienced (and therefore less expensive) consultant working under IMS supervision, which ensured quality work at a lower price.

Most of the issues were addressed simply by returning system configurations to recommended Microsoft best practice defaults. The extra Internet connection was disconnected physically from the system and DHCP and Active Directory were set to function properly again, immediately solving most of the client-side issues reported. The firewall was re-enabled and administrative passwords changes. Users were removed from the Domain Administrator security group and re-assigned to less powerful groups which still allowed them appropriate access to files and services. All non-essential applications and redundant file shares were pruned from the server, and the hyperactive database server process was toned down using documentation from the core engine developer's site and further disruption was reduced by tweaking the processor affinity of the process to occupy only one of the dual processor cores in the server.

The two WAPs were de-conflicted by changing their operating frequencies. With more appropriate placement in the office, it is probable that only one WAP could provide complete coverage, but having already purchased both, management was not inclined to dispose of either. However, to eliminate any other potential conflicts and reduce complexity, IMS set most laptops to only use one of the WAPs for access rather than either.

Backup storage requirements were calculated and it was determined that the most cost-effective, lowest-overhead solution was to use an online backup service across the high-speed DSL line each night. This gave instantly accessible, but safely off-site, backup for all business-critical files, and the selected backup provider's software is able to run in the background on the server and function with no staff input, providing error reporting and other updates directly to IMS as we manage the system on an on-going basis.

IMS also installed Microsoft's free WSUS patch management software at the server and configured all other computers to check there for patches and updates. WSUS provides an automated, highly configurable mechanism for obtaining software updates directly from Microsoft and re-distributing them to other machines in the company. It also provides detailed reporting options to check the patch status of all machines from a single location, dramatically reducing the time to verify the application of required patches. It also reduces bandwidth requirements considerably, since each patch need only be downloaded from the Internet once, instead of once for each machine should the more conventional Windows Automatic Update service be used.

IMS also configured the server to provide regular e-mail status reports back to our consultants, which are used to manage server and network issues proactively, before they begin to affect users directly. Also, as part of our standard practices, we assembled the extensive information we gathered during our troubleshooting and fixes and compiled it into a well-organized manual format, a hard copy of which was left at the client's site in the event they wish to attempt to correct further issues on their own, or through another consultant--saving valuable hours determining the basics of the site configuration which must be known before proceeding safely.

Computer related issues have all but disappeared from the radar for Workable employees. Support calls have fallen off to one or two per month after the initial settling-in period, and those are only rarely related to in-house systems--Internet related issues (which are both inevitable and outside of control) are more common now. With fewer support calls come lower costs, and less staff time lost to balky systems. Workable Solutions is well-situated to continue expanding, to focus now on their core business instead of incessant computer problems.