Library & Articles
Developing for Internal Database Systems

Library & Articles

Developing for Internal Database Systems

Most companies, given a choice, will favor a design-first software development approach for their projects. The idea is that you make a big design that has all the specifications and features written up in advance. Then you bid the project and try to get the best price for it, a fixed-price that the development team or firm is locked into.

With employee based teams, the idea is that you hire the expensive people up-front, use them to make a good design that you can then hand over to less expensive people who may be good at programming but may lack the experience to anticipate current and future design problems. The think behind this approach is that you can just “plug-in” technical people on an as-needed basis.

With external firms, the idea is that they will have to deliver that software for the specified price even if they massively underbid the project.

It sounds very good in theory, but it usually does not work out in practice. The statistics for successful completion of such projects are abysmal.

The problem with big up-front designs is that generates a mountain of documentation that must be prepared in advance before any programming is actually done.

The full formal documentation that lays out the design specifications such as maps, diagrams, specifications lists, technical requirements and features lists can take hundreds or even thousands of hours to prepare and almost as many hours to study it all and learn it.

Worse, even as the documentation is being prepared, the features and specifications may change due to new opportunities and changes in technology. Many companies never do completely finish this documentation process.

Often, it just becomes a massive waste of time and money. In today’s business environment the changes can roll in with astonishing speed, as the company jockeys its marketing strategies and service offers around in an attempt to take advantage of new opportunity or technology.

Management typically wants this kind of documentation up-front. The reason is that the documentation is essentially a translation of programming design concepts and source code routines into plain english. In source code, it takes a few lines of code and a single command to execute a sophisticated process or function. Translating this to diagrams, plain english and flow charts may take several pages of documentation. Every time you change the source code you have to change the corresponding documentation. Not only do you pay for having the source code written, you also pay for revising the documentation every time a change or addition is made.

The other problem with this type of documentation is that what works out very well on paper, in flow charts, diagrams and specifications lists very often does not work out programmatically. Once again, the documentation must be changed. Sometimes the problem is very fundamental and the changes to the documentation is extensive.

On top of all this, there is the documentation of the finished program and the interfaces. It is not unusual for this type of documentation and the training manuals to take as much time and investment as the programming of the application itself. Of course, any time there is a change in the program it must be changed as well.

The usual reason for the initial up-front specifications documentation is to try to control the programming process and make it possible to produce a fixed-bid which gives specific completion targets on a timeline. The contracted firm will then be required to perform on the contract or suffer various penalties.

Fixed-priced contracts can be workable for software applications that have a very specific purpose and can stand alone. As an example, this approach could work for developing a word processing software application. This is a stand-alone application with a very concrete set of features that do not change over time. There is no particular need to integrate into an ongoing and changing business system. The same is true for any kind of off-the-shelf software application that does not require integration.

Unless the the project is suited to up-front design, the usual sequence of events on fixed-price bids is:

• The company decides to start a development project.
• A request for proposal (RFP) which is a short description of requirements and specifications is made up and multiple bids are sought.
• The bid is awarded, generally to the lowest bidder.
• The winning bidder (development firm) starts development.
• The documentation process is carried out to set out the exact specifications and requirements.
• The development firm then finds out that the project is not as it was described, usually because the specifications are either inaccurate or incomplete.
• Modifications start in to produce the actual requirements.
• Price adjustments and time adjustments ensue. The project is now off budget and behind schedule.
• Adjustments and modifications occur continuously throughout the project.
• The development firm is not making money and therefore cuts corners and/or generates additional charges.
• The time schedules are extended again and again.
• Neither the development firm or the client is happy.
• The project may be abandoned due to cost overruns or time constraints.

For company wide systems that serve a growing company that is expecting to diversify, this approach is a train-wreck waiting to happen. By the time the development firm completes something, the needs have already changed. The contract itself is designed to resist and prevent change and is actually detrimental to the best interests of the company. Every change results in a change order, either generating a new bid or additional time and material billings.

A factor in all of this is that management has a very real and vital need to know and understand what the programming department is doing. And the programming department speaks and functions with languages that only programmers and systems personnel understand. At the very least, the documentation is supposed to provide a means of communicating needs to the programming department and it is supposed to exert a measure of control over their activities and costs.

Unfortunately, this approach does not work well for company wide systems. This because the very nature and role of software and information systems is very different from stand-alone applications. The internal software and information system of a company is like a backbone or central nervous system with nerve endings distributed throughout the entire organization in the form of user interfaces. It is dynamic by nature. You cannot just install it in and be done with it, for while you were installing it, technology changes and company growth continued unabated. As well, in our increasingly internetworked business environment, it must be flexible enough to be continually enhanced and upgraded to integrate and respond to connecting to partner systems, supplier systems, remote offices and online customer support systems, to name a few.

Successful and effective management and development of an internal system and its applications requires a very different perspective and approach. You have to evaluate it and cost it out much as you would any department of people in your company. With people, you expect to pay ongoing monthly salaries, invest in initial training and workstation costs, reorganize on a constant basis for greater efficiency, increase their knowledge and continually work to create a better and more cost-effective working “people machines”. This process never ends, must be continually managed and improvements require constant and continuous investment.

Internal software systems and applications are designed to both replace and facilitate people machines or the actual people in the people machines. That is really all they do. They are part and parcel of a people machine and because they are actually people-driven and people-derived, they require a similar level of continual development process and investment.

You could think of a company wide system as a person who is not very bright but will exactly and tirelessly carry out the precise instructions given to her. Maybe call her Bertha? Bertha grows and learns and must be cared for just as her human counterparts do. She just happens to be VERY literal and must be told every single thing to do in her own language. But once she is all set up and ready to go she never gets tired and never complains and never asks for a raise.

The best internal software system and application serves the needs of its people machines as quickly and efficiently and as accurately as possible. Not at some distant point in the future.

Its fundamental design and construction must accommodate flexibility, embrace changing conditions and facilitate expansion. Its core functionality will run through all of the people machines set up throughout the company, surfacing as user interfaces that service the individual people machines.

The actual cost justification for such a system is that a good system reduces redundancy, reduces errors and speeds execution of tasks. It also reduces the number of employees that are required to process information. As an example, implementing a full-scale accounting system that automates bill payment functions will allow a company to expand the area without having to hire and train additional bill payment employees. That’s where the savings are. It may cost $10,000 to build that module, but without that module, the company would have to pay 3 or 4 employees at the rate of around $2000 per month. The module pays for itself within 5 months if even one less employee is required.

Additionally, the system can and should be designed to prevent and detect errors, thus reducing the costs and possible loss of business that result from personnel errors.

The return on investment can be severely reduced or even completely wasted if the programming area is unskilled, weak on designing in a dynamic system, is under the supervision of individuals with vested interests or is not properly structured and managed.

Ensuring a good return requires setting up a high quality people machine that works continually to develop and maintain a quality internal system and applications as quickly as possible.

The command structure, organizational chart and operating basis for such a machine is very different from that of a sales department or billing department.

This is because the design and programming will very often affect the core or backbone processes and the overall flow and interchange of information as changes, improvements and specific departmental applications are added in or improved. As well, it has to be organized in such a way that security and confidentiality is maintained (crucial where the system integrates into a partner or outside firm) particularly where future plans or strategies that may be company secrets may be of critical importance.

This machine must have a function that coordinates, masterminds and interprets the needs of affected departments and core programming team or teams that will actually continually build and maintain the system.

Without such a function, non-technical managers are faced with the problems of attempting to manage an area that is highly technical and usually non-comprehensible.

The first and most common problem is that companies will attempt to hire inexpensive programmers. There are plenty of individuals who are somewhat trained technically who have learned that non-technical people do not have the knowledge or skill sets needed to evaluate their recommendations or credentials. Anyone can take a crash course in PHP or Visual Basic, perhaps learn Microsoft Access and then assert that they are a “programmer”. At best, they are beginner programmers. At worst, they are an active liability in a live programming environment. They are not usually ill intended. They are simply inexperienced

Beginner and intermediate programmers are useful for programming in small, non-critical projects or a very tightly supervised or mentored programming project. However, even though their hourly rate is low, they are still not generally cost-effective. There are several reasons for this:

• They are generally very slow either because of unfamiliarity or because they are still on a learning curve. You are not only paying to have them program, you are paying for their learning time.
• They generally write poor source code that becomes error-ridden over time and do not understand the underlying purpose of many of the functions they use.
• They cannot read other programmers source code and so cannot build into the existing source code. This results in existing code being scrapped and new code re-written. They will sometimes tell non-programmers that the code was “bad” or “unnecessary” or that it is faster to re-write. Sometimes their claims are correct, but very often they simply do not understand the existing code or cannot see the purpose of it.
• They do not know how to write source code cleanly and consistently so that it is possible to introduce new programmers to the project they are working on.
• They do not understand how to read and write generic code functions or libraries that consolidate source code into re-usable code. Consequently, they are essentially writing the same code over and over again, resulting in hundreds and even thousands of lines of code that are redundant.
• They generally have not worked on very many applications and do not actually know how the finished code will behave. They are also not aware of the vagaries of the programming language, particularly when it comes to multi-language or multi-database applications.
• They do not understand the underlying concepts of database and database applications. The result is poorly constructed databases that are slow, difficult to maintain and prone to error. In small, simple applications, this is not a problem but as the size of the database increases, the performance deteriorates and the errors escalate.

On the other end of the spectrum are the highly specialized programmers. These programmers have invested years into learning a specific language and database application and they can usually write top-notch, sophisticated code with their specialty. These programmers will generally resist the use of other languages or database development applications and will favor their particular brand. Where they have an established position in a company, they will seek to protect their own position and paycheck in several different ways:

• They will assert that their brand is superior and other brands should not be considered. They will generally have compelling arguments and their arguments should never be taken lightly. However, they may also fail to reveal the weaknesses of their particular specialty.
• If they feel threatened, they may write source code in such a way that only they actually know how it works or hide the true functionality deep inside layers of programming so it is very difficult to enhance or upgrade without them or their specific knowledge.
• They will refuse to consider alternatives they are unfamiliar with or refuse to upgrade their skills so alternatives can be used.
• They are generally quite expensive in terms of hourly rates, but if their specialty language is being used they are actually more cost-effective over the long run.

At the top end of the spectrum are programmers that have a strong systems background, can write in lower level languages such as C or C++ and are versed in several development languages or applications. These programmers are generally very expensive in terms of hourly rate, but utilized correctly they can be more cost-effective than specialty language or or beginner/intermediate programmers.

Their advantage is that they bring a broad skill set and a rich set of experiences with them. To be at this level at all, they have to have worked extensively with live code and installed systems. Since they are not focused on a specialty or restricted in their knowledge of what can and cannot be accomplished with higher-level functions or procedures, they will choose to use the language or tools that are going to be most efficient. They have a well-developed set of standard solutions they have already tested and they generally know where a program will fail or where problems will likely emerge. Most of them are accustomed to working in team programming situations and on live systems.

They also perform very well in today’s radidly changing technology environment. This is because systems and languages have become “transparent” to them, meaning that they understand that there are only so many things you can do with a computer. Every language must necessarily perform those tasks one way or another. The function names and the way the code must be written is different for each language but the functionality is always the same. Consequently, they can come up to speed very rapidly on new languages and system requirements. This is essential in today’s programming environment and especially true for Internet programming where continual and very rapid change is the norm.

Managing ongoing development of a company wide system can be best accomplished with short programming cycles that have specific objectives and that can be immediately tested and placed into production.

The overall strategy should be formulated by a top level management unit based on company needs that encompass the entire organization’s current and future needs. Then it should be targeted out in short, immediate projects that take between 3 to 6 weeks to accomplish. Only enough written documentation should be produced to effectively communicate the desired results. This should be a summary, most of the information should be obtained by actual inspection of the area to be addressed and face-to-face feedback and input. The top decision makers should be accessible and should actively participate in the overall strategy planning process.

The whole thing should be headed up by an individual who is a very good communicator, first and foremost and who understands business models and systems, including operations and accounting, sales and marketing, internet, computer networks and programming. This person acts as an interpreter, communicator, facilitator and coordinator and works directly with the people involved and with top management, gathering information and formulating strategy and generating short-term plans. This person must not have to answer to anyone but top management and must have the confidence of top management.

For the best possible results, the systems and programming area should be overseen by a very experienced, multi-platform programmer with excellent communication skills.

This person would work directly with programmers and systems personnel as a mentor, troubleshooter and would handle design issues that affect the core backbone processes of the entire system. Their purpose would be to ensure that the code stays clean and stable with each subsequent project and that programmers are writing good code that can be passed on. They essentially establish the core machine and eventually their knowledge is passed on to the lower level or specialized programmers as a kind of apprenticeship.

This person would not normally be doing a great deal of programming, although they should be able to assist programmers to write and debug code. They also make sure that the code and the documention is written in such a way that it can be passed forward to new or additional programmers.

Neither one of these two individuals may even need to be full-time employees and it may actually be advantageous to the company if they are contracted outside consultants who are willing to enter into a long term relationship with the company on what is essentially a retainer type payment structure based on their standard hourly rate. Then, since they are not seeking better employment opportunities (as many employees do) and their business focus is on providing professional services (rather like attorneys), they provide a much needed stability over time - while the employees come and go, they remain as a stable and informed arm of management.

The company should then use their expertise and knowledge to build a up a good basic programming and systems team that can produce an ongoing series of builds that add to the company’s ability to produce and to its profitability or cost-effectiveness.

To this team would be added any specialists required by specific builds as they are needed. If a particular specialty is consistently used, the company should consider investing in hiring such a person on a full time basis.

This method of development is lean and cost-effective. It does not produce mountains of largely useless documentation. It treats the internal database system as the dynamic and growing system that it is.

Programmers are generally very happy to work under such a system. Typically, the top programmer consultant will give them their projects and let them get on with it. When they get into trouble, they have someone to turn to for debugging and mentoring that is effective and from whom they can learn precious information that simply is not taught in college or in tutorials.

Additionally, there is considerable satisfaction in being able to complete projects in short time periods. Programmers who cannot ever complete anything lose morale very quickly. If they are forced to write to exact specifications or be penalized, they often will become apathetic and will not attempt to improve anything. They shrug their shoulders and write what they are told to write.

End users win on such a scheme. Their feedback is welcomed and their systems are tested and placed in production in such a way as to help them perform their tasks. They are not having to continually work around the restrictions of an off-the-shelf system or a system that missed the mark on actual required features or requirements.

The sales and marketing division gets a responsive system that builds towards their current needs. This in itself is very unusual.

Finally, management can keep planning fresh and current and can use their system and application as it was intended - to facilitate their company's tasks and reduce personnel costs and errors.

This type of system does require a continuous monthly budget for development. It may require a substantial investment to get it started.

But since it is not wasteful and it is always programming towards finite objectives that serve real and current company needs it will produce a much greater return on investment, not only in terms of source code and productivity but in increasingly valuable programming personnel who help each other and who contribute their best to the overall efforts of the company to be more successful.