October 6, 2010 at 8:37 am
The Holy One said:
> ... doubtless the same complaints were heard about the Ford assembly line:
>
> 1. This guy's gonna put us craftsmen/ex-bicyclemakers out of business
> 2. Yeah it's easy to make a black car on an assembly line
> but watch out when they ask him to make one in colour
> 3. This guy has some kind of hidden agenda; I don't
> care if this technique will save my company thousands
> of dollars -- I'm not going to be hooked.
>
> All of which makes me think this guy's on to something....
I agree with this. Although ORM tools still have a long way to go, it's obvious to me that they will put SQL and database design on a path that is parallel to assembly language programming. Which is a shame because I love writing SQL and thinking about table design.
I look forward to the rest of this series.
Hobbes
October 6, 2010 at 8:43 am
While I agree with the premise, and the benefits of standardization, I find one bit not considered. Given a template, and ridgid structure, people begin to look at the process and technolgy as the only way to do something. The critical thinking, the inventiveness and problem solving in the particular area fades. Human's in general are prone to falling in to such patterns of absent minded process.
In the analogy we are comparing developers to assembly line workers, but that is not quite fair to developers. The assembly line worker does a given set of steps over and over, without having to understand even what part of the product they are working on. The worker at station 42 needs to put in these four screws, that is all they need to know about the product.
Rather, I see developers as the design engineers, those who create the whole. They need to be skilled, to understand the entire process in a fair level of detail, and specific parts exceptionally. Developer or engineer needs to be the one who knows why the steps are being done, and which have meaning, as well as what changes in the end product when a change or different use is needed.
I see this sort of structure becoming the domain of super users, not developers. Does finance need to track some new data for reviews or audits? Well, here is the tool, which will build most of the end result automatically, and to the specifications set by the IT groups standards. In this type of use, it could be one of the great improvements of the IT job, the random user hand made solutions all in nice ordered and consistant formats.
October 6, 2010 at 8:49 am
It still takes smart people to write the generation systems - and a lot of other SQL-related things.
The industry isnt going to stop improving its methods just to save your job or mine. Its our job to improve ourselves and make ourselves relevant.
October 6, 2010 at 9:07 am
BTW, there's no actually no database in that diagram, although there's a box labeled "database". If you only individual tables, and only allow so-called "CRUD" operations, you've lost almost all the benefits of a RDBMS.
Sometimes that's all you need - but put them in files in that case.
Sometimes you need more. When that happens, if your programmers are limited to 'per table CRUD', they have to waste time writing whatever pieces of intersection, union, and difference code they require, and run it locally in memory, and handle concurrency themselves.
This is why we pay serious money for dataservers - for the astounding logic engines that can figure out the "how", if we just tell it "what".
Your product has merit, but I think you need to reconsider the limits of its usefulness. It is probably fine for certain classes of problems. Make sure you really understand what a RDBMS is before you build a new wheel for "business rules" - you may already have one perfectly good wheel.
Roger L Reid
October 6, 2010 at 9:09 am
My point in using the Indy 500 as an analogy is not to liken all software development to just this one race. Rather it is to point out that Honda and its other Asian siblings have dramatically changed the entire auto industry (not just racing) by enormously reducing variation in their auto manufacturing processes. The key to Honda's success, not just in racing but also in the garages of ordinary Americans, is to create cars that you can expect to drive to 200,000 miles with no major repairs. They do this by optimizing entire systems rather than leaving design decisions to be micromanaged, or perhaps not managed at all, at the individual auto-part level.
Someone is going to do something similar in software development. When they do, we are going to either be part of the movement or we are going to be unemployed. I'd prefer to lead this movement than to follow it or to be brushed out of the way.
October 6, 2010 at 9:30 am
I think the reason the Japanese are leaders in quality and productivity in manufacturing is not due to just "standardization". It is due to concepts that are usually described as "continous improvement", "deming methods", "kanban", etc.
These methods combine sophisticated statistical analysis along with maximizing "human capital" in the improvement process.
With that said, I think the concept of RAP as presented in the article has real merit and almost any software development project could benefit from some of the thinking presented and perhaps even a total adoption.
I sure as heck am going to try it and I thank the author for sharing his work.
p.s. I have been deep in the trenches of custom application development for almost 15 years...
October 6, 2010 at 10:03 am
dave.farmer (10/6/2010)
No-one seems to be mentioning the elephant in the room here...It may be more efficient, it may be good for the business, but are we really so selfless as to hasten the end of our careers for the good of the industry?
I have to agree with bernard.libbrecht. First of all this comment has no bearing on the article. Second of all, a comment like this assumes that SQL development and the developers themselves have been taken as far as they can, to the point that any optimization is a loss in jobs.
Somehow I refuse to believe that we have no ability to offer new ideas, take on the infinite problems that still lie in our domain and tackle the new ones coming down the road. I also refuse to believe that I cannot learn and adapt to the problems at hand.
It is thinking like the above that must have taken place in my current organization where senior management decided to handle layoffs based on lack seniority rather than skill. That left us with too many lazy people planning their retirement every day while those paid half their salaries with skills to flex are looking for other jobs. If I'm next on the chopping block I'll be looking for employment at a place that at least lets me fight for my job. I don't care if you have to lay me off, just let me fight. Don't fire me because some guy had my position once 5 years ago and has worked 2 months longer than I have. If he can't do my job better than I he has no logical reason to replace me. Of course, logical is the key word.
I will continue to jump at new opportunities to make my co-workers and customers jobs easier and better. There is always something to be improved around the corner and sharpen your skills if you are willing to still stretch your potential a bit.
October 6, 2010 at 10:07 am
Steve Jones - SSC Editor (10/6/2010)
jim7429 (10/6/2010)
The Indy 500 analogy is probably why I read the article, so I can't slam that, or the spirit of the article - sure standards make things better. I'm kind of lost on the author's perspective though. I don't know of any shop that doesn't work to or towards standards and I certainly don't know of any where the apps and db folks work in vacuums ....The analogy caught me when I edited this one.
However you're lucky. I've worked in quite a few shops where the DBAs did their own thing, and ignored or did not communicate with developers. I've also seen many developers that threw tables together without thinking about design, or without understanding normality and what advantages it brings (or disadvantages). Too often I've seen inconsistent development in all kinds of applications, in-house and commercial.
One way to avoid the communication breakdown is to follow an agile framework such as Scrum, XP, etc. where the technical team makes design decisions as a group incorporating ideas and information from representatives of multiple disciplines (e.g. Data Architects/Modelers, GUI Designers/Developers, Testers, and Business Users/Owners). The iterative approach to this method may span from analysis of the tools to be used to the broad and detailed architecture and design and finally to the development and deployment of the product. I worked on such a project to develop a data warehouse and data mart for HR reporting. In 5 months the team of 6:
It was amazing how much the team did in such a short time. The business owner of the project was very please with the direction and progress which was being made, the team was energized to perform the work, and the collaborative atmosphere ensured the best decisions were made with the most buy-in to keep the project moving toward its goals.
October 6, 2010 at 10:13 am
Charles Pockert (10/6/2010)
I think the Indy 500 analogy is flawed in that not all cars on the road are trying to win the Indy 500...
Many people are making the same statements and I'll just add mine.
What the author, David Ziffer, is proposing is a good idea in a sense and something that framework developers have been trying to tackle for years. It is not trivial, mainly because the problem keeps changing.
Like Charles said, if you confine development to a strict set of rules and a very limited set of user cases, like the Indy 500, then it would be very easy to churn out generated code based on a set of features and requirements. In fact, if it was that easy, we'd have a system that already does that dynamically. Just enter your list of fields, their constraints, and roles/permissions and click "Go". Honda practically does this already, they just haven't entirely taken out the human element yet and replaced it all with robots.
However, not only does the domain have practically unlimited user scenarios (including the ones we didn't realize in the first place) but the domain itself is always changing. Churning code has it's place, and leads to Go Daddy $1.99 websites. They are Ok for small businesses who like getting a stable and semi-dynamic website for cheap but it will look just like their competitors with no added value or "new" features.
In the last 5 years we've seen the development and wide-spread implementation of AJAX, REST, OData, WCF, Web Services, cloud services, hosted solutions, etc. While all of these were in some form of implementation before that it is only in the last few years that they have become mainstream and somewhat "required" for any apps developed today. This wouldn't have happened, nor would it be possible to produce, using rapid development/generated code.
I think generation of tried and true practices for specific domains of an application are definitely warranted. Why should a developer have to "design" the primary key structure of a standard Customer's table anymore? Churn out the 20% that is static, but I haven't found in my last few years of development that I can "outsource" more than 20% of my design like this. 60% still goes into designing the features to to work together in a quality efficient manner and the last %20 percent goes into the things I find that we never expected.
October 6, 2010 at 10:29 am
Steve Jones:
However you're lucky. I've worked in quite a few shops where the DBAs did their own thing, and ignored or did not communicate with developers. I've also seen many developers that threw tables together without thinking about design, or without understanding normality and what advantages it brings (or disadvantages). Too often I've seen inconsistent development in all kinds of applications, in-house and commercial.
To me this "software", all boils down to offering a band aid, and a poor one, even if free, for the failure of MANGAGMENT to direct and control the IT environment. To combine and coordinate all aspects of a project, including Users, GUI programmers, Developers, DBAs and what all. Plain and simple Poor Management
October 6, 2010 at 10:31 am
In a typical development the schema design is done by a database guy who is obsessed with micro-optimization of individual tables. He fusses over minimizing table size and deciding things like which tables should get synthetic keys or status fields. The result is a hodgepodge of table designs, all within the same app.
Typical? Not in my experience. The database guy, if you actually have a database guy and not just a developer who knows enough SQL to be dangerous, is usually focused on meeting the business requirements in the most efficient and flexible manner. This makes it sound like the DBA usually goes off into a corner somewhere and tinkers, like some mad inventor, producing a thing of theoretical beauty but no practical usefullness. In my experience, a good DBA talks to anyone and everyone to make sure he has a firm understanding of the entire business ecology before he even starts a design effort.
As a philosophical point, I'd also have to come down on the side of having the database integrity rules in the DB rather than the app. I've seen too many instances where the second and third app that hit a DB just didn't want to go through the layer that the first app went through and so they built a different layer with different rule. The eventual attempt at synchronization was a nightmare.
Just my 2 cents.
October 6, 2010 at 10:41 am
David Ziffer (10/6/2010)
Someone is going to do something similar in software development. When they do, we are going to either be part of the movement or we are going to be unemployed. I'd prefer to lead this movement than to follow it or to be brushed out of the way.
While some shops may move to this method, others are going to remain with the database developers do the database work, the app developers do the app work. I think the point I take from this, more than I need to go out and learn a new tool, is there is a communication issue between App and DB developers/DBAs. The issue is not the design of the system but rather the ego's involved. When you are on a team, you all need to work together. If a shop has issues with someone who is not a team player, that person should be let go so a team player can be brought in. I am lucky enough to work in a team oriented environment but have also been part of the "this is my server stay away" mentality as well.
Even if this becomes the new wave, for a solid developer there will still be work available. The tools are not advanced enough to performance tune what needs to be tuned. Not all problems are solved with WYSIWYG tools. Sometimes you have to dig into the source and when that happens you had better know what you are doing.
October 6, 2010 at 11:48 am
<<To me this "software", all boils down to offering a band aid, and a poor one, even if free, for the failure of MANGAGMENT to direct and control the IT environment. To combine and coordinate all aspects of a project, including Users, GUI programmers, Developers, DBAs and what all. Plain and simple Poor Management>>
I think that the above is a very, very good point as there has to be a top-down commitment to a unified architectural approach, and development standards and technques that includes documenting them, communicating them, assigning responsibility fror them, auditing their use and updating them on a continual cycle. Otherwise every project team whether in-house or contracted will follow their own best practices which will invariably be different. . some better, some worse, but with certainty they will be different.
October 6, 2010 at 12:17 pm
agree with some, but not all of the conlusions. There is value in standardizing the things that it makes sense to standardize.
With that said, first this is an attitude that can be taken too far easily, and second I think that some of the anologies used to draw the conclusions are flawed. As to the attitude for standarization, I certainly agree that the more that can be reasonably standarized the better. It helps everyone understand, it helps assimilate new objects rapidly, and it generally improves quality. But not everything can or should be standarized. You certainly could use an artificial key with the same name for all tables, but this comes at a price (Joe Celko discusses it many times). But if you try to use a natural key for all tables you will run into some where the natural key is composite which creates its own problems. I think it makes sense to decide, if not table by table, then certainly database by database whether you want to use artificial keys. I find the idea of standardized audit fields even worse since they do not make sense for all tables, and unused fields tend to create confusion and waste.
As to the comparison with the auto industry, I find it flawed. There the goal is precisely to mass produce large numbers of nearly identical products with interchangeable parts. It makes tremendous sense to standarize virtually everything in that and keep producing them. But no two software projects are absolutely identical (if they were, you would just copy the first one and be done with it). They must be produced adjusting for the unique situation at hand and the unique requirements of the task, unlike engines where you want thousands that are absolutely identical. That of course isn't to say you can't standarize and reuse a lot, must CRUD apps especially on the front end side are very similar, and good libraries are a move towards standardization, but you can't standardize everything for software.
Another thing to point out is that most assembly line workers are not the same as craftsmen. They just repeat the same thing over and over without trying to optimize for absolute performance. Where you see craftsmanship in that arena still is in custom jobs or after market customization. I recently had my motorcycle mechanic install modified foot rests on a motorcycle that wasn't really designed for them. As part of that, he custom built an extender to move the break pedal up and forward based on the bike, the foot rests I had chosen, and me. That (although simple in this case) was an example of custom craftsmanmship based on specific conditions. Software creation is much closer to that situation, then to assembly line workers repeating the same motions over and over.
---
Timothy A Wiseman
SQL Blog: http://timothyawiseman.wordpress.com/
October 6, 2010 at 12:27 pm
<<But if you try to use a natural key for all tables you will run into some where the natural key is composite which creates its own problems. >>
In my opinion the above statement is incorrect and a very common - perhaps widespread - fallacy. Here's why:
entity definition MUST be based on natural data. the only alternative is to have duplicates. I don't know about you, but I have seen TONS, if not the majority, of legacy databases with duplicates. And guess what. 99% have surrogate unique keys. These duplicates raise havoc with the systems, lower the confidence the business users have in the data quality, etc.
But if they have surrogate keys, then what is a duplicate? A duplicate is what the BUSINESS defines as a duplicate.
And the problem with the statement I have quoted above is that you can, and should, have BOTH.
Use a surrogate key as the PK. Use a unique index - compound most likely - as the NATURAL unique identifier.
Voila, no problems in any situation, AND NO DUPLICATES.
Viewing 15 posts - 31 through 45 (of 84 total)
You must be logged in to reply to this topic. Login to reply