Web publishing platforms face a conflict between putting everything in editors hands through web forms, while also attempting to be application development or integration platforms.
Lets face it, integrating with drupal often feels like we are in an epic novel that runs over 3 volumes when it could easily be edited back to one. We feel like short, hairy-footed characters that face all manner of surprising obstacles. However it is significantly better than other popular web publishing/web content management system platforms, Wordpress or Joomla!. The price is right, it has a strong brand and a large community. It also had a better power-to-weight ratio and more functionality, for the deployment overhead than other web publishing patforms we have used over the last fifteen years including commercial products that cost 5 or 6 figures per year.
We have experience in two types of sites, ones which follow a common pattern and require little custom work and the second type that might take months of development work to put together. The main recommendation is for the development of a culture of testing. This will help those cases where people are doing the driving using the web interface as there will be fewer surprises. It will help to an even greater extent from the perspective of this article, in undertaking custom integration.
Testing
Submitting a module to Drupal results in a bombarded with 'code quality' checks that are really no such thing. The reward for helping review modules process seems to have encouraged the development of automated programms that check indenting and whether you have @file at the top of your file. These things may have their place, but give me a break. Since we have development environments with templates restyling is just a matter of applying the code style template and selecting 'reformat' and voilà, the code style is applied. If you still have issues with my style, then raise a bug with the code template maker. We grew out of this programmer infatuation in the 90s. We like to code with style and we don't want our version management system clogged up with junk changes, but this misses the point.
No module should be able to be approved without test cases that clearly show fitness for purpose and safety of the module. Each module in the system should have a visible test report - even the PERL archive (CPAN) had this from the early 90s. Bug reporters should be encouraged to submit a test case. The collection of test cases should be actively developed into a comprehensive system over time. It is not that tests are non-existent (see http://qa.drupal.org/pifr/test/425658), just that it is too superficial and a culture does not exist that puts a focus on testing.
The test system needs to be able to be run from and IDE. Being able to debug and step through is a basic programmer productivity tool. Simpletest the Drupal standard for testing, does not cut it. We are not in the 70s or 80s where printing debugging with error_log() cuts it anymore - and dpm, dvm,dpr,dvr, print are all just derivatives of printing rubbish to error logs. Such output doesn't provide the contextual detail needed, it creates rubbish code and error logs that are forever filled with junk only meaningful to the programmer who wrote it. UPAL has put up a straw man that might help.
Who said it that any code without test cases was 'legacy'?
Quality
More important than superficial style checkers, is the stamping out of modules that don't work out of the box, or cause harm. I have installed modules that had bugs that tried to remove all the files on the system. And a page of 'me too' but no fix for 6 months. These modules and any changes to them that cause these behaviours need to be caught rapidly and the module flagged or quarantined. A million patches might create a lot of code, but without predators, evolution itself would cause disarray.
Test case reports can help force quality back into contributed modules. Rather than 'installed on avagadros number websites' the metric should be some measure of quality. This measure, test cases passed, number of reviewers, average time to bug fix etc should be prominently displayed. 1000 bugs open should be a source of shame, not pride (from both a quality and code complexity point of view). People should get a warning when they are installing the module that known bugs exist.
Oh and even drush has some features that typically 'fatally fail' when run. This core tool should not present options that fail or don't apply to a particular platform.
Error Checking
Drupal code has a poor discipline of error checking. Certainly outside of the core it is not common practice. This is in spite of just about every error - empty array elements or object properties - being fatal in php. It would be helpful if examples and tests focused on providing patterns that focused on catching errors earlier.
I have trouble with watchdog messages that are truncated, just before the function name and line where an error was caused and the lack of stack-traces. Yes I typically mangle my development version of drupal to change the last-resort error handler to identify hard to catch errors.
Interfaces
I think that the argument regarding flat functional/procedural programming versus some sort of hierarchical system (e.g. object oriented) is again very 1990s. I don't think it matters how this is done. We have Ruby on Rails, Grails and other systems that generate scaffolding required - controllers, views, tests etc. It just works. Yet with Drupal we have to spend half our time reproducing boiler-plate code. Menu code, theme code, hook code and so on that is relatively difficult to debug and maintain.
These patterns, e.g. how to define a web form, how to modify an existing one, how to save some data, should all be streamlined so that we can focus on the new things. It doesn't matter how this is done - templates in an IDE or inheritance - it is important that the every module is a CMS within the CMS and does things slightly differently approach is changed.
On the subject of interfaces, I haven't found any guidelines that indicate when a module should be moved from dev status to release status. Bugs outstanding might be one metric, but then views would not have a stable release. Stable interfaces may be another measure, but then organic groups would similarly not be stable. Harping back to test cases, we could build a definition of when code becomes 'stable' depending on both numbers of test cases added - stable interfaces - and the importance of the ones that fail.
Delineation
"You can do anything without programming". This is a lie. Programming is an explicit and effective way to do things. We build something by clicking on buttons, and then come back the next day and can't do it the same way again. Spend two days bending our heads around some combination of panels, views, blocks, permissions etc. With enough effort, it eventually can be forced to work, but it is much harder than it needs to be.
Similarly, the poor person that has never been trained in logical thinking goes about trying to build a website for someone with the notion that 'anything can be done by clicking'. It can't. I have watched people get excited by building a few forms and imagine "adaptive learning intranets" or even just "Amazon" and promise it to a client. We have to communicate what the limitations are before people head off making promises. We do ourselves a disservice in making promises that 'everything is easy'. Programming is often hard, it takes inspiration, time and commitment. Programming is undertaken in a complex part social and part technical environment a complex environment that is part social and part technical focus that can't just be plucked out of a web form.
Lets define some roles and define some things that can't be done without programming. At least then people don't try things and find out half way. A person with design knowledge should be able to run through the site without being directly exposed to relational logic, as is the case with views. A programmer should be able to build a module, have a generated view and move on, comfortable that a designer will be able to put gloss on it.
Traceable code versus web-form editing leads to general version management of system changes.
Persisting Code and Version Management
Version management should be in everything these days. That goes for you - nodes. Although we are talking about site building not content management. Not being ale to trace historical changes is a major problem. It takes one moment of experimentation or accidental clicking to create something we no longer have faith in how it works with changes we can't reverse. The second site takes just as long to build as the first one because it is hard to reproduce the steps and it is impossible to have development, integration, test, UAT, production sites. People like AGEIR have had a go at managing this problem, however being able to trace a build and configuration changes is integral to accurately building websites.
I'd kick views back out of the core - it goes way too far in catering for the "build anything without code crowd" - and put features in core. Everything configured via the interface should be able to be persisted as a file and version managed. This is the breakthrough that XML gave us - albeit in a wordy way. OK - let views back in but cut it back to 'list the last 5 things of type x' . Features adds to the reproducibility and tracking of solutions while views has a tendency to be an over-complicated noise and randomness generator.
Documentation
The module installation documentation is not documentation. Looking at the source code is great, but is is an extremely verbose text and one that doesn't actually show the intention of the programmer or the intention of how the code is to be used.The Drupal module pages are also a bit self referential for a first time user. The front-page description of how to install a module is often just the read-me file and the documentation link returns you back to the same content. Another web-design irk is the use of node numbers instead of module/fancybox in the URL.
A documentation template would be great for each module. Every module needs to have 'what this module does', 'getting started with the module', 'common configuration changes'. Rather than reading a great description 'this module will solve your life problems' then 'please install using the module installation instructions' and then no 'getting started' in which case you often have to start reading hook_menu() to work out where things are.
The documentation template could be part of a module infrastructure. I don't advocate texttile (wiki like markup engine) - but grails documentation engine provides some ideas on standardised sections and style. In fact Drupal is a CMS - perhaps when a module is enabled, it could come with an edit module documentation form. Save the Drupal-doc back to the module as per features. Perhaps even a drupal-doc-patch-submission tool built into module infrastructure.
There is a general problem with 'obsolete stuff'. Searching through drupal.org results in version 4 and 5 modules, comments and documentation. Yes - and comments on how to search drupal.org by putting site: into a google search. All this while drupal has the excellent solr module running the search and plenty of community filtering/voting solutions. Delivering relevant content is Drupal's core business.
Automatically closed -- issue fixed for 2 weeks with no activity.
Interfaces
Organic Groups has changed interfaces twice in the two years since we started working heavily with it. This has meant an almost complete reworking of all the code we wrote that interacts with it. Modules should be encouraged to provide an interface (list members of group) that is beyond arbitrary hooks. By stepping up a level from raw database access we will have less duplicated code and it should be less vulnerable to changes. So I might re-do the whole of Drupal as a messaging system - but it doesn't have to be that much of a technical change. Just providing an 'API' or Model-View-Controller (MVC) pattern and encouraging it's use - through making it easier to do than not would help. Again scaffolding, IDE tools or hierarchical code structure may help.
Modules are also deliberately seperate entities, in my experience some duplication is better in more fluid program types, of which web publishing platforms are. However, duplicating everything is madness and some dependency management systems show some promise e.g. Maven, groovy grape. Grooovy and grape have moved to apache in 2015. Being able to flag things as re-usable interfaces would help break the problem of 'how do I write the sql code for this'... again.
Patterns
Views is a CMS in the CMS. So is Biblio, as is Organic Groups, as is Ubercart. Everybody is reinventing the wheel. This design does affect the interface of the program. You need to go into Organic Groups to set permissions that aren't in the permissions section. The policy should be to strongly consider compliance with a structure and making it easier to follow the structure than not. Brooks says that re-using software is the only way to beat 'no silver bullet'.
Again this is not a point of I am smarter than the next guy because I can list all the gang-of-four-patterns. All too complicated, although drupal with only the odd reference to them is also hugely complicated. I am thinking more along the lines of an 'opinionated structure' where there are well trodden ways to do things and tools that support this, make it a no-brainer and increase productivity.
Separation of Concerns
This is also related to people being able to specialise in the development of a website. Designers should be able to design without having to build things, programmers to just build things and Search Engine Optimisation experts SEO, or Information Architects build the information architecture. Traditionally it was database/model, programmer, designer although recently we get the guy that puts it all together and a whole group of people that specialise in interactive elements. Drupal 7 is a lot better than 6 in this regard. Still we have interface (HTML) code bleeding into programs. We have trouble generating accurate html5 because we can't find all the places where HTML is generated. There is HTML everywhere. Viewing this from a level higher, unless we define roles explicitly, we are not able to separate them.
Conclusion
In conclusion, we aren't giving up on Drupal yet. We might cry in anguish at the some simple things that are made harder than they should be and the things that should be logical but aren't. The truth is that we cry at the interface, but no other system does better, or no other system can give us a chance at seeing so many interesting real-world problems being solved in innovative ways and being able to leverage from this. It has been a long slog, but we have been able to develop a bunch of solutions that can be rolled out efficiently and solve good problems that customers have. When it all looks too scary we have middleware solutions. We have used other systems, and the grass just isn't greener. It still all seems that it shouldn't be this hard.
My opinion is that if Drupal did nothing but focus on testing, making it easy, developing a culture and ranking everything by the level of tests provided Drupal would result in a far better experience, both for the 'build a website by clicking' crowd and those extending Drupal.