I've now narrowed my level generation methods down to two. Previously I was also using randomly placed rooms and BSP trees, but with one the corridors were too long, and the other suffered from terrible predictability.
The two I'm using are (1) cellular automata and (2) collision based room insertion.
In the maps above the D and E maps are type 1, a kind of cavern like level. There are several inputs like coverage and smoothness. They basically generate these two extremes, very open, joined caves, or sparse tunnels. I may add double thickness corridors to the (E) map as otherwise they are annoying to play (no room for maneuver).
Type 2 maps start by piling up a bunch of random rooms in the center of the map. Then they run collision detection to push them apart. (A) is a map with quite high padding value, which places the rooms far apart. (C) is the same method by a padding value of 1. The rooms are quite close together. Corridors are short, which is a good thing.
(B) and (F) have a "hole" specificed in the center of the map. Rooms won't be placed here and so get pushed further out for placement. What goes in this hole? A prefab!
I can specify the hole to be placed anywhere and any size, plop a prefabricated room or section in there and hook it up to the rest of the map. Now I have special puzzles, boss encounters, treasure vaults and many more possibilities. It would also be possible to generate a different kind of map there, like a cave system using generation method 1.