If you read a book on orchestration, you'll be lucky to see much about chamber groups, as these books typically are focused on the orchestra (it is called orchestration) or other large ensembles like concert band or a jazz big band. If they talk about chamber groups, they'll talk about the traditional ensembles like a string quartet (2 violins, viola, cello), woodwind quintet (flute, oboe, clarinet, horn, bassoon), and various subsets or augmentations of these ensembles (with a piano).
However, there are a wealth of other ensembles available, many of them just cultivated in the music of the 20th century. This page describes them (reporting on my own experience and information gathered from the Wikipedia discussion of ensembles).
I talk about the homophonic ensembles first (members come from the same orchestral family) before discussing some mixed ensembles (members come from different orchestral families).
Clarinet Quartet (3 [soprano] clarinets, 1 bass clarinet)
Four members of the clarinet family. Though the instruments above are perhaps the most common, in principle, there seems to be no reason why other kinds of clarinets such as sopranino (E-flat), alto, and contrabass couldn't be used in place of or in conjunction with the soprano clarinets (of course, specific groups might have specific requirements or restrictions). An ensemble with the same allotment of parts, but multiple players on each part is a Clarinet Choir.
Reed Trio (oboe, clarinet, bassoon)
An ensemble consisting of the orchestral woodwinds with either single or double reeds. Replacing the clarinet with an English horn yields a double reed trio – in my opinion, a much more homophonous group.
Saxophone Quartet (usually soprono, alto, tenor, baritone)
This is a relatively well-known group whose literature has grown substantially over the years. The saxophone's long-time association with more popular genres gives this ensemble ample opportunity for crossover between these genres and the classical tradition, in addition to exploiting the unique features of the instruments. It is worth noting that saxes are relatively loud and, thus, the sax quartet is less congenial to the delicate "ambience" than many other chamber ensembles – this could be a drawback or a desired quality, depending on the situation.
Horn Quartet (4 [french] horns)
The standard orchestral complement of horns isolated by themselves. The wide range of the horn offers quite possibly the most flexibility of a quartet of any one kind of single-line instrument.
Brass Quintet (2 trumpets, horn, trombone, tuba)
There is little reason why the brass quintet is not on a par with the string quartet and the woodwind quintet, except that (a) brass instruments are relatively loud and, thus, allegedly, not conducive to intimate setting of chamber music and (b) valved brass instruments are a relatively recent invention. But since World War II, these "drawbacks" have been dispelled and the brass quintet has been collecting an extensive literature, more than many of the other groups discussed here. Some brass quintets are augmented by a percussionist, often a drum set player.
Trombone Quartet (4 trombones [at least 1 bass trombone])
The standard complement of trombones in a concert band or jazz band, isolated by themselves. This ensemble offers much of the same sort of opportunities as a horn quartet, modulo features of the trombone. So this ensemble will &ndash almost by necessity – be lower-pitched, likely be brassier, and can utilize the (in)famous trombone glissando.
Tuba-Euphonium Quartet (2 euphoniums, 2 tubas)
In essence, just a quartet of tubas, as euphoniums function as the higher-pitched and more technically flexible member of the tuba family. Though these instruments function as bass brass instruments in larger ensembles, the flexibility and general neat mellow sound of this ensemble should not be underestimated.
Percussion Quartet (4 percussion)
The most open in actual instrumentation of any of the groups. Percussion parts in the Western classical tradition are generally "custom-made" for each piece and percussion quartets, in general, are no exception. For the best result, the composer should develop a setup of instruments for each of the four individual percussionists and then write accordingly. The grouping of glockenspiel, xylophone, vibraphone, and marimba gives the best percussion equivalent to a string quartet or brass quintet (that is, a collection of instruments of definite pitch that make a wide space of the pitch spectrum as well as the various types of production available), which could be treated in the style of these more traditional ensembles. However, the plethora of other percussion instruments, both of definite pitch and indefinite pitch, offers a wealth of timbral opportunities that the composer should not whimsically ignore.
Sometimes percussion quartet pieces are just marimba quartets, which can be economically written (with certain range limitations) on 2 instruments: 2 players per instrument. Pieces for the famous NEXUS percussion ensemble are written for percussion quintet. A feature of percussion ensembles generally is that an additional player does not present nearly the same kind of balance and blend issues as an added string or woodwind player would to a more traditional ensemble.
Piano Duo plus Percussion (2 piano, 2 percussion)
Like several of the ensembles mentioned on this page, the piano duo plus percussion ensemble has its roots in a specific piece where this instrumentation was custom-chosen for it. The "genesis piece" for this ensemble is Bela Bartók's Sonata for Two Pianos and Percussion. However, George Crumb's Music for a Summer Evening is another widely-known example of the genre.
It is debatable if this ensemble is as homophonous as the other ensembles in this homophonic ensembles section. Certainly percussion and piano blend together in a less complete way than an ensemble like the double reed trio. However, composers have generally treated the pianos very percussively in pieces with this grouping, so I think this ensemble is more homophonous than one might initially think.
Bartók's original utilizes what might be considered the traditional orchestral percussion instruments and he centers one percussion part around timpani and the other around the xylophone. However, new entries to this genre need not follow that mold and may instead follow the lead of Crumb, where the entire palette of percussion colors is open for deployment in a piece for this ensemble.
Guitar Quartet (4 guitars)
Much like the horn quartet, the guitar quartet features four very flexible instruments. Since guitars can play both single lines and chords, the guitar quartet can easily suggest both an intimate small ensemble or a much larger ensemble, as the musical circumstances warrant. Furthermore, with a large range (encompassing the entire middle part of the traditional Western pitch spectrum), the guitars allow for a wide variety of pitch distributions, including spacing of larger distance, less available in other quartets of a single instrument.
Pierrot Ensemble (flute, clarinet, violin, cello, piano, percussion)
This is the chamber ensemble par excellence of the 20th century — just about every composer born after 1970 has written for this ensemble and quite a few composers of the previous generation have also written for it. The ensemble derives from Arnold Schönberg's piece Pierrot Lunaire, hence the name. The percussionist was not present in Schönberg's original lineup and a singer was: some pieces either lack a percussionist and/or add a vocalist. Yet with the instrumentation as given above, this chamber ensemble has an equal number of winds, strings, and percussive instruments. Though works for the ensemble date just from the 20th century, the ensemble bears a certain resemblance to Renaissance ensembles known as "broken consorts," which also had a mix of winds, bowed strings, and percussive strings. The Pierrot ensemble also conventionally has certain instrumentalist doublings available, dating from Schönberg's original: the flute with piccolo and alto flute, the clarinet with bass clarinet, and, somewhat unusually, the violin with viola.
Subparts of this ensemble have also been utilized: the Violin-Clarinet-Piano trio has a literature, and Messaien used this ensemble minus flute and percussion in his Quartet for the End of the Time.
Flute-Viola-Harp Trio (as the name says)
Like the Pierrot Ensemble, this ensemble derives from an original "genesis piece": in this case, Claude Debussy's Sonata for Flute, Viola, and Harp. The lineup suggests a light feel to the orchestration (though an interesting challenge would be to make it heavier). Reputed to have the second largest 20th century literature, after the Pierrot Ensemble.
Flexible Music Ensemble (tenor saxophone, guitar, piano, percussion)
This ensemble is the line-up from the contemporary group Flexible Music, who, in turn, derive their instrumentation from the piece Hout [Wood], by Louis Andriessen. For this piece, Andriessen uses marimba and woodblocks for the percussion, but other composers have utilized other percussion setups. This ensemble is very similar to the standard jazz quartet (cf. the above with sax, piano, bass, drums), easily allowing for crossover or genre-bending writing, in addition to new explorations within the classical tradition.
Soldat Ensemble (clarinet, bassoon, trumpet, trombone, violin, bass, percussion)
The instrumental forces of Igor Stravinsky's Histoire du Soldat. Though not as famous as the Pierrot Ensemble (in fact, I know of no pieces which precisely "re-use" Stravinsky's instrumentation), the principle is similar: a selection of instruments from each orchestral family. Stravinsky's grouping specifically involves high and low instruments from each family, except the percussion (which arguably has high and low instruments, too; they are just played by one individual). William Walton's Façade utilizes a very similar kind of group (flute/piccolo, alto sax, trumpet, clarinet/bass clarinet, cello, and percussion). In both pieces, the percussion part is either literally to be executed on a drum set or written to evoke one.
Le Marteau Ensemble (voice, alto flute, viola, guitar, vibraphone, xylophone)
A proposed ensemble using the "genesis piece" of Pierre Boulez' Le Marteau Sans Maître [The Hammer Without Master] (I know of no other piece that uses this group). The grouping is interesting because it focuses on the middle register and each instrument shares a certain characteristic with a pair of the other instruments (as nicely depicted here). In the original, Boulez employs these instruments in typical Boulezian style, but I imagine that this grouping could give rise to numerous different kinds of musical expression.
Sinfonietta (flute, oboe, clarinet, [saxophone: optional], bassoon, horn, trumpet, trombone, 1 or 2 percussion, piano, [guitar: optional], violin I, violin II, viola, cello, bass)
In essence just a small symphony orchestra – the instruments are roughly those standard in the early Romantic orchestra, except one to a part. In terms of parts, it's about as much work as an orchestra piece, but, of course, it occupies a distinctly different "sound space," due to the reduction in actual forces. Quite a few contemporary organizations exist with this instrumentation, so it appears to be a genre ripe for future expansion.