I spent the night in bed with R.A. Fisher's The Genetical Theory of Natural Selection. The book was published in 1930; I got my hands on a 2000 re-print plucked from the RPI garbage bin. Its forward and annotations by H.A. Bennett as well as the book's former owner were also enlightening -- I review those at the end.
Sexual selection is an expansive topic. It is also one with a complicated history and fits messily into rigorous empirical research. Though progress in formalization of sexual selection theory has been significant within the past 25 years, the major issues and concepts were sketched out by Fisher.
Prior to Fisher sexual selection was discussed quite extensively by Charles Darwin, but unlike natural selection it was roundly rejected. Both Thomas Huxley and Alfred Russel Wallace attacked it as implausible and inconsequential, and many contemporaries found the idea of female mate selection as a driver of evolutionary change ludicrous. Fisher's discussion of the topic was one of the few serious examinations of the topic before the contemporary period subsequent to Darwin. The whole sordid tale of the culturally rooted rejection of sexual selection theory is detailed in The Mating Mind, Geoffrey Miller's attempt to reinterpret human evolutionary history through the lens of this process.
There are multiple forms of sexual selection. First, one must distinguish between intrasexual selection, and intersexual selection. The former consists primarily of competition between males which results in differential access to, or superior utilization of, mating resources. A common example might be ritual combat which determines the extent of a male's territory, with that territory being proportional to the number of resident females whom the "owner" of the territory may have an opportunity to mate with. Another example of intra-sexual selection is sperm competition, that is, differences in the nature of sperm (e.g., in motility or seminal viscosity) which results in different likelihoods of fertilization. Inter-sexual competition consists primarily of female mate choices which result in differential mating success across the males within the population. The main reason that females are the focus of inter-sexual selection is that in most species of interest they are the limiting sex in relation to natural increase, i.e., a small number of males can inseminate innumerable females. This does not mean that inter-sexual selection is exclusively female to male, and some have argued that humans are one of the primary exceptions to this rule of thumb, as in our species females are subject to mate choice as much as the inverse situation.
The reason that sexual selection theory emerged was the existence of costly ornamentation among male birds which seemed to have no natural selective advantage, and might even imply disadvantage. Charles Darwin hypothesized that female choice for these extravagant traits as secondary sexual characteristics in males was the root cause of this phenomenon. There are several rationales for inter-sexual selection:
1) Good genes. That is, the individual with extravagant traits is advertising their genetic fitness. This is tied to the Handicap Principle, whereby possession of a fitness reducing trait in an adult may indicate that their underlying superiority in bearing the cost of the trait. The Handicap Principle's logic is that if a trait is highly costly it serves as an "honest" indicator of genetic fitness because less fit individuals would not be able to afford the risk which that trait would entail.
2) Sensory bias. This refers to the concept that a bias exists for particular shapes or colors because of other adaptive behaviors. For example, a fruit eater might be drawn to bright colors (reds, yellows, etc.) which stand out against the green vegetative background. These fruit eaters might then also exhibit some preference for conspecifics with similar traits, and these visual biases might be utilized in identification of conspecifics and so reduce the likelihood of mistaken matings with closely related species which exhibit similar phenotypes.
3) Finally, there is the dynamic of Runaway processes which draw from small initial arbitrary differences. Again, note that sexual selection's initial root in secondary sexual characteristics of a trivial nature might simply be a way to conveniently differentiate closely related species. Remember that mistaken matings or hybridizations severely reduce the fitness of an individual, so such preferences would be transparently beneficial. But, over time an initial preference for trait x by females begins to take off as males with the trait mate with females who prefer it. The offspring then carry the trait (sons) or the preference (females). The rate of increase of the increase within the population is proportional to the maximum extent of the male trait, so males which exhibit trait to a more extravagant extent sustain further fitness advantages vis-a-vis their less extravagant peers. Though the initial advantage was likely tied to fitness in a straightforward manner (as in the conspecific identification rationale above), subsequent increases in fitness for males who carry the trait are driven by a feedback loop, as males with the trait and females with the preference continue to produce broods. If the trait begins to reduce an individual's environmental fitness (e.g., the trait results in easier capture by predators), at an advanced stage of runaway the female preference for the trait may become so strong that males who lack the trait, even if their intrinsic fitness is higher, will not be able to find a female mate and so will not capitalize on this advantage. Similarly, females that prefer males without the trait will produce sons who are disadvantaged in a population where the vast majority of females prefer the trait that they lack. Of course, ultimately his runaway process will be constrained by natural selection, and one may imagine a situation of metastable equilibrium as the mortality of hyper-extravagant males is balanced by their reproductive successes in their short lifetimes, vs. the lower mortality of dull males who nevertheless are characterized by a lower rate of matings.
Fisher's conception of sexual selection implies that these traits may emerge in short and quick bursts, and then stabilize as natural selection constrains further development toward a phenotypic extreme. Additionally, there is the consideration of sexual dimorphism, as one notes that in many (most) bird species if the male is characterized by bright plumage the female is rather dull. Since aside from sex chromosomes there is no average sequence level genetic difference between males and females the contrast must be due to sex hormones which mediate the developmental path of phenotypic expression. One can imagine a situation where bright plumage tends to reduce environmental fitness due to increased predation risk, females who prefer said plumage will tend to have male and female offspring which are at increased predation risk. Since females are the limiting sex in terms of natural increase of a population this is a natural break on the process of phenotypic evolution, as the optimal state is one where females are dull (so at lower predation risk) while males express the trait and so only the fittest may evade predation (in other words, reproductive skew is increased). Ultimately the existence of sexual dimorphism suggests that genetic modifiers emerge which induce a dependence of plumage development on sex hormones. But, the necessity of supplemental mutations and evolution acts as a break on the velocity of sexual selection as these modifiers tend to come to the fore only slowly. Some have even suggested that the evolution of dimorphism is, on average, about 1/10 the "speed" of conventional phenotypic evolution that affects both sexes.
Sexual selection has recently become a very active area of research. H.A. Bennett refers to Malte Andersson's work and Russ Lande's formal models of the process. Annotations of the book's former owner bring up Peter Frost, who has proposed that male mate choice is one of the primary determinants of the emergence of blonde hair in Europe. Nevertheless, we should be cautious of the ubiquitous use of sexual selection, it often becomes a deus ex machina which can explain every mysterious character with a wave of the hand. After all, runaway in particular is arbitrary and capricious, and so exhibits the scattershot characteristic of genetic drift.
An empirical exploration of sexual selection needs to accompany theoretical models, but this is easier said than done as gauging the fitness differentials due to variation in a character is often difficult in an ecological context, and supplementing this with an understanding of that character's genetic architecture (e.g., is the character heritable?) is often less tenable. And yet the reality is that a deep genetic understanding of the nature of the dynamics we posit is essential to putting the genie in the bottle.