Caveat lector: I will by no means try to give the complete story. I am especially not interested in exhuming the very early papers written by prophets, whose meaning and depth are only now appreciated.
Instead of doing this, I will try to give a broad outline of the changes in the theoretical fashions that the vast majority of physicists have followed over time.
The very idea of a scalar field appeared to be trivial after the development of special relativity and a clear understanding of the nature of the electromagnetic field. If we know that the electric and magnetic field together form a tensor in Minkowski space and that the electromagnetic potential is a vector in Minkowski space, why shouldn't we try to develop a theory of scalar fields, since this is the simplest case of a relativistic field?! In most of the popular textbooks of 40 or 50 years ago, such as, for example, Wentzel's Quantum Theory of Fields (Wentzel, 1943), the scalar field is discussed right at the beginning, because it is the simplest example, and had no authors' names associated with it. The real triumph and excitement came when the Japanese physicist Yukawa proposed a scalar field with a rest mass in order to explain the nuclear forces. This proposal aroused enthusiasm because the two main properties of the nuclear forces that distinguish them from the electromagnetic forces were immediately explained: the attraction of like particles (mutual attraction of the nucleons in the nucleus) and the short range of the nuclear forces. This range, as estimated from nucleon scattering and the size of the nucleus, corresponded to a meson mass on the order of 100-200 MeV (for comparison, mec2 = 0.5 MeV for the electron and mpc2 = 938 MeV for the proton); so, Yukawa had predicted the existence of intermediate-mass particles - mesons.
The enthusiasm turned into euphoria when mesons of approximately
the required mass were discovered in cosmic rays. Soon thereafter came
a comic moment: the mesons that were discovered first had a very small
cross-section for interaction with nucleons. They were found to be
µ-mesons, which are "heavy electrons". However, the required
-mesons were found very soon
thereafter. Their mass was approximately
140 MeV, exactly within the range predicted by Yukawa!
The next development was connected with Kemmer's extension of
the similarity of neutrons and protons (the particle physics name for
this is isotopic invariance) to mesons. It turned out that three
kinds of
mesons were needed: +,
-, and
0. The charged mesons decay into
a muon and neutrino (this is precisely what caused the initial error).
The neutral 0 meson must
decay into two photons. This prediction,
together with an approximate probability calculation, was made by
Robert Oppenheimer just before he went into neutron stars, black
holes, and atomic weapons. The
0 ->
2
decay was
experimentally
observed. It seemed that the sky was cloudless for the scalar (or
pseudo-scalar - the difference is unimportant for us) theory of nuclear forces,
and a wide field was opened up for theoretical and experimental
work.
However, this turned out not to be the case. The detailed predictions of the scalar theory were experimentally disproved.
On the other hand, many unexpected new particles were discovered: first, the strange particles (and, later, the charmed, beautiful, and, in the eighties, flavored particles), and then the "resonances," which one can call excited states of particles. All of this led to the modern picture of QCD (quantum chromodynamics) with quarks and gluons mentioned above.
What is important for us is that the pions were dethroned - they were composed of one quark and one antiquark. The large number of different "elementary" particles is merely an indication of the fact that they are composite particles. Many of them are not different combinations of quarks, but various excited states of the same combination of quarks.
The attitude toward the nuclear force has also changed. The protons and neutrons are analogous to noble gas (helium, neon, argon, . . . ) atoms, with the gluonic forces holding the quarks together in the same way that the electrostatic force holds the electrons together in an atom. Continuing this analogy, one can say that the nuclear forces are like the van der Waals forces that hold these atoms together in a liquid droplet (liquid helium, neon, ...). This analogy is confirmed by the relative smallness of the nuclear forces: the binding energy of the protons and neutrons in the nucleus is of the order of 10-20 MeV - very large compared to chemical energies (electron volts), but small compared to mpc2 = 940 MeV.
We shall now return to the scalar field. For some time, it came into disrepute. The theoreticians had become enamored of vector fields; in particular, those associated with gluons in the strong nuclear interaction and with the W+ and Z0 particles in the weak interaction. No experimental evidence was seen for genuine scalar fields; the pions were bosons, but not elementary bosons - they were like alpha particles (helium-4 nuclei).
The fact that 4He is a boson was apparent from, for example, the superfluidity of helium-4, but nobody was crazy enough to treat helium-4 as an elementary field and put its wave function in the fundamental Lagrangian of elementary particles.
I must repeat that my version of the story of modern physics is highly individual, and the contrasts are strongly exaggerated. A more accurate and cold-blooded observer would find some papers on vector fields published during the reign of the Yukawa theory and, likewise, some papers on scalar fields published during the excitement about vector fields (gauge fields, symmetries, etc.). But I only intended to describe the general tendencies.
The new wave of interest in scalar fields was not a continuation of Yukawa's idea: it had a very different ideological basis.