Without knowing what Google's real expectations of the Nexus One are, I will speculate that big sales numbers aren't necessary for the Nexus One to be considered a success. In marketing, the halo effect is a term coined to explain how one product's positive traits translate into increased positive perception of the whole brand. Many companies produce such "halo products", especially in the automotive industry where virtually every car company spends a lot of time and money developing models which sell in tiny numbers. For example, Ford built just over 4000 Ford GTs over several years, while they sell over 2,000,000 cars per year.
While the Nexus One isn't (relatively speaking) as high end as the Ford GT, looking at it as a halo product makes it look fairly successful. Namely, it has brought of a lot of positive press and attention towards the Android platform, while acting as an excellent showcase of what the platform is capable of. This strategy also makes sense when you consider that Google relies heavily on their partners to sell devices. By positioning the Nexus One as a halo product, Google can generate a lot of positive brand awareness for Android and help their partners sell more units, while at the same time pushing their partners to keep up with the rapid evolution of the platform.
For a software company, it is a pretty good strategy.