Privately-held companies are - no surprise here - privately held. This means that, in most cases, the company is owned by the company's founders, management or a group of private investors. A public company, on the other hand, is a company that has sold a portion of itself to the public via an initial public offering of some of its stock, meaning shareholders have claim to part of the company's assets and profits.
One of the biggest differences between the two types of companies deals with public disclosure. If it's a public U.S. company, which means it is trading on a U.S. stock exchange, it is typically required to file quarterly earnings reports (among other things) with the Securities and Exchange Commission (SEC). This information is also made available to shareholders and the public. Private companies, however, are not required to disclose their financial information to anyone since they do not trade stock on a stock exchange.
The main advantage public companies have is their ability to tap the financial markets by selling stock (equity) or bonds (debt) to raise capital (i.e. cash) for expansion and projects. The main advantage to private companies is that management doesn't have to answer to stockholders and isn't required to file disclosure statements with the SEC. However, a private company can't dip into the public capital markets and must therefore turn to private funding, which can boost the cost of capital and may limit expansion. It has been said often that private companies seek to minimize the tax bite, while public companies seek to increase profits for shareholders.
The popular misconception is that privately-held companies are small and of little interest. In fact, there are many big-name companies that are also privately held - check out the Forbes.com list of the largest private companies in 2006.
(For further reading on this subject, check out Policing The Securities Market: An Overview Of The SEC.)