Though it now seems to be legal, I still don't like it. But, like income tax, I don't have much choice in the matter.

In 1979, the supreme Court ruled that there is no expectation of privacy regarding the numbers being called. It appears, this would be equivalent to staking out a potential crackhouse. No warrant is required to record who goes in or to take pictures of them. If the cops want to search inside - analogous to recording conversations - then a warrant would be needed. But, to merely watch the outside - or see who's calling what number - it seems the court has decided that's OK.

The actual supreme Court case is Smith v. Maryland, 442 U.S. 735 (1979)

Here is the relevant part;

===========================
Smith v. Maryland, 442 U.S. 735 (1979)

Surveillance Power: Expanding the Scope of the Government?s Surveillance Power

PART C: Access to Business Records Held in Third Party Storage

The police installed a pen register, without a warrant, at the central telephone system in order to obtain phone numbers that petitioner was dialing from his home. A robbery victim had been receiving calls from the alleged robber. The police found that the petitioner had called the victim?s home and charged petitioner with robbery. Petitioner contended that the use of pen registers violated his Fourth Amendment rights. However, the state court did not agree with this contention. Likewise, the Supreme Court held that pen registers do not violate the Fourth Amendment. They may be obtained without a warrant because pen registers are not a search within the traditional sense. Pen registers help uncover numbers that someone has dialed and individuals should not reasonably expect that this information is private since the dialed numbers are automatically shared with the telephone company. Therefore, the Court held that the pen register was a proper form of investigation.
===========================

And, here we have the complete decision.