Carlyle initiated the discussions and was in early stages of due diligence when differences emerged, bringing the talks to an end, a source familiar with the matter told Reuters on Monday.
Management at Nasdaq feels that the company is undervalued compared with its peers, two separate sources said. Nasdaq's board has a fiduciary responsibility to consider all offers.
The talks, first reported by Fox Business Network, occurred about three weeks ago.
Nasdaq spokesman Joseph Christinat said the company does not comment on market rumours or speculation. A spokesman for Carlyle declined to comment.
The talks came just weeks after Nasdaq rival NYSE Euronext said it was being bought by IntercontinentalExchange Inc in a cash and stock deal valued at $8.2 billion when it was announced in December.
Warren Buffett's Berkshire Hathaway in November also bid for the Big Board parent. The Berkshire offer fell short, but it highlighted the interest in the sector.
Of the four private equity firms that trade publicly, Carlyle is the only one listed on the Nasdaq. KKR & Co LP, Apollo Global Management LLC and Blackstone Group LP are listed on the New York Stock Exchange.
Carlyle CFO Adena Friedman joined Nasdaq in early 2011. Prior to that, she had been at Nasdaq since 1993, where she held several posts, including CFO. When she did so, market players immediately speculated that Carlyle would go public on the Nasdaq, which it did in May of last year.
Shares of Nasdaq were up 3.5 per cent at $30.49 on Monday afternoon.
Copyright @ Thomson Reuters 2013