Of those applications still open for review, 160 cases were open from 206 days to more than three years -- through two election cycles.
Among the recommendations made by the Treasury inspector general: The IRS better document reasons why applications are chosen for review, develop a process to track requests for assistance, develop and provide training to employees before each election cycle and immediately resolve outstanding cases.
The report also called on the Department of Treasury to develop guidelines to explain social welfare activity -- the primary factor in obtaining tax-exempt status.
"Although the IRS has taken some action, it will need to do more so that the public has reasonable assurance that applications are processed without unreasonable delay in a fair and impartial manner in the future," the report said.
Lois Lerner, director of tax-exempt organizations for the IRS, acknowledged Friday that the IRS had targeted some groups for further review because they had those words in their names.
She said the activity took place at the IRS office in Cincinnati, which handles applications for 501(c)(4) status.
But documents suggest at least three other IRS offices did the same.
Letters provided to CNN show IRS officials in Washington and California contacted conservative groups to demand more information before approving the groups' requests for tax-exempt status.
The American Center for Law and Justice, a legal group representing numerous conservative organizations, provided CNN with four such letters: one each from IRS offices in Washington; Cincinnati; El Monte, California, and Laguna Niguel, California.
The IRS did not respond to CNN's request for comment regarding the letters.
White House spokesman Jay Carney told reporters Tuesday that he's "confident" no one at the White House was involved in the practice.
Asked about complaints by some Republican lawmakers for the past couple of years that conservative groups were being unfairly targeted, Carney said he is "sure some people knew about the stories. But we were not aware of any activity or any review by the inspector general."
Some Republicans criticized the president for not speaking out on the issue immediately. He said he learned about it through news reports Friday.
The Republican-led House Ways and Means Committee, which oversees the IRS, announced it will hold a hearing Friday. Slated to testify are Steve Miller, the acting IRS commissioner; and the Treasury inspector general investigating the complaints, J. Russell George.
Sen. Carl Levin, D-Michigan, said the Permanent Subcommittee on Investigations -- which he chairs -- now needs to expand an investigation already under way.
That one has focused on the IRS's "failure to enforce the law requiring that tax-exempt 501(c)4s be engaged exclusively in social welfare activities, not partisan politics," Levin said in a statement. The IRS' announcement about targeting of some conservative groups raises questions over its impartiality in doing so, he added.