He told the House Ways and Means Committee that the IRS division handling requests for tax exempt status was overwhelmed by a surge that followed the Supreme Court's 2010 Citizens United decision.
"I think that what happened here was that foolish mistakes were made by people who were trying to be more efficient in their workload selection," Miller said, calling the practices described in the inspector general's report as "intolerable" and a "mistake," but "not an act of partisanship."
He apologized for what he later called "horrible customer service," but he also stubbornly rejected any accusation that it amounted to politicizing the work of the IRS.
However, Republicans noted the increased requests for tax exempt status didn't kick in until 2011, months after the targeting began, according to the inspector general's report.
Rep. Dave Camp, chairman of the Republican-led panel, and other GOP members sought to depict the controversy as indicative of government gone wild, with the IRS abusing conservative groups and other political foes of the administration.
Democrats on the committee also expressed outrage at the targeting of conservative groups seeking tax-exempt status, but they pointed out that the top IRS official at the time was appointed by Republican President George W. Bush, not Obama.
They also noted that the inspector general's report stated there was no evidence of any political motivation for what happened, or influence from outside the IRS.
The Treasury Department oversees the quasi-independent IRS. Some Republicans are trying to find a link between the Obama administration and the IRS targeting.
According to the inspector general's report, the IRS developed and followed a faulty policy to determine whether the applicants were engaged in political activities, which would disqualify the groups from receiving tax-exempt status.
The controversial move began in early 2010 and continued for more than 18 months, the report said, declaring that "the IRS used inappropriate criteria that identified for review Tea Party and other organizations applying for tax-exempt status based upon their names or policy positions instead of indications of potential political campaign intervention."
Among the criteria used by IRS officials to flag applications was a "Be On the Look Out" list, which was discontinued in 2012, the report said.
Conservative groups complain their requests were delayed for months or even years through the targeting that sought to prevent ineligible political groups from getting tax exempt status. Miller testified Friday that determining the political nature of groups was one of the hardest tasks of IRS officers tasked with assessing requests for tax exempt status.
The investigation by the Treasury inspector general for tax administration was initiated after congressional complaints began to surface in the media in 2012 that the IRS was targeting conservative groups and holding up applications.
In a written response included in the report, the IRS commissioner of the Tax Exempt and Government Entities Division said there was no criminal behavior behind the actions of the agents, but rather inefficient management.
Obama called the inspector general's findings outrageous and forced Miller's resignation. In addition, the commissioner of the IRS' Tax Exempt and Government Entities Division also announced his retirement Thursday. Joseph Grant will leave in June, according to an internal IRS memo provided to CNN. Miller also is scheduled to exit then.
Obama has appointed Danny Werfel, a White House budget office official who has served in both Democratic and Republican administrations, to succeed Miller through the end of the fiscal year on September 30.