The Boston Planning and Development Agency’s board voted 3-1 Thursday to allow the nonprofit owner of the Harriet Tubman House in the South End to sell the building to a condo developer.

The board lifted a deed restriction, which dates to the urban renewal era, on the historic building at the corner of Columbus and Massachusetts Avenues. The United South End Settlements, a nonprofit social service agency, owns the Tubman House. The board’s evaluation of the measure was peppered with outbursts from audience members frustrated that public officials were allowed to express opinions while general public comments were prohibited.

Board Treasurer Carol Downs asked Michael Sinatra, a BPDA project manager, to summarize the opposition’s concerns.

“A lot of folks feel that the organization, because they cut some services, has forgotten about certain parts of the community. So that’s pretty much it,” Sinatra said to an eruption of outrage over what many opponents called an inaccurate summarization. Board Vice Chair Pricscilla Rojas gaveled the crowd back to order, reminding attendees “this is not a public hearing.”

Up until the vote, Tubman House was restricted to being used for offices and recreation space. In June 1970, the city conveyed the land beneath it to the agency, also known as USES, for the construction of a community service center.

Maicharia Weir Lytle, president and CEO of USES, has said the building’s sale is necessary to sustain the organization into the future.

“Today, over 350 children and their families who rely on United South End Settlements for affordable early childhood education, out of school time programming (including a summer sleepaway camp), family engagement, and workforce development services can rest easy knowing that these important services will continue,” Lytle said in a written statement. “Selling 566 Columbus Avenue is our last financial lifeline. Now that the development is approved, we can move forward and know that the proceeds from the sale will be directly reinvested into the community to expand programs for whole families.”

The developer poised to take possession of the site is New Boston Ventures. Documents filed with the BPDA show the company’s plans to raise a six-story building with 66 condos. The complex would include 11 income restricted one- and two-bedroom units priced between $186,400 and $288,700. The proposal also includes preservation or relocation of the building’s multi-panel mural, a “social enterprise café” to be named after a former area jazz club, and a public art gallery.

Activists who protested the sale were disappointed with the board’s decision, storming out of the board’s 9th-floor room in City Hall on Thursday.

“Number one, Harriet Tubman is a very important historical figure in the United States, so if you destroy something in her name, you destroy her image and likeness,” Roxbury resident Janis L. McManus said after the hearing. “When I first came to Boston, I took classes there. So it was a community building. It’s not an open building anymore.”

Community activists insist the building’s nonprofit owner did not try hard enough to find a way to right itself financially while also preserving its longtime function. The sale, they say, amounts to an erasure of another community-friendly space forged by people of color in the era of the federal urban renewal program.

The BPDA board vote closed a controversy punctuated with passion, protests and petitions — including one from Change.org, which, as of Thursday afternoon, had more than 56,000 opposing signatures.

The debate began with USES’ announcement of the sale back in January and came to a raucous end with the final public engagement meeting at the Tubman House Monday evening. There, opponents obstructed the development team’s presentation with drumming and megaphone-amplified chants of “Keep the restrictions,” and “Condo’s for who? Not the families YOU serve.”

Boston City Councilor Kim Janey and Massachusetts Rep. Jon Santiago, who both represent the area, announced their support for the sale in a joint statement Monday.

“While we understand the frustration of some in our community, and as difficult as this situation is, we support USES in their mission to save their organization from financial ruin by selling their building,” the statement read. “If the sale of 566 Columbus does not go through, USES will be forced to shut down and close its doors permanently. That would only exacerbate the gentrification we seek to curtail. Without USES, we would lose our largest local provider of social services to low-income families.”

Documents from Thursday’s board meeting show that the BPDA also allowed itself to charge a fee “to capture the value for the creation of the residential portion of the project.” When the market-rate housing units are sold, the BPDA will collect a 4 percent cut of the net sale price. Any subsequent resale would also garner the city agency a 2 percent fee.

Correction: A previous version of this story misidentified the board’s vote tally.