HAILEY, Idaho — (CNN) -- No one here knew exactly where he was or what he was going through, but this tiny city in Idaho never forgot about Sgt. Bowe Bergdahl.
Yellow balloons and American flags line the sidewalks of Hailey, population 8,000. Residents planted one tree for each of the five years Bergdahl was held captive in Afghanistan.
Now, uncertainty has given way to elation as the Army sergeant has been released from his captors -- in exchange for freeing five Taliban members held at Guantanamo Bay.
"There were times where we wondered, but (parents) Jani and Bob Bergdahl never once gave up faith that their son was coming home to them," family friend Stefanie O'Neill said.
Bergdahl's parents said the battle is far from over.
"The recovery and reintegration of Bowe Bergdahl is a work in progress," Bob Bergdahl told reporters Sunday, one day after his son was freed.
"It isn't over for us. In many ways, it's just beginning for Jani and I, and our family. There's a long process here."
So what's next for Sgt. Bowe Bergdahl and the Obama administration, which has come under criticism from some lawmakers for the secretive nature of the swap?
WHAT'S NEXT FOR BERGDAHL
Recovery in Germany
Bergdahl is recovering at Landstuhl Regional Medical Center in Germany, a U.S. Defense official said Sunday. He is also expected to be debriefed at Landstuhl.
"He had lost ... a good bit of weight," National Security Adviser Susan Rice told CNN's "State of the Union With Candy Crowley."
Believing that his health was deteriorating, the United States acted quickly to save Bergdahl's life. U.S. special operations forces recovered Bergdahl without incident Saturday at a pickup point in eastern Afghanistan, near the border with Pakistan.
American officials said the government of Qatar brokered the deal.
"We were very concerned that time was not ... something we could play with -- that we needed to act when we had the opportunity," Rice said.
Bergdahl's condition on Monday was stable, and he was "receiving treatment for conditions that require hospitalization," a U.S. defense official told CNN.
People at Landstuhl were trying to build Bergdahl's trust in them, as he hasn't been able to trust anyone around him for five years, the official said.
A senior Defense official said Bergdahl was having trouble speaking English, but the reasons for that were not clear, given the trauma he's been through.
Doctors in Germany are evaluating his health and whether he's ready to be transported, said Arwen Consaul, a spokeswoman for U.S. Army South.
Once doctors give the OK, a receiving team will travel to Germany to facilitate his transport to Lackland Air Force Base in San Antonio, Texas.
The recovery team has been training for the trip for years, practicing Bergdahl's return trip 10 times with drills eventually involving hundreds of personnel, including stand-ins for the soldier and his family, she said.
From Lackland, Bergdahl will head to the San Antonio Military Medical Center, where a room is ready for him and a support team is standing by, Consaul said.
Speaking to CNN, White House spokesman Jay Carney declined to discuss Bergdahl's condition, saying he should be allowed to recover privately.
"His process of repatriation has begun, and reintegration into society," Carney said. "I'm sure it will not be an easy one, but we're overjoyed on behalf of his parents and his friends and family that he's returning home."
Transitioning to normal life
The first meeting between Bergdahl and his parents may only last minutes, Consaul said, depending on what psychologists recommend.
Once in the United States, she said, Bergdahl's daily routine will focus on four key areas: medical care, psychological support, debriefings and family support.
"This is to help a person who has had no control of their own life for years now regain that control step by step," she said.
There's no set time limit for how long the reintegration process will take, she said, because the case for each returnee is different.
Bergdahl was believed to be held by operatives from the Haqqani network, which is affiliated with the Taliban and al Qaeda.
An Afghan Taliban commander not authorized to speak to the media confirmed to CNN that Bergdahl was captured by insurgents with links to the Haqqani network in Pakistan.
Over the years, the captive was transferred back and forth between Afghanistan and Pakistan, the Taliban commander said.
David Rhode, a former New York Times journalist who was held captive by the Taliban for seven months, said Bergdahl faces a long road ahead.
"Psychologically, there will be a long debrief," said Rhode, who was taken in Afghanistan and held in Pakistan until he managed to escape. "They're actually going to want intelligence (about) who held him."
But there's a chance Bergdahl might not know much about Taliban commanders.
"I was kept with very young guards, most of the time, and I didn't really meet very senior commanders that often," Rhode said.
Army human resources officials will determine where Bergdahl goes after he completes the reintegration process in San Antonio, Consaul said. He'll receive follow up care and have access to a psychiatrist for the rest of his life.
Bergdahl, who's still on active duty in the Army, could return to his unit in Alaska or be reassigned, she said.
One question that remains unanswered is how Bergdahl disappeared in the first place.
Several veterans and soldiers call him a deserter who walked off his base on June 30, 2009. At least six soldiers were killed in subsequent searches for him.
"Bowe Bergdahl deserted during a time of war and his fellow Americans lost their lives searching for him," said former Sgt. Matt Vierkant, a member of Bergdahl's platoon.
Vierkant said Bergdahl needs to not only acknowledge his actions publicly but face a military trial for desertion under the Uniform Code of Military Justice.
When asked Sunday whether Bergdahl had left his post without permission or deserted, Defense Secretary Chuck Hagel didn't answer directly.
"Our first priority is assuring his well-being and his health and getting him reunited with his family," Hagel said. "Other circumstances that may develop and questions, those will be dealt with later."
A senior Defense official said Bergdahl will not likely face any punishment.
"Five years is enough," he told CNN on condition of anonymity.
Carney said questions about the circumstances of Bergdahl's disappearance didn't play into efforts to retrieve him.
"Here's what matters. He was a prisoner in an armed conflict, a member of the military, and in that situation the United States does not leave its men and women behind," Carney told CNN.
WHAT'S NEXT FOR THE ADMINISTRATION
Defending the swap
Some lawmakers are openly wondering whether Bergdahl's release sets a dangerous precedent for future kidnappings of Americans.
"The methodology in what we used is very troublesome," said Rep. Mike Rogers, chairman of the U.S. House Intelligence Committee.
"We have a changing footprint in Afghanistan, which would put our soldiers at risk for this notion that if I can get one, I can get five Taliban released."
Rice defended the Obama administration's efforts.
"If we got into a situation where we said, 'Because of who has captured an American soldier on the battlefield, we will leave that person behind,' we would be in a whole new era for the safety of our personnel and for the nature of our commitment to our men and women in uniform," she said on CNN's "State of the Union."
"Because it was the Taliban that had him did not mean that we had any less of an obligation to bring him back."
Some members of Congress are trying to understand why they didn't know what was happening.
Rep. Buck McKeon, chairman of the Armed Services Committee, said he still hasn't been briefed by administration officials on the detainee swap.
The California Republican said he plans to hold hearings about the Bergdahl exchange, he told CNN Chief Congressional Correspondent Dana Bash on Sunday.
"We're getting our information -- other than a very brief notification -- we're getting most of our information now from the media," he said in an exclusive interview on "Newsroom."
McKeon said the Defense Department notified his staff on Saturday, after the exchange took place. But he pointed to a law that requires the administration to notify Congress 30 days before detainees are released from the facilities at Guantanamo Bay.
Sen. Dianne Feinstein, chairwoman of the Senate Intelligence Committee, told reporters on Capitol Hill Monday that lawmakers on her committee should have been given a heads-up about the prisoner swap.
"We had participated in a number of briefings some time ago, and there (were) considerable concerns," the California Democrat said.
Rep. Mike Rogers, chairman the House Intelligence Committee, said he planned to look into whether Obama broke the law by not notifying Congress 30 days in advance.
"I think it certainly merits further review, and that's what I'm going to do to make that determination," the Michigan Republican told CNN's Wolf Blitzer. "It certainly doesn't smell right to me."
Tony Blinken, Obama's deputy national security adviser, told CNN Monday that administration officials didn't notify Congress in advance because they faced an "urgent situation" with deep concerns about Bergdahl's health.
"We had information that it was deteriorating. We had an opportunity that was very fast-moving to close this deal, to bring him back. And based on the law, we had the flexibility to do that. ... We had an urgent matter, and we had an opportunity, and we seized it," he said.
After consulting with the Justice Department to make sure the move was legal, he said, the administration decided to go ahead with the prisoner swap to avoid the risk of the Taliban changing its mind or the information leaking out.
"In the days ahead, there will be full briefings of the relevant committees in Congress, and indeed, of every member who wants it," Blinken said.
CNN's Nick Valencia contributed to this report from Hailey, Idaho. CNN's Alexandra Field, Steve Almasy, Jake Tapper, Barbara Starr, Sophia Saifi, Victoria Eastwood, Ashley Killough, Martin Savidge, Wolf Blitzer and Catherine E. Shoichet and journalist Zahir Shah Sherazi also contributed to this report.