In its 6-3 decision, the Texas Court of Criminal Appeals said that recording deliberations would violate an "ancient and centuries-old rule that jury deliberations should be private and confidential."
The court in Austin determined that a video camera would create an external influence and put pressure on the jury in the trial of 17-year-old Cedric Harrison, accused of murdering a man during an attempted carjacking.
If convicted, the minor could face the death penalty.
During arguments last month, one of the judges questioned whether the taping -- which would have been a first in a capital punishment case -- would turn jury deliberations into "reality TV, like Survivor."
The PBS series Frontline asked for permission to videotape the entire trial proceedings -- including jury deliberations -- for a documentary on capital punishment.
Last November, State District Judge Ted Poe of Houston granted their request. Harris County prosecutors appealed Poe's decision, calling it a violation of jury confidentiality.
Cedric Harrison's attorneys supported Poe's decision, saying the documentary could "shed light" on death penalty trials.
"We have in this country and this world a huge debate over the propriety of putting people to death, and Texas and Harris County are right in the middle of it," Harrison's attorney, Chip Babcock, told the appeals court.
Babcock appeared on the NewsHour last November, saying the Frontline documentary would help broaden viewers' understanding of the justice system.
"[S]eeing how a jury reaches this very important life-and-death decision, it seems to me will aid in the understanding, the educational understanding of that process," Babcock said on the NewsHour.
Harris County's assistant district attorney Warren Diepraam, the lead prosecutor in the Harrison case, countered during the program that jurors would "tend to act for the camera and change their personality for the camera."
He also contended that jurors could later be identified and possibly targeted in retaliation for their actions.
A bill to prohibit the recording of jury deliberations is now before the Texas state legislature.