The Kadima party won the most votes but had little chance of building enough support for a coalition; whereas Netanyahu can get the needed support, but analysts say the resulting coalition would be dysfunctional, Reuters reported.
Nearly final results gave Netanyahu's party 27 seats in the 120-seat Knesset and Livni's party 28 seats -- both well short of the 61 needed to form a government. The Likud party appeared to be leading in opinion polls prior to the election.
The ultra-nationalist Yisrael Beiteinu party came in third with 15 seats, and Defense Minister Ehud Barak's Labor party got 13 seats, according to The New York Times.
It would be the first time in Israel's 60-year history that the winner of the election ended up sitting in opposition.
Livni said she would be prime minister and invited Netanyahu to join a "unity government," but Netanyahu said he would lead the "nationalist camp" in Parliament and control 64 seats.
President Shimon Peres must decide whether to call on Livni or Netanyahu, who then has 42 days to form a government.
Coalition negotiations are expected to continue for weeks.
"Tzipi Livni has only the slightest chance, or none at all, of forming a government under her leadership," said Abraham Diskin, a political scientist at Jerusalem's Hebrew University, quoted Reuters.
One of the country's newspaper commentator said the split result showed Israel's political process doesn't work. "One thing is clear to all Israeli voters," wrote Eitan Haber of the newspaper Yedioth Ahronoth. "The political system is shattered."