Manco had retreated to the other side of the Andes in a far away outpost called Ollantaytambo. Manco had several thousand troops with a quantity of captured Spanish weapons and several horses. (He had forced his Spanish prisoners to make gunpowder.)

The Spanish forced their way through the town right to the foot of the fortress, while Manco rode a Spanish horse high up on the terraces, waving a Spanish sword and urging his men on. An attempt by Spanish commanders to outflank the position was beaten off, and Pizarro's brother, Hernando, was finally forced to admit defeat and retreated to Cuzco. It was the first time the Spanish had been worsted in open battle. But it was only a temporary reprieve for the Incas.

Manco had hoped to make a new Cuzco here, but he seems to have realized now that, in the long term, he had no chance of stopping the Spanish. His people had been devastated by disease, and, although Manco may have been able to hold on to his mountain hideouts, the sea was controlled by the Spanish who were now being strengthened every day. As Titsu Cusi says: "At this time an infinite number of people arrived from Spain, having seen the great riches taken to Spain from Cajamarca. Every day ships from Panama, Mexico and Nicaragua put into Lima." So Manco decided to retreat over the mountains into the cloud forest bordering Amazonia, where he hoped to keep an Inca state going out of the Spaniards' reach.

Before he left, he made a speech telling his followers not to forget what the ancient rulers had done for them, and, most important, not to forget their rituals. The following was recorded by his son, Titsu Cusi:

"I know that some day by force or deceit, they will make you worship what they worship. When that time comes, when you can no longer resist, do it in front of them, but on the other hand do not forget our ceremonies. And if they tell you to break your shrines, and force you to do so, reveal only what you have to, and keep the rest hidden, close to your hearts."