Hitler wanted peace on his own terms. Chamberlain was prepared to make concessions to Hitler on most of his points and even considered giving Germany colonial territories in Africa, but it involved negotiation and working within the existing international system. Up until the Polish crisis, we were far more serious about peace than they were.
Hitler (more so Ribbentrop, who was very hostile to Britain) didn't take this route. He assumed we weren't really willing to go to war (influenced by faulty intelligence on our military preparedness and diplomatic intentions), and then put us and France in a position where we either had to back up our earlier threats, or suffer a massive loss in prestige and international influence.
If we backed down at that point, there's no doubt we would have lost the empire. It was already close to breaking point, Italy and Japan would have been emboldened by our show of weakness and would take advantage to unsettle our positions in Africa and Asia.
Whether we would have been economically undamaged by peace depends on whether you think the empire was an economic benefit or not.
That's up for debate - places like India consumed more resources than we got back, and the preference system helped countries like Australia and Canada but not really us. There's a good argument to be made that the empire did sod all for the average Brit, and mostly helped people with business interests in those countries and those with cushy civil service posts abroad. Once the rest of the world closed the huge lead we'd had technologically and economically in earlier centuries, empire was becoming a liability.