throw the baby out with the bathwater

Từ điển Oxford Advanced Learner 8th

throw the baby out with the ˈbathwater idiom
(informal)to lose sth that you want at the same time as you are trying to get rid of sth that you do not want
Main entry: ↑baby_idiom