I have not quite understood how template changer works. I mean I can use it, but what is the Compile template button?
Our CSS is generated from SCSS files. So it has to compile to generate that CSS. After making your color changes you need to compile then save the template. That's all the necessary steps. In the future we'll probably tie this process into the save button.
And what happens when you make changes in the bootstrap template tab? Will that change actually get reflect in the file itself or is the change in database?
None of this is database based. Changes there will directly modify their respective CSS files as the description of the tab states, but you'll lose those modifications if you compile.