Appending compiler flags to a file with CMake

How do I add a compiler flag (I want to APPEND it, not overwrite the others) to a single translation unit with cmake?

I tried with

set_source_files_properties(MyFile.cpp PROPERTIES CMAKE_CXX_FLAGS "-msse4.1")

but it isn't working.. any advice on how to do that?