A great example of what I'm looking for is Evernote. Its text editor supports text formatting and allows you to insert images in-line.
I've tried LibreOffice, but it's rather bloated and complicated to achieve my rather simple note-taking needs.
I would suggest taking a look at RedNotebook it is primarily a journal keeping application but works equally well for note taking:
yum install rednotebook
apt-get install rednotebook