A great example of what I'm looking for is Evernote. Its text editor supports text formatting and allows you to insert images in-line.
I've tried LibreOffice, but it's rather bloated and complicated to achieve my rather simple note-taking needs.
Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.Sign up to join this community
I would suggest taking a look at RedNotebook it is primarily a journal keeping application but works equally well for note taking:
yum install rednotebook
apt-get install rednotebook