Does it look bad after you upload it to YouTube or does it look bad on your computer already? Does the quality get noticeably worse after you upload it to YouTube?
You should probably switch to a software that supports H.264 encoding, which is the recommended format for YouTube. Try capturing and uploading very short videos first, increase the bitrate until you're satisfied with the quality. You may even try setting it to lossless when capturing, so that when YouTube re-encodes your video, the quality is lost only once (instead of twice: when you're capturing and then when you're uploading).
Plenty of software support H.264, including some free software, for example, OBS.