I want to create a small videos in Windows from still images, and trying to find some open source software for the same.
FFMPeg is the Swiss Army Knife of the Video world and it is free, open source & cross platform including Windows.
If you are not afraid of the command like then you can do what you need quite handily.
ffmpeg -framerate 1/5 -start_number 126 -i img%03d.png -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4
I would also suggest checking out MoviePy, which uses ffmpeg under the hood, with several other tools, and allows things like creating a video clip from a single of image as simple as:
myclip = ImageClip("some_picture.jpeg", duration=10)
or from a sequence:
clip = ImageSequenceClip(['image_file1.jpeg', ...], fps=24)
or from text:
clip = TextClip("Hello !", font="Amiri-Bold", fontsize=70, color="black")