20

After some research, we've found:

  • Watin (http://watin.org/)
  • CodedUI (Microsoft)
  • Coypu (Selenium based)
  • TestStack.Seleno (also Selenium based)

But with so many options, decision is also difficult to make.

Testing experts in Stackoverflow, can you please share with us your experience with those different frameworks? Thank you.

26

Fist off, if you want to test web UI like end users, then it probably shouldn't matter if your application is built using ASP.NET MVC or not. (It only implies you are looking for a .NET UI testing framework I assume.)

I'm not sure why Selenium WebDriver itself was not listed by you, which will be the one I'd recommend. Because it dominates the market, where you can get all support/resources about it. It's a highly active open source project that many other frameworks rely on. Watin and CodedUI are alternatives, while Coypu and TestStack.Seleno are just high level frameworks that wrap around Selenium. It supports Java, .NET, Ruby, Python, JavaScript bindings, code from other languages can be easily ported into C#.

  • Watin. The project is inactive. Back in the days when Selenium WebDriver was just released, they were both popular and you might have a difficult decision to make. But not anymore, Watin has not released anything for three years, which should be never considered as one of your options now. At the time, IE was supported to 9, Firefox was 4 and Chrome was 11.

  • CodeUI. Not recommended. The advantage is that it supports testing more than just web UI and is created by Microsoft themselves and strongly tied within Visual Studio. But it is not open sourced, with a small community with limited resources and functionalities more focused on Windows applications.

  • Coypu is an open source framework that wraps both Selenium WebDriver and Watin (IE only in Coypu). It's more like Capybara in a way, but for .NET. Personally I found this idea brilliant.

  • TestStack.Seleno can also be a good choice that creates some kind of DSL to hide Selenium WebDriver commands and provides Page Objects which is essential in web UI testing project. Furthermore, it can be tied closely with your ASP.NET MVC application. But these are things in a level above Selenium. It helps you writing code Selenium, which you might want to implement yourself anyway.

Therefore I'd say Coypu and TestStack.Seleno would be worth looking at, but just be careful.

  1. Those frameworks are dependent of Selenium, which might be a constraint. For example, your tests might be broken if there is critical bug in Selenium. Then Coypu developers need to wait it to be fixed in Selenium first and update Coypu to fix the issue for you. The time might vary depending on how active the project is. During the process, more bugs might be introduced that only affect Coypu users. You don't really need DSL/Page Objects from them, but implement your own unique way to support testing your application.

  2. Even though they are open sourced, they might not have the community as large as Selenium. Anything happens above browser automating level (i.e. code written Coypu or TestStack.Seleno) would be hard for you to get help. For instance, there are heaps of other frameworks wrapping Selenium, like Capybara, Watir WebDriver, etc., people struggle to get answers soon enough on StackOverflow, because there are not many people use them.

  3. Nobody knows how long these projects can keep active. There was another .NET UI testing framework called White, which was a good project for testing Web/WinForm/Silverlight applications, but deprecated and taken over by TestStack, which is now known as TestStack.White. You can image that if someone developed a large testing project using White a while back, they might get into trouble.

  4. DSL and page objects are things you should master yourself. To some developers, implementing their own would be better/more reliable than using third party libraries, simply they have full control of what's under the hood.

Outro:

The best way to evaluate frameworks is to create a demo for your application using each of the framework. Then you might have a preference that would help you make a decision.

In general, my suggestion is: to automate web UI, the option would be using Selenium WebDriver, not Watin or CodedUI. For high level supporting frameworks, personally I'd go for none (i.e. use pure Selenium WebDriver library and write my own supporting classes inside testing project), but Coypu and TestStack.Seleno can be considered if you like them.

  • The problem with Coded UI is that it treats the browser just like any other application. Selenium is tailor made for web, which is better in many aspects. Even things as simple as firing up the browser can go wrong because Coded UI will look for something hardcoded along the lines of C:\Program Files\FireFox\firefox.exe. Worse, it relies on the internal UI tree of the browser (you'll see stuff in the vein of FireFoxMainPane and inside FireFoxDisplayBorder etc. until you reach the actual DOM). That can break with new versions of the browser. You also can't use JS queries to find elements... – Ohad Schneider Apr 27 '16 at 10:32
  • Very nice summary. I wish I had read this before we started using Seleno, which is effectively abandoned (last commit from May 6, 2016) and because of Selenium changes (and removal of obsolete code) it does not work now. – jumbo Apr 2 '19 at 7:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.