Cross-runtime .NET disassembly with BenchmarkDotNet

BenchmarkDotNet is a cool tool for benchmarking. It has a lot of useful features that help you with performance investigations. However, you can use these features even if you are not actually going to benchmark something. One of these features is DisassemblyDiagnoser. It shows you a disassembly listing of your code for all required runtimes. In this post, I will show you how to get disassembly listing for .NET Framework, .NET Core, and Mono with one click! You can do it with a very small code snippet like this:

[DryCoreJob, DryMonoJob, DryClrJob(Platform.X86)]
public class IntroDisasm
    public double Sum()
        double res = 0;
        for (int i = 0; i < 64; i++)
            res += i;
        return res;

Read more

BenchmarkDotNet v0.10.14

BenchmarkDotNet v0.10.14 has been released! This release includes:

  • Per-method parameterization (Read more)
  • Console histograms and multimodal disribution detection (A blog post is coming soon)
  • Many improvements for Mono disassembly support on Windows (A blog post is coming soon)
  • Many bugfixes

In the v0.10.14 scope, 8 issues were resolved and 11 pull requests where merged. This release includes 47 commits by 8 contributors.

Read more

BenchmarkDotNet v0.10.13

BenchmarkDotNet v0.10.13 has been released! This release includes:

  • Mono Support for DisassemblyDiagnoser: Now you can easily get an assembly listing not only on .NET Framework/.NET Core, but also on Mono. It works on Linux, macOS, and Windows (Windows requires installed cygwin with obj and as). (See #541)
  • Support ANY CoreFX and CoreCLR builds: BenchmarkDotNet allows the users to run their benchmarks against ANY CoreCLR and CoreFX builds. You can compare your local build vs MyGet feed or Debug vs Release or one version vs another. (See #651)
  • C# 7.2 support (See #643)
  • .NET 4.7.1 support (See 28aa94)
  • Support Visual Basic project files (.vbroj) targeting .NET Core (See #626)
  • DisassemblyDiagnoser now supports generic types (See #640)
  • Now it's possible to benchmark both Mono and .NET Core from the same app (See #653)
  • Many bug fixes (See details below)

Read more

Analyzing distribution of Mono GC collections

Sometimes I want to understand the GC performance impact on an application quickly. I know that there are many powerful diagnostic tools and approaches, but I'm a fan of the "right tool for the job" idea. In simple cases, I prefer simple noninvasive approaches which provide a quick way to get an overview of the current situation (if everything is terrible, I always can switch to an advanced approach). Today I want to share with you my favorite way to quickly get statistics of GC pauses in Mono and generate nice plots like this:

Read more

BenchmarkDotNet v0.10.12

BenchmarkDotNet v0.10.12 has been released! This release includes:

  • Improved DisassemblyDiagnoser: BenchmarkDotNet contains an embedded disassembler so that it can print assembly code for all benchmarks; it's not easy, but the disassembler evolves in every release.
  • Improved MemoryDiagnoser: it has a better precision level, and it takes less time to evaluate memory allocations in a benchmark.
  • New TailCallDiagnoser: now you get notifications when JIT applies the tail call optimizations to your methods.
  • Better environment info: when your share performance results, it's very important to share information about your environment. The library generates the environment summary for you by default. Now it contains information about the amount of physical CPU, physical cores, and logic cores. If you run a benchmark on a virtual machine, you will get the name of the hypervisor (e.g., Hyper-V, VMware, or VirtualBox).
  • Better summary table: one of the greatest features of BenchmarkDotNet is the summary table. It shows all important information about results in a compact and understandable form. Now it has better customization options: you can display relative performance of different environments (e.g., compare .NET Framework and .NET Core) and group benchmarks by categories.
  • New GC settings: now we support NoAffinitize, HeapAffinitizeMask, HeapCount.
  • Other minor improvements and bug fixes

Read more

BenchmarkDotNet v0.10.10

BenchmarkDotNet v0.10.10 has been released! This release includes many new features like Disassembly Diagnoser, ParamsSources, .NET Core x86 support, Environment variables, and more!

Read more

Reflecting on performance testing

Performance is an important feature for many projects. Unfortunately, it's an all too common situation when a developer accidentally spoils the performance adding some new code. After a series of such incidents, people often start to think about performance regression testing.

As developers, we write unit tests all the time. These tests check that our business logic work as designed and that new features don't break existing code. It looks like a good idea to write some perf tests as well, which will verify that we don't have any performance regressions.

Turns out this is harder than it sounds. A lot of developers don't write perf tests at all. Some teams write perf tests, but almost all of them use their own infrastructure for analysis (which is not a bad thing in general because it's usually designed for specific projects and requirements). There are a lot of books about test-driven development (TDD), but there are no books about performance-driven development (PDD). There are well-known libraries for unit-testing (like xUnit/NUnit/MSTest for .NET), but there are almost no libraries for performance regression testing. Yeah, of course, there are some libraries which you can use. But there are troubles with well-known all recognized libraries, approaches, and tools. Ask your colleagues about it: some of them will give you different answers, the rest of them will start Googling it.

There is no common understanding of what performance testing should look like. This situation exists because it's really hard to develop a solution which solves all problems for all kind of projects. However, it doesn't mean that we shouldn't try. And we should try, we should share our experience and discuss best practices.

Read more

Measuring Performance Improvements in .NET Core with BenchmarkDotNet (Part 1)

A few days ago Stephen Toub published a great post at the Microsoft .NET Blog: Performance Improvements in .NET Core. He showed some significant performance changes in .NET Core 2.0 Preview 1 (compared with .NET Framework 4.7). The .NET Core uses RyuJIT for generating assembly code. When I first tried RyuJIT (e.g., CTP2, CTP5, 2014), I wasn't excited about this: the preview versions had some bugs, and it worked slowly on my applications. However, the idea of a rethought and open-source JIT-compiler was a huge step forward and investment in the future. RyuJIT had been developed very actively in recent years: not only by Microsoft but with the help of the community. I'm still not happy about the generated assembly code in some methods, but I have to admit that the RyuJIT (as a part of .NET Core) works pretty well today: it shows a good performance level not only on artificial benchmarks but also on real user code. Also, there are a lot of changes not only in dotnet/coreclr (the .NET Core runtime), but also in dotnet/corefx (the .NET Core foundational libraries). It's very nice to watch how the community helps to optimize well-used classes which have not changed for years.

Now let's talk about benchmarks. For the demonstration, Stephen wrote a set of handwritten benchmarks. A few people (in comments and on HackerNews) asked about BenchmarkDotNet regarding these samples (as a better tool for performance measurements). So, I decided to try all these benchmarks on BenchmarkDotNet.

In this post, we will discuss how can BenchmarkDotNet help in such performance investigations, which benchmarking approaches (and when) are better to use, and how can we improve these measurements.

Read more

BenchmarkDotNet v0.10.7

BenchmarkDotNet v0.10.7 has been released. In this post, I will briefly cover the following features:

  • LINQPad support
  • Filters and categories
  • Updated Setup/Cleanup attributes
  • Better Value Types support
  • Building Sources on Linux

Read more

65535 interfaces ought to be enough for anybody

It was a bright, sunny morning. There were no signs of trouble. I came to work, opened Slack, and received many messages from my coworkers about failed tests.

After a few hours of investigation, the situation became clear:

  • I'm responsible for the unit tests subsystem in Rider, and only tests from this subsystem were failing.
  • I didn't commit anything to the subsystem for a week because I worked with a local branch. Other developers also didn't touch this code.
  • The unit tests subsystem is completely independent. It's hard to imagine a situation when only the corresponded tests would fail, thousands of other tests pass, and there are no changes in the source code.
  • git blame helped to find the "bad commit": it didn't include anything suspicious, only a few additional classes in other subsystems.
  • Only tests on Linux and MacOS were red. On Windows, everything was ok.
  • Stacktraces in failed tests were completely random. We had a new stack trace in each test from different subsystems. There was no connection between these stack traces, unit tests source code, and the changes in the "bad commit." There was no clue where we should look for a problem.

So, what was special about this "bad commit"? Spoiler: after these changes, we sometimes have more than 65535 interface implementations at runtime.

Read more

More posts in Russian

Subscribe: RSS Atom