Skip to main content

Posts tagged with 'testing'

This is a repost that originally appeared on the Couchbase Blog: Performance Testing and Load Testing Couchbase with Pillowfight.

Performance testing and load testing are important processes to help you make sure you are production ready. For testing Couchbase Server clusters, there is an open-source command line utility called "cbc-pillowfight". It’s part of libcouchbase.

Before you begin

You’ll need a Couchbase Server cluster up and running. You can try it out directly on your local machine (download Couchbase for Linux, Windows, and Mac) or in a Docker container.

If you’re just trying out pillowfight, you may want to create a bucket on your cluster just for that purpose. I created a bucket called "pillow".

After you have Couchbase Server installed, you’ll need to download and install libcouchbase:

  • Mac: brew install libcouchbase

  • Windows: download a zip file (latest at the time of writing is libcouchbase-2.8.1)

For more information, including Linux instructions, check out the libcouchbase release notes.

Pillow fight for Performance Testing

If you used homebrew to install on a Mac, you can type cbc-pillowfight --help straight away for the command line help screen.

On Windows, unzip the libcouchbase zip file wherever you’d like. You’ll find cbc-pillowfight.exe in the bin folder.

Performance testing tools

The simplest pillowfight you can run is:

.\cbc-pillowfight.exe -U couchbase://localhost/pillow -u Administrator -P password

This is for a Windows Powershell command line, but it will be very similar on other OSes.

A pillow fight will start for the cluster running on your local machine (localhost), with the "Administrator" user that has a password of "password" (your username and password may be different).

Starting pillowfight

You should see a message like "Thread 0 has finished populating".

What is a pillow fight?

At this point, the pillowfight is going to start creating, updating, and reading documents from the "pillow" bucket. It’s going to do all these operations ("ops") according to the command line settings you specify (or fall back to the defaults).

For instance, with the -I flag, you can specify how many total documents you want to operate on. The default is 1000. So, if you run the above command, you will soon see 1000 documents show up in the pillow bucket.

It doesn’t just create 1000 documents and quit. Pillowfight will keep "getting" and "updating" those documents until you terminate the process. It’s called a "pillowfight" because it will put your Couchbase Cluster into battle (with actual exertion), but it’s really more of a battle simulation.

While the fight is happening, you can monitor bucket statistics to see how your cluster is performing under load.

Performance testing monitoring

As I type this, the fan on my laptop is whirring to life as I stress test the single node Couchbase cluster that I’ve installed on it. (I suspect my home desktop would create a much more impressive set of charts, but I am traveling a lot this month).

There are a lot of statistics available for you to look at on a bucket level. Check out the Couchbase Server documentation on Monitoring Statistics for more details.

Options for performance testing

The default pillowfight settings may not be optimal for the type of application that you’ll be using with Couchbase. There are many ways to adjust your pillow fight to make it better fit your use cases. For the full list of options, type cbc-pillowfight --help at the command line.

But here are some notable options you might want to try out:

  • -I or --num-items with a number, to specify how many documents you want to operate on.

  • --json to use JSON payloads in the documents. By default, documents are created with non-JSON payloads, but you may want to have real JSON documents in order to test other aspects of performance while the pillow fight is running.

  • -e to expire documents after a certain period of time. If you are using Couchbase as a cache or short term storage, you will want to use this setting to monitor the effect of documents expiring.

  • --subdoc to use the subdocument API. Not every operation will need to be on an entire document.

  • -M or --max-size to set a ceiling on the size of the documents. You may want to adjust this to tailor a more realistic document size for your system. There’s a corresponding -m and --min-size too.

Here’s another example using the above options:

.\cbc-pillowfight.exe -U couchbase://localhost/pillow -u Administrator -P password -I 10000 --json -e 10 --subdoc -M 1024

This will start a pillowfight using 10000 JSON documents, that expire after 10 seconds, uses the sub-document API, and has a max document size of 1024 bytes.

Note: there is a -t --num-threads option. Currently, if you're using Windows (like me), you are limited to a single thread (see this code).

Summary

Couchbase is committed to performance. We do extensive performance testing to make sure that we are delivering the speed you expect. Check out recent blog posts on our Plasma storage engine and N1QL enhancements. But no one knows your use case and infrastructure better than you. With pillowfight, you have a tool to help you do performance testing, load testing, and stress testing.

Thanks go out to Sergey Avseyev for helping with this blog post, and his contributions to libcouchbase.

Please reach out with questions on Couchbase by leaving a comment below or finding me on Twitter @mgroves.

Eric Elliott is increasing code quality by leveraging pure functions and fast feedback.

Editors note: sorry for the audio quality; I had to use the Skype audio recording, and it's not as good as I'd like. But it's too good of an episode to just throw out!

Show Notes:

Eric Elliott is on Twitter. Special thanks to JS Cheerleader, who set up this interview!

Want to be on the next episode? You can! All you need is the willingness to talk about something technical.

Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.

Jeremy Clark is writing unit tests.

Show Notes:

Jeremy Clark is on Twitter.

Want to be on the next episode? You can! All you need is the willingness to talk about something technical.

Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.

Michael Bowman is writing functional tests with F# and Canopy.

Show Notes:

Michael Bowman is on Twitter

Want to be on the next episode? You can! All you need is the willingness to talk about something technical.

Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.

This is a repost that originally appeared on the Couchbase Blog: Visual Studio Live Unit Testing: New to Visual Studio 2017.

Visual Studio 2017 was just officially released. It comes with a lot of new, great stuff, but one of my favorite new features is built-in Visual Studio Live Unit Testing (available in Visual Studio 2017 Enterprise, not yet available for .NET Core projects).

In this post, I’m going to show you how Visual Studio Live Unit Testing works, as well as some thoughts around using unit tests vs integration tests. You can follow along by getting the full source code for this Live Unit Testing example on GitHub.

Visual Studio Live Unit Testing with NUnit

NUnit is perhaps the most popular testing tool for C#/.NET developers. Visual Studio Live Unit Testing can also work with xUnit and MSTest, but for this post I’m going to just cover NUnit.

To use NUnit, you add it with NuGet, just as normal. To use Visual Studio Live Testing, you’ll also need to add the NUnit Test Adapter (Install-Package NUnite3TestAdapter).

Next, start Live Testing by clicking Test → Live Unit Testing → Start.

Start Visual Studio Live Unit Testing

Writing a Unit Test

We’ll need some unit tests to demonstrate. We could just do Assert.That(1, Is.EqualTo(1)), but where’s the fun in that? Let’s create a shopping cart class.

public class ShoppingCart
{
    public string UserName { get; set; }
    public DateTime LastUpdated { get; set; }
    public List<Item> Items { get; set; }
    public decimal Total
    {
        get { return Items.Sum(i => i.Price); }
    }
}

This shopping cart has a couple properties, and a collection of items in it. Notice the Total property. Astute readers may already notice some problems with it, but let’s start with a single, simple unit test to make sure it calculates a total.

[Test]
public void ShoppingCart_Total_Should_Sum_Up_the_Item_Prices()
{
    // arrange: create shopping cart with 2 items and figure out the expected total
    var item1 = new Item { Name = "Large Pepperoni Pizza", Price = 14.99M };
    var item2 = new Item { Name = "Cheese Sticks", Price = 4.99M };
    var expectedTotal = item1.Price + item2.Price;
    var cart = new ShoppingCart { Items = new List<Item> { item1, item2 } };

    // act: user the Total method on ShoppingCart
    var actualTotal = cart.Total;

    // assert: totals should match
    Assert.That(actualTotal, Is.EqualTo(expectedTotal));
}

If Live Unit Testing is turned on, then the test is being automatically run in the background by Visual Studio. You should see some green checkmarks appear.

Visual Studio Live Unit Testing in action

The green checkmarks will also appear wherever the code that is under test is covered.

Visual Studio Live Unit Testing code under test

With Visual Studio Live Unit Testing, you don’t have to stop to run the tests. As you are coding, the tests will run, and give you immediate feedback on whether your code is making tests fail (or whether you’ve written enough code to make your test pass).

Most of all, this will encourage you to write more tests.

What are Integration Tests?

When writing unit tests, you are meant to test a small piece of code on its own. For code that interacts with some external service (a web service, a database, a file system, etc), you often mock those pieces out, so that you can focus on the unit.

You may also write integration tests with NUnit. Integration tests that are meant to go beyond testing a single unit of code, and test that systems work together. Let’s write a method that writes a record to Couchbase Server. This test will use a real database, therefore we can consider it an integration test.

public void SaveShoppingCart(ShoppingCart cart)
{
    _bucket.Insert(new Document<ShoppingCart>
    {
        Id = Guid.NewGuid().ToString(),
        Content = cart
    });
}

This method should save a shopping cart to a document in Couchbase Server. To make sure it’s working, we can write an NUnit test.

[Test]
public void Repo_Can_Save_a_New_Shopping_Cart_to_Database()
{
    // arrange: create a shopping cart
    var cart = new ShoppingCart
    {
        UserName = "Matthew " + Guid.NewGuid().ToString(),
        LastUpdated = DateTime.Now
    };

    // act: save shopping cart to database
    Repo.SaveShoppingCart(cart);

    // assert: check that the cart was saved
    var cartBackOut = Repo.GetCartByUserName(cart.UserName);
    Assert.That(cartBackOut, Is.Not.Null);
    Assert.That(cartBackOut.UserName, Is.EqualTo(cart.UserName));
}

Note: To keep this post simple, I omitted some of the repository details, and test setup. You can view all of this in the GitHub repository.

Integration Tests with Visual Studio Live Unit Testing

Visual Studio Live Unit Testing will happily run this unit test. You may not want these types of tests to be running in the background automatically because:

  1. If you don’t have Couchbase Server installed, or a bucket created and indexed, then they will fail.

  2. If you have a lot of tests that rely on external components, they could slow down the tests (reading/writing documents in Couchbase is very fast, but setting up a Cluster object for each test or test fixture is not).

  3. These tests could add a lot of unnecessary junk test data to your database.

Excluding Integration Tests from Visual Studio Live Unit Testing

To exclude tests from Live Unit Testing, you can simply right-click on the test file and select "Exclude" from the context menu.

Exclude from Live Unit Testing

After this, none of the tests in that file will be executed by Live Unit Testing. You can also exclude an entire project. So, if you organize unit tests and integration tests into separate projects, then you are all set.

If you don’t organize them into separate projects, then this process could be a bit tedious. Further, the Include/Exclude information is a local setting that can’t (as of the time I’m writing this, and to the best of my knowledge) be committed to source control.

So, after asking about Live Testing exclusion on StackOverflow, I created an attribute that you can place on tests to exclude them from Live Testing.

public class IgnoreForLiveTesting : Attribute, ITestAction
{
    readonly string _ignoreReason;

    public IgnoreForLiveTesting(string ignoreReason = null)
    {
        _ignoreReason = ignoreReason;
    }

    public ActionTargets Targets { get; set; }

    public void AfterTest(ITest test) { }

    public void BeforeTest(ITest test)
    {
        var isLiveTesting = AppDomain.CurrentDomain.GetAssemblies()
            .Any(a => a.GetName().Name == "Microsoft.CodeAnalysis.LiveUnitTesting.Runtime");
        if (isLiveTesting)
            Assert.Ignore(_ignoreReason ?? "Ignoring this test");
    }
}

This attribute implements the ITestAction interface (which is kinda like Aspect-Oriented Programming/AOP for NUnit, but that’s a topic for a whole other blog post). It will check to see if it’s being run by a LiveUnitTesting process. If it is, it instructs NUnit to ignore the test.

Furthermore, I added an optional ignoreReason to the constructor, so that you can add a helpful note to other people on your team to explain why this test should not be run with Live Unit Testing. You can use it on an integration test like so:

[IgnoreForLiveTesting("Integration Test")]

Summary

I’m not terribly pleased with this method, as it’s NUnit specific, and it’s not quite exactly what I was hoping for with Visual Studio Live Unit Testing. But right now I think "the juice is worth the squeeze". Live Unit Testing is such a great feature for writing code, especially Test-Driven Development (TDD), that it’s worth it to have to write and use a special NUnit attribute.

By all means, if you know of a better way to achieve this, I want to know about it. Please leave a comment below or ping me on Twitter @mgroves.

If you have questions about the Couchbase code you saw in this post, I’d be happy to help. Or, you can check out the responsive and knowledgeable community on the Couchbase .NET SDK forum. If you want to learn more about Couchbase, check out the Couchbase Developer Portal.

Matthew D. Groves

About the Author

Matthew D. Groves lives in Central Ohio. He works remotely, loves to code, and is a Microsoft MVP.

Latest Comments

Twitter