curl -XPUT -H "Content-Type: application/json" http://localhost:8094/api/index/medical-condition -d '{ ... json payload ...}'Posts tagged with 'f'
This is a repost that originally appeared on the Couchbase Blog: Powershell with the Couchbase REST API.
PowerShell is a scripting environment / command line that comes with Windows and is also available for Linux and within Azure.
Maybe you’ve used Postman or Fiddler to make HTTP requests. Those are great, but not necessarily the right tools for automation or scripting.
You may have heard of curl before. It’s a command line tool for making HTTP requests.
If you’re a .NET/Windows developer (like me), maybe you aren’t familiar with curl. I use PowerShell as my default command line every day (though I still consider myself a PowerShell neophyte). In this post, I’ll show you how you can use PowerShell’s Invoke-WebRequest to make HTTP requests (which you can use within PowerShell scripts for automation).
You can check out the PowerShell script I created on GitHub. Note: as of the time of writing this post, I’m using PowerShell 5.1 on Windows 10.
Couchbase REST API
Couchbase Server has a an extensive REST API that you can use to manage and administrate just about every aspect of Couchbase Server. For this blog post, I’m going to focus on the Full Text Search (FTS) API. I’m going to show this because:
-
Creating an FTS index is something you’ll eventually want to automate
-
You will probably want to share an FTS index you created with your team and/or check it into source control
-
Couchbase Console already shows you exactly how to do it with curl.
I’m not going to cover FTS in detail: I invite you to check out past blog posts on FTS, and this short video demonstrating full text search.
Full Text Search review
When you initially create an FTS index, you will probably use the built-in FTS UI in the Couchbase Console. This is fine when you are doing the initial development, but it’s not practical if you want to share this index with your team, automate deployment, or take advantage of source control.
Fortunately, you can use the "Show index definition JSON" feature to see the JSON data that makes up the index definition. You can also have Couchbase Console generate the curl method for you.
Well, if you’re using curl, that’s very convenient. Here’s an example:
You can copy/paste that into a script, and check the script into source control. But what if you don’t use curl?
PowerShell version: Invoke-WebRequest
First, create a new PowerShell script. I called mine createFtsIndex.ps1. All this PowerShell script is going to do is create an FTS index on an existing bucket.
You can start by pasting the "curl" command into this file. The bulk of this command is the JSON definition, which will be exactly the same.
Let’s breakdown the rest of the curl command to see what’s happening:
-
-XPUT- This is telling curl to use the PUT verb with the HTTP request -
-H "Content-Type: application/json"- Use a Content-Type header. -
http://localhost:8094/api/index/medical-condition- This is the URL of the REST endpoint. The "localhost" will vary based on where Couchbase is running, and the "medical-condition" part is just the name of the FTS index. -
-d '…json payload…'- The body of content that will be included in the HTTP request.
PowerShell’s Invoke-WebRequest can do all this stuff too, but the syntax is a bit different. Let’s step through the equivalents:
-
-Method PUT- This is telling Invoke-WebRequest to use the PUT verb with the HTTP request, so you can replace-XPUT -
-Header @{ … }- Specify headers to use with the request (more on this later) -
-Uri http://localhost:8094/api/index/medical-condition"- You just need to add "-Uri" in front -
-Body '…json payload…'- The body of content is included this way instead of using curl’s-d
Headers
PowerShell expects a "dictionary" that contains headers. The syntax for a literal dictionary in PowerShell is:
@{"key1"="value1"; "key2"="value2"}
So then, to specify Content-Type:
-Headers @{"Content-Type"="application/json"}
One thing that the curl output did not generate is the authentication information that you need to make a request to the API. With curl, you can specify basic authentication by adding the username/password to the URL. It will then translate it into the appropriate Basic Auth headers.
With PowerShell, it appears you have to do that yourself. My local Couchbase Server has credentials "Administrator" and "password" (please don’t use those in production). Those need to be encoded into Base64 and added to the headers.
Then the full Headers dictionary looks like this:
-Headers @{"Authorization" = "Basic "+[System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("Administrator:password")); "Content-Type"="application/json"}
You might think that’s a bit noisy, and I agree. If you know a cleaner way to do this, I’m dying to know. Please leave a comment.
Execute the PowerShell script
To execute the script, simply type .\createFtsIndex.ps1 at the PowerShell command line.
You’re now ready to make this a part of your deployment.
Summary
PowerShell is a handy tool for scripting and automation. It’s used by Azure, Octopus Deploy, and anywhere Windows is running. Use PowerShell to automate Couchbase REST API calls for things like Full Text Search indexes, and your team will thank you.
Need help? Reach out to me with questions by leaving a comment below or finding me on Twitter @mgroves.
This is a repost that originally appeared on the Couchbase Blog: Tooling Improvements in Couchbase 5.0 Beta.
Tooling improvements have come to Couchbase Server 5.0 Beta. In this blog post, I’m going to show you some of the tooling improvements in:
-
Query plan visualization - to better understand how a query is going to execute
-
Query monitoring - to see how a query is actually executing
-
Improved UX - highlighting the new Couchbase Web Console
-
Import/export - the new cbimport and cbexport tooling
Some of these topics have been covered in earlier blog posts for the developer builds (but not the Beta). For your reference:
Query Plan Visualization tooling
In order to help you write efficient queries, the tooling in Couchbase Server 5.0 has been enhanced to give you a Visual Query Plan when writing N1QL queries. If you’ve ever used the Execution Plan feature in SQL Server Management Studio, this should feel familiar to you.
As a quick example, I’ll write a UNION query against Couchbase’s travel-sample bucket (optional sample data that ships with Couchbase Server). First, I’ll click "Query" to bring up the Couchbase Query Workbench. Then, I’ll enter a query into the Query Editor.
This is a relatively complex query that involves the following steps (and more):
-
Identify and scan the correct index(es)
-
Fetch the corresponding data
-
Project the fields named in the
SELECTclause -
Find distinct results
-
UNIONthe results together -
Stream the results back to the web console
In Couchbase Server 4.x, you could use the EXPLAIN N1QL command to get an idea of the query plan. Now, in Couchbase Server 5.0 beta, you can view the plan visually.
This tooling shows you, at a glance, the costliest parts of the query, which can help you to identify improvements.
Query monitoring
It’s important to have tooling to monitor your queries in action. Couchbase Server 5.0 beta has tooling to monitor active, completed, and prepared queries. In addition, you have the ability to cancel queries that are in progress.
Start by clicking "Query" on the Web Console menu, and then click "Query Monitor". You’ll see the "Active", "Completed", and "Prepared" options at the top of the page.
Let’s look at the "Completed" queries page. The query text and other information about the query is displayed in a table.
Next, you can sort the table to see which query took the longest to run (duration), return the most results (result count), and so on. Finally, if you click "edit", you’ll be taken to the Query Workbench with the text of that query.
New Couchbase Web Console
If you’ve been following along, you’ve probably already noticed the new Couchbase Web Console. The UI has been given an overhaul in Couchbase Server 5.0. The goal is to improve navigation and optimize the UI.
This new design maximizes usability of existing features from Server 4.x, while leaving room to expand the feature set of 5.0 and beyond.
cbimport and cbexport
New command line tooling includes cbimport and cbexport for moving data around.
cbimport supports importing both CSV and JSON data. The documentation on cbimport should tell you all you want to know, but I want to highlight a couple things:
-
Load data from a URI by using the
-d,--dataset <uri>flags -
Generate keys according to a template by using the
-g,--generate-key <key_expr>flags. This gives you a powerful templating system to generate unique keys that fit your data model and access patterns -
Specify a variety of JSON formats when importing: JSON per line (
lines), JSON list/array (list), JSON ZIP file/folder containing multiple files (sample). So no matter what format you receive JSON in, cbimport can handle it.
For more about cbimport in action, check out Using cbimport to import Wikibase data to JSON documents.
cbexport exports data from Couchbase to file(s). Currently, only the JSON format is supported. Again, the documentation on cbexport will tell you what you want to know. A couple things to point out:
-
Include the document key in your export by using the
--include-key <key>flag. -
Export to either "lines" or "list" format (see above).
Here’s an example of cbexport in action (I’m using Powershell on Windows, but it will be very similar on Mac/Linux):
PS C:\Program Files\Couchbase\Server\bin> .\cbexport.exe json -c localhost -u Administrator -p password -b mybucketname -f list -o c:\exportdirectory\cbexporttest.json --include-key _id
Json exported to `c:\exportdirectory\cbexporttest.json` successfully
PS C:\Program Files\Couchbase\Server\bin> type C:\exportdirectory\cbexporttest.json
[
{"_id":"463f8111-2000-48cc-bb69-e2ba07defa37","body":"Eveniet sed unde officiis dignissimos.","type":"Update"},
{"_id":"e39375ab-2cdf-4dc4-9659-6c19b39e377d","name":"Jack Johnston","type":"User"}
]Notice that the key was included in an "_id" field.
Summary
Tooling for Couchbase Server 5.0 beta is designed to make your life easier. These tools will help you whether you’re writing queries, integrating with data, monitoring, or performing administrative tasks.
We’re always looking for feedback. Inside of the Web Console, there is a feedback icon at the bottom right of the screen. You can click that to send us feedback about the tooling directly. Or, feel free to leave a comment below, or reach out to me on Twitter @mgroves.
This is a special crossover episode of Cross Cutting Concerns with the Eat Sleep Code podcast, hosted by Ed Charbeneau (Microsoft MVP). This was recorded at the Stir Trek conference.
Show Notes:
- Eric Brewer: One of his recent blog posts was about Cloud Spanner and the CAP Theorum
- Check out the blogs at Telerik, and check out Ed on Telerik's developer portal
- Couchcase: Github repo, blog posts
- Ed's website, EdCharbeneau.com
- Machine Learning for Developers
- This episode was published to Microsoft's Channel 9 and also Telerik's Develper Portal
Want to be on the next episode? You can! All you need is the willingness to talk about something technical.
Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.
Michael Bowman is writing functional tests with F# and Canopy.
Show Notes:
Want to be on the next episode? You can! All you need is the willingness to talk about something technical.
Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.
Arthur Doler talks about retrospectives and how to make them better.
Note that this was recording at the Indy.Code() conference in a hallway, so the audio may be a bit noisier than usual.
Also, SPECIAL THANKS to the great David Giard (who has been on the show before: Episode 6 and Episode 15, and he's also the host of the excellent Techology and Friends show, of which my podcast is a pale imitation) who gave me some new podcasting equipment that I used in this episode. I am extremely grateful, but I'm still trying to figure out how best to use this equipment (which may be obvious in this episode).
Show Notes:
- Book: Thinking, Fast and Slow by Daniel Kahneman
- Hidden Brain by Shankar Vedantam
- Hidden Brain podcast on NPR
- Hidden Brain book
- Book: You Are No So Smart by David McRaney
- Arthur was kind enough to give his email address in the podcast if you want to contact him.
Want to be on the next episode? You can! All you need is the willingness to talk about something technical.
Theme music is "Crosscutting Concerns" by The Dirty Truckers, check out their music on Amazon or iTunes.
