Announcing the New Rockset Developer Tools

We are excited to release a new ecosystem of developer tools intended to help advanced users edit, execute, and deploy Query Lambdas from a local development environment, while integrating seamlessly with Version Control and CI/CD systems. Right now, we are releasing three tools into an Open Beta:

1. Rockset CLI
2. Rockset VS Code Extension
3. Rockset Developer UI

In this blog, we will explore best practices for using these tools together. For more detailed documentation on usage of each individual tool, please see the README for each tool. Check out the video below for a detailed walk-through of the new tools.

Embedded content: https://www.youtube.com/watch?v=WhRueFicCUg&t=2s

Before you start

Before you start using the new Rockset Developer Tools, we recommend that you try the following from the Rockset Console:

1. Set up a Rockset Account in the Rockset Console
2. Create a Query Lambda using the Console Query Editor
3. Create a tag for your Query Lambda
4. Execute a Query Lambda by tag from its API endpoint or from one of our SDKs

The High Level Workflow

The new Rockset Developer Tools enable you to move the source of truth for your Query Lambdas to your Version Control System (VCS). They also provide robust tools for developing Rockset SQL and deploying your Query Lambdas to Rockset from your local development environment.

We recommend the following high level workflow for using the new developer tools. This workflow closely parallels industry-standard continuous development flows.

1. Set up a local Rockset project in your source code.

   a. Initialize a new project in your repository

   b. [CLI] Download your Query Lambdas and check them into your VCS of choice

2. Develop your Query Lambdas

   a. [CLI] Create / delete Query Lambdas

   b. [VS Code] Edit Query Lambdas

   c. [Dev UI] Execute your local Query Lambdas

   d. Iterate

   e. Commit and push your code

3. Deploy your Query Lambdas (CI/CD)

   a. [CLI] Deploy your Query Lambdas to Rockset with Query Lambda Tags

4. Hit your Query Lambda from your application by its Query Lambda Tag

5. If necessary, roll back your Query Lambdas

   a. [CLI] Re-deploy the Query Lambdas associated with the previous tagged commit

A Hello World Example

0. Install the Developer Tools

Install the Rockset CLI tool. Installation is currently supported on Linux and MacOS only.

# Install 
$ curl https://rockset-cli-artifacts.s3-us-west-2.amazonaws.com/install-standalone.sh | bash

# Restart your terminal to complete installation
# Verify
$ rockset --version

# Install CLI autocomplete
$ rockset autocomplete

# Add your authentication information
$ rockset auth:add APIKEY

# Add shell autocomplete support (bash and zsh only)
$ rockset autocomplete

You can create an API key in the Rockset Console. Install the Rockset VS Code plugin through the VS Code marketplace. Please see the VS Code documentation for more details about how to install the extension.

1. Set up a project

# Navigate to an empty folder
$ cd ~/projects/rockset

# Initialize your local project
$ rockset local:init -y

# Add a Query Lambda to your project
$ rockset local:queryLambda:add commons.helloWorld

That was easy! But what’s going on under the hood here? Let’s view the file system and take a closer look.

$ tree
.
├── rockset.config.json
└── src
    └── commons
        ├── __sql
        │   └── helloWorld.sql
        └── helloWorld.lambda.json

We have created 2 files: a SQL file, and a Query Lambda definition file. The Lambda definition file includes information such as default parameters and the description of your Lambda. You can view more information about these files in the CLI Documentation.

2. Develop your Query Lambdas

We recommend editing your Query Lambdas in VS Code for the best experience. To get started, open the root directory of your project (the directory containing rockset.config.json).

Open src/commons/__sql/helloWorld.sql, and paste in the following.

SELECT
    'Hello, World' AS "Hello World"

If you have the Rockset SQL extension installed, you should see full syntax highlighting, as well as autocomplete functionality.

You can also execute this SQL directly in VS Code by running Execute Rockset Query from the command palette (Open the command palette with Ctrl-Shift-P or Cmd-Shift-P). You can also execute your query using the CLI.

3. Using Parameters

Let’s make our Query Lambda more interesting — let’s add a parameter! Reopen the source for your Lambda, and paste in the following.

SELECT
    CONCAT('Hello ', :name) AS "Hello World"

This query will now say hello to you by name — but how do we execute it with parameters? For more complex queries that include parameters, we recommend using the Rockset Developer UI. To get started, open the Developer UI using

$ rockset local:serve -p 3001 # port

This should automatically open a webpage in your default browser. Click “commons.helloWorld” to view the execution page for this Lambda.

Home Page:

Execution Page:

Clicking the execute button from this page will automatically execute the latest SQL text that you have saved. But if we click execute right now, we see an error — we haven’t specified a value for the “name” parameter yet! To add parameters, click the Parameters tab. From here, you can add execution parameters that will be passed to along with your SQL during execution. Add your name as a parameter called “name”, with type “string”, and execute your Lambda.

Add Parameters:

Execute with Parameters:

It works!

4. Deploying and Executing your Query Lambdas from an Application

Deploying your Query Lambdas consists of two steps. First, we will deploy a new Query Lambda Version. This bundles the Query Lambda, along with its configuration, and uploads it to Rockset under a version hash. Next, we tag our Query Lambda so that we can refer to it from our application.

By using Query Lambdas in conjunction with Tags, we can version and update our Query Lambdas without having to modify our application code.

# Upload new versions, then tag them with the development tag
$ rockset project:deploy -t dev

# Deploy to production
$ rockset project:deploy -t prod
Successfully updated commons.helloWorld — version 061293cecfb67e36
Successfully tagged commons.helloWorld version 061293cecfb67e36 with tag "prod"

Your Query Lambda is now live, and can be executed from your application!

We will use the cURL to test the endpoint. You can also execute a Query Lambda by Tag from any of our Language Clients or SDKs.

# Execute the newly deployed CLI
$ rockset api:queryLambdas:execute commons helloWorld prod
[INFO]: POST: /v1/orgs/self/ws/commons/lambdas/helloWorld/tags/prod

You can also find your Query Lambda in the Rockset Console. When you select your Query Lambda, you can see example snippets for executing your Query Lambda from your application in the language of your choice.

Conclusion

We’re incredibly excited to announce these new tools. We hope that by incorporating them into your workflow will enable you to

• Check Query Lambdas into version control
• Develop Query Lambdas in your local development environment
• Manage production Query Lambdas from CI/CD

Happy Hacking!