Special Fields

This page describes system-generated fields with unique roles and behaviors in Rockset documents.

Overview

Special fields are automatically generated by Rockset during data ingestion and are added as new fields to each document. They can be queried like any other field, and you may also apply transformations to them using field mappings.

For instance, you might execute the following query on any collection to view its special fields:

SELECT

    _id,

    _meta,

    _event_time

FROM

    mycollection

LIMIT

    5;

+---------------------------------------------------------------------------------+----------------+----------------------------------------+
| _meta                                                                           | _event_time    | _id                                    |
|---------------------------------------------------------------------------------+----------------+----------------------------------------|
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '47304-1709397620935'}}  | 1535068823524  | '5d99f42c-44ec-2024-a032-ac14d1cbf44d' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '46242-1709397472272'}}  | 1535068823426  | 'e5d4c136-8d56-ac39-38f5-2f795ae38007' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '46242-1503238554088'}}  | 1535068823326  | '71df712e-04bc-7c9b-0114-97b19741b215' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '47302-1151051235692'}}  | 1535068822852  | '517412d2-1a23-1b49-fc3b-441c630ed4ff' |
| {'s3': {'bucket': 'mycollection', 'offset': 0, 'path': '45189-1709397393868'}}  | 1535068822527  | 'd16bf924-f758-21db-c832-4a25323b2938' |
+---------------------------------------------------------------------------------+----------------+----------------------------------------+

The _id Field

Every document in a Rockset collection is uniquely identified by its _id field.

• If the document source does not already have an _id field, Rockset populates it with an automatically generated uuid.
• If the document source has _id specified, or a field mapping outputs an _id field, its value is preserved. A newly ingested document will overwrite any existing document with the same _id value.

The _meta Field

Metadata regarding each document is stored in a _meta field of object type.

If the source of a document specifies a _meta field, Rockset will ignore the field. Currently, _meta holds information about the source from which the document was inserted into the collection (such as the bucket name and path in case of S3). If Rockset is unable to parse the source of a document, it will create a document without any of the source's fields and will have _meta with a nested field named bad.

This field is never populated for rollups.

The _event_time Field

Rockset associates a timestamp with each document in a field named _event_time, recorded as microseconds since the Unix epoch. Queries on this field are significantly faster than similar queries on regularly-indexed fields.

A user can select a field of the source of a document to be copied into the _event_time field at collection creation. This is done by specifying:

• the name of the selected field, which should be of type int or timestamp
• the format with which to interpret the field, which can be either:
  • milliseconds_since_epoch
  • seconds_since_epoch
• the default time zone, specified as one of the IANA time zone values

If an _event_time configuration is not specified, Rockset will first check for a pre-existing _event_time field (must be type int or timestamp) and, if nonexistent, populates it with the time of document insertion. This field is immutable after document creation.

If your collection has rollups and your rollup query does not contain an _event_time mapping, this field is populated with the initial insertion time of the rolled up document. It does not change as more input documents are aggregated into the rolled up document.