Fields are a powerful way to add structure to unstructured events and allow the manipulation of both the textual and visual representation of data. Log Insight

Fields are one of the most important items in a content pack as they can be used in different ways including aggregations and constraints. Aggregations allow you to apply functions and groupings to fields. Constraints allow you to perform operations over fields.

You must extract any part of a log message that might be applicable to a query or aggregation. Fields are a type of regular expression query and are useful for complex pattern matching so you do not need to know, remember, or learn complicated regular expressions.

Field Context Value

Definition

Regex before value

Iinclude as many keywords as possible. If this field is empty or only contains special characters, then the Regex after value must include keywords.

Regex after value

Include as many keywords as possible. If this field is empty or only contains special characters, then the Regex before value must include keywords.

Name

Only use alphanumeric characters. Ensure all characters are lower case and use underscores instead of spaces as this makes fields easier to view. Keep in mind that names for content pack fields and user fields can be the same, though content pack fields will have a namespace in parenthesis to the right of the field name. Prefix content pack fields with an abbreviation, for example vmw_, to avoid confusion.

In addition to the various components that make up a field, several best practices apply.

Only create fields for regular expression patterns. If a field can be queried using keyword queries, or will only ever return a single value, then use keyword queries instead of a pre-defined field. If a field will only return two values then consider constructing individual queries instead of extracting a field. Fields are meant to add structure to unstructured data as well as provide a way to query over specific parts of an event.

Only create fields for regular expression patterns that return a fraction of the total events. Fields that will match most events and/or return a very large number of results are not a good candidate for field extraction. The regular expression will need to be applied to a large quantity of events resulting in a resource intensive operation. If possible add additional keywords to reduce the number of results returned and optimize the query.

If a field contains keywords within regular expression syntax, then add such keywords as a constraint without regular expression syntax. For example, if the value or the context of a field contains keywords within regular expression syntax such as this|that, then add the keywords as a text constraint to optimize the query like text contains this, that.