# Variable manipulation
## Introduction
The `string-eval` step can perform string based evaluations to transform data. It enables numeric calculations, casting to different data types, splits, and more.
The format will feel familiar to those with database and/or JavaScript experience.
An example use case may look like the following:
```yaml
- id: string-eval
params:
param:
bind: radius_miles
expression:
constant: 'FLOAT(radius_km)/1.6'
condition: radius_km
```
This example takes an input variable `radius_km`, evaluates the expression, and then writes the output variable to the `radius_miles` parameter.
The condition ensures that the step only runs if the `radius_km` variable exists.
Alternatively, you can also write the output back to the same variable, which is useful if you want to augment the existing parameters. For example:
```yaml
- id: string-eval
params:
param:
bind: radius
expression:
constant: 'FLOAT(radius)/1.6'
condition: radius
```
Variables are always assumed to be strings, so you need to convert them using `INT()` or `FLOAT()` casting to use them in mathematical expressions. See [#casting](#casting "mention") below.
### Referencing fields in record pipeline
When using the string-eval step in the record pipeline, you can read input or write output directly onto the field using "fields." prefix before the fieldname.
To transform a record using string-eval step before indexing, see the following example:
```yaml
- id: string-eval
params:
field:
constant: title
expression:
constant: 'TRIM(title)'
condition: fields.title
```
The above step checks if the incoming record contains the 'title' field. The expression removes all trailing and leading whitespaces from the 'title' field. The output from the expression replaces the 'title' field.
### Operators
The expression supports `AND`, `OR` and `XOR` operators as well as parentheses. The logic also supports the `IF()` function much like you would see in databases and/or Excel (see Conditional functions below).
## Functions
This step supports a range of functions that can be used in the expression. These functions fall into several groups:
### Casting
Convert a string into different data types. This is common for numeric operations.
```yaml
INT(item)
```
Int attempts to convert the item to an integer. If the type is a float it will be rounded to the nearest integer.
```yaml
FLOAT(item)
```
Float attempts to convert the item into a float.
```yaml
STRING(item)
```
String converts the item to a string.
```yaml
BOOL(item)
```
Bool attempts to convert the item to a Boolean. A zero will convert to `FALSE`, other numeric values will convert to `TRUE`. The string versions of `TRUE` and `FALSE` will convert as expected.
```
TIME(item)
```
Time attempts to convert the `item` to a time. The default time layout is `2006-01-02 15:04:05`.
```
DURATION(item)
```
Duration attempts to convert the `item` to a duration. `item` should be a duration string, which is a signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns" (nanosecond), "us" or "µs" (microsecond), "ms" (millisecond), "s" (second), "m" (minute), "h" (hour).
### Conditional
```yaml
IF(bool, item_if_true, item_if_false)
```
This takes a bool value (which could be the result of an expression), and returns `item_if_true` OR `item_if_false` for true/false values respectively.
### Arrays
As previously shown above you can create and join lists using the bracket notation, e.g. `[a]+[b]` produces a list `[a, b]`
### String functions
String functions allow for strings to be transformed, joined, quoted, trimmed and more.
```yaml
JOIN(elements, separator)
```
Join concatenates the `elements` of its first argument to create a single string. The `separator` string defaults to `""` if not set, and is placed between elements in the resulting string.
```yaml
SPLIT(string, separator)
```
Split the string by separator, returning a list of strings.
```yaml
QUOTE(item)
```
Quote a string, or list of strings. The returned string(s) uses escape sequences (\t, \n, \xFF, \u0100) for control characters and non-printable characters.
```yaml
LENGTH(item)
```
Length returns the number of runes (UTF-8) if the item is a `string`, or the number of elements if the item is an array.
```yaml
TRIM(string)
```
Trim returns a slice of the `string` with all leading and trailing Unicode code points representing whitespace.
```yaml
LTRIM(string)
```
LTrim returns a slice of the `string` with all leading Unicode code points representing whitespace.
```yaml
RTRIM(string)
```
RTrim returns a slice of the `string` with all trailing Unicode code points representing whitespace.
```yaml
RIGHT(string, num_of_chars)
```
Right extracts a number of characters `num_of_chars` from `string` (starting from right)
```yaml
LEFT(string, num_of_chars)
```
Left extracts a number of characters `num_of_chars` from `string` (starting from left)
```
SUBSTRING(string, start_offset, [num_of_chars])
```
Substring extracts a number of characters (`num_of_chars`) from a `string` (starting from `start_offset`). If `num_of_chars` is not provided, the rest of the `string` starting from `start_offset` is returned.
```yaml
UPPER(string)
```
Upper converts `string` to upper-case.
```yaml
LOWER(string)
```
Lower converts a string to lower-case.
### Numeric functions
Numeric functions can be used to perform numeric transforms on inbound parameters.
```yaml
POWER(base, exponent)
```
Power takes a `base` number and raises it to the power of the `exponent`.
```yaml
ABS(number)
```
Abs returns the absolute value of `number`.
```yaml
EXP(number)
```
Exp returns `e` raised to the power of `number`. The constant `e` (2.718281...), is the base of natural logarithms.
```yaml
CEIL(number)
```
Ceil returns the smallest integer value that is larger than or equal to `number`.
```yaml
FLOOR(number)
```
Floor returns the largest integer value that is smaller than or equal to `number`.
```yaml
SQRT(number)
```
Sqrt returns the square root of `number`.
```yaml
LN(number)
```
Ln returns the natural logarithm of `number`.
```yaml
MIN(number1,number2)
```
Min returns the smaller of `number1` and `number2`.
```yaml
MAX(number1,number2)
```
Max returns the larger of `number1` and `number2`.
### Time functions
Time functions can be used to measure time.
```
NOW()
```
Returns the current time. (in UTC timezone)
```
SINCE(time)
```
Returns the time elapsed (duration) since `time`.
```
UNTIL(time)
```
Returns the length of time (duration) until `time`.
### Geo functions
```yaml
HAVERSINE(lat1,lng1,lat2,lng2)
```
Calculates the great circle distance in kilometers between points `(lat1,lng1)` and `(lat2,lng2)`.
For example, to calculate the distance between Sydney (-33.865143, 151.209900) and Melbourne (-37.840935, 144.946457) and store the output in variable `distance`:
```yaml
- id: string-eval
params:
param:
bind: distance
expression:
constant: HAVERSINE(-33.865143,151.209900,-37.840935,144.946457)
```
If the coordinates are stored in variables then they need to be cast to a `FLOAT`:
```yaml
- id: string-eval
params:
param:
bind: distance
expression:
constant: HAVERSINE(FLOAT(sydneyLat),FLOAT(sydneyLng),FLOAT(melbourneLat),FLOAT(melbourneLng))
```
## Examples
### String concatenation
Much like JavaScript, strings can be concatenated using the `+` symbol. For example:
```yaml
- id: string-eval
params:
param:
bind: concat_output
expression:
constant: firstName + ' ' + lastName
```
The above function would take two inputs and concatenate them with a space in between.
You can also use the `JOIN()` function for this:
```yaml
- id: string-eval
params:
param:
bind: concat_output
expression:
constant: JOIN([firstName, middleName, lastName], " ")
```
The above function concatenates all variables (firstName,middleName,lastName) separated by a space. The variables are closed in brackets indicating that it is a list of elements that are being joined.
### Combining filter expressions
You can also combine filters using variables that may not exist in the request. Take the below example, if the `bathFilter` and `priceFilter` are `NULL` and the `bedsFilter = “beds = 3”`, then the output will be `filterExpr = “beds = 3”`
```yaml
- id: string-eval
params:
param:
bind: filterExpr
expression:
constant: JOIN([bedsFilter, bathFilter, priceFilter], " AND ")
```
Much like JavaScript, if you don’t convert strings, operators may not do what you are expecting, e.g. `a+b = 56` if `a = “5”` and `b = “6”` are both `string` values, you might instead want `INT(a)+INT(b) = 11`.
### Calculating square root of a number
For example, you may want to calculate the square root of an input parameter (string):
```yaml
- id: string-eval
params:
param:
bind: my_number
expression:
constant: SQRT(FLOAT(my_number))
condition: my_number
```
In the above example, the input `my_number` must exist for the step to execute. If it does, it is converted from a string to a float, then the square root is calculated and written back into the variable. You can also write the output back to the same variable if needed.
## Advanced examples
Combining `string-eval` with other steps such as regular expressions allows highly complex logic to be compiled quickly. Below are some examples:
### Filtering using multiple nested variants
A common situation in ecommerce is to filter on multiple variant fields. This can be done using the `ARRAY_MATCH(expression)` filter syntax where `expression` is a filter expression targeting multiple variant fields to ensure they all match the expression.
Consider a JSON record:
```json
{
"id" = "123",
"product_name": "BC679 Premium Ceramic Disc Brake Pads"
"year": ["2010,2006,2010"],
"make": ["ford,toyota,ford"],
"model": ["falcon,corolla,bronco"],
...
}
```
Now assume that the end-user makes the following search request:
```json
{
"q": "2005 ford bronco brake pads"
}
```
In this case, let's assume we have transformed the query into the following using either NLP or regular expression steps:
```json
{
"q": "brake pads",
"yearFilter": "year = 2005",
"makeFilter": "make = 'ford'",
"modelFilter": "model = 'bronco'"
}
```
Now, we want to take all of these filter expressions and combine these to get this param:
```json
"comboFilterExpr": "ARRAY_MATCH(make = 'ford' AND year = 2005 AND model = 'bronco')"
```
Using the above, we want to target the sub-variants of each record to ensure they match the filters. Below is one way to do this:
```yaml
- id: string-eval
params:
expression:
constant: STRING("ARRAY_MATCH(" + JOIN([makeFilter, yearFilter, modelFilter], " AND ") + ")")
param:
bind: comboFilterExpr
condition: makeFilter OR yearFilter OR modelFilter
```
The above step uses `string-eval` to collapse the filters into a single `ARRAY_MATCH()` filter expression. The output from the above step will be:
```json
"comboFilterExpr": "ARRAY_MATCH(make = 'ford' AND year = 2005 AND model = 'bronco')",
```
Now, we want to use this param to filter results. We can do this using the `add-filter` step.
```yaml
- id: add-filter
params:
filter:
bind: comboFilterExpr
condition: comboFilterExpr
```
The output from the above step will be:
```json
"filter": "ARRAY_MATCH(make = 'ford' AND year = 2005 AND model = 'bronco')"
```
Notes:
* If none of `makeFilter`, `yearFilter` or `modelFilter` exist, the string-eval step would not work as expected, so we add the `condition: makeFilter OR yearFilter OR modelFilter` to the step to ensure at least one of these filter expressions exists
* Assuming at least one filter exists, the `string-eval` step joins the filters, wraps them in an `ARRAY_MATCH()` function and binds the output to the `comboFilterExpr` variable
* The `add-filter` step uses `condition: comboFilterExpr` to ensure the filter expression exists, if so it is added to the query
### Selecting best-matched variant from multiple arrays
This [post-step](/developer-documentation/fundamentals/pipelines/steps.md#pre-steps-and-post-steps) is used for choosing the best variant match within a record where product variants are stored in multiple arrays. Typical use case would be displaying the correct image based on a colour match in the colour array, e.g. a record contains colour array \[“blue”, “green”, “red”], and another array with the image URLs that correspond to each colour based on index position. If a user searches “green” we want to return the green image in index position “1” so we display the correct image to the user.
This must be used in the Post-step as the index position must be written back to each record returned in the search response.
```yaml
# this step is optional but useful to have to see which index position is being matched on
- id: string-eval
params:
expression:
constant: IBM(fields.variant_colours,values.q)
field:
constant: colour_output
condition: q != ''
# this step will check if there is an index match
# if matched then check the variant_images array at the matched index position
# take the value and write it to a new "image" field
# if no match return the first image in the image array
- id: string-eval
params:
expression:
constant: IF((INT(IBM(fields.variant_colours,values.q)) >= 0), fields.variant_images[IBM(fields.variant_colours,values.q)],fields.variant_images[0])
field:
constant: image
```
### Split values before records are indexed
The string-eval step can also be used in the record pipelines. A common use case for string-eval step in record pipelines is to split values before storing values on a repeated field using delimiter or separator.
Let's assume that you want to index a webpage that contains 'features' and 'category' fields which include multiple values, e.g.,
```json
{
"id" = "123",
"title": "BC679 Premium Ceramic Disc Brake Pads"
"features": "Gray, Copper-free, Reduced noise",
"category": "Automotive|Cars|Brakes",
...
}
```
If this page was indexed, the value of features will be stored as one value, e.g. "Gray, Copper-free, Reduced noise", which is not ideal if you want to use these fields in facets.
To split out each feature and category item and store it as an array with items, you can use a string-eval step. See the example below:
```yaml
- id: string-eval
params:
expression:
constant: SPLIT(fields.features,', ')
field:
constant: features
```
The above step takes the 'features' value from the incoming record, splits all values using the separator ", ", and stores items as a list in the 'features' field. You can use the same step to split the values of the 'category' field using the delimiter "|". The prefix "fields." before the fieldname is required when you want to reference the field values rather than the variables.
Note that the output from the step is being added to the field directly rather than assigning it to a variable. To split the value from the 'features' field into a variable, use:
```yaml
- id: string-eval
params:
expression:
constant: SPLIT(fields.features,', ')
param:
bind: features
```
The output from the above step will be:
```json
"features": ["Gray","Copper-free","Reduced noise"]
```
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.search.io/developer-documentation/fundamentals/pipelines/variable-manipulation.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.