Previously, the precedence of `as` binding was higher than the binary
operators. This results in unexpected behaviors, `1 + 2 as $x | -$x`
was `-1` instead of `-3`. The commit 92daac90f6 (PR #3053) changed
the behavior to emit syntax error on this case. Now, this commit allows
binary operators with `as` binding. This fixes#2446, and fixes#3284.
This commit improves the parser and compiler error messages, making
it more easier to understand where in the filter the problem is.
Like modern compilers, jq will show the line in the filter where the
error occurred, will display squiggly line under the invalid token.
This commit improves the parser to fix some unintuitive results of the unary
operator `-` (e.g. `[-1 as $x | 1,$x]` is now `[1,-1]`, previously `[-1,-1]`).
Also, this commit now allows to use binary operator expressions, `reduce`,
`foreach`, `if`, and `try` expressions for object values.
This patch exports all the binary operator builtins functions from
builtin.c and uses them for constant folding in the parser, allowing
constant folding to work will all kinds and combinations of constants.
Now string*number, $ARGS+$ARGS, string/string, etc will also be
constant folded and the implementation of constant folded operators and
runtime operators will be the same.
And thanks to the new ERRORK bytecode operation, errors are constant
folded too! (e.g. 1 / 0 [] * {} etc)
Use a StringStart component that is either FORMAT QQSTRING_START or
QQSTRING_START instead of having two similar rules for String.
This is simpler and avoids having to use an untyped mid-rule action
component to copy FORMAT at the top of the stack before QQString, and
having to use jv_free($<literal>3) instead of jv_free($1) just to make
bison not complain about the "unused" mid-rule component.
Previously constant folding of zero division (e.x. 1/0) produces a
compile error. This was incorrectly implemented by checking if the
division result is infinite, so produces wrong results compared to the
query where no constant folding is processed (e.x. 1e308/0.1). This
patch delays the operation when the divisor is zero. This makes the
results more consistent, but changes the exit code on zero division from
3 to 5. Also 0/0 now produces the zero division error, not NaN.
This patch also fixes the modulo operation. Previously constant folding
logic does not take care of the % operator, but now it folds if the both
dividend and divisor are safe numbers to cast to the integer type, and
the divisor is not zero. This patch also fixes some code that relies on
undefined cast behaviors in C. The modulo operation produces NaN if
either the dividend or divisor is NaN.
It seems that bison doesn't call destructors for mid-rule action
components on error, since it does not know their type.
A mid-rule action was used to allocate the "text" string used as format
by string literals without a format, which would leak on error.
This patch replaces it with a new NoFormat component of type <literal>.
Now bison will call jv_free() on that string after a syntax error.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60836
$ ./jq '"'
jq: error: syntax error, unexpected end of file, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at <top-level>, line 1:
"
jq: 1 compile error
=================================================================
==1495450==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 21 byte(s) in 1 object(s) allocated from:
#0 0x7fc21aee1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x557ccb6ff608 in jv_mem_alloc src/jv_alloc.c:141
SUMMARY: AddressSanitizer: 21 byte(s) leaked in 1 allocation(s).
Close#1885, #2140, #2011, #2220, #2485, #2073
Rename the FORK_OPT opcode to TRY_BEGIN, add a TRY_END opcode, and wrap
errors when raising through a TRY_END so that they will not be caught by
the matching TRY_BEGIN.
Now a `try exp catch handler` expression generates code like:
TRY_BEGIN handler
<exp>
TRY_END
JUMP past_handler
handler: <handler>
past_handler:
...
On backtrack through TRY_BEGIN it just backtracks.
If anything past the whole thing raises when <exp> produced a value,
then the TRY_END will catch the error, wrap it in another, and
backtrack. The TRY_BEGIN will see a wrapped error and then it will
unwrap and re-raise the error.
If <exp> raises, then TRY_BEGIN will catch the error and jump to the
handler, but the TRY_BEGIN will not stack_save() in that case, so on
raise/backtrack the TRY_BEGIN will not execute again (nor will the
TRY_END).
n.b.: { $__loc__: "foo" } is still not allowed.
$__loc__ cannot be overwritten, and it is always an object.
Using { $x: "foo" } where $x is not a string is an error, so there is
not much of a point in supporting that.
Previously, bindings were parsed as the combination of two tokens:
'$' IDENT
This meant that using a keyword as variable name (e.g. $then, $label)
did not work.
Attempts were made to allow $keyword to work by adding some '$' Keyword
rules in the parser, but this did not allow $keyword in all places:
jq --arg label foo -n '$label' # ok
jq -n '"foo" as $label | .' # error
Treating $foo as a single token is much simpler, in my opinion.
This patch also changes how LOC is lexed: "$__loc__" instead of as
"__loc__" that gets combined with '$' in the parser.
This patch also disallows having spaces after '$' when recalling a
variable `$ foo' since that was probably just an unintentional side
effect of the implementation, and it was not documented.
Fixes#2675
We need to be careful to not grab an extra reference when mutating data
structures because that means we make extra copies. Doing that every
time in `_modify` is really painful, as that function implements `|=`
and all modify-assign operators.
`jv_setpath()` also grabs additional references, so this is not the only
fix needed for the modify-assign operators to not be accidentally
quadratic.
We have to use `LOADVN` in order to make this fix possible, so we should
really byte-code `_modify` rather than jq-code it. However, as a
stop-gap to make this fix easier, I'm adding syntax for referring to a
`$binding` such that you get `LOADVN` instead of `LOADV`.
This syntax is not meant to be used outside jq's internals! Therefore
it is not documented. I promise to break it later, so don't use it!
TBD, but later:
- Optimize `_modify` for the case where `update` outputs more than one
value.
This is trivial: add a `setpath($p; null)` in the middle of `_modify`
before calling `update`. But unfortunately the VM retains a
reference to `value_at_path` for path expression checking, and fixing
that will require more work.
Extend jv_number to use decNumber for storing number literals. Any math
operations on the numbers will truncate them to double precision.
Comparisons when both numbers are literal numbers will compare them
without truncation.
Delay conversion of numbers to doubles until a math operation is performed,
to preserve precision. A literal jv_number will only need conversion to
double once, and will reuse the resultant double on subsequent
conversions.
Outputting literal jv_numbers preserves the original precision.
Add strong pthread requirement to manage contexts/allocations for
converting numbers between their decNumber, string, and double formats.
Defaults the else clause to the identity filter "." if an else clause is
missing.
The following two jq programs are considered equivalent:
```
if .foo then
.foo = "bar"
else
.
end
```
```
if .foo then
.foo = "bar"
end
```
Avoid doing the internal binding of top-level symbols in the parser,
leaving that work to be done in a post-processing step. For builtins,
this lets us do a reference-aware bind step (block_bind_incremental)
*after* generating builtins/0.
Libraries are a bit trickier since they may be bound multiple times, so
instead of thinking through the implications I added (block_bind_self)
to resolve all internal symbols immediately.
Users are often surprised by the requirement to parenthesize any
non-trivial object key expressions in object constructors. E.g.,
{"a"+"b":1}. This commit adds one more kind of key expression besides
literals and idents: variable references.
A common use case for this is jq programs as JSON templates to fill in
with variables computed from inputs or passed in on the command-line.
E.g., {some_key:$value}. Now users can also use, e.g., {$key:$value}.
This and the restrictions on key and value expressions in object
constructors are now clarified a bit in the documentation.
itchyny