A format without %d left the day of the month at 0, which mktime and
todate read as the last day of the previous month, and the weekday was
wrong for dates in January and February of 1900 and 2100.
Start parsing from the 1st, so a format without a day now gives the
first of the month, and a time-only format no longer outputs the 8 and
367 placeholders for the weekday and day of the year. This also fixes
the weekday of dates in January and February of 1900 and 2100.
Both were defined as explode | map(...) | implode, which took 4 seconds
on a 10 MB string, about 90 times as long as parsing it.
Change the case of A-Z or a-z a byte at a time in C, copying multibyte
UTF-8 sequences unchanged, so the output stays the same and the 10 MB
case now takes about 60 ms including parsing. On a non-string input the
error now names the function, "ascii_downcase input must be a string",
instead of "explode input must be a string".
The manual says halt_error prints its input with no decoration, not
even a newline, but non-string values such as {"a":"xyz"} were printed
with a trailing newline.
Dump non-string values to JSON and print them the same way as strings,
so {"a":"xyz"} | halt_error(1) now writes exactly {"a":"xyz"} to stderr.
The tutorial is where most people meet jq, but it was missing key
filters like select and abbreviated commands in ways that are hard for
beginners to follow.
Grow the page from 7 examples to 24, newly covering -r and -c, chained
paths and slicing, map, sort_by, group_by, select, test, to_entries,
from_entries, with_entries, string interpolation and @tsv. The sample
data is pinned with the API's until parameter and the outputs avoid
GitHub account data, so running the commands gives exactly the output
shown.
The manual's "Defining Functions" section shows only one-argument
functions, so it never says how to pass more than one. This adds a short
paragraph: arguments are separated by `;` in both `def f(a; b):` and
`f(.x; .y)`, and `f(.x, .y)` instead calls a one-argument `f` with the
filter `.x, .y`. It also adds an example for each. `tests/man.test` and
`jq.1.prebuilt` are regenerated.
Fixes#2163
The manual said `a op= b` is equivalent to `a |= . op b`. It is actually
equivalent to `(b as $b | a |= . op $b)`: the right-hand side gets `.`
as input, not the value being updated, and each value it outputs gives a
separate result.
This updates the description and adds two examples, `.a += .b` and `.a
+= (1,2)`. `tests/man.test` and `jq.1.prebuilt` are regenerated.
Fixes#2119, fixes#2407
Nothing tested that jq --run-tests exits with status 1 when the
tests fail, so a failure could have been reported as a pass.
Add shell tests with incorrect, missing and extra output that check
both the exit status and the reported message, such as "Insufficient
results", along with a passing control. Under Valgrind, memory errors
exit with status 99 so they are not mistaken for a test failure.
The manual recommended backslash-escaped double quotes inside the jq
program for all PowerShell versions, but since PowerShell 7.3 native
command arguments are passed with standard quoting by default, so the
backslashes reach jq literally.
Describe the quoting rules separately for Unix shells and PowerShell
7.3+, for older PowerShell and Windows PowerShell, and for cmd.exe, and
update the examples to match.
Since jq 1.8.0, negating a number literal computed 0 - x, so -0 and
0 | -. printed 0 while (0+0) | -. printed -0.
Flip the sign bit with decNumberCopyNegate() instead, so -0 prints -0
again as it did in jq 1.7, and -(-0) is still 0.
jvp_utf8_next() had a fast path for ASCII, but it is not inlined into
its callers, so every byte of a string cost a function call.
Move the end check and the ASCII case into a static inline function in
jv_unicode.h and keep the rest out of line. This speeds up parsing of
string-heavy input by up to 1.17x and string functions such as length,
explode and @html, with the same output for every input.
Enable colored output by default only when the TERM environment variable
is set to a value other than dumb, as git and grep do, so that users no
longer need to pass -M or set NO_COLOR on terminals that cannot display
escape sequences. Colored output is now disabled not only for TERM=dumb
but also when TERM is unset or empty. The -C option still enables colors
in all these cases. On Windows, colored output is still decided by
ANSICON and the console mode, and TERM is not checked.
These functions failed with "unknown system failure" whenever strftime()
returned 0 for a non-empty format, but 0 also means the format expands to
an empty string, as %p does in a locale without AM/PM.
Prepend a character to the format and trim it from the result, so that 0
can only mean the result is too long. Now 0 | strflocaltime("%p") gives
"" in fr_FR.UTF-8, and a result that is still too long fails with
"result too long" error instead. The size limit is unchanged.
In --seq mode, a parse error was reported as "need RS to resync", but
parsing continued in the middle of the broken text, so fragments of it
were reported as more errors or output as values.
Set the parser state to wait for the next RS after resetting the parser,
instead of before, so that the rest of the broken text is skipped as
RFC 7464 recommends.
Printing a string wrote each character with a separate call, one
fwrite() per character, or one jv_string_append_buf() for tojson and
tostring, which made output of string-heavy data slow.
Scan ahead for the next byte that needs escaping and write the run of
bytes before it with a single call, without changing the output.
Printing strings is now significantly faster, most of all for long
strings with few characters to escape, and the same applies to tojson
and tostring. With --ascii-output, text that is almost all non-ASCII is
about the same speed, since nearly every byte is escaped anyway.
A call passing more than 4095 arguments at the start of a nested
function body aborted jq after reporting the compile errors, with
"Assertion failed: (nemb > 0 && sz > 0), function jv_mem_calloc" and
exit status 134.
Stop compiling a function before its bytecode is allocated when it has
compile errors, since the bytecode is only generated and used when
compilation succeeds. jq now prints the same compile errors followed by
"jq: 2 compile errors" and exits with status 3, as it does for other
compile errors, instead of crashing.
halt_error with a negative exit code did not exit with that code. The
code collided with the internal status values for null or false output
and for no output, so halt_error(-1) exited with 0, or with 1 under
--exit-status.
Keep the low 8 bits of a negative code, as exit(3) does on POSIX
systems, so halt_error(-1) now exits with 255 and halt_error(-2) with
254, the same with or without --exit-status. Under --exit-status this
replaces the previous absolute value, so halt_error(-2) used to exit
with 2 and halt_error(-255) with 255, and they now exit with 254 and 1.
Converting a double outside the int range to int is undefined behavior,
and has() and del() did it on array indices like infinite or 1e10, which
UBSan reports as "inf is outside the range of representable values of
type 'int'". Only NaN was guarded against.
Clamp the index to the int range in has(), as getting and setting array
elements already do, and drop out-of-range indices in del() before they
are converted. Such indices can never refer to an element, so has()
returns false and del() leaves the array unchanged, the same as for
indices past the end of the array. Valid indices deleted together with
out-of-range ones, as in del(.[0, 1e10]), are still deleted.
Add a BSD workflow with FreeBSD and NetBSD jobs that run the same build
and tests as the linux job, each inside a VM booted under QEMU on an
ubuntu-latest runner. The tests/shtest $NO_COLOR pty checks are now
skipped on NetBSD, as they already are on SunOS.
The UTF-8 BOM was stripped only at the start of the parser stream, but jq
feeds all input files into a single parser, so a BOM at the start of the
second and later files reached the parser as content and failed with a
confusing "Invalid numeric literal" error. Strip it in the file readers
instead, at the start of each input file including stdin, and in
jv_load_file.
This also strips the BOM for raw input, so -R and --rawfile no longer
yield a leading U+FEFF, and a BOM at the start of a -f program file or an
imported module is now accepted rather than reported as a syntax error.
In exchange, "\uFEFF1" | fromjson is now a parse error, since a U+FEFF
inside a string is content rather than an encoding marker, and the
dedicated "Malformed BOM" error is gone.
The date in the .TH line of jq.1.prebuilt changed whenever the manual was
regenerated in a new month, causing merge conflicts between pull requests
that touch the manual, and it did not reflect the release date anyway.
Generate the committed man page without a date.
Stamp the date into jq.1 at build time instead, taking it from the current
commit, or from SOURCE_DATE_EPOCH where that is set, so that release
tarballs and distribution builds still carry one and the behaviour added
in #3514 is kept. Write it in UTC and in ISO 8601 rather than the previous
"%B %Y", for which mandoc reports "cannot parse date, using it verbatim".
`fromdateiso8601` was `strptime("%Y-%m-%dT%H:%M:%SZ") | mktime`, which
accepts only one of the many representations ISO 8601 permits. strptime
cannot be extended to cover the rest portably, since it has no specifier
for fractional seconds and glibc discards `%G` and `%V`.
Replace it with a parser in src/iso8601.c that handles basic and extended
representations, ordinal and week dates, fractional seconds, offsets and
date-only input. It uses integer arithmetic instead of `time_t`, so the
results no longer depend on the platform.
This is stricter than before. Impossible dates like `2015-02-30`, which
`mktime` used to normalize into March, and unpadded components like
`2015-3-5T6:00:00Z` are now errors. So is any trailing input, including
whitespace. Leading whitespace is rejected too; before, glibc skipped it
and macOS did not. A leap second now maps to the second before it rather
than the next minute. The error message has changed, and the result can
now be non-integer.
The lexer returned a single ALTERNATION token for `?//`, so any `?`
followed by `//` was consumed by it, and `.name?//"unknown"` failed
to parse even though they read as the optional operator followed by
the alternative operator.
Drop the lexer rule and spell the destructuring alternative operator
as `'?' "//"` instead. So `x ?// y` now parses as `x? // y`, while
`. as [$a] ?// [$b] | ...` keeps its meaning. Note that `?` and `//`
may now be separated by whitespaces inside a pattern.
The push trigger had no branch filter, so pull requests from branches
in this repository (e.g. Dependabot) ran the workflow twice, once for
push and once for pull_request.
join/1 folded each element as `(acc + $x) + elem`, with the element
branch ending in `.//""`. Both leave a second live reference to the
accumulator: the nested + feeds the same `.` to both operands, and the
`//` leaves a forkpoint below it on the stack. `jvp_string_append`
therefore cannot grow the accumulator in place, and copies the whole
string once per element.
Pipe `acc + $x` into `. + elem` and drop the `//`, since adding null to
a string is already a no-op. join/1 is now linear: joining 8000
strings of 1000 chars goes from 6.5s to 0.06s, with output and error
messages unchanged.
glibc's arc4random() aborts with "Fatal glibc error: cannot get entropy
for arc4random" when the kernel lacks getrandom(2), so jq died on
startup on older kernels once 0c7d133 began seeding the hash on every
run. getentropy() merely returns -1/ENOSYS there, so prefer it, and
route its failure through the existing `/dev/urandom` path - which works
on precisely those kernels -- instead of a guessable `getpid() ^ time()`.
`reduce f as $x (init; update)` evaluated `f` against a stack slot
that forkpoints created while evaluating `init` restore. Both the
`DUPN` opening the reduction loop and the `LOADVN` retrieving the
result pop that slot destructively, nulling it out, so an `init`
that backtracks (e.g. `0, 0`) made `f` see `null` and fail with
"Cannot iterate over null".
Duplicate the input before the `FORK` instead of inside the loop,
and drop the leftover copy with `POP` once the loop is exhausted.
The slot left live at the `FORK` is then a block pushed after
`init` completed, which no earlier forkpoint can reach, so nulling
it is harmless. This is the same approach `gen_foreach` has used
since c70ae1a.
`DUPN` had no other user, so remove the opcode.
Define `from_entries` as `add(.[] | f) // {}` instead of
`map(f) | add // {}`. Since `add(f)` is `reduce f as $x (null; . + $x)`,
both fold the same values in the same order from the same seed, so the
only difference is that the array of single-key objects built by `map`
is never materialized.
Measured with both the generator and the result suppressed, so that only
the `from_entries` work differs: 6-15% less user time across input shapes,
which is 10-24% of what `from_entries` itself costs. For inputs above
roughly a thousand entries, peak RSS drops by 37%, around 80% of the
memory `from_entries` adds on top of its input; a 1,000,000 entry array
goes from 1064 MiB to 672 MiB.
block_mark_referenced walks the joined block backwards and skips the
body of any definition not yet marked referenced, so a library must
precede its importers. f58787c41 registered a library before walking
its dependencies, to have somewhere to keep the `loading` flag, which
reversed that order and made block_drop_unreferenced free definitions
that are still referenced through bound_by.
Keep the libraries being loaded on a stack threaded through the
load_library frames instead, so registration can move back after the
dependency walk.
jq dumps JSON to the Windows console as UTF-16 with WriteConsoleW(),
but priv_fwrite() still writes the raw UTF-8 bytes with WriteFile(),
which the console decodes with its output code page. So `jq .k` prints
non-ASCII correctly while `jq -r .k` prints mojibake, and the same goes
for -j, --raw-output0, --seq, halt_error and the `stderr` builtin.
The tty flag is computed once, from stdout, and passed to writes aimed
at stderr too, so ask the handle instead: with stdout on a console and
stderr redirected, WriteConsoleW() failed on the file handle and the
output was dropped.
One byte-level change: with stdout on a console and stderr redirected,
halt_error and the `stderr` builtin now go through the CRT, so bare LF
becomes CRLF and an existing CRLF becomes CR CRLF. Use -b for
byte-exact output on that stream.
The ownership comment in `jv.h` currently says every `jv_*` function
except `jv_copy` consumes its input. Several accessors and
object-iteration functions borrow their arguments instead, which makes
that guidance misleading for callers and language bindings.
Document the borrowing exceptions beside their declarations, including
numeric helpers and `jv_show`. Clarify that returned `jv` values are
owned, while the pointers returned by `jv_string_value` and
`jv_number_get_literal` belong to their input values. This changes
comments only.
The whole seconds were truncated toward zero while the fractional part
was measured from the floor, so any negative non-integral timestamp was
rendered one second late; gmtime was not monotonic across the epoch.
Only the latest release stays visible in the binary, checksum and
signature, and source tarball lists; older versions move into
collapsible `<details>` sections. Condense the repetitive per-version
checksum and signature sentences into a single list.
State explicitly in the README and download page that the
platform-specific prebuilt files are standalone executables, not
archives. Include the rename and executable-permission step for macOS
and Linux in the README.
Addresses #2956. Source archives are not described as executables.
Validation: the official `docs/build_website.py` completed and the
generated download page contains the new guidance. Used a Python venv
with the site's required rendering packages; restored the Windows
checkout's manual symlink in a temporary build copy only. jq itself was
not built or tested for this documentation-only change.
Co-authored-by: Hugohong258 <282228220+Hugohong258@users.noreply.github.com>
The bundle generated by actions/attest is currently only retrievable
via the GitHub attestations API, which requires authentication.
Uploading it as jq-attestation.json lets downstream consumers
(distro packagers, unauthenticated CI) verify provenance offline.
The bound added in CVE-2026-32316 (e47e56d22) still allowed
`sizeof(jvp_string) + (currlen + len) * 2 + 1` to wrap `size_t` on
32-bit platforms. Tighten the threshold so the final allocation
fits in 32-bit `size_t`.
Also break out of `jv_string_implode` and `f_string_implode` once
`jv_string_append_codepoint` returns an invalid `jv`; otherwise the
next iteration triggers the assertion in `jvp_string_ptr` (or
invokes undefined behavior under `-DNDEBUG`).
Fixes CVE-2026-54679.
Co-authored-by: Dirk Müller <dirk@dmllr.de>
When `jv_string_append_buf` overflows the string length limit,
it returns an invalid `jv`; `jv_load_file` then re-entered it
on the invalid value and overran the heap. Break out of the loop
once the value is invalid.
Fixes CVE-2026-49839.
The table was declared with 255 entries but indexed by an unsigned
char, so the decoder would read one byte past the array for input
bytes equal to 0xFF. Through jq's normal input paths the over-read
is unreachable because invalid UTF-8 bytes are normalized to U+FFFD
before reaching @base64d, but the table is still wrong at the C
level and reachable by libjq callers that construct strings via
the C API without UTF-8 validation.
jv_equal and jv_cmp overflows the C stack on deeply nested
input. Cap recursion at 10000 with -1 / INT_MIN sentinels;
operators that compose user expressions surface this as
"Equality check too deep" / "Comparison too deep".
Fixes CVE-2026-47770.
jq used to recurse without bound on mutual or self-referential
`import` declarations, exhausting the stack. Track each library's
load state with a `loading` flag set before its dependencies are
processed; a recursive reference to an in-progress library now
reports "circular import of X".
Fixes CVE-2026-44777.
jq accepts embedded NUL bytes at the language level but resolves
module import paths through NUL-terminated C strings, so the path
validated by policy or audit code could differ from the on-disk
path jq actually opens. Pass jv through gen_import so the AST
preserves the original bytes, and reject embedded NULs in
validate_relpath.
Fixes CVE-2026-43895.
A signed-int overflow in decNumber's D2U macro lets huge literals
write attacker-controlled bytes past a stack buffer. Cap the length
before calling decNumberFromString, and pre-slice long strings in
jv_dump_string_trunc so the resulting error message doesn't itself
allocate a multi-GiB buffer.
Fixes CVE-2026-43894.
Deeply nested path arrays can cause unbounded recursion in
`jv_setpath`, `jv_getpath`, and `jv_delpaths`, leading to
stack overflow. Add a depth limit of 10000 to match the
existing `tojson` depth limit. This fixes CVE-2026-33947.
The hash function used a fixed seed, allowing attackers to craft colliding keys
and cause O(n^2) object parsing performance. Initialize the seed from a random
source at process startup to prevent the attack. This fixes CVE-2026-40164.
Co-authored-by: Asaf Meizner <asafmeizner@gmail.com>
In `jvp_string_append`, the allocation size `(currlen + len) * 2` could
overflow `uint32_t` when `currlen + len` exceeds `INT_MAX`, causing a small
allocation followed by a large `memcpy`.
In `jvp_string_copy_replace_bad`, the output buffer size calculation
`length * 3 + 1` could overflow `uint32_t`, again resulting in a small
allocation followed by a large write.
Add overflow checks to both functions to return an error for strings
that would exceed `INT_MAX` in length. Fixes CVE-2026-32316.
I love jq :) So while updating the version we ship in Solaris I have
tried to modify stuff to make everything work out of the box. Just
```
autoreconf -fiv
./configure
gmake
gmake check
```
I also went ahead and created Solaris test action should you be
interested in having it too. The changes are not extensive and I have
tried to comment them nicely in each commit. Hopefully that's enough.
Thanks for jq!
If --disable-maintainer-mode is enabled, then the rules for generating
parser.[ch] and lexer.[ch] did nothing. This worked fine if the source
and build directories are the same as the pre-generated parser.c and
lexer.c files would suffice. However, if the build directory is not the
same as the source directory, then the rest of the Make rules expect
parser.[ch] and lexer.[ch] to have been created in the build directory
if their source files (parser.y and lexer.l) are newer than the target
files, which can happen in case the source is fetched using Git.
Avoid the problem by copying the files to the build directory if needed.
Co-authored-by: Peter Kjellerstedt <pkj@axis.com>
- Fix segfault when --skip/--take is the last argument.
- Fix FILE handle leak: opened test files were never closed.
- Support running multiple test files in one invocation.
Copy `kk` and `vk` before passing to jv_object_get(), which consumes
its arguments. Without copies, subsequent loop iterations read freed
memory (CWE-416) and post-loop jv_free() double-frees (CWE-415).
This bug was introduced in `b279713e` when array processing was hoisted
out of
the loop into locals without accounting for jv_object_get() semantics.
Fixes GHSA-gf4g-95wj-4q4r.
When a module fails to parse as a library, `load_library` cached the
invalid block. A second import/include of the same module would find
the cached invalid block and pass it to `block_bind_library`,
triggering the assertion failure. Skip caching the block when there
are parse errors to avoid this problem.
This commit improves performance by refactoring `@uri` and `@urid`
to pre-alloacte buffers and build the `jv` string at the end.
Also, this fixes `@urid` to prevent multi-byte UTF-8 corruption,
and to preserve embedded null bytes.
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Was detect by AFL build but it's unsure if possible to trigger in non-fuzz build.
Oniguruma has a default limit of 4096 but this is not be low enough to
protect from stack-overflows in some builds like with AFL where it seems
to use more stack space. A limit of 1024 should be enough usage-wise
and also give more margin.
The `jv_string_empty` did not properly null-terminate the string data,
which could lead to a heap buffer overflow. The test case of
GHSA-p7rr-28xf-3m5w (`0[""*0]`) was fixed by the commit dc849e9bb7,
but another case (`0[[]|implode]`) was still vulnerable. This commit
ensures string data is properly null-terminated, and fixes CVE-2025-48060.
* Handle input errors for --indent
Handle malformed and overflowing arguments. Also, forbid leading and
trailing spaces to match the behavior of tonumber from commit ce0e788
(improve tonumber/0 performance by parsing input as number literal,
2024-03-02).
* Do not skip leading whitespace in jvp_strtod
`jvp_strtod` skips leading whitespace, but its decnum counterpart
`decNumberFromString` (called within `jv_number_with_literal`) does not.
Those two are called interchangeably, so it leads to inconsistent
behavior depending on whether the decnum feature is enabled.
Additionally, `classify`, used in the token scanner, only considers
[ \t\n\r] to be whitespace, but `jvp_strtod` consumes the larger set
[ \t\n\v\f\r], so those extra characters are considered literals.
Changing this deviates from the behavior of `strdod` from <stdlib.h> and
is technically a breaking API change, since it is a public symbol.
Previously, the precedence of `as` binding was higher than the binary
operators. This results in unexpected behaviors, `1 + 2 as $x | -$x`
was `-1` instead of `-3`. The commit 92daac90f6 (PR #3053) changed
the behavior to emit syntax error on this case. Now, this commit allows
binary operators with `as` binding. This fixes#2446, and fixes#3284.
This commit fixes signed integer overflow and SEGV issues on growing
arrays and objects. The size of arrays and objects is now limited to
`536870912` (`0x20000000`). This fixes CVE-2024-23337 and fixes#3262.
This commit improves the parser and compiler error messages, making
it more easier to understand where in the filter the problem is.
Like modern compilers, jq will show the line in the filter where the
error occurred, will display squiggly line under the invalid token.
This commit improves the parser to fix some unintuitive results of the unary
operator `-` (e.g. `[-1 as $x | 1,$x]` is now `[1,-1]`, previously `[-1,-1]`).
Also, this commit now allows to use binary operator expressions, `reduce`,
`foreach`, `if`, and `try` expressions for object values.
This commit refactors `splits/2` implementation using `foreach` syntax
and reduces intermediate arrays. This refactoring removes an unnecessary
private (and undocumented) filter `_nwise` and closes#3148.
The introduction says all the usage examples (e.g. of truncate_stream,
etc) in the section use [0,[1]] as input, but they actually use [1,[2]].
["a",["b"]] seems better than [1,[2]] as the string values won't be
confusable with the indexes in <path>.
Simplifies the compilation of `foreach` to avoid the problem in #3227 by
removing unnecessary `FORK ... BACKTRACK`.
Does not address `reduce`, which is less trivial.
This commit drops support for parsing NaN with payload in JSON like
`NaN123` and fixes CVE-2024-53427. Other JSON extensions like `NaN` and
`Infinity` are still supported. Fixes#3023, fixes#3196, fixes#3246.
Using `calloc` function with a count of zero elements is implementation
defined and it may cause allocation errors on some platforms (ref #3277).
This commit fixes `jv_mem_calloc` callers to avoid undefined behavior.
When we use a large format string to `strftime/1` and `strflocaltime/1`,
we get segmentation fault. This happens because the result buffer is
allocated on the stack using `alloca` function, and `strftime` function
writes out of the space when the size is not enough to format time.
I fixed the segmentation fault issue by allocating the result buffer on
the heap using `malloc` function, because `alloca` function does not
provide a way to detect insufficient space.
This commit improves the performance of repeating strings, by copying
the result string instead of the string being repeated. Also it adds
an error message when the result string is too long.
Previously, the configure script automatically enables Valgrind during
testing when the valgrind command is found. However, running tests with
Valgrind is quite slow, so it is better to disable it by default, and
to enable it only when the user specifies the `--enable-valgrind` flag.
On OpenBSD, some prototypes are hidden behind `_BSD_SOURCE`.
Defining this variable fixes#3252, and also fixes#3276.
This can be seen as a regression of #2631.
Previously, `unique` and `unique_by` filters are implemented using
`group_by` and then `map(.[0])`. This commit re-implements in C, to
avoid unnecessary boxing of grouping, and improves the performance.
As discussed in https://github.com/jqlang/jq/discussions/3244, we are
moving jqplay to the jqlang GitHub organization. This updates all
references from jqplay.org to play.jqlang.org.
Whether `char` has a sign is implementation dependent, and `c >= 0` may
result in a compiler warning. We can use bitwise operator to check
whether the character is a control character, regardless of the sign.
The `jv2tm` function was zeroing fields of `struct tm` that were not
specified by the standard. However, depending on the libc this produced
incorrect timezone data when used together with formatting functions.
This change tries to fill the timezone data using either `mktime`,
`timegm`, or manually.
Apple's Libc implementation contains a bug which causes it to ignore
the offset data present in the `struct tm` in favor of the older heuristic
needed by legacy standards. This workaround temporarily sets the global
timezone so it gets picked up during formatting.
The code of `jv_number_value` is intended to cache the double value of
literal numbers, but it does not work because it accepts the `jv` struct
by value. This patch fixes the behavior by checking if the double value
is `NaN`, which indicates the unconverted value. This patch improves the
performance of major use cases; e.g. `range(1000000)` runs 25% faster.
We no longer use the `--root` option of `build_website.py` since we have
moved to jqlang.org (from jqlang.github.io/jq where the root was `/jq`).
We don't need the option on local development either since `http.server`
has the `-d` option to specify the directory to serve from.
The -static-libtool-libs flag causes GNU libtool to add a rpath to the .libs
build directory, which is intended for internal use by the libtool
implementation and is not installed in the resulting package causing Gentoo to
print a QA notice. See Gentoo issue: https://bugs.gentoo.org/945698.
* Create `src/config.h` instead of passing everything on the compiler
command line.
* To reduce the magnitude of this change, add `--include src/config.h`
to CFLAGS instead of adding `#include "src/config.h"` at the top of
each source file. Not all compilers support this, but I don't think
we care about anything other than gcc and clang at this point.
* Rather than generate `src/config_opts.inc`, emit `JQ_CONFIG` directly
into `src/config.h`.
* Rather than generate `src/version.h`, define `JQ_VERSION` as an alias
for the standard `PACKAGE_VERSION`.
Adding https://github.com/actions/attest-build-provenance to the ci builds so
that the release assets and docker image for the next release tag generate
signed build provenance attestations for workflow artifacts.
In C23 (default C standard used by GCC 15), jv (*fptr)(); has become
equivalent to jv (*fptr)(void); so we can no longer assign builtin
implemenations directly to the fptr member of cfunctions without
generating a compile error.
Since there does not seem to be any straight-forward way to tell
autoconf to force the compiler to use C99 short of explicitly adding
-std=c99 to CFLAGS, it is probably a cleaner solution to just make the
code C23 compatible.
A possible solution could have been to just redeclare cfunction.fptr
as void*, but then the functions' return type would not have been type
checked (e.g. if you tried to add a {printf, "printf", 2}, where printf
is a function that does not return jv, the compiler wouldn't have
complained.)
We were already not typechecking the arguments of the functions, so e.g.
{binop_plus, "_plus", 3}, /* instead of {f_plus, "_plus, 3}, */
{f_setpath, "setpath", 4}, /* instead of {f_setpath, "setpath", 3}, */
compile without errors despite not having the correct prototype.
So I thought of instead improving the situation by redefining
cfunction.fptr as a union of function pointers with the prototypes that
the jq bytecode interpreter can call, and use a macro to add the builtin
functions to function_list using to the arity argument to assign the
implementation function to the appropriate union member.
Now the code won't compile if the wrong arity, or an arity not supported
by the bytecode interpreter (>5 = 1input+4arguments), or a prototype not
jallable by the bytecode interpreter (e.g. binop_plus that doesn't
expect a jq_state* argument).
Also, the code now compiles with gcc -std=c23.
Fixes#3206
When the UPDATE query in a reduce or foreach syntax does not emit
a value, it should simply skip updating the state variable. But
currently it resets the state variable to null.
* Improve reliability of NO_COLOR tests.
Use `env` to ensure that `NO_COLOR` is set to the desired value
regardless of any initialization that may occur when `script`
forks a shell to run jq in.
Note that `env -u` is not portable, so there is no good solution
for the first test case.
* Always use /bin/sh, not the user's login shell.
This comment predates the addition of $ARGS in jq 1.6, so there was no
way to access named arguments with an invalid identifier. Rather than
forbid such names, update the manual to clarify.
When jq is not supplied with a program, it usually exits with an error
and usage text, except for when jq is not connected to a TTY (i.e., not
interactive), in which case a default program of `.` is used. Combined
with `-f`, it then tries to read `.` as the filename.
$ jq
jq - commandline JSON processor [version 1.7.1]
…
$ jq -f
jq - commandline JSON processor [version 1.7.1]
…
$ jq | cat
^C
$ jq -f | cat
jq: Could not open .: It's a directory
Fixes:
* Short options cannot be repeated in the same group:
e.g., `-nn` fails, but `--null-input --null-input` works
* Short options are processed in an arbitrary fixed order, not the order
given by the user: e.g., `-hV`, `-Vh`, `-h -V`, and `--help --version`
are processed as `-h`, but `-V -h` and `--version --help` are
processed as `-V`
* `-L` cannot be a part of a short option group (as the last one):
e.g., `-nLlib` and `-nL lib` fail
I also changed the behavior of limit/2 with negative count to emit an error, like nth/2 does.
Also, I redefined nth/2 using skip/2 to minimize the impact of growth of builtin filters.
This check was redundant and useless since jv2tm() used to require 8
elements, and errored anyway if not enough elements were provided.
After commit bc96146, this check also prevents mktime/0 from accepting
now supported inputs with less than 8 elements.
Ref: https://github.com/jqlang/jq/pull/3070#issuecomment-2299902072
Reported-by: itchyny <itchyny@cybozu.co.jp>
Fixes issue with confusion about features that has not been
released yet.
Move deveopment manual.yml to dev/manual.yml
Symlink manual.yml to v1.7/manual.yml
Refactor to share history header with links to other manual versions.
Change man.test and man page generation to use dev/manual.yml
Related to #3078#3127
`jq_reset` calls `jv_free` on the `exit_code` and the `error_message` stored on the jq state.
However, it doesn't replace the actual instance of those members. This means that subsequent
calls to `jq_reset` will call `jv_free` again on those members, which in turn may call `free`
on the same pointer multiple times. Freeing the same pointer multiple times is undefined
behavior and can cause heap corruption, which is how I spotted this issue.
In practice, this issue only occurs when using a program that may `halt_error`, because that
is when the `exit_code` and `error_message` are set to values other than `jv_invalid`.
Subsequent attempts to call `jq_start` (which calls `jq_reset` internally) after hitting a
`halt_error` can cause you to run into this issue.
The changes simply reset the `exit_code` and the `error_message` to `jv_invalid` (the initial
value set in `jq_init`) after they are freed.
`-b`/`--binary` should be accepted on all platforms so that they can be used in
code that is meant to be portable.
Signed-off-by: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
When I tried to build a new version in the same working directory I'd used to build a previous version, I saw build errors. After a few attempts, I remembered that "make clean" might get past the problem. It seems like a nice thing to suggest to everyone who might fall into the same trap.
arrays[arrays] is a special case of "INDEX" that actually returns an
array containing the indices in which the array that is being indexed
contains the start of the key array.
So array keys, for array values, are a kind of key that can be "got",
but not "set". jv_setpath() was not freeing the value it "got" from
indexing that key, in case the following "set" on that key failed,
resulting in a leak.
$ ./jq -n '[] | setpath([[1]]; 1)'
jq: error (at <unknown>): Cannot update field at array index of array
=================================================================
==953483==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 272 byte(s) in 1 object(s) allocated from:
#0 0x725f4d4e1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x5ec17b1a7438 in jv_mem_alloc src/jv_alloc.c:141
SUMMARY: AddressSanitizer: 272 byte(s) leaked in 1 allocation(s).
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66061
Now, time functions accept array inputs even if they don't have all the
elements, 0 will be assumed if a value is not present.
Also, jv2tm now properly clamps large number values to a signed 32-bit
integer and rejects nan.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65885
tests/mantest were failing on Raspberry Pi OS 11:
==10205== Conditional jump or move depends on uninitialised value(s)
==10205== at 0x48806F8: checkfail (jq_test.c:58)
==10205== by 0x48806F8: run_jq_tests (jq_test.c:99)
==10205== by 0x4880FCB: jq_testsuite (jq_test.c:39)
==10205== by 0x10B37F: main (main.c:576)
==10205==
==10205== Conditional jump or move depends on uninitialised value(s)
==10205== at 0x4880714: checkfail (jq_test.c:58)
==10205== by 0x4880714: run_jq_tests (jq_test.c:99)
==10205== by 0x4880FCB: jq_testsuite (jq_test.c:39)
==10205== by 0x10B37F: main (main.c:576)
In windows, time_t is a signed 32-bit integer type, so TIME_MAX needs to
be declared as INT32_MAX instead of INT64_MAX.
Also bump NetBSD's strptime to revision 1.65 from 1.63 to fix undefined
behaviour (signed integer overflow) bugs.
Related NetBSD problem report:
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58041
Noticed thanks to a compiler warning in the windows build CI.
I declared the d variable as time_t instead of as unsigned to ensure
that the signedness/size of TIME_MAX-d in the sse > TIME_MAX-d check
is always correct, and to prevent -Wsign-compare warnings from
triggering in the windows build.
There was a incorrect else, that caused jq to not ensure that the
argument to strftime/1 is a string when the input is a number; this ends
up calling jv_string_value on a non-string value, which does not work,
and causes an assert failure.
Also fix same bug in strflocaltime/1.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67403
Previously, the tonumber/0 filter parses the input as JSON values, but
this is less-performant on large non-number strings. Parsing the input
string as number literal fixes the performance issue. Also, this fix
changes the filter to reject numbers with white spaces.
* Fix typo in documentation for map_values
"map_value(f)" -> "map_values(f)"
* map_value => map_values also in the devolpment manual
---------
Co-authored-by: Emanuele Torre <torreemanuele6@gmail.com>
Before, arrays would not reset colors after `[` and `,`, but objects
would; it would reset colors twice before `]` and `}`; and some cases of
indentation would have colors applied. Now, colors are reset immediately
after any token that is colored, before any indentation. This makes the
formatting consistent, for the benefit of custom `JQ_COLORS`.
The README mentions only the MIT license but in fact part of `jq`
is under ICU license and the documentation is under CC.
Also jq.spec claimed that it is licensed under BSD.
C99 only allows string literals long at most 4095 characters.
jq_builtins was a lot longer than that.
I rewrote all the optional libm error stubs in C so the value of
jq_builtins is not build dependent.
I replaced the command that generates builtin.inc with a POSIX compliant
od|sed command that encodes builtin.jq as a comma delimited list of
octal numbers (that can be embedded in C using a {} literal).
I also added -Woverlength-strings to AM_CFLAGS to verify that the
problem is fixed.
Fixes#1481
Previously, ltrimstr/rtrimstr would just let the input pass through for
non-string inputs or arguments.
That was happening because, they were leaking the errors returned by
startswith/endswith treating them as if they were jv_false().
The leak was resolved by #2977 for 1.7.1
This patch rewrites ltrimstr and rtrimstr in jq, and makes them not
ignore startswith and endswith errors anymore.
* Always build docker image on CI
* Revert "Bump actions/download-artifact from 3 to 4"
This reverts commit 9e2695e117.
* Group Dependabot PRs for official actions
decNumberToString calls for a buffer that can hold a string of digits+14
characters, not a buffer of size digits+14.
We need to allocate an extra byte for the NUL byte.
-10E-1000010001, for example, will be stringified as -1.0E-1000010000
and decNumberToString will currently write an extra NUL byte after the
allocated buffer in the heap.
Originally reported by @SEU-SSL on GitHub.
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64574
Fixes GHSA-686w-5m7m-54vc
The unit allocated for decNumberCompare was accidentally removed by
commit 680baeffeb (PR #2804)
This caused a stack overflow when comparing a nan with a payload of 1000
or more.
This bug was found by OSS-fuzz.
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64771
Fixes GHSA-7hmr-442f-qc8j
It also fixes 1e999999999 > 1e-1147483646 triggering UBSAN errors
Fixes#2968
This commit temporarily reverts the commit that allows #\ "tcl-style"
comments everywhere and documents them, for the 1.7.1 patch release
cca1f7d18f.
\r is removed from the list of characters not allowed in a comment to
preserve that bugfix.
ltrimstr/rtrimstr was ignoring and leaking the error returned by
f_startswith()/f_endswith().
This also means that they just let the input pass through for non-string
inputs or arguments.
Only fix the leak for now; in the next release, #2969 will make them
rethrow the error returned by startswith/endswith.
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946
The "macos (arm64)" runner on github does not have nproc.
/Users/runner/work/_temp/f44f0d9f-19eb-4a23-860e-26533d7efdfa.sh: line 10: nproc: command not found
It does not matter much since they most likely just get multiplied
together, but some compilers would complain about this if these were
calls to calloc.
Replace deprecated test(1) parentheses and -a logical operator with two
tests command.
Replace deprecated tail -1 with tail -n1.
Replace non-standard egrep(1) command with grep -E ; this also
prevents obsolescence warnings on GNU systems.
The locale test was using ./jq intead of $JQ.
I also removed the use of obsolete egrep instead of grep -E that
triggers warnings on GNU systems, and the use of deprecated head -1
instead of head -n1.
Also removed the unnecessary hiding of strptime/1 errors with
? // false.
The description of the Alternative operator `//` was hard for me to grasp in its wording.
I suggest dividing the looong sentence into two parts. Since it is actually an alternative formulation, the sentence can be divided into the first formulation, and the alternative formulation.
"\(" is implementation-defined in C++, and both clang and gcc treat it
as equivalent to "(", not "\\(".
This patch replaces "\(" with "\\(" so that tests with string
interpolation actually use string interpolation.
If using libjq from C++ it would be nice to not need to do this at the
import site, so just extern "C" to the public headers for libjq
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
`recurse/0` already handles traversing objects and arrays, so it is more
consistent to use that.
For `paths/1` it is easier to use the actual value returned by
`recurse` instead of querying that value with `getpath/1` afterwards.
* bugfix: comments were incorrectly being terminated by CR; for example
jq -n $'1 #foo\r'
fails to compile because the CR character terminates the comment, and
CR is not a valid character in jq syntax.
* improvement: comments fully support Tcl-style line continuation.
Previously this was only "supported" in `-f' scripts, whose first line
starts with "#!", and second line starts with # and ends with \, only
for the comment on the second line, only for one extra line.
* man: document comment syntax, which was previously undocumented.
* tests: add regression tests for the bugfix, and some tests for line
continuation in comments.
Otherwise `AGRS` and `program_arguments` remain allocated/unfreed in the
early (extremely unlikely) pledge(2) failure case.
Move their allocation before jq_init(), the first case of jumping to
`out` where they are cleaned up, where it also seems to logically fit
better than above between function entry, locale setup and OpenBSD
specific pledge.
Use pledge(2)[0] to limit jq(1) to reading files.
It does not change files and only writes to standard output/error.
It never deals with TTY, network, process management or other subsystems.
This is to reduce jq's attack surface and potential damage.
OpenBSD is carrying a local patch[1] in its official jq port/package
since 2016. An improved version:
- drop no longer needed "getpw" promise
f1c4947 "Avoid getpwuid for static linking" removed getpwuid(3) usage
- pledge before jq_init() to simplify the error path
- use perror(3) to print errno(2)
No behaviour change in tests or real world usage observed on
OpenBSD/amd64 7.4.
0: https://man.openbsd.org/pledge.2
1: https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/textproc/jq/patches/patch-main_c
This reverts commit 0e70f7a57e.
There is no reason to revert this change.
In #2922, I only disagreed with the commit message suggesting that
LC_CTYPE=C od -t c is equivalent to od -c
The only documented differences are that -tc is required to be
influenced by -N and -j, while -c is not, and that -c is required to
only support a subset of the backslash sequences that -tc should
support.
jq previously only allowed passing the inline script before -- (as if
they were options) even though one would expect the inline script to be
a positional argument.
Since jq previously also refused to run with a usage error if the script
was passed after -- (It was not assuming . as script as it does when
no arguments are passed), and positional arguments are allowed before --
and even before other options, it should not be a breaking change to
change that weird behaviour, and allow the script to appear after --.
It also simplifies the option parsing code a bunch.
Fixes#2918
In process there is a suspicious options |= EXIT_STATUS_EXACT that
is run when the jq script is terminated by halt, or halt_error.
That line of code acutally does nothing because options is a local
argument variable, and is not passed as a pointer. It was probably meant
to be a *options |= EXIT_STATUS_EXACT with the options argument
passed as a int*.
In any case, we do not want to run the code in main() that was supposed
to run if EXIT_STATUS_EXACT is set (but didn't since it is never added
to options); as far as I can tell, we only want to run that code when
the --exit-status/-e option is passed.
So I removed EXIT_STATUS_EXACT completely, and the useless assignment,
instead of fixing it since it was not used for anything else.
Useless assignment detected by clang-tidy.
A very large program can cause these leaks:
==758838== 7,820 (16 direct, 7,804 indirect) bytes in 2 blocks are definitely lost in loss record 17 of 28
==758838== at 0x4848A23: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==758838== by 0x125D30: jv_mem_calloc (jv_alloc.c:153)
==758838== by 0x162ADE: compile (compile.c:1286)
==758838== by 0x162D4B: compile (compile.c:1304)
==758838== by 0x163697: block_compile (compile.c:1381)
==758838== by 0x11B5CA: jq_compile_args (execute.c:1245)
==758838== by 0x115E20: main (main.c:691)
==758838==
==758838== 1,674,694 (103,576 direct, 1,571,118 indirect) bytes in 1,177 blocks are definitely lost in loss record 28 of 28
==758838== at 0x4843839: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==758838== by 0x125CD0: jv_mem_alloc (jv_alloc.c:141)
==758838== by 0x162B19: compile (compile.c:1289)
==758838== by 0x163697: block_compile (compile.c:1381)
==758838== by 0x11B5CA: jq_compile_args (execute.c:1245)
==758838== by 0x115E20: main (main.c:691)
This commit fixes that.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=61349
For the jq_state used by the jq utility, the JQ_LIBRARY_PATH attribute
will always be set, but, in general, it is possible that it might not
be.
If it is not set, jq_get_lib_dirs() will return jv_invalid().
That is not good, because some code in linker.c expects it to always
returns an array.
This patch makes jq_get_lib_dirs() return an empty array if
JQ_LIBRARY_PATH is not set to prevent problems.
This issue made OSS fuzz trigger failed assertions every time it tried
to compile a script that uses "include".
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=61796
Although #2839 fixed the overflow of exponent subtraction,
there still is possibility of overflow in the `D2U` macro.
This patch fixes the overflow in the `D2U` macro, and also
truncates the maximum digits to `DEC_MAX_DIGITS`.
The decNumber library subtracts the exponents of two numbers,
we make sure to limit the number of digits not to make it overflows.
Since the maximum adjusted exponent is `emax` and the minimum is
`emin - digits + 1`, we follow `emax - (emin - digits + 1) <= INT32_MAX`.
This reuses the existing `block_list_funcs` capability and adds an extra field on the `modulemeta` output, called `defs`, containing that list of functions.
* Build windows 64bit binary using UCRT64
Is the default and recommended msystem setting. Will produce
binaries that are compatible with windows 10 and later.
Also run tests for 32bit build.
Related to #2831
* Use jq -b in tests/shtest
* Add Windows strptime
* Make Windows-optional tests not run on Windows again
---------
Co-authored-by: Nicolas Williams <nico@cryptonector.com>
This patch removes the weird behaviour of jv_invalid_with_msg(jv_null())
that returns jv_invalid() (i.e. empty), instead of a boxed jv_null().
The previous behaviour of null|error was obviously unintentional, and
allowing is jv_invalid_with_msg() to return values on which you can't
call jv_invalid_get_msg() is only error prone.
This patch exports all the binary operator builtins functions from
builtin.c and uses them for constant folding in the parser, allowing
constant folding to work will all kinds and combinations of constants.
Now string*number, $ARGS+$ARGS, string/string, etc will also be
constant folded and the implementation of constant folded operators and
runtime operators will be the same.
And thanks to the new ERRORK bytecode operation, errors are constant
folded too! (e.g. 1 / 0 [] * {} etc)
Use a StringStart component that is either FORMAT QQSTRING_START or
QQSTRING_START instead of having two similar rules for String.
This is simpler and avoids having to use an untyped mid-rule action
component to copy FORMAT at the top of the stack before QQString, and
having to use jv_free($<literal>3) instead of jv_free($1) just to make
bison not complain about the "unused" mid-rule component.
For some reason, the env vars are set correctly without refering the variable in `matrix` case-sensitvely:
```
2023-08-01T05:05:58.4626090Z env:
2023-08-01T05:05:58.4626367Z AR: arm-linux-gnueabihf-ar
2023-08-01T05:05:58.4626690Z CHOST: arm-linux-gnueabihf
2023-08-01T05:05:58.4626961Z CC: arm-linux-gnueabihf-gcc
2023-08-01T05:05:58.4627332Z CPP: arm-linux-gnueabihf-cpp
2023-08-01T05:05:58.4627662Z CXX: arm-linux-gnueabihf-g++
```
For clarify, we update them to match the definitions. Besides, `CXX` is
not needed and removed.
Previously constant folding of zero division (e.x. 1/0) produces a
compile error. This was incorrectly implemented by checking if the
division result is infinite, so produces wrong results compared to the
query where no constant folding is processed (e.x. 1e308/0.1). This
patch delays the operation when the divisor is zero. This makes the
results more consistent, but changes the exit code on zero division from
3 to 5. Also 0/0 now produces the zero division error, not NaN.
This patch also fixes the modulo operation. Previously constant folding
logic does not take care of the % operator, but now it folds if the both
dividend and divisor are safe numbers to cast to the integer type, and
the divisor is not zero. This patch also fixes some code that relies on
undefined cast behaviors in C. The modulo operation produces NaN if
either the dividend or divisor is NaN.
You only need to specify the return type in a function pointer
declaration in C.
If you use () in the declaration, the function pointer can be called
with any arguments, and the type of the arguments is decided for each
function call based on the types of the arguments used for the call.
(To declare a function pointer for a function with no arguments, you use
`(void)'.)
Since all the cfunction structs have a fptr that points to a functions
that return jv, not void, we can we can just declare cfunction.fptr as
jv (*)() and avoid having those annoying and technically not C-standard
compliant casts everywhere.
* Bump up Bootstrap to v5.3.1, Bootstrap Icon to v1.10.5.
* Use autoComplete.js to drop dependency on jQuery and typeahead.js.
* Support dark mode.
* New svg logo and icon with responsive color mode support.
* Normalize section ids to lower kebab-case for easiness of linking.
* Use relative paths for links for local development (--root /output).
* Various markup cleanups and accessibility improvements.
The option naming --nul-output was confusing, especially when we have a
similar option for input stream in the future (--nul-input vs --null-input).
Based on the observation of other command line tools, we rename the option
to --raw-output0. We also drop the short option -0 to avoid confusion on
introducing the NUL-delimited input option.
Unlike the other command line tools outputting file names with NUL delimiter,
jq deals with JSON, and its strings may contain NUL character. To protect
users from the risk of injection attacks, we abort the program and print an
error message before outputting strings including NUL character. Closes#2683.
RS should only only have special meaning when parsing json-seq.
Before this fix, RS was sometimes treated as a value terminator, causing
weird results.
Bug discovered by OSS fuzz.
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60835
It seems that bison doesn't call destructors for mid-rule action
components on error, since it does not know their type.
A mid-rule action was used to allocate the "text" string used as format
by string literals without a format, which would leak on error.
This patch replaces it with a new NoFormat component of type <literal>.
Now bison will call jv_free() on that string after a syntax error.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60836
$ ./jq '"'
jq: error: syntax error, unexpected end of file, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at <top-level>, line 1:
"
jq: 1 compile error
=================================================================
==1495450==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 21 byte(s) in 1 object(s) allocated from:
#0 0x7fc21aee1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x557ccb6ff608 in jv_mem_alloc src/jv_alloc.c:141
SUMMARY: AddressSanitizer: 21 byte(s) leaked in 1 allocation(s).
Error on non-number and nan codepoint, would asserd before
Replace negative codepoint and surrogate range with unicode replacement character, would assert before
Fixes#1160
In previous code nomem_handler in jv_alloc.c is called only once
and therefore the structure is not initialized for second and
following threads.
This leads to segmentation fault in multi-threading environment.
This patch moves initialization of nomem_handler out of the
pthread_once call.
This makes the standard build instructions a bit easier,
just ./configure, and also requires less tools installed (bison).
Also i think few people probably want to generate the lexer and paser code.
valgrind (3.19.0) locally on macOS docker container seems to detect this leak but not CI.
$ cat modulemeta.test
modulemeta | .deps |= length
"c"
{"whatever":null,"deps":6}
$ valgrind --error-exitcode=1 --leak-check=full -q ./jq --run-tests < modulemeta.test
Test #1: 'modulemeta | .deps |= length' at line number 1
*** Insufficient results for test at line number 3: modulemeta | .deps |= length
0 of 1 tests passed (0 malformed, 0 skipped)
==14129== 482 (392 direct, 90 indirect) bytes in 1 blocks are definitely lost in loss record 7 of 8
==14129== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==14129== by 0x13BC78: jv_mem_alloc (jv_alloc.c:122)
==14129== by 0x1371C6: jvp_object_new (jv.c:1521)
==14129== by 0x145840: parse_token (jv_parse.c:167)
==14129== by 0x145840: scan (jv_parse.c:684)
==14129== by 0x145840: jv_parser_next (jv_parse.c:793)
==14129== by 0x1463CB: jv_parse_sized (jv_parse.c:865)
==14129== by 0x136BD9: run_jq_tests (jq_test.c:183)
==14129== by 0x1370FA: jq_testsuite (jq_test.c:34)
==14129== by 0x13135C: main (main.c:582)
==14129==
This is unnecessary since we are using AC_USE_SYSTEM_EXTENSIONS in
configure.ac, which makes the compiler always run with -D_GNU_SOURCE=1
when building for GNU/Linux.
Also, the #define _GNU_SOURCE in src/util.c, and in
src/inject_errors.c (--enable-inject-errors), were not guarded by a
#ifndef _GNU_SOURCE so they were just triggering redefinition warnings.
- Add error/0 and mentions null input behavior (close#2231)
- Explain value iterator suffix syntax .foo[] (close#1047)
- Mention array slicing is also zero-based (close#2094)
- Add examples of input and inputs filters (close#2216, close#2470)
- Improve sort_by about multiple values (close#2103, close#2467, close#2474)
- Improve foreach section and simplify examples (close#1148, close#2169)
- Fix recurse/1 document on how it is identical using recurse/2 (close#2036, close#2412)
- Add non-string examples of index/1, rindex/1 (close#1422)
- Simplify the example of truncate_stream/1 (close#1736)
Close#1885, #2140, #2011, #2220, #2485, #2073
Rename the FORK_OPT opcode to TRY_BEGIN, add a TRY_END opcode, and wrap
errors when raising through a TRY_END so that they will not be caught by
the matching TRY_BEGIN.
Now a `try exp catch handler` expression generates code like:
TRY_BEGIN handler
<exp>
TRY_END
JUMP past_handler
handler: <handler>
past_handler:
...
On backtrack through TRY_BEGIN it just backtracks.
If anything past the whole thing raises when <exp> produced a value,
then the TRY_END will catch the error, wrap it in another, and
backtrack. The TRY_BEGIN will see a wrapped error and then it will
unwrap and re-raise the error.
If <exp> raises, then TRY_BEGIN will catch the error and jump to the
handler, but the TRY_BEGIN will not stack_save() in that case, so on
raise/backtrack the TRY_BEGIN will not execute again (nor will the
TRY_END).
Mostly clarifications w.r.t. numbers.
The anomalous behavior of leading minus signs is documented, with an explicit notice that this will probably change.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60691
$ ./jq <<< 0E2202012202
src/decNumber/decNumber.c:598:18: runtime error: signed integer overflow: 440402440 + 1761609760 cannot be represented in type 'int'
0E+999999999
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60683
$ ./jq <<< 5E7657765755
src/decNumber/decNumber.c:598:18: runtime error: left shift of 765776575 by 3 places cannot be represented in type 'int'
src/decNumber/decNumber.c:598:18: runtime error: signed integer overflow: 1531553150 + 1831245304 cannot be represented in type 'int'
1.7976931348623157e+308
Only examples defined in the "examples" field are generated, if they are
defined in "example", they are just ignored.
Also fix a bunch of incorrect tests.
Instead of just using {"captures":[]}.
sub functions are use captures for replacement expressions.
If we don't populate, captures for empty matches, the replacement
expression is run with an empty object as input instead of an object
containing the named captures with "" as value:
* before:
$ jq -n '"123foo456bar" | gsub("[^a-z]*(?<x>[a-z]*)"; "Z\(.x)")'
"ZfooZbarZnull"
* after:
$ jq -n '"123foo456bar" | gsub("[^a-z]*(?<x>[a-z]*)"; "Z\(.x)")'
"ZfooZbarZ"
---
I also removed a redundant
result = NULL;
if (result) {
...
}
To make debug/0 not call an uninitialised function pointer when using
--run-tests or when using a jq_state on which jq_set_debug_cb() has not
been called.
To avoid causing segmentation faults when input/1 is called in a
jq_state on which jq_set_input_cb() has not been called; e.g. the one
used by jq --run-tests.
That segfault could also be fixed in run_jq_tests() by calling:
jq_set_input_cb(jq, NULL, NULL);
But I think it makes sense to just make jq_init() initialise those
values to NULL.
Ref: https://github.com/jqlang/jq/pull/2717#discussion_r1264338841
* Fix memory leak for has(nan)
jv_array_get() used to be responsible of freeing the input array, but
since b5c4c3d67d, it is no longer called
if the key is nan. We need to free it manually to avoid leaking the
array.
* Parse nan in JSON as NaN instead of triggering a parse error
Fixes#2021
The markdown python changed its API for markdown extensions.
Bumping the markdown package version in previous commits broke
docs/build_manpage.py since it was using registering an extension with
the old API.
n.b.: { $__loc__: "foo" } is still not allowed.
$__loc__ cannot be overwritten, and it is always an object.
Using { $x: "foo" } where $x is not a string is an error, so there is
not much of a point in supporting that.
Previously, bindings were parsed as the combination of two tokens:
'$' IDENT
This meant that using a keyword as variable name (e.g. $then, $label)
did not work.
Attempts were made to allow $keyword to work by adding some '$' Keyword
rules in the parser, but this did not allow $keyword in all places:
jq --arg label foo -n '$label' # ok
jq -n '"foo" as $label | .' # error
Treating $foo as a single token is much simpler, in my opinion.
This patch also changes how LOC is lexed: "$__loc__" instead of as
"__loc__" that gets combined with '$' in the parser.
This patch also disallows having spaces after '$' when recalling a
variable `$ foo' since that was probably just an unintentional side
effect of the implementation, and it was not documented.
Fixes#2675
The primary purpose of this commit is to rectify most problems with
`gsub` (and also `sub` with the `g` option), in particular fix#1425 ('\b'),
fix#2354 (lookahead), and fix#2532 (regex == `"^(?!cd ).*$|^cd "`).
This commit also partly resolves#2148 and resolves#1206 in that
`gsub` no longer loops infinitely; however, because the new `gsub`
depends critically on `match/2`, the behavior when regex == `""` is
sometimes non-standard.
The documentation has been updated to reflect the fact that `sub`
and `gsub` are intended to be regular in the second argument.
Also, `_nwise/1` has been tweaked to take advantage of TCO.
* Release pipeline
* Group CI builds for different OSes into `ci.yml`.
* Add release job to release `jq` when tag is in the format of v*.
* Use `clang` as the only compiler on CI.
* Provide extensible matrix for future cross-compile builds, e.g.
for https://github.com/jqlang/jq/pull/2618.
* Locate bison for Windows build
* Also install flex for Windows build
* Add matrix to test all available GH Actions images
* Enable all tests for Windows
* Run `brew update-reset` when `brew update` fails
`brew update` can fail for "Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask failed!"
Ref: https://github.com/owenthereal/jq/actions/runs/5432314910/jobs/9879266028#step:3:19
* Select gcc binaries to release
* Only enable CI when push to master & tag
* Try out statick build
* No need to enforce the same `AM_INIT_AUTOMAKE` version for MacOS
* Disable static build for ubuntu-20.04
See https://github.com/jqlang/jq/pull/2620#discussion_r1248936822
Specified with `--docdir=<path>`, the `$(docdir)` variable is already
available, and has the same default as what it was being set to.
By specifying it here, it overrides what the `--docdir` option
specifies.
* configure.ac: Enable system extensions
Use AC_USE_SYSTEM_EXTENSIONS to build with _GNU_SOURCE (and similar
macros on non-GNU systems), to avoid an implicit declaration of the
strptime function. This improves compatibility with future compilers
which do not support implicit function declarations.
* configure.ac: gettimeofday lives in <sys/time.h>
The gettimeofday function is declared in <sys/time.h>, not <time.h>,
according to POSIX and actual systems. This avoids a configure
probe failure with compilers which do not support implicit function
declarations.
* Try making some GitHub Actions
* GHA: Add OS X, Windows, and Linux w/ scan-build builds
* Add link to discord server
Add link to discord server
* Update Ubuntu to 22.04 for GitHub Actions
* Remove if condition for Ubuntu build
* Fix MacOS OS on GitHub Actions
* Fix python3 package on GitHub Actions
* Disalbe valgrind
Getting failed tests: https://github.com/jqlang/jq/actions/runs/5113866588/jobs/9193542140#step:9:341
* Format file
* Fix macos build
* Fix syntax errors of windows build
https://github.com/jqlang/jq/actions/runs/5114162556/workflow
* Install windows package with choco
* Pin oniguruma to 5a24a49d710a9e3bb8ff11d12e1eae5a9f9df40c
MacOS build [fails](https://github.com/jqlang/jq/actions/runs/5114162555/jobs/9194126677#step:6:3160) due to
```
posix.c:94:3: error: implicit declaration of function 'onig_end' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
onig_end();
```
The current `oniguruma` revision
6fa38f4084b448592888ed9ee43c6e90a46b5f5c (15 Mar 2017) lacks the following explicit declaration in src/onigposix.h:
```
ONIG_EXTERN int onig_end P_((void));
```
This was added to oniguruma in revision 5a24a49d710a9e3bb8ff11d12e1eae5a9f9df40c (8 Sep 2017).
Ref: https://github.com/jqlang/jq/issues/2381
* Revert windows build to use pacman
* Don't zip jq.exe
* Fixing windows build
* Update .gitattriutes for eol on Windows
* Skip workflow_dispatch for windows build
* Clean up builds
* Use LF line endings for all text
Ref: https://github.com/actions/checkout/issues/135
* Fix test that has rounding error
https://github.com/jqlang/jq/pull/2596#issuecomment-1568498993
* Enable CI for all
---------
Co-authored-by: Nicolas Williams <nico@cryptonector.com>
When the string transformation to number failed, all following
transformation failed too.
This happend because status in decNumberFromString function is
updated just in error case. Reusing the DEC_CONTEXT that failed
before results into error even if the string is valid number.
We need to be careful to not grab an extra reference when mutating data
structures because that means we make extra copies. Doing that every
time in `_modify` is really painful, as that function implements `|=`
and all modify-assign operators.
`jv_setpath()` also grabs additional references, so this is not the only
fix needed for the modify-assign operators to not be accidentally
quadratic.
We have to use `LOADVN` in order to make this fix possible, so we should
really byte-code `_modify` rather than jq-code it. However, as a
stop-gap to make this fix easier, I'm adding syntax for referring to a
`$binding` such that you get `LOADVN` instead of `LOADV`.
This syntax is not meant to be used outside jq's internals! Therefore
it is not documented. I promise to break it later, so don't use it!
TBD, but later:
- Optimize `_modify` for the case where `update` outputs more than one
value.
This is trivial: add a `setpath($p; null)` in the middle of `_modify`
before calling `update`. But unfortunately the VM retains a
reference to `value_at_path` for path expression checking, and fixing
that will require more work.
We need to be careful to not grab an extra reference when mutating
data structures because that means we make extra copies. Doing that
every time in `jv_setpath()` is really painful, as that function is used
in `_modify/2`, which is used in `|=` and all modify-assign operators.
`_modify` also grabs additional references, so this is not the only fix
needed for the modify-assign operators to not be accidentally quadratic.
Keep a cached copy of the man tests that we can use when no manpage
changes are made. This allows automated systems that might not have
easy access to a pipenv to build and run tests.
It has no effect after the change c4524da.
Bug reported by the clang static analyzer.
Description: Value stored to 'value' is never read
File: jq/src/util.c
Line: 439
Test Plan:
To verify that this is in fact the behavior:
```
$ jq --indent 7 -n '[1]'
[
1
]
$ jq --indent 8 -n '[1]'
jq: --indent takes a number between -1 and 7
Use jq --help for help with command-line options,
or see the jq manpage, or online docs at https://stedolan.github.io/jq
```
This patch was generated by running `git grep "no more than"` and fixing
up appropriate results.
wchargin-branch: docs-fix-indent-bounds
Allow building jq in a mode that doesn't use decnumber for benchmarking
purposes. decnumber support is enabled by default, and this option is
meant to be removed once we're happy with the performance.
Extend jv_number to use decNumber for storing number literals. Any math
operations on the numbers will truncate them to double precision.
Comparisons when both numbers are literal numbers will compare them
without truncation.
Delay conversion of numbers to doubles until a math operation is performed,
to preserve precision. A literal jv_number will only need conversion to
double once, and will reuse the resultant double on subsequent
conversions.
Outputting literal jv_numbers preserves the original precision.
Add strong pthread requirement to manage contexts/allocations for
converting numbers between their decNumber, string, and double formats.
The current instructions tell users to perform two actions:
- update the package database
- install the jq package
The only thing users need to or should be doing is actually installing
the jq package -- regardless of which version is being installed.
Guidelines on how to perform system updates are massively out of scope.
In the case of partially performing a system update as a prerequisite
for installing jq, the official guidance from Arch Linux is not to do
this: partial updates are not supported, we refuse to support them, and
anyone who does try to perform them anyway is assumed to know so much
about their system that they clearly do not ever need help from anyone
else (which is a good thing since they won't get it). The result is a
frankensteined system that can only ever be supported by the person who
frankensteined it to begin with. The only reason the package manager
even allows it to occur in the first place is because other
distributions using pacman might have different LTS policies, and
because it would prevent expert users from being in control of their
system, as per the traditional Unix philosophy:
"Unix was not designed to stop you from doing stupid things, because
that would also stop you from doing clever things."
Consequences of performing partial updates without understanding the
ramifications in extensive detail can include breaking the partially
updated application (jq), breaking any application that shares a mutual
dependency with the partially updated application (which jq is *lucky*
to only depend on the ever-backwards-compatible glibc), or breaking the
entire operating system by leaving armed traps behind for the next time
a `pacman -S new-package` is executed and thereby breaks *its* cascading
dependencies.
See:
https://wiki.archlinux.org/index.php/System_maintenance#Partial_upgrades_are_unsupported
The current paragraph is not complete, since a digit is not a special character.
Changing it to:
If the key contains special characters or starts with a digit,
you need to surround it with double quotes like this:
`."foo$"`, or else `.["foo$"]`.
Run `make uninstall` for oniguruma before running the recursive
distclean that will remove oniguruma's Makefile and cause a build error
due to missing make target.
Fetch oniguruma using git submodule instead of a release tarball. It
will fix a build problem, caused by jq's autotools configuration, trying
to run `make distclean` recursively in an empty modules directory. This
will also improve the maintainability of the Dockerfile.
Expressions of the form `path(EXPR) | select(GENERATOR)`, where `EXPR`
is a path expression and `GENERATOR` is a generator conditional
expression (e.g., `has("a"), has("b")`) cause an assertion if the
jq_state VM is torn down too soon. That assert() was only correct if
assuming that the conditional is not a generator.
If the conditional is generator, then what we see is that when
backtracking a SUBEXP_END is executed without a corresponding
SUBEXP_BEGIN because the entire conditional is bracketed with
SUBEXP_BEGIN and SUBEXP_END, and since it's resumed in the middle, in
between the brackets.
Rather than assert that the jq->path_len being restored has some
particular value, we can simply re-compute it from the restored
jq->path.
The behavior of memmem for an empty needle is inconsistent between
implementations of libc.
Our tests imply that we want an empty string needle to be true,
so check for an empty needle before calling memmem.
block_bind_incremental is block_bind_referenced in a loop backwards. For
an 1-inst block, it does the same thing and isn't too much more
expensive, so it's not really useful to keep both.
Also, block_bind_referenced was a better name for the function.
Since 605bfb3, block_drop_unreferenced actually reverses the order of
instructions in the block it's run against. This bug was hidden by the
fact that normally it's run *twice* against the main program, flipping
it back, and that order of function definitionss doesn't really matter
after symbol resolution.
Stop losing \0s from the end of files because we don't know how much
fgets() read into our buffer. If we prefill the buffer with anything
other than zeroes, we can actually tell the difference between the
terminator and unused space in the buffer.
This is possibly more expensive, but at least it should be correct.
Fixes#1504
Defaults the else clause to the identity filter "." if an else clause is
missing.
The following two jq programs are considered equivalent:
```
if .foo then
.foo = "bar"
else
.
end
```
```
if .foo then
.foo = "bar"
end
```
Avoid doing the internal binding of top-level symbols in the parser,
leaving that work to be done in a post-processing step. For builtins,
this lets us do a reference-aware bind step (block_bind_incremental)
*after* generating builtins/0.
Libraries are a bit trickier since they may be bound multiple times, so
instead of thinking through the implications I added (block_bind_self)
to resolve all internal symbols immediately.
Remove the special code which loads ~/.jq in builtin.c, and instead glue
an optional include which points to the same file onto the main program
in linker.c.
Fixes a minor bug where errors in ~/.jq would be labelled <builtin>.
Only the second and subsequent path components were being checked, which
I guess is theoretically security-relevant.
There's no apparent point to reconstructing the path after splitting it
by adding /s back in, either.
"Module path must be a string" is not a useful error message when the
reason the module path isn't a string is because the string it was got
replaced with an invalid with an error message for some other reason.
Also fixes a few memory leaks on early exits.
The mingw-w64 runtime expands wildcards, which causes a discrepancy
between `main`'s `argv` (multiple file names) and Windows' `wargv` (a
single wildcard). Use `wmain` to retrieve the wide character args.
Windows regards the `NUL` device as tty, which causes `WriteFile` to
fail (the bytes written pointer cannot be `NULL` in this case). Tweak
the color test to ensure tty is accurate.
Windows 10 supports color output, but it must be explicitly enabled;
ANSICON provides support for other versions. Enable color if either of
those are available. Resolves#1494.
The parser generates a call to error/0 on break. Having that exported
from C directly removes a language dependency on builtin.jq.
In any case, error/0 seems to be the more fundamental operation, seeing
as the old C impementation of error/1 did nothing useful with its input.
The distinction between input/0 and _input/0 has been unnecessary since
a9c613..7dc34b, which moved end-of-input handling into C and replaced
the jq definition of input with `def input: _input;`
If these values do not match exactly, it is because the UTF-8 is invalid
anyway and we counted codepoints differently than oniguruma did. Perhaps it
would be better to error out here, but at least one similar loop already
uses < vs != and since we're off the rails anyway this might be OK. It is
certainly better than overruning the buffer.
Resolves https://github.com/stedolan/jq/issues/1804
my_timegm was introduced in 1900c7 to add a mktime fallback used in
4a6241, the subsequently removed in c53823. As a result of this code
shuffling, the jq mktime function (f_mktime) wound up using a fallback
even on systems where timegm is available, with incorrect results in
some time zones with DST, e.g.
% TZ=America/New_York jq -n '"2018-08-31T00:00:00Z"|fromdate|todate'
"2018-08-31T01:00:00Z"
This undoes 1900c7 and moves the workaround it added into the #else of
my_mktime.
Replace uses of the nonstandard $^ variable by explicitly specifying the
source filenames. This is the only obvious thing preventing the build
from running correctly on non-GNU make.
"Undefined symbols are not allowed in x86_64-w64-mingw32 shared libraries"
This was also applied in the official MINGW-packages repo:
Alexpux/MINGW-packages@b5566c58a
* Set default error code to -4 in main(), Fixes#1142
* fix --exit-code with more than one object in input, Fixes#1139
- Return code 1 or 4 based on last output, not last input.
Users are often surprised by the requirement to parenthesize any
non-trivial object key expressions in object constructors. E.g.,
{"a"+"b":1}. This commit adds one more kind of key expression besides
literals and idents: variable references.
A common use case for this is jq programs as JSON templates to fill in
with variables computed from inputs or passed in on the command-line.
E.g., {some_key:$value}. Now users can also use, e.g., {$key:$value}.
This and the restrictions on key and value expressions in object
constructors are now clarified a bit in the documentation.
We had config machinery that determined which math functions are
available in libc. If a c math function was missing on the host system,
then the corresponding jq function would be removed from the source,
enabling the build to proceed anyway. The detection machinery was broken
in a subtle way, as was shown after glibc updated to 2.27, dropping the
`pow10` function. This caused compilation to fail.
The essential problem was that we detected whether a math function was
available by compiling and linking a small program evaluating that
function on constants. However, since gcc's optimization machinery has
special knowledge of some math functions (e.g. `pow10`), it can
optimize them away, even if they don't exist in the library and are not
linkable. That is, the following example compiles and links against
glibc 2.27, even though `pow10` has been removed:
```
int main () {
printf("%f", pow10(0.5));
return 0;
}
```
What?!
On the other hand, this program does not link:
```
int main () {
double f;
printf("%f", &f);
printf("%f", pow10(f));
return 0;
}
```
In the first program the call to `pow10` can be optimized away as a
constant expression. This requires GCC to know about `pow10` (which it
does!), but it does not require `pow10` to be in the library (and
actually linkable).
The solution is to use autoconf's machinery for detecting function
presence, instead of our own (buggy) machinery. This has the added
benefit of simplifying the code.
The bug was reported in issue #1659
JV_PRINT_COLOUR was part of the public libjq headers and was removed as
part of 2d05b54. While JV_PRINT_COLOR is definitely the preferred
spelling this side of the pond, we shouldn't just remove otherwise
exposed enum values.
Attempting to use the existing FORK_OPT opcode resulted in difficulty
knowing when to pop an error message off the stack and when not to. This
commit makes DESTRUCTURE_ALT a real opcode that is identical to
FORK_OPT, except for never pushing the error message onto the stack when
continuing from an error backtrack.
Some small changes were necessary to the DUP/POP behavior surrounding
destructuring to accomodate this.
Not all stdio implementations disallow one to open a directory with
fopen(3) and so we specifically check for directories as it is also
in the standard search path.
This reverts commit e24af3c78e.
While the semantics are desirable, there is no way to implement them
efficiently. The reason is that in order to handle backtracking (empty)
from the state update expression, we have to retain a reference to the
reduction state value in order to restore it upon backtracking.
Retaining a reference to the reduction state kills performance by
causing lots of additional memory allocations and garbage because the
input to the update expression will always have at least two references,
thus no changes to it can be done in-place, and all changes end up being
CoW changes.
Avoiding this is the very reason for the LOADVN instruction (leaving
`null` in the variable loaded from).
This has been a complicated issue to fix for a number of reasons.
The core of it is that the behavior is different between different
versions of macOS, some of which set possible-but-incorrect values.
This commit addresses the issue by always using our computation for
tm_wday and tm_yday on macOS. As a side-effect, strptime format
strings that specify %u and %j will no longer work on macOS.
As noted by @nicowilliams, `tostream` used `keys`,
which sorts the keys in alphabetical order, instead
of `keys_unsorted`, which preserves the parsing order.
Fixes#1541.
Dates in 1900 are before the Unix epoch. We shouldn't make any promises
about how well they are supported, especially given that our time
support is a thin wrapper over the libc functions.
This changes the test to use dates after the epoch, which should fit
within both a signed and an unsigned 32-bit time_t.
strptime() on OS X and *BSDs (reputedly) does not set tm_wday and
tm_yday unless corresponding %U and %j format specifiers were used.
That can be... surprising when one parsed year, month, and day anyways.
Glibc's strptime() conveniently sets tm_wday and tm_yday in those cases,
but OS X's does not, ignoring them completely.
This commit makes jq compute those where possible, though the day of
week computation may be wrong for dates before 1900-03-01 or after
2099-12-31.
OS X (and *BSD) strptime() does not set tm_wday nor tm_yday unless
corresponding format options are used. That means we must call timegm()
to set them.
The conditional expression in if-then-elif-else-end cannot contribute to
path expressions because it doesn't change the input to any of the then/
elif/else expressions. These must be generated via gen_subexp().
See also #1366.
It needs to be possible to do something like
getpath($paths[]) += 1
meaning: increment all the paths in . that are listed in $paths[].
In order to do this getpath() needs to update the jq->path and
jq->value_at_path as necessary.
Rather than testing that a constant expression produces so many
instructions, test that a variety of of such expressions produce the
same number of instructions. This will make future changes in the
compiler less likely to break this test.
This allows one to force the use of jq's builtin copy of Oniguruma
(builtin as a git submodule).
We still need to fix make dist to include that copy of Oniguruma.
Configure should still allow use of prebuilt onigurumas (whether
system-installed or in a special prefix). If these are not found, and
configure was not called with `--without-oniguruma`, then use the vendored
oniguruma module. If configure was called with `--without-oniguruma`, then we
do not build regex functionality into jq.
Now that #1313 is fixed, |= no longer outputs null when the RHS update
expression outputs empty.
When a user wants to keep the current value of the LHS they would have
the RHS update expression output `.`, so having `empty` achieve the same
thing would be redundant. The obvious thing to do is to delete the LHS
when the RHS update outputs `empty` (i.e., doesn't output any values).
It's reasonable to think that existing programs won't be broken by this
change, because reduce and |= not handling empty well is clearly a bug.
(Though it's possible that some programs were using empty to quickly
terminate reduce or |=, it's not likely. They should use label/break
instead.)
Prior to this change |= would use the _last_ value output by the RHS
update expression. With this change |= will use the _first_ value
instead. This change _is_ a minor backwards-incompatible change. It
may or may not be acceptable; we'll see. It is a useful change in that
it makes |= faster when the update expression produces multiple values.
first(g) was extracting two values, which, if g is slow, made first(g)
slow. And if the second extraction were to throw an error, then
first(g) would throw that error, which is clearly not the right thing to
do. Besides that, first(g) was allocating garbage, which it no longer
does.
Use %%FAIL IGNORE MSG for error messages that differ on Windows, and
move the sort_by/group_by tests that fail qhen qsort() is not stable.
Now only tests/sorttest and tests/shtest are disabled on Windows.
Allow a continuation on a comment immediately after a shebang to make
this traditional hack possible:
#!/bin/sh
# this next line is ignored by jq \
exec jq -f "$0" "$@"
# jq code follows
But continue only on the first line following a shebang, and only if
it's a comment.
./jq --run-tests < tests/onig.test
has been run successfully:
...
Testing 'gsub("a";"b")' at line number 70
Testing 'gsub( "(.*)"; ""; "x")' at line number 74
...
22 of 22 tests passed (0 malformed)
strptime() is an XPG4v2 function, on OpenBSD the prototype
is not included in time.h unless visibility is set, it needs
either "#define _XOPEN_SOURCE_EXTENDED 1" or _XOPEN_SOURCE
>= 500. Without this the prototype isn't included and
strptime's returned pointer is truncated to an int, causing
SEGV on 64-bit arches when using it.
Thanks @sthen
This was being done 5 different way in 5 places:
- indent 3 spaces, then tabs to column 73
- indent 2 spaces, then tabs to column 73
- indent 2 spaces, then a single tab
- indent with spaces up to equal sign, then tabs to column 73
- indent 1 tab, then tabs to column 73
I standardized on 8 spaces, then spaces to column 73. The 8
spaces lines up with common browsers' indentation of the
rest of the makefile, which is indented by single tabs. The
spaces to column 73 make sure the trailing backslashes line
up regardless of the tabstop setting.
This gets rid of a linker warning when compiling on OpenBSD:
./.libs/libjq.a(jv_print.o)(.text+0x2fc): In function `jvp_dump_string':
/root/jq-1.5/jv_print.c:118: warning: sprintf() is often misused, please use snprintf()
I updated some of these gems for a different project, and
suddenly `rake build` was giving very different output
compared to the current gh-pages. This commit makes sure
everyone updating gh-pages is using reasonably similar
versions, so we avoid spurious diffs.
This caused problems because liquid >=3.0.0 does not allow a
variable and a template with the same name. Here we had a
variable and a template both called footer.
This was an important conflict. In the following expression:
def a: 0; . | a
Bison needs to decide between these two equally valid
parses:
(def a: 0; .) | a
def a: 0; (. | a)
For jq we want the second one, because the first results in
"a/0 is not defined". In the current parser the first parse
is a reduce and the second parse is a shift. Since Bison
prefers to shift in shift/reduce conflicts, we accidentally
got the correct behavior.
This commit adds a precedence level FUNCDEF which is lower
precedence than '|', meaning we explicitly choose the
correct parse.
Of course many unit tests already cover this case, but I
added one specifically for it.
* Remove excess layer creation
This is an antipattern in Docker.
* Remove build tools once build is complete
Leaving them around is not necessary and expands the image size 5x
* Make check works with libonig installed
* Compiles statically (needs libonig to be built rather than installed)
* Valgrind doesn't work with the use of TLS in jq so it's disabled -- this
might be a FIXME situation; I'm not familiar enough with valgrind to
say whether this is expected
* Make entrypoint be the jq binary so that the image can also be used
to run jq in environments where you don't want to or can't install
jq (such as CoreOS).
Previously, with clang++:
jq.h:46:37: error: typedef redefinition with different
types ('struct jq_util_input_state *' vs 'jq_util_input_state')
With g++:
jq.h:46:37: error: conflicting declaration
‘typedef struct jq_util_input_state* jq_util_input_state’
This typedef was added to libjq by commit 0d41447 which was
after the 1.4 release, so although it is a public API, this
is not a backcompat break because it has never been in a
release.
Specifying the "*" at all uses of jq_util_input_state is
slightly tedious, but jq_state already works that way, so at
least it will be consistent.
The CHECK_MATH_FUNC() m4 macro needs work. It should use
AC_RUN_IFELSE(), not just AC_LINK_IFELSE(), and it should #define
_GNU_SOURCE and/or other such feature macros in the prologue to get
non-standard math functions (alternatvely jq should only support
standard math functions).
When reporting an error to the user, add information about the offending
object/value (possibly truncated).
The goal is to give a user some context regarding which input object
caused the runtime error.
Examples:
$ echo '"hello"' | ./jq '-.'
jq: error: string ("hello") cannot be negated
$ echo '"very-long-string"' | ./jq '-.'
jq: error: string ("very-long-...) cannot be negated
$ echo '["1",2]' | ./jq '.|join(",")'
jq: error: string (",") and number (2) cannot be added
$ echo '["1","2",{"a":{"b":{"c":33}}}]' | ./jq '.|join(",")'
jq: error: string (",") and object ({"a":{"b":{...) cannot be added
$ echo '{"a":{"b":{"c":33}}}' | ./jq '.a | @tsv'
jq: error: object ({"b":{"c":33}}) cannot be tsv-formatted, only array
(Fix#754)
With this patch, jq run-time errors printed to stderr will contain the
filename and line of the offending input.
But note that the JSON text parser does (yet) print the input filename
on parse error. A jq run-time error is a jq program error, not
including syntax errors nor JSON text input format errors.
Examples:
With stdin and multiple lines:
$ printf '{"a":43}\n{"a":{"b":66}}\n' | ./jq '.a+1'
44
jq: error (at stdin:2): object and number cannot be added
With multiple files:
$ printf '{"a":43}' > 1.json
$ printf '{"a":"hello"}\n' > 2.json
$ printf '{"a":{"b":66}}\n' > 3.json
$ ./jq '[.a]|@tsv' 1.json 2.json 3.json
"43"
"hello"
jq: error (at 3.json:1): object is not valid in a csv row
With very long lines (spanning multiple `fgets` calls):
$ ( printf '{"a":43}\n' ;
printf '{"a":{"b":[' ; seq 10000 | paste -d, -s | tr -d '\n' ;
printf ']}}\n' ;
printf '{"a":"hello"}\n' ) | ./jq '[.a] | @tsv'
"43"
jq: error (at stdin:2): object is not valid in a csv row
"hello"
With raw input:
$ seq 1000 | ./jq --raw-input 'select(.=="700") | . + 10'
jq: error (at stdin:700): string and number cannot be added
Caveat:
The reported line will be the last line of the (valid) parsed JSON data.
Example:
$ printf '{\n"a":\n"hello"\n\n\n}\n' | ./jq '.a+4'
jq: error (at stdin:6): string and number cannot be added
We can't know how many bytes fgets() read when we reach EOF and fgets()
didn't see a newline; we can only assume that at least strlen(buf) bytes
were read. This is quite obnoxious if one wants to use NULs in raw
input, but at least we can make reading "a\0b\0c\0" with no newline
yield "a\0b\0c", losing only the final sequence of NULs.
We can't use getline() either, since it will want to allocate a buffer
big enough for an entire line, and we might not have any newlines in our
input. A complete fix will have to use getc() or read(), preferably the
latter.
When trying to use the new regex functions (match/test/sub/capture/etc)
in a JQ that was compiled without the ONIGURAMA regex library,
report an informative error message instead of a 'not defined' error.
Before:
$ echo '"foo"' | ./jq-old 'test("foo")'
jq: error: test/1 is not defined at <top-level>, line 1:
test("foo")
jq: 1 compile error
After:
$ echo '"foo"' | ./jq 'test("foo")'
jq: error: jq was compiled without ONIGURAMA regex libary. match/test/sub and related functions are not available.
When running `make` I ran into a couple of problems building the manual. While
I'm not entirely sure that this is the root cause, it appears to have been
related to the fact that ruby 2.0 dropped syck completely in favor of psych
(which was introduced in 1.9.2) for YAML processing. I'm currently using ruby
2.1.0p0.
I'm assuming that the fact that the YAML engine was explicitly set to syck in
the Rakefile was an attempt to work around some incompatibility between the two
libraries, so I looked into what would be necessary to get it to work with the
newer one. The changes to `manual.yml` ended up being pretty minor: I ran it
through `iconv` to convert some ISO-8859-1 characters to UTF-8 and added some
quotes in places (apparently you can't start a string value with '`').
When escaping string (e.g. for `@tsv` or `@csv` outputs),
escape NULs as '\0'.
Existing behaviour, unchanged by this patch:
$ echo '"a\u0000b"' | ./jq '.'
"a\u0000b"
$ echo '"a\u0000b"' | ./jq -r '.' | od -a
0000000 a nul b nl
0000004
When using `@tsv`, escape NUL to `\0`:
$ echo '"a\u0000b"' | ./jq -r '[.]|@tsv'
a\0b
$ echo '"a\u0000b"' | ./jq '[.]|@tsv'
"a\\0b"
When using '@tsv' output format with '-r' option,
escape \r, \n and \\ in addition to \t.
Example:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' | jq .
{
"id": "hello\ttab\nworld",
"x": 43
}
Before: newlines are not escaped, generating invalid TSV output:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-old -r '[.id,.x] | @tsv'
hello\ttab
world 43
After: newlines are properly escaped:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-new -r '[.id, .x] | @tsv'
hello\ttab\nworld 43
Before: backslashes themselves are not escaped, so there's no way to
distinguish between escaped characters and 'real' backslashes
(in the example below, there is should not be newline, despite the
output containing "\n".
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-old -r '[.x]|@tsv'
hello\ttab\new world
After: backslashes are escaped:
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-new -r '[.x]|@tsv'
hello\ttab\\new world
Detect output errors when the program exits.
Currently:
$ echo '{}' | jq . > /dev/full && echo ok
ok
with the patch:
$ echo '{}' | jq . > /dev/full && echo ok
Error: writing output failed: No space left on device
also apply to hardware/network/other I/O errors.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Improve robustness in automated system when using exit code in shell scripts,
by exiting with code 2 if there was any input error (even overriding other
possible error exit codes).
Exit code 2 is already used to indicate system errors.
Without the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
0
With the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
2
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
With this change, runtime exceptions are propagated to non-zero exit
code of 'jq', allow better scripting and automation. The new exit code
value is 5.
This allows using the shell's and/or operations ('&&' and '||') to
detect input runtime exceptions.
Before:
runtime exceptions are printed to STDERR, but not reported as non-zero exit-code:
$ echo '"hello"' | jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
0
After:
$ echo '"hello"' | ./jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Note that there's a subtle interplay when using "-e" option.
The value of the non-zero exit code changes from 4 to 5, but it still
indicates a 'failure' or non-true value.
Before:
$ echo '"hello"' | jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
4
After:
$ echo '"hello"' | ./jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Also flex is now optional.
The outputs of flex and bison are now committed. By default they get
built, but users who want to build from git can now
./configure --disable-maintainer-mode
to turn off the dependency on bison and flex.
Maintainers must, of course, commit the bison and/or flex outputs when
they make changes to parser.y and/or lexer.l, respectively.
Tests won't pass if built without Oniguruma. We don't have a way to
make a test optional yet. That will come later. For now the ability to
reduce build-time dependencies could really help some users.
Much of this could be in libjq. Eventually all of the work of reading
from files and looping over `jq_next()` should move into libjq, with
`main()` mostly doing all the command-line option processing.
To import a module now use:
# Import module.jq file:
import "relative/path/to/module" as foo;
# Use the module's defs as foo::<def-name>
To import a JSON file:
# Read file.json:
import "relative/path/to/file" as $foo;
#
# Use as $foo::foo
Using `-L` now drops the builtin library path and appends the requested
path to the empty array (or the result of an earlier `-L`).
Support for the `$JQ_LIBRARY_PATH` environment variable has been
removed.
This is to fix the problem where `break` is dynamic, not lexical.
With this it should be possible to do this sort of thing:
label $break | inputs | if ... then $break|error else . end
This is a backwards-incompatible change for master, but the previous
`break` hadn't shipped yet.
Still needed:
- testing
This will allow moving tests of some tests from tests/run to
tests/all.test. It will also be useful for more extensive testing of
language features where we want nice error messages, or errors at all,
to be produced, particularly language features where the errors arise at
compile-, codegen-, or link-time rather than parse-time.
This does not check for run-time errors though -- those we can already
check for with try/catch.
$ jq --run-tests
%%FAIL
break
jq: error: break used outside labeled control structure
^D
1 of 1 tests passed (0 malformed)
$
Print the line number and program text on the same output line where a
failure is reported, for all failures that were already reported on
lines starting with '***'. This makes poring over test logs with
failures much easier.
This will be useful for the upcoming test-erroneous-programs improvement
to --run-tests, so we can switch between the default error reporting
method (print to stderr) to a method internal to --run-tests, and back.
The idea is that when testing programs that are expected to compile (and
link), it'd be nice if errors continue going to stderr, while when
testing programs that must fail to compile (or link), the error has to
be captured so it can be compared to the error expected by the test.
Streaming means that outputs are produced as soon as possible. With the
`foreach` syntax one can write programs which reduce portions of the
streaming parse of a large input (reduce into proper JSON values, for
example), and discard the rest, processing incrementally.
This:
$ jq -c --stream .
should produce the same output as this:
$ jq -c '. as $dot | path(..) as $p | $dot | getpath($p) | [$p,.]'
The output of `jq --stream .` should be a sequence of`[[<path>],<leaf>]`
and `[[<path>]]` values. The latter indicate that the array/object at
that path ended.
Scalars and empty arrays and objects are leaf values for this purpose.
For example, a truncated input produces a path as soon as possible, then
later the error:
$ printf '[0,\n'|./jq -c --stream .
[[0],0]
parse error: Unfinished JSON term at EOF at line 3, column 0
$
If the caller is at EOF and has no more bytes to feed the parser, how is
the parser to be told about the EOF condition? A small fix to allow
zero-length buffers in jv_parser_set_buf() fixes this problem (it also
makes it easier to deal with async I/O: feed the parser whatever is
available, including nothing).
The search path listed in an import directive can now be an array. The
top-level search path is appended. Null and empty strings in the path
terminate any search. The "." in "." and "./*" is replaced with the
directory containing the file doing the import (for command-line
programs this is the current directory, though that may be a bad idea).
No version numbers or anything of the sort are gratuitously added to the
search paths.
All this makes external package managers possible by allowing
dependencies to be installed local to dependents.
It is not very straight forward how to select objects by their attributes from an array. This example provides the basic use case of how to select some objects based key values.
Now we have the ability to define a generator in jq:
def for(cond; update):
def _for:
if cond then ., (update | _for) else . end;
_for;
for(. < 10; . + 1) # generates numbers between `.` and 10
Running this by hand with --debug-dump-disasm (with a fix for that
coming up next) we can see that the call to _for is optimized:
_for:0:
0000 DUP
0001 CALL_JQ cond:0^1
0005 JUMP_F 0022
0007 POP
0008 FORK 0012
0010 JUMP 0020
0012 CALL_JQ update:1^1
0016 TAIL_CALL_JQ _for:0^1
0020 JUMP 0023
0022 POP
0023 RET
And timing this with 1000, 10000, 100000 iterations shows that
indeed we must be applying TCO; otherwise, without TCO, this gets
very slow very quickly.
Close#446.
Currently tested by disassembling and tracing various recursive jq
programs by hand under valgrind. An improved test framework that
can test for errors and specific bytecode patterns is in
development.
instead of lowercase ones.
According to RFC 3986,
The uppercase hexadecimal digits 'A' through 'F' are equivalent to
the lowercase digits 'a' through 'f', respectively. If two URIs
differ only in the case of hexadecimal digits used in percent-encoded
octets, they are equivalent. For consistency, URI producers and
normalizers should use uppercase hexadecimal digits for all percent-
encodings.
See https://github.com/stedolan/jq/issues/451 for details.
Test suite and manual are also updated to reflect this change.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Instead of checking for self-recursion check that the thing we're
calling is a function and not a closure, therefore the new frame will
have the same env as the current frame.
jq now depends on oniguruma for regex support.
Modified configure.ac accordingly.
Added valgrind suppression file for oniguruma to prevent one-time and bounded
leaks from causing tests to fail.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
jv_get() doesn't know if it's being called in the context of a jq
program or not, so it can't produce a very useful error message when the
types of the to-be-indexed value and the key don't agree. For now,
including the key (when it is a short string) in the error message is as
significant an improvement as is easy to make.
Heuristic IEEE754 endianness autoconf detection
Use AC_C_BIGENDIAN, though it's not really the correct approach. Autoconf ought to have provided a test specifically for IEEE754 endianness, since it can differ from integer/pointer endianness!
This reverts commit 77936a594d.
There are too many odd bugs in this mode, and it turns out to be a bad
idea anyways. Instead, in the future a better option will be to pursue
alternative parsers, such as:
- streaming parser that outputs only when a new leaf value is added or
an array/object is opened/closed; options here include whether to
include a path in each output;
- parsers for binary JSON encodings (there's a variety of them).
Then one might run jq with a streaming parser and use `reduce` to
coalesce inputs from some depth down (instead of from one level down as
the reverted commit had intended).
Besides, a fully streaming parser is desirable in some cases, therefore
we should have such a thing as an option.
I've explored modifying the current parser to support a streaming
option, but it only makes the code very difficult to follow, which is
one reason that alternate parsers makes sense. At any rate, this is all
for the future. For now there's no streaming of individual texts, just
text sequences.
And Solaris 8 and 9 too, no doubt. The problem was that non-standard
vsnprintf()s that return -1 when the buffer is too small were not
properly supported.
For the Mingw build we don't want to pull in the pthread DLL just
because we can autodetect pthread support. That would make the jq.exe
binary not self-contained.
Arrays, objects, numbers, strings, booleans, nulls, values (non-nulls)
-- these builtins filter out those inputs that don't match the name of
the builtin.
This fixes#322 and #324.
This commit adds a jv_object_merge_recursive function, that performs
recursive object merging, and binds it to multiply when applied to
two objects.
Added docs and tests.
Closes#320
--unbuffered was only affecting the normal output case, not the --raw-output case. Make the two of them play together.
This also makes sure that the output is flushed *after* printing the newline, so a consumer doesn't lag a line behind.
Make XPath-like `//a/b` recursive structure traversal easier in jq,
which then becomes:
..|.a?.b?
The `?` operator suppresses errors about . not being an array or object.
The `..` operator is equivalent to calling the new `recurse_down`
built-in, which in turn is equivalent to
recurse(.[]?)
Note that `..a` is not supported; neither is `...a`. That could be add
added, but it doesn't seem worth the trouble of saving the need to type
a '|'.
when using a newer automake, the autoreconf step fails with warnings:
"linking libtool libraries using a non-POSIX archiver requires 'AM_PROG_AR' in 'configure.ac' "
This happens for instance on ubuntu 13.10.
Doing just that, adding 'AM_PROG_AR' to configure.ac fixes the problem.
- Re-remove VERSION file and all traces of it
- Pass a decent version indicator to AC_INIT (using git describe)
When we tag 1.4 this will produce 1.4 as the version string when
building on the HEAD of that tag.
jv_dumpf() takes a FILE *.
jv_show() is intended for use in debuggers, so it dumps the jv to stderr
and it does not jv_free() the jv, so it's safe to
"call jv_show(some_jv, -1)" in a debugger. If flags == -1 then the jv
will be shown pretty-printed and in color.
Previously, the @uri example didn't match the actual behavior of the current jq, as exclamation marks do not need to be encoded in URIs.
Replace the example with an input that needs encoding, and is encoded by jq.
From wikipedia:
Travis CI is a hosted, distributed continuous integration service used
to build and test projects hosted at GitHub.
Travis CI is configured by adding a file named .travis.yml, which is a
YAML format text file, to the root directory of the GitHub repository.
Travis CI automatically detects when a commit has been made and pushed
to a GitHub repository that is using Travis CI, and each time this
happens, it will try to build the project and run tests. This includes
commits to all branches, not just to the master branch. When that
process has completed, it will notify a developer in the way it has been
configured to do so — for example, by sending an email containing the
test results (showing success or failure), or by posting a message on an
IRC channel. It can be configured to run the tests on a range of
different machines, with different software installed (such as older
versions of a programming language, to test for compatibility).
jv_string_empty()
-> return an empty string with given allocated length (for fast
appends)
jv_string_append_codepoint
-> append a single codepoint (int) to the given string
jv_string_explode
-> return an array of codepoints making up a string
jv_string_implode
-> return the UTF-8 encoding of an array of codepoint numbers
This lets valgrind find more bugs - if a field isn't given a
well-defined value valgrind will now find it instead of seeing it
set to zero with memset.
This is useful when one has a database (in JSON form) to query using jq
input data.
% echo '{"a":1, "c":5}' > db.json
% echo '"c"'|./jq --argfile f /tmp/a '$f[.]'
5
% echo '"a"'|./jq --argfile f /tmp/a '$f[.]'
1
% echo '"b"'|./jq --argfile f /tmp/a '$f[.]'
null
%
If you have a usage question, please ask us on [Stack Overflow](https://stackoverflow.com/questions/tagged/jq), in the [#jq channel](https://web.libera.chat/#jq) on [Libera.Chat](https://libera.chat/), or in our [Discord server](https://discord.gg/yg6yjNmgAC).
-->
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Provide a minimal test case to reproduce the behavior.
If the input is large, either attach it as a file, or [create a gist](https://gist.github.com) and link to it here.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment (please complete the following information):**
- OS and Version: [e.g. macOS, Windows, Linux (please specify distro)]
- Convert decnum to binary64 (double) instead of decimal64. This makes jq behave like the JSON specification suggests and more similar to other languages. @wader @leonid-s-usov #2949
- Fix memory leaks on invalid input for `ltrimstr/1` and `rtrimstr/1`. @emanuele6 #2977
- Fix memory leak on failed get for `setpath/2`. @emanuele6 #2970
- Fix nan from json parsing also for nans with payload that start with 'n'. @emanuele6 #2985
- Allow carriage return characters in comments. @emanuele6 #2942 #2984
## Documentation changes
- Generate links in the man page. @emanuele6 #2931
- Standardize arch types to AMD64 & ARM64 from index page download dropdown. @owenthereal #2884
## libjq
- Add extern C for C++. @rockwotj #2953
## Build and test changes
- Fix incorrect syntax for checksum file. @kamontat @wader #2899
- Remove `-dirty` version suffix for windows release build. @itchyny #2888
- Make use of `od` in tests more compatible. @nabijaczleweli @emanuele6 @nicowilliams #2922
- Add dependabot. @yeikel #2889
- Extend fuzzing setup to fuzz parser and JSON serializer. @DavidKorczynski @emanuele6 #2952
- Keep releasing executables with legacy names. @itchyny #2951
# 1.7
After a five year hiatus we're back with a GitHub organization, with new admins and new maintainers who have brought a great deal of energy to make a long-awaited and long-needed new release. We're very grateful for all the new owners, admins, and maintainers. Special thanks go to Owen Ou (@owenthereal) for pushing to set up a new GitHub organization for jq, Stephen Dolan (@stedolan) for transferring the jq repository to the new organization, @itchyny for doing a great deal of work to get the release done, Mattias Wadman (@wader) and Emanuele Torre (@emanuele6) for many PRs and code reviews. Many others also contributed PRs, issues, and code reviews as well, and you can find their contributions in the Git log and on the [closed issues and PRs page](https://github.com/jqlang/jq/issues?q=is%3Aclosed+sort%3Aupdated-desc).
Since the last stable release many things have happened:
- jq now lives at <https://github.com/jqlang>
- New maintainers, admins, and owners have been recruited.
- A list of [current maintainers](https://github.com/jqlang/jq/blob/jq-1.7/AUTHORS#L4-L14)
- NEWS file is replaced by NEWS.md with Markdown format. @wader #2599
- CI, scan builds, release, website builds etc now use GitHub actions. @owenthereal @wader @itchyny #2596 #2603 #2620 #2723
- Lots of documentation improvements and fixes.
- Website updated with new section search box, better section ids for linking, dark mode, etc. @itchyny #2628
- Release builds for:
- Linux `amd64`, `arm64`, `armel`, `armhf`, `i386`, `mips`, `mips64`, `mips64el`, `mips64r6`, `mips64r6el`, `mipsel`, `mipsr6`, `mipsr6el`, `powerpc`, `ppc64el`, `riscv64` and `s390x`
- macOS `amd64` and `arm64`
- Windows `i386` and `amd64`
- Docker `linux/386`, `linux/amd64`, `linux/arm64`, `linux/mips64le`, `linux/ppc64le`, `linux/riscv64` and `linux/s390x`
- More details see @owenthereal #2665
- Docker images are now available from `ghcr.io/jqlang/jq` instead of Docker Hub. @itchyny #2652 #2686
- OSS-fuzz. @DavidKorczynski #2760 #2762
Full commit log can be found at <https://github.com/jqlang/jq/compare/jq-1.6...jq-1.7> but here are some highlights:
## CLI changes
- Make object key color configurable using `JQ_COLORS` environment variable. @itchyny @haguenau @ericpruitt #2703
```sh
# this would make "field" bold yellow (`1;33`, the last value)
- Use decimal number literals to preserve precision. Comparison operations respects precision but arithmetic operations might truncate. @leonid-s-usov #1752
```sh
# precision is preserved
$ echo '100000000000000000' | jq .
100000000000000000
# comparison respects precision (this is false in JavaScript)
- Adds new builtin `scan($re; $flags)`. Was documented but not implemented. @itchyny #1961
```sh
# look for pattern "ab" in "abAB" ignoring casing
$ jq -n '"abAB" | scan("ab"; "i")'
"ab"
"AB"
```
- Adds new builtin `abs` to get absolute value. This potentially allows the literal value of numbers to be preserved as `length` and `fabs` convert to float. @pkoppstein #2767
- Allow `if` without `else`-branch. When skipped the `else`-branch will be `.` (identity). @chancez @wader #1825 #2481
```sh
# convert 1 to "one" otherwise keep as is
$ jq -n '1,2 | if . == 1 then "one" end'
"one"
2
# behaves the same as
$ jq -n '1,2 | if . == 1 then "one" else . end'
"one"
2
# also works with elif
$ jq -n '1,2,3 | if . == 1 then "one" elif . == 2 then "two" end
"one"
"two"
3
```
- Allow use of `$binding` as key in object literals. @nicowilliams 8ea4a55
```sh
$ jq -n '"a" as $key | {$key: 123}'
{
"a": 123
}
# previously parentheses were needed
$ jq -n '"a" as $key | {($key): 123}'
{
"a": 123
}
```
- Allow dot between chained indexes when using `.["index"]` @nicowilliams #1168
```sh
$ jq -n '{"a": {"b": 123}} | .a["b"]'
123
# now this also works
$ jq -n '{"a": {"b": 123}} | .a.["b"]'
123
```
- Allow dot for chained value iterator `.[]`, `.[]?` @wader #2650
```sh
$ jq -n '{"a": [123]} | .a[]'
123
# now this also works
$ jq -n '{"a": [123]} | .a.[]'
123
```
- Fix try/catch catches more than it should. @nicowilliams #2750
- Speed up and refactor some builtins, also remove `scalars_or_empty/0`. @muhmuhten #1845
- Now `halt` and `halt_error` exit immediately instead of continuing to the next input. @emanuele6 #2667
- Fix issue converting string to number after previous convert error. @thalman #2400
- Fix issue representing large numbers on some platforms causing invalid JSON output. @itchyny #2661
- Fix deletion using assigning empty against arrays. @itchyny #2133
```sh
# now this works as expected, filter out all values over 2 by assigning empty
`jq` is a lightweight and flexible command-line JSON processor akin to `sed`,`awk`,`grep`, and friends for JSON data. It's written in portable C and has zero runtime dependencies, allowing you to easily slice, filter, map, and transform structured data.
If you want to learn to use jq, read the documentation at
[http://stedolan.github.com/jq](http://stedolan.github.com/jq). This
documentation is generated from the docs/ folder of this repository.
## Documentation
If you want to hack on jq, feel free, but be warned that its internals
are not well-documented at the moment. Bring a hard hat and a
shovel. Also, read the wiki: http://github.com/stedolan/jq/wiki
git submodule update --init # if building from git to get oniguruma
autoreconf -i # if building from git
./configure --with-oniguruma=builtin
make clean # if upgrading from a version previously built from source
make -j8
make check
sudo make install
sudo ldconfig # on Linux, if jq cannot find libjq.so
```
Build a statically linked version:
```console
make clean
make LDFLAGS=-all-static
```
If you're not using the latest git version but instead building a released tarball (available on the release page), skip the `autoreconf` step, and flex or bison won't be needed.
##### Cross-Compilation
For details on cross-compilation, check out the [GitHub Actions file](.github/workflows/ci.yml) and the [cross-compilation wiki page](https://github.com/jqlang/jq/wiki/Cross-compilation).
GitHub has a [mechanism for private disclosure of vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability) to repository owners and authorized persons such as maintainers. The `jqlang/jq` repository now has this feature enabled.
## Reporting a Vulnerability
See [Privately Reporting a Security Vulnerability](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability). Click on [`jqlang/jq`](https://github.com/jqlang/jq)'s [Security page](https://github.com/jqlang/jq/security) and click on [Report a vulnerability](https://github.com/jqlang/jq/security/advisories/new). This will notify the owners and maintainers. After submitting you'll get an option to start a private clone of `jqlang/jq` for collaboration with the maintainers.
{"completed_in":0.108,"max_id":247677287108067328,"max_id_str":"247677287108067328","next_page":"?page=2&max_id=247677287108067328&q=json&rpp=5&include_entities=1","page":1,"query":"json","refresh_url":"?since_id=247677287108067328&q=json&include_entities=1","results":[{"created_at":"Mon, 17 Sep 2012 12:44:01 +0000","entities":{"hashtags":[],"urls":[{"url":"http:\/\/t.co\/XRvh1ZVw","expanded_url":"http:\/\/jase.im\/Ri7I0M","display_url":"jase.im\/Ri7I0M","indices":[112,132]}],"user_mentions":[{"screen_name":"imagemechanics","name":"Jason Cotterell","id":57271393,"id_str":"57271393","indices":[3,18]}]},"from_user":"_AaronNg","from_user_id":79771704,"from_user_id_str":"79771704","from_user_name":"NgChenChong","geo":null,"id":247677287108067328,"id_str":"247677287108067328","iso_language_code":"en","metadata":{"result_type":"recent"},"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2523558403\/ek8mo4j4beq84iw28gjl_normal.jpeg","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2523558403\/ek8mo4j4beq84iw28gjl_normal.jpeg","source":"<a href="http:\/\/twitter.com\/">web<\/a>","text":"RT @imagemechanics: iPhone 5 website teardown: How Apple compresses video using JPEG, JSON, and <canvas>.\nhttp:\/\/t.co\/XRvh1ZVw","to_user":null,"to_user_id":0,"to_user_id_str":"0","to_user_name":null}, ...
- text:|
There's lots of info and no whitespace, so to make it a bit more
legible we pipe it through jq, telling jq to just spit the input
"message": "build(deps): bump markdown from 3.10.2 to 3.10.3 in /docs (#3622)",
"author": "dependabot[bot]"
}
{
"message": "build(deps): bump lxml from 6.1.1 to 6.1.2 in /docs (#3621)",
"author": "dependabot[bot]"
}
{
"message": "Fix a handful of typos (#3605)",
"author": "Anna Rift"
}
{
"message": "Fix strftime-related functions on NetBSD. (#3579)",
"author": "Thomas Klausner"
}
{
"message": "Reject raw NUL bytes in the parser instead of silently truncating tokens (#3606)",
"author": "august"
}
{
"message": "docs: `in` example use [3,4] instead of [0,1] (#3599)",
"author": "A4-Tacks"
}
{
"message": "Fix typo 'you may wish [to] use' (#3604)\n\nFix all instances of the phrase \"you may wish use `fabs`\" -> \"you may\nwish _to_ use `fabs`\" (emphasis mine) in the manual.",
"author": "Anna Rift"
}
{
"message": "build(deps): bump actions/setup-python from 6 to 7 in the official-actions group (#3595)",
"author": "dependabot[bot]"
}
{
"message": "docs: clarify join behavior on objects (#3555)",
"author": "ded-furby"
}
{
"message": "Advance past the whole codepoint after a zero-width regex match (#3586)",
"author": "WinkleMad"
}
{
"message": "shtest: skip locale test if date lacks locale support (#3583)",
"author": "pavel kuliaka"
}
{
"message": "Cast char to unsigned char before passing to isspace (#3574)",
"author": "Thomas Klausner"
}
{
"message": "Add a download link for Windows arm64",
"author": "itchyny"
}
{
"message": "Update jq documentation for jq 1.8.2 release",
"author": "itchyny"
}
{
"message": "Update signatures of 1.8.2 (#3566)\n\nCo-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>",
"author": "github-actions[bot]"
}
{
"message": "build(deps): bump actions/checkout from 6 to 7 in the official-actions group (#3565)",
"author": "dependabot[bot]"
}
{
"message": "Update NEWS.md and AUTHORS for 1.8.2 (#3529)",
"author": "itchyny"
}
{
"message": "Upload attestation bundle as a release artifact (#3563)\n\nThe bundle generated by actions/attest is currently only retrievable\nvia the GitHub attestations API, which requires authentication.\nUploading it as jq-attestation.json lets downstream consumers\n(distro packagers, unauthenticated CI) verify provenance offline.",
"author": "itchyny"
}
{
"message": "Tighten string length bounds and propagate invalid jv in implode\n\nThe bound added in CVE-2026-32316 (e47e56d22) still allowed\n`sizeof(jvp_string) + (currlen + len) * 2 + 1` to wrap `size_t` on\n32-bit platforms. Tighten the threshold so the final allocation\nfits in 32-bit `size_t`.\n\nAlso break out of `jv_string_implode` and `f_string_implode` once\n`jv_string_append_codepoint` returns an invalid `jv`; otherwise the\nnext iteration triggers the assertion in `jvp_string_ptr` (or\ninvokes undefined behavior under `-DNDEBUG`).\n\nFixes CVE-2026-54679.\n\nCo-authored-by: Dirk Müller <dirk@dmllr.de>",
"author": "itchyny"
}
{
"message": "Fix heap-buffer-overflow in raw file loading\n\nWhen `jv_string_append_buf` overflows the string length limit,\nit returns an invalid `jv`; `jv_load_file` then re-entered it\non the invalid value and overran the heap. Break out of the loop\nonce the value is invalid.\n\nFixes CVE-2026-49839.",
"author": "itchyny"
}
{
"message": "build(deps): bump lxml from 6.1.0 to 6.1.1 in /docs (#3553)",
"author": "dependabot[bot]"
}
{
"message": "Fix undefined pointer arithmetic in UTF-8 helpers\n\nFixes GHSA-ggc9-rpv2-xgpm and GHSA-gvwx-xj9r-3frq.",
"author": "theyoucheng"
}
{
"message": "Fix one-byte over-read in BASE64_DECODE_TABLE (#3547)\n\nThe table was declared with 255 entries but indexed by an unsigned\nchar, so the decoder would read one byte past the array for input\nbytes equal to 0xFF. Through jq's normal input paths the over-read\nis unreachable because invalid UTF-8 bytes are normalized to U+FFFD\nbefore reaching @base64d, but the table is still wrong at the C\nlevel and reachable by libjq callers that construct strings via\nthe C API without UTF-8 validation.",
"author": "itchyny"
}
{
"message": "Guard deep structural equality and comparison recursion (#3539)\n\njv_equal and jv_cmp overflows the C stack on deeply nested\ninput. Cap recursion at 10000 with -1 / INT_MIN sentinels;\noperators that compose user expressions surface this as\n\"Equality check too deep\" / \"Comparison too deep\".\n\nFixes CVE-2026-47770.",
"author": "Yu-Fu Fu"
}
{
"message": "Detect circular module imports to prevent stack overflow\n\njq used to recurse without bound on mutual or self-referential\n`import` declarations, exhausting the stack. Track each library's\nload state with a `loading` flag set before its dependencies are\nprocessed; a recursive reference to an in-progress library now\nreports \"circular import of X\".\n\nFixes CVE-2026-44777.",
"author": "itchyny"
}
{
"message": "Reject embedded NUL bytes in module import paths\n\njq accepts embedded NUL bytes at the language level but resolves\nmodule import paths through NUL-terminated C strings, so the path\nvalidated by policy or audit code could differ from the on-disk\npath jq actually opens. Pass jv through gen_import so the AST\npreserves the original bytes, and reject embedded NULs in\nvalidate_relpath.\n\nFixes CVE-2026-43895.",
"author": "itchyny"
}
{
"message": "Reject numeric literals longer than DEC_MAX_DIGITS (999999999)\n\nA signed-int overflow in decNumber's D2U macro lets huge literals\nwrite attacker-controlled bytes past a stack buffer. Cap the length\nbefore calling decNumberFromString, and pre-slice long strings in\njv_dump_string_trunc so the resulting error message doesn't itself\nallocate a multi-GiB buffer.\n\nFixes CVE-2026-43894.",
{"message":"build(deps): bump markdown from 3.10.2 to 3.10.3 in /docs (#3622)","author":"dependabot[bot]"}
{"message":"build(deps): bump lxml from 6.1.1 to 6.1.2 in /docs (#3621)","author":"dependabot[bot]"}
{"message":"Fix a handful of typos (#3605)","author":"Anna Rift"}
{"message":"Fix strftime-related functions on NetBSD. (#3579)","author":"Thomas Klausner"}
{"message":"Reject raw NUL bytes in the parser instead of silently truncating tokens (#3606)","author":"august"}
{"message":"docs: `in` example use [3,4] instead of [0,1] (#3599)","author":"A4-Tacks"}
{"message":"Fix typo 'you may wish [to] use' (#3604)\n\nFix all instances of the phrase \"you may wish use `fabs`\" -> \"you may\nwish _to_ use `fabs`\" (emphasis mine) in the manual.","author":"Anna Rift"}
{"message":"build(deps): bump actions/setup-python from 6 to 7 in the official-actions group (#3595)","author":"dependabot[bot]"}
{"message":"docs: clarify join behavior on objects (#3555)","author":"ded-furby"}
{"message":"Advance past the whole codepoint after a zero-width regex match (#3586)","author":"WinkleMad"}
{"message":"shtest: skip locale test if date lacks locale support (#3583)","author":"pavel kuliaka"}
{"message":"Cast char to unsigned char before passing to isspace (#3574)","author":"Thomas Klausner"}
{"message":"Add a download link for Windows arm64","author":"itchyny"}
{"message":"Update jq documentation for jq 1.8.2 release","author":"itchyny"}
{"message":"Update signatures of 1.8.2 (#3566)\n\nCo-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>","author":"github-actions[bot]"}
{"message":"build(deps): bump actions/checkout from 6 to 7 in the official-actions group (#3565)","author":"dependabot[bot]"}
{"message":"Update NEWS.md and AUTHORS for 1.8.2 (#3529)","author":"itchyny"}
{"message":"Upload attestation bundle as a release artifact (#3563)\n\nThe bundle generated by actions/attest is currently only retrievable\nvia the GitHub attestations API, which requires authentication.\nUploading it as jq-attestation.json lets downstream consumers\n(distro packagers, unauthenticated CI) verify provenance offline.","author":"itchyny"}
{"message":"Tighten string length bounds and propagate invalid jv in implode\n\nThe bound added in CVE-2026-32316 (e47e56d22) still allowed\n`sizeof(jvp_string) + (currlen + len) * 2 + 1` to wrap `size_t` on\n32-bit platforms. Tighten the threshold so the final allocation\nfits in 32-bit `size_t`.\n\nAlso break out of `jv_string_implode` and `f_string_implode` once\n`jv_string_append_codepoint` returns an invalid `jv`; otherwise the\nnext iteration triggers the assertion in `jvp_string_ptr` (or\ninvokes undefined behavior under `-DNDEBUG`).\n\nFixes CVE-2026-54679.\n\nCo-authored-by: Dirk Müller <dirk@dmllr.de>","author":"itchyny"}
{"message":"Fix heap-buffer-overflow in raw file loading\n\nWhen `jv_string_append_buf` overflows the string length limit,\nit returns an invalid `jv`; `jv_load_file` then re-entered it\non the invalid value and overran the heap. Break out of the loop\nonce the value is invalid.\n\nFixes CVE-2026-49839.","author":"itchyny"}
{"message":"build(deps): bump lxml from 6.1.0 to 6.1.1 in /docs (#3553)","author":"dependabot[bot]"}
{"message":"Fix undefined pointer arithmetic in UTF-8 helpers\n\nFixes GHSA-ggc9-rpv2-xgpm and GHSA-gvwx-xj9r-3frq.","author":"theyoucheng"}
{"message":"Fix one-byte over-read in BASE64_DECODE_TABLE (#3547)\n\nThe table was declared with 255 entries but indexed by an unsigned\nchar, so the decoder would read one byte past the array for input\nbytes equal to 0xFF. Through jq's normal input paths the over-read\nis unreachable because invalid UTF-8 bytes are normalized to U+FFFD\nbefore reaching @base64d, but the table is still wrong at the C\nlevel and reachable by libjq callers that construct strings via\nthe C API without UTF-8 validation.","author":"itchyny"}
{"message":"Guard deep structural equality and comparison recursion (#3539)\n\njv_equal and jv_cmp overflows the C stack on deeply nested\ninput. Cap recursion at 10000 with -1 / INT_MIN sentinels;\noperators that compose user expressions surface this as\n\"Equality check too deep\" / \"Comparison too deep\".\n\nFixes CVE-2026-47770.","author":"Yu-Fu Fu"}
{"message":"Detect circular module imports to prevent stack overflow\n\njq used to recurse without bound on mutual or self-referential\n`import` declarations, exhausting the stack. Track each library's\nload state with a `loading` flag set before its dependencies are\nprocessed; a recursive reference to an in-progress library now\nreports \"circular import of X\".\n\nFixes CVE-2026-44777.","author":"itchyny"}
{"message":"Reject embedded NUL bytes in module import paths\n\njq accepts embedded NUL bytes at the language level but resolves\nmodule import paths through NUL-terminated C strings, so the path\nvalidated by policy or audit code could differ from the on-disk\npath jq actually opens. Pass jv through gen_import so the AST\npreserves the original bytes, and reject embedded NULs in\nvalidate_relpath.\n\nFixes CVE-2026-43895.","author":"itchyny"}
{"message":"Reject numeric literals longer than DEC_MAX_DIGITS (999999999)\n\nA signed-int overflow in decNumber's D2U macro lets huge literals\nwrite attacker-controlled bytes past a stack buffer. Cap the length\nbefore calling decNumberFromString, and pre-slice long strings in\njv_dump_string_trunc so the resulting error message doesn't itself\nallocate a multi-GiB buffer.\n\nFixes CVE-2026-43894.","author":"itchyny"}
<li class="nav-item d-md-none"><a class="nav-link{% if not navitem %} active{% endif %}"{% if not navitem %} aria-current="page"{% endif %} href="/">Top page</a></li>
{%- for item in ['tutorial', 'download', 'manual'] %}
Loaded 100 of 495 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Jay Hemnani
