Previously, bindings were parsed as the combination of two tokens:
'$' IDENT
This meant that using a keyword as variable name (e.g. $then, $label)
did not work.
Attempts were made to allow $keyword to work by adding some '$' Keyword
rules in the parser, but this did not allow $keyword in all places:
jq --arg label foo -n '$label' # ok
jq -n '"foo" as $label | .' # error
Treating $foo as a single token is much simpler, in my opinion.
This patch also changes how LOC is lexed: "$__loc__" instead of as
"__loc__" that gets combined with '$' in the parser.
This patch also disallows having spaces after '$' when recalling a
variable `$ foo' since that was probably just an unintentional side
effect of the implementation, and it was not documented.
Fixes#2675
The primary purpose of this commit is to rectify most problems with
`gsub` (and also `sub` with the `g` option), in particular fix#1425 ('\b'),
fix#2354 (lookahead), and fix#2532 (regex == `"^(?!cd ).*$|^cd "`).
This commit also partly resolves#2148 and resolves#1206 in that
`gsub` no longer loops infinitely; however, because the new `gsub`
depends critically on `match/2`, the behavior when regex == `""` is
sometimes non-standard.
The documentation has been updated to reflect the fact that `sub`
and `gsub` are intended to be regular in the second argument.
Also, `_nwise/1` has been tweaked to take advantage of TCO.
* Release pipeline
* Group CI builds for different OSes into `ci.yml`.
* Add release job to release `jq` when tag is in the format of v*.
* Use `clang` as the only compiler on CI.
* Provide extensible matrix for future cross-compile builds, e.g.
for https://github.com/jqlang/jq/pull/2618.
* Locate bison for Windows build
* Also install flex for Windows build
* Add matrix to test all available GH Actions images
* Enable all tests for Windows
* Run `brew update-reset` when `brew update` fails
`brew update` can fail for "Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask failed!"
Ref: https://github.com/owenthereal/jq/actions/runs/5432314910/jobs/9879266028#step:3:19
* Select gcc binaries to release
* Only enable CI when push to master & tag
* Try out statick build
* No need to enforce the same `AM_INIT_AUTOMAKE` version for MacOS
* Disable static build for ubuntu-20.04
See https://github.com/jqlang/jq/pull/2620#discussion_r1248936822
Specified with `--docdir=<path>`, the `$(docdir)` variable is already
available, and has the same default as what it was being set to.
By specifying it here, it overrides what the `--docdir` option
specifies.
* configure.ac: Enable system extensions
Use AC_USE_SYSTEM_EXTENSIONS to build with _GNU_SOURCE (and similar
macros on non-GNU systems), to avoid an implicit declaration of the
strptime function. This improves compatibility with future compilers
which do not support implicit function declarations.
* configure.ac: gettimeofday lives in <sys/time.h>
The gettimeofday function is declared in <sys/time.h>, not <time.h>,
according to POSIX and actual systems. This avoids a configure
probe failure with compilers which do not support implicit function
declarations.
* Try making some GitHub Actions
* GHA: Add OS X, Windows, and Linux w/ scan-build builds
* Add link to discord server
Add link to discord server
* Update Ubuntu to 22.04 for GitHub Actions
* Remove if condition for Ubuntu build
* Fix MacOS OS on GitHub Actions
* Fix python3 package on GitHub Actions
* Disalbe valgrind
Getting failed tests: https://github.com/jqlang/jq/actions/runs/5113866588/jobs/9193542140#step:9:341
* Format file
* Fix macos build
* Fix syntax errors of windows build
https://github.com/jqlang/jq/actions/runs/5114162556/workflow
* Install windows package with choco
* Pin oniguruma to 5a24a49d710a9e3bb8ff11d12e1eae5a9f9df40c
MacOS build [fails](https://github.com/jqlang/jq/actions/runs/5114162555/jobs/9194126677#step:6:3160) due to
```
posix.c:94:3: error: implicit declaration of function 'onig_end' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
onig_end();
```
The current `oniguruma` revision
6fa38f4084b448592888ed9ee43c6e90a46b5f5c (15 Mar 2017) lacks the following explicit declaration in src/onigposix.h:
```
ONIG_EXTERN int onig_end P_((void));
```
This was added to oniguruma in revision 5a24a49d710a9e3bb8ff11d12e1eae5a9f9df40c (8 Sep 2017).
Ref: https://github.com/jqlang/jq/issues/2381
* Revert windows build to use pacman
* Don't zip jq.exe
* Fixing windows build
* Update .gitattriutes for eol on Windows
* Skip workflow_dispatch for windows build
* Clean up builds
* Use LF line endings for all text
Ref: https://github.com/actions/checkout/issues/135
* Fix test that has rounding error
https://github.com/jqlang/jq/pull/2596#issuecomment-1568498993
* Enable CI for all
---------
Co-authored-by: Nicolas Williams <nico@cryptonector.com>
When the string transformation to number failed, all following
transformation failed too.
This happend because status in decNumberFromString function is
updated just in error case. Reusing the DEC_CONTEXT that failed
before results into error even if the string is valid number.
We need to be careful to not grab an extra reference when mutating data
structures because that means we make extra copies. Doing that every
time in `_modify` is really painful, as that function implements `|=`
and all modify-assign operators.
`jv_setpath()` also grabs additional references, so this is not the only
fix needed for the modify-assign operators to not be accidentally
quadratic.
We have to use `LOADVN` in order to make this fix possible, so we should
really byte-code `_modify` rather than jq-code it. However, as a
stop-gap to make this fix easier, I'm adding syntax for referring to a
`$binding` such that you get `LOADVN` instead of `LOADV`.
This syntax is not meant to be used outside jq's internals! Therefore
it is not documented. I promise to break it later, so don't use it!
TBD, but later:
- Optimize `_modify` for the case where `update` outputs more than one
value.
This is trivial: add a `setpath($p; null)` in the middle of `_modify`
before calling `update`. But unfortunately the VM retains a
reference to `value_at_path` for path expression checking, and fixing
that will require more work.
We need to be careful to not grab an extra reference when mutating
data structures because that means we make extra copies. Doing that
every time in `jv_setpath()` is really painful, as that function is used
in `_modify/2`, which is used in `|=` and all modify-assign operators.
`_modify` also grabs additional references, so this is not the only fix
needed for the modify-assign operators to not be accidentally quadratic.
Keep a cached copy of the man tests that we can use when no manpage
changes are made. This allows automated systems that might not have
easy access to a pipenv to build and run tests.
It has no effect after the change c4524da.
Bug reported by the clang static analyzer.
Description: Value stored to 'value' is never read
File: jq/src/util.c
Line: 439
Test Plan:
To verify that this is in fact the behavior:
```
$ jq --indent 7 -n '[1]'
[
1
]
$ jq --indent 8 -n '[1]'
jq: --indent takes a number between -1 and 7
Use jq --help for help with command-line options,
or see the jq manpage, or online docs at https://stedolan.github.io/jq
```
This patch was generated by running `git grep "no more than"` and fixing
up appropriate results.
wchargin-branch: docs-fix-indent-bounds
Allow building jq in a mode that doesn't use decnumber for benchmarking
purposes. decnumber support is enabled by default, and this option is
meant to be removed once we're happy with the performance.
Extend jv_number to use decNumber for storing number literals. Any math
operations on the numbers will truncate them to double precision.
Comparisons when both numbers are literal numbers will compare them
without truncation.
Delay conversion of numbers to doubles until a math operation is performed,
to preserve precision. A literal jv_number will only need conversion to
double once, and will reuse the resultant double on subsequent
conversions.
Outputting literal jv_numbers preserves the original precision.
Add strong pthread requirement to manage contexts/allocations for
converting numbers between their decNumber, string, and double formats.
The current instructions tell users to perform two actions:
- update the package database
- install the jq package
The only thing users need to or should be doing is actually installing
the jq package -- regardless of which version is being installed.
Guidelines on how to perform system updates are massively out of scope.
In the case of partially performing a system update as a prerequisite
for installing jq, the official guidance from Arch Linux is not to do
this: partial updates are not supported, we refuse to support them, and
anyone who does try to perform them anyway is assumed to know so much
about their system that they clearly do not ever need help from anyone
else (which is a good thing since they won't get it). The result is a
frankensteined system that can only ever be supported by the person who
frankensteined it to begin with. The only reason the package manager
even allows it to occur in the first place is because other
distributions using pacman might have different LTS policies, and
because it would prevent expert users from being in control of their
system, as per the traditional Unix philosophy:
"Unix was not designed to stop you from doing stupid things, because
that would also stop you from doing clever things."
Consequences of performing partial updates without understanding the
ramifications in extensive detail can include breaking the partially
updated application (jq), breaking any application that shares a mutual
dependency with the partially updated application (which jq is *lucky*
to only depend on the ever-backwards-compatible glibc), or breaking the
entire operating system by leaving armed traps behind for the next time
a `pacman -S new-package` is executed and thereby breaks *its* cascading
dependencies.
See:
https://wiki.archlinux.org/index.php/System_maintenance#Partial_upgrades_are_unsupported
The current paragraph is not complete, since a digit is not a special character.
Changing it to:
If the key contains special characters or starts with a digit,
you need to surround it with double quotes like this:
`."foo$"`, or else `.["foo$"]`.
Run `make uninstall` for oniguruma before running the recursive
distclean that will remove oniguruma's Makefile and cause a build error
due to missing make target.
Fetch oniguruma using git submodule instead of a release tarball. It
will fix a build problem, caused by jq's autotools configuration, trying
to run `make distclean` recursively in an empty modules directory. This
will also improve the maintainability of the Dockerfile.
Expressions of the form `path(EXPR) | select(GENERATOR)`, where `EXPR`
is a path expression and `GENERATOR` is a generator conditional
expression (e.g., `has("a"), has("b")`) cause an assertion if the
jq_state VM is torn down too soon. That assert() was only correct if
assuming that the conditional is not a generator.
If the conditional is generator, then what we see is that when
backtracking a SUBEXP_END is executed without a corresponding
SUBEXP_BEGIN because the entire conditional is bracketed with
SUBEXP_BEGIN and SUBEXP_END, and since it's resumed in the middle, in
between the brackets.
Rather than assert that the jq->path_len being restored has some
particular value, we can simply re-compute it from the restored
jq->path.
The behavior of memmem for an empty needle is inconsistent between
implementations of libc.
Our tests imply that we want an empty string needle to be true,
so check for an empty needle before calling memmem.
block_bind_incremental is block_bind_referenced in a loop backwards. For
an 1-inst block, it does the same thing and isn't too much more
expensive, so it's not really useful to keep both.
Also, block_bind_referenced was a better name for the function.
Since 605bfb3, block_drop_unreferenced actually reverses the order of
instructions in the block it's run against. This bug was hidden by the
fact that normally it's run *twice* against the main program, flipping
it back, and that order of function definitionss doesn't really matter
after symbol resolution.
Stop losing \0s from the end of files because we don't know how much
fgets() read into our buffer. If we prefill the buffer with anything
other than zeroes, we can actually tell the difference between the
terminator and unused space in the buffer.
This is possibly more expensive, but at least it should be correct.
Fixes#1504
Defaults the else clause to the identity filter "." if an else clause is
missing.
The following two jq programs are considered equivalent:
```
if .foo then
.foo = "bar"
else
.
end
```
```
if .foo then
.foo = "bar"
end
```
Avoid doing the internal binding of top-level symbols in the parser,
leaving that work to be done in a post-processing step. For builtins,
this lets us do a reference-aware bind step (block_bind_incremental)
*after* generating builtins/0.
Libraries are a bit trickier since they may be bound multiple times, so
instead of thinking through the implications I added (block_bind_self)
to resolve all internal symbols immediately.
Remove the special code which loads ~/.jq in builtin.c, and instead glue
an optional include which points to the same file onto the main program
in linker.c.
Fixes a minor bug where errors in ~/.jq would be labelled <builtin>.
Only the second and subsequent path components were being checked, which
I guess is theoretically security-relevant.
There's no apparent point to reconstructing the path after splitting it
by adding /s back in, either.
"Module path must be a string" is not a useful error message when the
reason the module path isn't a string is because the string it was got
replaced with an invalid with an error message for some other reason.
Also fixes a few memory leaks on early exits.
The mingw-w64 runtime expands wildcards, which causes a discrepancy
between `main`'s `argv` (multiple file names) and Windows' `wargv` (a
single wildcard). Use `wmain` to retrieve the wide character args.
Windows regards the `NUL` device as tty, which causes `WriteFile` to
fail (the bytes written pointer cannot be `NULL` in this case). Tweak
the color test to ensure tty is accurate.
Windows 10 supports color output, but it must be explicitly enabled;
ANSICON provides support for other versions. Enable color if either of
those are available. Resolves#1494.
The parser generates a call to error/0 on break. Having that exported
from C directly removes a language dependency on builtin.jq.
In any case, error/0 seems to be the more fundamental operation, seeing
as the old C impementation of error/1 did nothing useful with its input.
The distinction between input/0 and _input/0 has been unnecessary since
a9c613..7dc34b, which moved end-of-input handling into C and replaced
the jq definition of input with `def input: _input;`
If these values do not match exactly, it is because the UTF-8 is invalid
anyway and we counted codepoints differently than oniguruma did. Perhaps it
would be better to error out here, but at least one similar loop already
uses < vs != and since we're off the rails anyway this might be OK. It is
certainly better than overruning the buffer.
Resolves https://github.com/stedolan/jq/issues/1804
my_timegm was introduced in 1900c7 to add a mktime fallback used in
4a6241, the subsequently removed in c53823. As a result of this code
shuffling, the jq mktime function (f_mktime) wound up using a fallback
even on systems where timegm is available, with incorrect results in
some time zones with DST, e.g.
% TZ=America/New_York jq -n '"2018-08-31T00:00:00Z"|fromdate|todate'
"2018-08-31T01:00:00Z"
This undoes 1900c7 and moves the workaround it added into the #else of
my_mktime.
Replace uses of the nonstandard $^ variable by explicitly specifying the
source filenames. This is the only obvious thing preventing the build
from running correctly on non-GNU make.
"Undefined symbols are not allowed in x86_64-w64-mingw32 shared libraries"
This was also applied in the official MINGW-packages repo:
Alexpux/MINGW-packages@b5566c58a
* Set default error code to -4 in main(), Fixes#1142
* fix --exit-code with more than one object in input, Fixes#1139
- Return code 1 or 4 based on last output, not last input.
Users are often surprised by the requirement to parenthesize any
non-trivial object key expressions in object constructors. E.g.,
{"a"+"b":1}. This commit adds one more kind of key expression besides
literals and idents: variable references.
A common use case for this is jq programs as JSON templates to fill in
with variables computed from inputs or passed in on the command-line.
E.g., {some_key:$value}. Now users can also use, e.g., {$key:$value}.
This and the restrictions on key and value expressions in object
constructors are now clarified a bit in the documentation.
We had config machinery that determined which math functions are
available in libc. If a c math function was missing on the host system,
then the corresponding jq function would be removed from the source,
enabling the build to proceed anyway. The detection machinery was broken
in a subtle way, as was shown after glibc updated to 2.27, dropping the
`pow10` function. This caused compilation to fail.
The essential problem was that we detected whether a math function was
available by compiling and linking a small program evaluating that
function on constants. However, since gcc's optimization machinery has
special knowledge of some math functions (e.g. `pow10`), it can
optimize them away, even if they don't exist in the library and are not
linkable. That is, the following example compiles and links against
glibc 2.27, even though `pow10` has been removed:
```
int main () {
printf("%f", pow10(0.5));
return 0;
}
```
What?!
On the other hand, this program does not link:
```
int main () {
double f;
printf("%f", &f);
printf("%f", pow10(f));
return 0;
}
```
In the first program the call to `pow10` can be optimized away as a
constant expression. This requires GCC to know about `pow10` (which it
does!), but it does not require `pow10` to be in the library (and
actually linkable).
The solution is to use autoconf's machinery for detecting function
presence, instead of our own (buggy) machinery. This has the added
benefit of simplifying the code.
The bug was reported in issue #1659
JV_PRINT_COLOUR was part of the public libjq headers and was removed as
part of 2d05b54. While JV_PRINT_COLOR is definitely the preferred
spelling this side of the pond, we shouldn't just remove otherwise
exposed enum values.
Attempting to use the existing FORK_OPT opcode resulted in difficulty
knowing when to pop an error message off the stack and when not to. This
commit makes DESTRUCTURE_ALT a real opcode that is identical to
FORK_OPT, except for never pushing the error message onto the stack when
continuing from an error backtrack.
Some small changes were necessary to the DUP/POP behavior surrounding
destructuring to accomodate this.
Not all stdio implementations disallow one to open a directory with
fopen(3) and so we specifically check for directories as it is also
in the standard search path.
This reverts commit e24af3c78e.
While the semantics are desirable, there is no way to implement them
efficiently. The reason is that in order to handle backtracking (empty)
from the state update expression, we have to retain a reference to the
reduction state value in order to restore it upon backtracking.
Retaining a reference to the reduction state kills performance by
causing lots of additional memory allocations and garbage because the
input to the update expression will always have at least two references,
thus no changes to it can be done in-place, and all changes end up being
CoW changes.
Avoiding this is the very reason for the LOADVN instruction (leaving
`null` in the variable loaded from).
This has been a complicated issue to fix for a number of reasons.
The core of it is that the behavior is different between different
versions of macOS, some of which set possible-but-incorrect values.
This commit addresses the issue by always using our computation for
tm_wday and tm_yday on macOS. As a side-effect, strptime format
strings that specify %u and %j will no longer work on macOS.
As noted by @nicowilliams, `tostream` used `keys`,
which sorts the keys in alphabetical order, instead
of `keys_unsorted`, which preserves the parsing order.
Fixes#1541.
Dates in 1900 are before the Unix epoch. We shouldn't make any promises
about how well they are supported, especially given that our time
support is a thin wrapper over the libc functions.
This changes the test to use dates after the epoch, which should fit
within both a signed and an unsigned 32-bit time_t.
strptime() on OS X and *BSDs (reputedly) does not set tm_wday and
tm_yday unless corresponding %U and %j format specifiers were used.
That can be... surprising when one parsed year, month, and day anyways.
Glibc's strptime() conveniently sets tm_wday and tm_yday in those cases,
but OS X's does not, ignoring them completely.
This commit makes jq compute those where possible, though the day of
week computation may be wrong for dates before 1900-03-01 or after
2099-12-31.
OS X (and *BSD) strptime() does not set tm_wday nor tm_yday unless
corresponding format options are used. That means we must call timegm()
to set them.
The conditional expression in if-then-elif-else-end cannot contribute to
path expressions because it doesn't change the input to any of the then/
elif/else expressions. These must be generated via gen_subexp().
See also #1366.
It needs to be possible to do something like
getpath($paths[]) += 1
meaning: increment all the paths in . that are listed in $paths[].
In order to do this getpath() needs to update the jq->path and
jq->value_at_path as necessary.
Rather than testing that a constant expression produces so many
instructions, test that a variety of of such expressions produce the
same number of instructions. This will make future changes in the
compiler less likely to break this test.
This allows one to force the use of jq's builtin copy of Oniguruma
(builtin as a git submodule).
We still need to fix make dist to include that copy of Oniguruma.
Configure should still allow use of prebuilt onigurumas (whether
system-installed or in a special prefix). If these are not found, and
configure was not called with `--without-oniguruma`, then use the vendored
oniguruma module. If configure was called with `--without-oniguruma`, then we
do not build regex functionality into jq.
Now that #1313 is fixed, |= no longer outputs null when the RHS update
expression outputs empty.
When a user wants to keep the current value of the LHS they would have
the RHS update expression output `.`, so having `empty` achieve the same
thing would be redundant. The obvious thing to do is to delete the LHS
when the RHS update outputs `empty` (i.e., doesn't output any values).
It's reasonable to think that existing programs won't be broken by this
change, because reduce and |= not handling empty well is clearly a bug.
(Though it's possible that some programs were using empty to quickly
terminate reduce or |=, it's not likely. They should use label/break
instead.)
Prior to this change |= would use the _last_ value output by the RHS
update expression. With this change |= will use the _first_ value
instead. This change _is_ a minor backwards-incompatible change. It
may or may not be acceptable; we'll see. It is a useful change in that
it makes |= faster when the update expression produces multiple values.
first(g) was extracting two values, which, if g is slow, made first(g)
slow. And if the second extraction were to throw an error, then
first(g) would throw that error, which is clearly not the right thing to
do. Besides that, first(g) was allocating garbage, which it no longer
does.
Use %%FAIL IGNORE MSG for error messages that differ on Windows, and
move the sort_by/group_by tests that fail qhen qsort() is not stable.
Now only tests/sorttest and tests/shtest are disabled on Windows.
Allow a continuation on a comment immediately after a shebang to make
this traditional hack possible:
#!/bin/sh
# this next line is ignored by jq \
exec jq -f "$0" "$@"
# jq code follows
But continue only on the first line following a shebang, and only if
it's a comment.
./jq --run-tests < tests/onig.test
has been run successfully:
...
Testing 'gsub("a";"b")' at line number 70
Testing 'gsub( "(.*)"; ""; "x")' at line number 74
...
22 of 22 tests passed (0 malformed)
strptime() is an XPG4v2 function, on OpenBSD the prototype
is not included in time.h unless visibility is set, it needs
either "#define _XOPEN_SOURCE_EXTENDED 1" or _XOPEN_SOURCE
>= 500. Without this the prototype isn't included and
strptime's returned pointer is truncated to an int, causing
SEGV on 64-bit arches when using it.
Thanks @sthen
This was being done 5 different way in 5 places:
- indent 3 spaces, then tabs to column 73
- indent 2 spaces, then tabs to column 73
- indent 2 spaces, then a single tab
- indent with spaces up to equal sign, then tabs to column 73
- indent 1 tab, then tabs to column 73
I standardized on 8 spaces, then spaces to column 73. The 8
spaces lines up with common browsers' indentation of the
rest of the makefile, which is indented by single tabs. The
spaces to column 73 make sure the trailing backslashes line
up regardless of the tabstop setting.
This gets rid of a linker warning when compiling on OpenBSD:
./.libs/libjq.a(jv_print.o)(.text+0x2fc): In function `jvp_dump_string':
/root/jq-1.5/jv_print.c:118: warning: sprintf() is often misused, please use snprintf()
I updated some of these gems for a different project, and
suddenly `rake build` was giving very different output
compared to the current gh-pages. This commit makes sure
everyone updating gh-pages is using reasonably similar
versions, so we avoid spurious diffs.
This caused problems because liquid >=3.0.0 does not allow a
variable and a template with the same name. Here we had a
variable and a template both called footer.
This was an important conflict. In the following expression:
def a: 0; . | a
Bison needs to decide between these two equally valid
parses:
(def a: 0; .) | a
def a: 0; (. | a)
For jq we want the second one, because the first results in
"a/0 is not defined". In the current parser the first parse
is a reduce and the second parse is a shift. Since Bison
prefers to shift in shift/reduce conflicts, we accidentally
got the correct behavior.
This commit adds a precedence level FUNCDEF which is lower
precedence than '|', meaning we explicitly choose the
correct parse.
Of course many unit tests already cover this case, but I
added one specifically for it.
* Remove excess layer creation
This is an antipattern in Docker.
* Remove build tools once build is complete
Leaving them around is not necessary and expands the image size 5x
* Make check works with libonig installed
* Compiles statically (needs libonig to be built rather than installed)
* Valgrind doesn't work with the use of TLS in jq so it's disabled -- this
might be a FIXME situation; I'm not familiar enough with valgrind to
say whether this is expected
* Make entrypoint be the jq binary so that the image can also be used
to run jq in environments where you don't want to or can't install
jq (such as CoreOS).
Previously, with clang++:
jq.h:46:37: error: typedef redefinition with different
types ('struct jq_util_input_state *' vs 'jq_util_input_state')
With g++:
jq.h:46:37: error: conflicting declaration
‘typedef struct jq_util_input_state* jq_util_input_state’
This typedef was added to libjq by commit 0d41447 which was
after the 1.4 release, so although it is a public API, this
is not a backcompat break because it has never been in a
release.
Specifying the "*" at all uses of jq_util_input_state is
slightly tedious, but jq_state already works that way, so at
least it will be consistent.
The CHECK_MATH_FUNC() m4 macro needs work. It should use
AC_RUN_IFELSE(), not just AC_LINK_IFELSE(), and it should #define
_GNU_SOURCE and/or other such feature macros in the prologue to get
non-standard math functions (alternatvely jq should only support
standard math functions).
When reporting an error to the user, add information about the offending
object/value (possibly truncated).
The goal is to give a user some context regarding which input object
caused the runtime error.
Examples:
$ echo '"hello"' | ./jq '-.'
jq: error: string ("hello") cannot be negated
$ echo '"very-long-string"' | ./jq '-.'
jq: error: string ("very-long-...) cannot be negated
$ echo '["1",2]' | ./jq '.|join(",")'
jq: error: string (",") and number (2) cannot be added
$ echo '["1","2",{"a":{"b":{"c":33}}}]' | ./jq '.|join(",")'
jq: error: string (",") and object ({"a":{"b":{...) cannot be added
$ echo '{"a":{"b":{"c":33}}}' | ./jq '.a | @tsv'
jq: error: object ({"b":{"c":33}}) cannot be tsv-formatted, only array
(Fix#754)
With this patch, jq run-time errors printed to stderr will contain the
filename and line of the offending input.
But note that the JSON text parser does (yet) print the input filename
on parse error. A jq run-time error is a jq program error, not
including syntax errors nor JSON text input format errors.
Examples:
With stdin and multiple lines:
$ printf '{"a":43}\n{"a":{"b":66}}\n' | ./jq '.a+1'
44
jq: error (at stdin:2): object and number cannot be added
With multiple files:
$ printf '{"a":43}' > 1.json
$ printf '{"a":"hello"}\n' > 2.json
$ printf '{"a":{"b":66}}\n' > 3.json
$ ./jq '[.a]|@tsv' 1.json 2.json 3.json
"43"
"hello"
jq: error (at 3.json:1): object is not valid in a csv row
With very long lines (spanning multiple `fgets` calls):
$ ( printf '{"a":43}\n' ;
printf '{"a":{"b":[' ; seq 10000 | paste -d, -s | tr -d '\n' ;
printf ']}}\n' ;
printf '{"a":"hello"}\n' ) | ./jq '[.a] | @tsv'
"43"
jq: error (at stdin:2): object is not valid in a csv row
"hello"
With raw input:
$ seq 1000 | ./jq --raw-input 'select(.=="700") | . + 10'
jq: error (at stdin:700): string and number cannot be added
Caveat:
The reported line will be the last line of the (valid) parsed JSON data.
Example:
$ printf '{\n"a":\n"hello"\n\n\n}\n' | ./jq '.a+4'
jq: error (at stdin:6): string and number cannot be added
We can't know how many bytes fgets() read when we reach EOF and fgets()
didn't see a newline; we can only assume that at least strlen(buf) bytes
were read. This is quite obnoxious if one wants to use NULs in raw
input, but at least we can make reading "a\0b\0c\0" with no newline
yield "a\0b\0c", losing only the final sequence of NULs.
We can't use getline() either, since it will want to allocate a buffer
big enough for an entire line, and we might not have any newlines in our
input. A complete fix will have to use getc() or read(), preferably the
latter.
When trying to use the new regex functions (match/test/sub/capture/etc)
in a JQ that was compiled without the ONIGURAMA regex library,
report an informative error message instead of a 'not defined' error.
Before:
$ echo '"foo"' | ./jq-old 'test("foo")'
jq: error: test/1 is not defined at <top-level>, line 1:
test("foo")
jq: 1 compile error
After:
$ echo '"foo"' | ./jq 'test("foo")'
jq: error: jq was compiled without ONIGURAMA regex libary. match/test/sub and related functions are not available.
When running `make` I ran into a couple of problems building the manual. While
I'm not entirely sure that this is the root cause, it appears to have been
related to the fact that ruby 2.0 dropped syck completely in favor of psych
(which was introduced in 1.9.2) for YAML processing. I'm currently using ruby
2.1.0p0.
I'm assuming that the fact that the YAML engine was explicitly set to syck in
the Rakefile was an attempt to work around some incompatibility between the two
libraries, so I looked into what would be necessary to get it to work with the
newer one. The changes to `manual.yml` ended up being pretty minor: I ran it
through `iconv` to convert some ISO-8859-1 characters to UTF-8 and added some
quotes in places (apparently you can't start a string value with '`').
When escaping string (e.g. for `@tsv` or `@csv` outputs),
escape NULs as '\0'.
Existing behaviour, unchanged by this patch:
$ echo '"a\u0000b"' | ./jq '.'
"a\u0000b"
$ echo '"a\u0000b"' | ./jq -r '.' | od -a
0000000 a nul b nl
0000004
When using `@tsv`, escape NUL to `\0`:
$ echo '"a\u0000b"' | ./jq -r '[.]|@tsv'
a\0b
$ echo '"a\u0000b"' | ./jq '[.]|@tsv'
"a\\0b"
When using '@tsv' output format with '-r' option,
escape \r, \n and \\ in addition to \t.
Example:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' | jq .
{
"id": "hello\ttab\nworld",
"x": 43
}
Before: newlines are not escaped, generating invalid TSV output:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-old -r '[.id,.x] | @tsv'
hello\ttab
world 43
After: newlines are properly escaped:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-new -r '[.id, .x] | @tsv'
hello\ttab\nworld 43
Before: backslashes themselves are not escaped, so there's no way to
distinguish between escaped characters and 'real' backslashes
(in the example below, there is should not be newline, despite the
output containing "\n".
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-old -r '[.x]|@tsv'
hello\ttab\new world
After: backslashes are escaped:
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-new -r '[.x]|@tsv'
hello\ttab\\new world
Detect output errors when the program exits.
Currently:
$ echo '{}' | jq . > /dev/full && echo ok
ok
with the patch:
$ echo '{}' | jq . > /dev/full && echo ok
Error: writing output failed: No space left on device
also apply to hardware/network/other I/O errors.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Improve robustness in automated system when using exit code in shell scripts,
by exiting with code 2 if there was any input error (even overriding other
possible error exit codes).
Exit code 2 is already used to indicate system errors.
Without the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
0
With the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
2
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
With this change, runtime exceptions are propagated to non-zero exit
code of 'jq', allow better scripting and automation. The new exit code
value is 5.
This allows using the shell's and/or operations ('&&' and '||') to
detect input runtime exceptions.
Before:
runtime exceptions are printed to STDERR, but not reported as non-zero exit-code:
$ echo '"hello"' | jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
0
After:
$ echo '"hello"' | ./jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Note that there's a subtle interplay when using "-e" option.
The value of the non-zero exit code changes from 4 to 5, but it still
indicates a 'failure' or non-true value.
Before:
$ echo '"hello"' | jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
4
After:
$ echo '"hello"' | ./jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Also flex is now optional.
The outputs of flex and bison are now committed. By default they get
built, but users who want to build from git can now
./configure --disable-maintainer-mode
to turn off the dependency on bison and flex.
Maintainers must, of course, commit the bison and/or flex outputs when
they make changes to parser.y and/or lexer.l, respectively.
Tests won't pass if built without Oniguruma. We don't have a way to
make a test optional yet. That will come later. For now the ability to
reduce build-time dependencies could really help some users.
Much of this could be in libjq. Eventually all of the work of reading
from files and looping over `jq_next()` should move into libjq, with
`main()` mostly doing all the command-line option processing.
To import a module now use:
# Import module.jq file:
import "relative/path/to/module" as foo;
# Use the module's defs as foo::<def-name>
To import a JSON file:
# Read file.json:
import "relative/path/to/file" as $foo;
#
# Use as $foo::foo
Using `-L` now drops the builtin library path and appends the requested
path to the empty array (or the result of an earlier `-L`).
Support for the `$JQ_LIBRARY_PATH` environment variable has been
removed.
This is to fix the problem where `break` is dynamic, not lexical.
With this it should be possible to do this sort of thing:
label $break | inputs | if ... then $break|error else . end
This is a backwards-incompatible change for master, but the previous
`break` hadn't shipped yet.
Still needed:
- testing
This will allow moving tests of some tests from tests/run to
tests/all.test. It will also be useful for more extensive testing of
language features where we want nice error messages, or errors at all,
to be produced, particularly language features where the errors arise at
compile-, codegen-, or link-time rather than parse-time.
This does not check for run-time errors though -- those we can already
check for with try/catch.
$ jq --run-tests
%%FAIL
break
jq: error: break used outside labeled control structure
^D
1 of 1 tests passed (0 malformed)
$
Print the line number and program text on the same output line where a
failure is reported, for all failures that were already reported on
lines starting with '***'. This makes poring over test logs with
failures much easier.
This will be useful for the upcoming test-erroneous-programs improvement
to --run-tests, so we can switch between the default error reporting
method (print to stderr) to a method internal to --run-tests, and back.
The idea is that when testing programs that are expected to compile (and
link), it'd be nice if errors continue going to stderr, while when
testing programs that must fail to compile (or link), the error has to
be captured so it can be compared to the error expected by the test.
Streaming means that outputs are produced as soon as possible. With the
`foreach` syntax one can write programs which reduce portions of the
streaming parse of a large input (reduce into proper JSON values, for
example), and discard the rest, processing incrementally.
This:
$ jq -c --stream .
should produce the same output as this:
$ jq -c '. as $dot | path(..) as $p | $dot | getpath($p) | [$p,.]'
The output of `jq --stream .` should be a sequence of`[[<path>],<leaf>]`
and `[[<path>]]` values. The latter indicate that the array/object at
that path ended.
Scalars and empty arrays and objects are leaf values for this purpose.
For example, a truncated input produces a path as soon as possible, then
later the error:
$ printf '[0,\n'|./jq -c --stream .
[[0],0]
parse error: Unfinished JSON term at EOF at line 3, column 0
$
If the caller is at EOF and has no more bytes to feed the parser, how is
the parser to be told about the EOF condition? A small fix to allow
zero-length buffers in jv_parser_set_buf() fixes this problem (it also
makes it easier to deal with async I/O: feed the parser whatever is
available, including nothing).
The search path listed in an import directive can now be an array. The
top-level search path is appended. Null and empty strings in the path
terminate any search. The "." in "." and "./*" is replaced with the
directory containing the file doing the import (for command-line
programs this is the current directory, though that may be a bad idea).
No version numbers or anything of the sort are gratuitously added to the
search paths.
All this makes external package managers possible by allowing
dependencies to be installed local to dependents.
It is not very straight forward how to select objects by their attributes from an array. This example provides the basic use case of how to select some objects based key values.
Now we have the ability to define a generator in jq:
def for(cond; update):
def _for:
if cond then ., (update | _for) else . end;
_for;
for(. < 10; . + 1) # generates numbers between `.` and 10
Running this by hand with --debug-dump-disasm (with a fix for that
coming up next) we can see that the call to _for is optimized:
_for:0:
0000 DUP
0001 CALL_JQ cond:0^1
0005 JUMP_F 0022
0007 POP
0008 FORK 0012
0010 JUMP 0020
0012 CALL_JQ update:1^1
0016 TAIL_CALL_JQ _for:0^1
0020 JUMP 0023
0022 POP
0023 RET
And timing this with 1000, 10000, 100000 iterations shows that
indeed we must be applying TCO; otherwise, without TCO, this gets
very slow very quickly.
Close#446.
Currently tested by disassembling and tracing various recursive jq
programs by hand under valgrind. An improved test framework that
can test for errors and specific bytecode patterns is in
development.
instead of lowercase ones.
According to RFC 3986,
The uppercase hexadecimal digits 'A' through 'F' are equivalent to
the lowercase digits 'a' through 'f', respectively. If two URIs
differ only in the case of hexadecimal digits used in percent-encoded
octets, they are equivalent. For consistency, URI producers and
normalizers should use uppercase hexadecimal digits for all percent-
encodings.
See https://github.com/stedolan/jq/issues/451 for details.
Test suite and manual are also updated to reflect this change.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Instead of checking for self-recursion check that the thing we're
calling is a function and not a closure, therefore the new frame will
have the same env as the current frame.
jq now depends on oniguruma for regex support.
Modified configure.ac accordingly.
Added valgrind suppression file for oniguruma to prevent one-time and bounded
leaks from causing tests to fail.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
jv_get() doesn't know if it's being called in the context of a jq
program or not, so it can't produce a very useful error message when the
types of the to-be-indexed value and the key don't agree. For now,
including the key (when it is a short string) in the error message is as
significant an improvement as is easy to make.
Heuristic IEEE754 endianness autoconf detection
Use AC_C_BIGENDIAN, though it's not really the correct approach. Autoconf ought to have provided a test specifically for IEEE754 endianness, since it can differ from integer/pointer endianness!
This reverts commit 77936a594d.
There are too many odd bugs in this mode, and it turns out to be a bad
idea anyways. Instead, in the future a better option will be to pursue
alternative parsers, such as:
- streaming parser that outputs only when a new leaf value is added or
an array/object is opened/closed; options here include whether to
include a path in each output;
- parsers for binary JSON encodings (there's a variety of them).
Then one might run jq with a streaming parser and use `reduce` to
coalesce inputs from some depth down (instead of from one level down as
the reverted commit had intended).
Besides, a fully streaming parser is desirable in some cases, therefore
we should have such a thing as an option.
I've explored modifying the current parser to support a streaming
option, but it only makes the code very difficult to follow, which is
one reason that alternate parsers makes sense. At any rate, this is all
for the future. For now there's no streaming of individual texts, just
text sequences.
And Solaris 8 and 9 too, no doubt. The problem was that non-standard
vsnprintf()s that return -1 when the buffer is too small were not
properly supported.
For the Mingw build we don't want to pull in the pthread DLL just
because we can autodetect pthread support. That would make the jq.exe
binary not self-contained.
Arrays, objects, numbers, strings, booleans, nulls, values (non-nulls)
-- these builtins filter out those inputs that don't match the name of
the builtin.
This fixes#322 and #324.
This commit adds a jv_object_merge_recursive function, that performs
recursive object merging, and binds it to multiply when applied to
two objects.
Added docs and tests.
Closes#320
--unbuffered was only affecting the normal output case, not the --raw-output case. Make the two of them play together.
This also makes sure that the output is flushed *after* printing the newline, so a consumer doesn't lag a line behind.
Make XPath-like `//a/b` recursive structure traversal easier in jq,
which then becomes:
..|.a?.b?
The `?` operator suppresses errors about . not being an array or object.
The `..` operator is equivalent to calling the new `recurse_down`
built-in, which in turn is equivalent to
recurse(.[]?)
Note that `..a` is not supported; neither is `...a`. That could be add
added, but it doesn't seem worth the trouble of saving the need to type
a '|'.
when using a newer automake, the autoreconf step fails with warnings:
"linking libtool libraries using a non-POSIX archiver requires 'AM_PROG_AR' in 'configure.ac' "
This happens for instance on ubuntu 13.10.
Doing just that, adding 'AM_PROG_AR' to configure.ac fixes the problem.
- Re-remove VERSION file and all traces of it
- Pass a decent version indicator to AC_INIT (using git describe)
When we tag 1.4 this will produce 1.4 as the version string when
building on the HEAD of that tag.
jv_dumpf() takes a FILE *.
jv_show() is intended for use in debuggers, so it dumps the jv to stderr
and it does not jv_free() the jv, so it's safe to
"call jv_show(some_jv, -1)" in a debugger. If flags == -1 then the jv
will be shown pretty-printed and in color.
Previously, the @uri example didn't match the actual behavior of the current jq, as exclamation marks do not need to be encoded in URIs.
Replace the example with an input that needs encoding, and is encoded by jq.
From wikipedia:
Travis CI is a hosted, distributed continuous integration service used
to build and test projects hosted at GitHub.
Travis CI is configured by adding a file named .travis.yml, which is a
YAML format text file, to the root directory of the GitHub repository.
Travis CI automatically detects when a commit has been made and pushed
to a GitHub repository that is using Travis CI, and each time this
happens, it will try to build the project and run tests. This includes
commits to all branches, not just to the master branch. When that
process has completed, it will notify a developer in the way it has been
configured to do so — for example, by sending an email containing the
test results (showing success or failure), or by posting a message on an
IRC channel. It can be configured to run the tests on a range of
different machines, with different software installed (such as older
versions of a programming language, to test for compatibility).
jv_string_empty()
-> return an empty string with given allocated length (for fast
appends)
jv_string_append_codepoint
-> append a single codepoint (int) to the given string
jv_string_explode
-> return an array of codepoints making up a string
jv_string_implode
-> return the UTF-8 encoding of an array of codepoint numbers
This lets valgrind find more bugs - if a field isn't given a
well-defined value valgrind will now find it instead of seeing it
set to zero with memset.
This is useful when one has a database (in JSON form) to query using jq
input data.
% echo '{"a":1, "c":5}' > db.json
% echo '"c"'|./jq --argfile f /tmp/a '$f[.]'
5
% echo '"a"'|./jq --argfile f /tmp/a '$f[.]'
1
% echo '"b"'|./jq --argfile f /tmp/a '$f[.]'
null
%
Pulling some I/O out to a function meant that buf changed from
"char buf[4096]" to "char* buf", and "sizeof(buf)" got a lot
less interesting. The upshot of this is that jq read input eight
bytes at a time, which is not the fastest.
After something is popped from a stack, we overwrite the memory
with uninitialised data (if JQ_DEBUG is on). This means that
valgrind reports use-after-pop as an uninitialised memory error.
- Build binaries for multiple platforms
- Make a manpage out of the manual (see #19)
- Extract more tests from the documentation
- Fix a few documentation bugs uncovered by above.
This prevents the circuluar dependency between parser.gen.h and
lexer.gen.h. Newer versions of bison add a prototype for yyparse() to
parser.gen.h that include the as-yet-undeclared yyscan_t type.
IDENT syntax now includes ASCII letters and underscores, so
'.foo_bar' now works. Non-ASCII letters won't work in IDENT
tokens (it's impossible to tell which non-ascii characters are
"letters" without full unicode tables), so '.données' is still
a syntax error (the workaround is '.["données"]', since you can
put anything you like in a string).
This might be the most philosophical commit. I shall ponder it.
true, false, null, empty now defined as bytecoded builtins rather
than calls out to C functions.
Requires a lexer hack, but a surprisingly un-hideous one. The lexer
maintains a stack of bracket characters so that it can match
parens/brackets/braces to tell where a subexp nested in a string
actually ends.
- Change memory allocation to thread a context structure through
the code rather than using globals (which would require a
-lpthread and some locking to become threadsafe).
- Remove a few configuration options (the "private memory" space,
and K&R style headers).
- Prefix exported symbols with jvp_ to prevent libc conflicts.
- Add jvp_dtoa.h
- Include netlib's g_fmt.c into dtoa.c
- Add a few not-strictly-necessary initialisations in order to
compile cleanly with gcc -Wall
If you have a usage question, please ask us on either Stack Overflow (https://stackoverflow.com/questions/tagged/jq) or in the #jq channel (https://web.libera.chat/#jq) on Libera.Chat (https://libera.chat/).
-->
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Provide a minimal test case to reproduce the behavior.
If the input is large, either attach it as a file, or [create a gist](https://gist.github.com) and link to it here.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment (please complete the following information):**
- OS and Version: [e.g. macOS, Windows, Linux (please specify distro)]
GitHub has a [mechanism for private disclosure of vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability) to repository owners and authorized persons such as maintainers. The `jqlang/jq` repository now has this feature enabled.
## Reporting a Vulnerability
See [Privately Reporting a Security Vulnerability](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability). Click on [`jqlang/jq`](https://github.com/jqlang/jq)'s [Security page](https://github.com/jqlang/jq/security) and click on [Report a vulnerability](https://github.com/jqlang/jq/security/advisories/new). This will notify the owners and maintainers. After submitting you'll get an option to start a private clone of `jqlang/jq` for collaboration with the maintainers.
"message": "Mention -n in IO-section and for input/inputs",
"name": "Nico Williams"
}
{
"message": "Fix iterration problem for non decimal string\n\nWhen the string transformation to number failed, all following\ntransformation failed too.\n\nThis happend because status in decNumberFromString function is\nupdated just in error case. Reusing the DEC_CONTEXT that failed\nbefore results into error even if the string is valid number.",
"name": "Nico Williams"
}
{
"message": "docs: point to Libera.Chat instead of Freenode",
"name": "Nico Williams"
}
{
"message": "Missing \"va_end\" call. This was found by running the cppcheck static analysis where it shows as error.",
"name": "Nico Williams"
}
- text:|
`.[]` returns each element of the array returned in the response, one at a
"message": "Mention -n in IO-section and for input/inputs",
"name": "Nico Williams"
},
{
"message": "Fix iterration problem for non decimal string\n\nWhen the string transformation to number failed, all following\ntransformation failed too.\n\nThis happend because status in decNumberFromString function is\nupdated just in error case. Reusing the DEC_CONTEXT that failed\nbefore results into error even if the string is valid number.",
"name": "Nico Williams"
},
{
"message": "docs: point to Libera.Chat instead of Freenode",
"name": "Nico Williams"
},
{
"message": "Missing \"va_end\" call. This was found by running the cppcheck static analysis where it shows as error.",
"name": "Nico Williams"
}
]
- text:|
- - -
Next, let's try getting the URLs of the parent commits out of the
API results as well. In each commit, the GitHub API includes information
"message": "Fix iterration problem for non decimal string\n\nWhen the string transformation to number failed, all following\ntransformation failed too.\n\nThis happend because status in decNumberFromString function is\nupdated just in error case. Reusing the DEC_CONTEXT that failed\nbefore results into error even if the string is valid number.",
Loaded 100 of 222 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
liquidaty
