diff --git a/README.md b/README.md index 4287d941..2df0357f 100644 --- a/README.md +++ b/README.md @@ -196,7 +196,7 @@ Let both `"foo 'is' bar"` and `' foo "was" bar'` become `» foo ... bar «` inde | `-b` | **`--keep-backup`** Keep a backup file of the original content. [boolean] | | `-m` | **`--output-match`** Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean] | | `-T` | **`--trim-pipe`** Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] | -| `-P` | **`--void-pipe`** Ignore any pipe data - even if stdin is open. Can not be combined with -R or -T. [boolean] | +| `-P` | **`--void-pipe`** Ignore any pipe data - even if stdin is open. Can not be combined with -R or -T. Can also be set via the environment variable RR_VOID_PIPE. [boolean] | | `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | | `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command (null if no piped data), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | | `-x` | **`--exclude-re`** Exclude files with a path that matches this regular expression. Will follow same regex flags and setup as the main search. Can be used multiple times. [string] | @@ -224,10 +224,12 @@ Let both `"foo 'is' bar"` and `' foo "was" bar'` become `» foo ... bar «` inde - The `€` and `§` aliases are only for what you write as pattern, replacement (also with `--replacement-js`) and `--exclude-re`. File globs, `--exclude-glob` and piped data (both content to search and a replacement given with `--replacement-pipe`) are used exactly as they are, so a file named `price€.txt` is found without any flags. -- RexReplace reads from stdin whenever stdin is not a terminal, and waits for it to close. Some environments (CI runners, cron, `docker exec`, editors spawning processes) leave stdin open without ever writing to it, which makes the command hang. Add the `-P` flag (`--void-pipe`) to ignore piped data (it can not be combined with `-R` or `-T`): +- RexReplace reads from stdin whenever stdin is not a terminal, and waits for it to close. Some environments (CI runners, cron, `docker exec`, editors spawning processes) leave stdin open without ever writing to it, which makes the command hang. Add the `--void-pipe` flag (`-P`) to ignore piped data. Can also be set with the environment variable `RR_VOID_PIPE`: ```bash > rexreplace 'foo' 'bar' myfile.md -P + +> RR_VOID_PIPE=1 rexreplace 'foo' 'bar' myfile.md ``` ### Limitations @@ -364,7 +366,6 @@ _.oO(What should "sed" have looked like by now?)_ - Let Pattern, replacement, and globs come from a file - Let pattern and glob be javascript code returning a string as the result - Auto string search / replace if no regex magic is used (and verify that speed is better) -- Error != warning - Flag for String-literal (no regex, no special chars, no escape chars) to avoid backslashes or remembering which characters needs to be escaped - Check if https://github.com/eugeneware/replacestream is good to rely on - Check if regex engine from spider monkey can be wrapped in something that does not need node @@ -373,6 +374,7 @@ _.oO(What should "sed" have looked like by now?)_ - Set engine to run on streams with https://www.npmjs.com/package/replacestream - Implement in go so that all platforms can be supported with no need for node (might be based on) - Let https://github.com/dthree/vorpal deal with the interface? Or maybe https://www.npmjs.com/package/pretty-cli +- Load env. vars from prefix setting before loading data. Check conflict also. ## Related projects diff --git a/npm/rexreplace-bun/package.json b/npm/rexreplace-bun/package.json index 9d78a47d..f1ce6988 100644 --- a/npm/rexreplace-bun/package.json +++ b/npm/rexreplace-bun/package.json @@ -1,6 +1,6 @@ { "name": "rexreplace-bun", - "version": "7.1.14+bun", + "version": "8.0.0-dev+bun", "description": "RexReplace running on Bun: Search & replace across files with a CLI tool that makes you trust what you are doing.", "author": "Mathias Rangel Wulff", "license": "MIT", @@ -33,6 +33,6 @@ "bun" ], "dependencies": { - "rexreplace": "7.1.14" + "rexreplace": "8.0.0-dev" } } diff --git a/package.json b/package.json index 2f772431..38d8261d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "rexreplace", - "version": "7.1.14", + "version": "8.0.0-dev", "description": "Search & replace across files with a CLI tool that makes you trust what you are doing.", "author": "Mathias Rangel Wulff", "license": "MIT", diff --git a/src/cli.ts b/src/cli.ts index b22934cb..46c7661a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,7 +1,7 @@ // CLI interface for rexreplace import * as rexreplace from './engine.ts'; -import {outputConfig, die} from './output.ts'; +import {outputConfig, die, warn} from './output.ts'; // Make it visible when Bun is the runtime (like with rexreplace-bun) as the regex engine is then not the one from Node. // Added as semver build metadata (`+bun`) as a `-bun` would make it a prerelease. @@ -193,7 +193,8 @@ const yargs = require('yargs') .alias('P', 'void-pipe') .describe( 'P', - `Ignore any pipe data - even if stdin is open. Can not be combined with -R or -T.` + `Ignore any pipe data - even if stdin is open. Can not be combined with -R or -T. ` + + `Can also be set via the environment variable RR_VOID_PIPE.` ) .conflicts('P', 'R') .conflicts('P', 'T') @@ -393,6 +394,8 @@ function unescapeString(str = '') { } }); + outputConfig(config); + let pipeInUse = false; let pipeData = ''; @@ -417,7 +420,6 @@ function unescapeString(str = '') { config.voidSection ? replacement : replacement.replace(/§/g, '\\') ); } catch (e) { - outputConfig(config); return die('Invalid escape sequence in replacement:', replacement); } } @@ -425,10 +427,15 @@ function unescapeString(str = '') { /*if(Boolean(process.stdout.isTTY)){ config.output = true; }*/ + + if (process.env.RR_VOID_PIPE) { + const conflict = ' conflicts with the env. var RR_VOID_PIPE as no data will be piped.'; + config.replacementPipe && warn('The flag --replacement-pipe' + conflict); + config.trimPipe && warn('The flag --trim-pipe' + conflict); + config.voidPipe = true; + } + if (config.voidPipe || Boolean(process.stdin.isTTY)) { - if (config.replacementPipe) { - return backOut(); - } rexreplace.engine(config); } else { process.stdin.setEncoding(config.encoding); diff --git a/src/engine.ts b/src/engine.ts index 73b04528..7726d03f 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -6,7 +6,7 @@ const globs = require('fast-glob'); const now = new Date(); -import {outputConfig, step, debug, chat, info, error, die} from './output.ts'; +import {outputConfig, step, debug, chat, info, warn, error, die} from './output.ts'; const re = { euro: /€/g, @@ -128,7 +128,7 @@ function doReplacement(_file_rr: string, _config_rr: any, _data_rr: string) { } //Make sure data is always on disk - const oriFile = path.normalize(path.join(process.cwd(), _file_rr)); + const oriFile = path.resolve(_file_rr); const salt = new Date().toISOString().replace(re.colon, '_').replace('Z', ''); const backupFile = oriFile + '.' + salt + '.backup'; @@ -156,8 +156,14 @@ function handlePipedData(config) { step('Check Piped Data'); if (config.includeGlob.length) { - if (!config.replacementJs && config.pipedData) { - chat('Piped data never used.'); + // Empty piped data (or only a line break) is not worth a warning + if ( + !config.replacementJs && + !config.pipedDataUsed && + config.pipedData && + '\n' !== config.pipedData + ) { + warn('Data is piped in, but no flag tells what to do with it. It is not used.'); } return false; @@ -439,7 +445,7 @@ function dynamicReplacement(_file_rr, _config_rr, _data_rr) { const betterToReadfromFile = needsByteOrSize && 50000000 < _text.length; // around 50 Mb will lead to reading filezise from file instead of copying into buffer if (!_config_rr.dataIsPiped) { - _file = path.normalize(path.join(_cwd, _file_rr)); + _file = path.resolve(_cwd, _file_rr); _file_rel = path.relative(_cwd, _file); const pathInfo = path.parse(_file); _dirpath = pathInfo.dir; diff --git a/src/output.ts b/src/output.ts index e3f8d6ab..b1847c73 100644 --- a/src/output.ts +++ b/src/output.ts @@ -16,6 +16,11 @@ export const info = function (msg, data = '') { console.error(font.gray(msg), data); }; +export const warn = function (msg) { + // The line break gives an empty line under the warning + info(' 🟡 ' + msg, '\n'); +}; + export const chat = function (msg, data = '') { if (config.verbose) { info(msg, data); @@ -28,7 +33,8 @@ export const die = function (msg = '', data = '', displayHelp = false) { if (displayHelp && !config.quietTotal) { config.showHelp(); } - msg && error(' ❌ ' + msg, data); + // The line break gives an empty line under the error + msg && error(' ❌ ' + msg, data + '\n'); kill(); }; diff --git a/test/cli/global.run.node.sh b/test/cli/global.run.node.sh index 3d6ef704..bc37a5ae 100644 --- a/test/cli/global.run.node.sh +++ b/test/cli/global.run.node.sh @@ -12,19 +12,19 @@ npm uninstall --location=global rexreplace rexreplace-bun npm install --location=global . -assert_matches "rr --version" "[0-9]$" +assert_matches "rr --version" "(dev|[0-9])$" echo 'foobar' > my.file rr o x my.file assert "cat my.file" "fxxbar" -assert_matches "rexreplace --version" "[0-9]$" +assert_matches "rexreplace --version" "(dev|[0-9])$" echo 'foobar' > my.file rexreplace o x my.file assert "cat my.file" "fxxbar" -assert_matches "rexreplace-bun --version" "[0-9]\+bun$" +assert_matches "rexreplace-bun --version" "(dev|[0-9])\+bun$" echo 'foobar' > my.file rexreplace-bun o x my.file assert "cat my.file" "fxxbar" diff --git a/test/cli/run.sh b/test/cli/run.sh index bca55aaa..6abf25af 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -294,6 +294,34 @@ assert_failure "printf x | rexreplace 'b' _ my.file -P -R" assert_failure "printf x | rexreplace 'b' _ my.file -P -T" +# Warning when piped data is not used +reset +assert "printf x | rexreplace 'o' 'x' my.file 2>&1 | grep 'Data is piped in'" ' 🟡 Data is piped in, but no flag tells what to do with it. It is not used. ' +assert "cat my.file" 'fxxbar' +assert_failure "printf x | rexreplace 'o' 'x' my.file -q 2>&1 | grep 'Data is piped in'" +assert_failure "printf x | rexreplace 'o' 'x' my.file -P 2>&1 | grep 'Data is piped in'" +assert_failure "printf x | rexreplace 'b' _ my.file -R 2>&1 | grep 'Data is piped in'" +assert_failure "printf x | rexreplace 'b' 'pipe' my.file -j 2>&1 | grep 'Data is piped in'" + + +# RR_VOID_PIPE +reset +assert "printf x | RR_VOID_PIPE=1 rexreplace 'foo' 'pipe' my.file -o --replacement-js 2>&1" 'nullbar' +assert "printf x | RR_VOID_PIPE= rexreplace 'foo' 'pipe' my.file -o --replacement-js" 'xbar' +assert "printf x | RR_VOID_PIPE=1 rexreplace 'foo' 'pipe' my.file -o --replacement-js -T" 'nullbar' +assert "printf x | RR_VOID_PIPE=1 rexreplace 'foo' 'pipe' my.file -o --replacement-js -T 2>&1 >/dev/null" ' 🟡 The flag --trim-pipe conflicts with the env. var RR_VOID_PIPE as no data will be piped. ' +assert "printf x | RR_VOID_PIPE=1 rexreplace 'foo' 'pipe' my.file -o --replacement-js -T -q 2>&1" 'nullbar' +assert "printf x | RR_VOID_PIPE=1 rexreplace 'foo' 'pipe' my.file -o --replacement-js -T -Q 2>&1" 'nullbar' +assert_failure "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R" +assert "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R 2>&1 | grep -E 'RR_VOID_PIPE as|No data piped'" ' 🟡 The flag --replacement-pipe conflicts with the env. var RR_VOID_PIPE as no data will be piped. \n ❌ No data piped into replacement ' +assert "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R -T 2>&1 | grep -E 'RR_VOID_PIPE as|No data piped'" ' 🟡 The flag --replacement-pipe conflicts with the env. var RR_VOID_PIPE as no data will be piped. \n 🟡 The flag --trim-pipe conflicts with the env. var RR_VOID_PIPE as no data will be piped. \n ❌ No data piped into replacement ' +assert_failure "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R -q" +assert "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R -q 2>&1 | grep -v 'exited with code'" ' ❌ No data piped into replacement ' +assert_failure "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R -Q" +assert "printf x | RR_VOID_PIPE=1 rexreplace 'b' _ my.file -R -Q 2>&1 | grep -v 'exited with code'" '' +assert "cat my.file" 'foobar' + + # -L diff --git a/test/js/fix/691.test.js b/test/js/fix/691.test.js index 943bd56e..c6f66bbc 100644 --- a/test/js/fix/691.test.js +++ b/test/js/fix/691.test.js @@ -53,7 +53,7 @@ describe('#691 replacement that used to crash the CLI', () => { test('an invalid escape sequence gives an error message and no stack trace', () => { const result = replaceO('\\x4'); expect(result.out).toBe(''); - expect(result.err).toBe(' ❌ Invalid escape sequence in replacement: \\x4\n'); + expect(result.err).toBe(' ❌ Invalid escape sequence in replacement: \\x4\n\n'); expect(result.exit).toBe(1); expect(replaceO('\\x4', '--quiet-total')).toEqual({out: '', err: '', exit: 1}); }); diff --git a/test/js/fix/700.test.js b/test/js/fix/700.test.js index 8ee54f4b..57568dc7 100644 --- a/test/js/fix/700.test.js +++ b/test/js/fix/700.test.js @@ -47,7 +47,7 @@ describe('#700 § is a full substitute for \\ in the replacement', () => { test('an invalid escape sequence gives the same error as with a backslash', () => { expect(replaceB('§x4')).toEqual({ out: '', - err: ' ❌ Invalid escape sequence in replacement: §x4\n', + err: ' ❌ Invalid escape sequence in replacement: §x4\n\n', exit: 1, }); }); diff --git a/test/js/fix/719.test.js b/test/js/fix/719.test.js new file mode 100644 index 00000000..ea46053c --- /dev/null +++ b/test/js/fix/719.test.js @@ -0,0 +1,50 @@ +// https://github.com/mathiasrw/rexreplace/issues/719 +// Data piped in together with files was not used, and nothing told about it. +import {describe, test, expect} from 'bun:test'; +import {rr} from '../helper.js'; + +const warning = ' 🟡 Data is piped in, but no flag tells what to do with it. It is not used. \n\n'; + +// Run on my.file (with the content `foobar`) with `x` piped in +async function piped(flags = {}, replacement = '_') { + await Bun.write('my.file', 'foobar'); + return rr('b', replacement, {includeGlob: ['my.file'], pipedData: 'x', ...flags}); +} + +describe('#719 a warning when piped data is not used', () => { + test('is given when data is piped in and files are also given', async () => { + expect(await piped()).toEqual({out: '', err: warning + 'my.file \n', exit: null}); + expect(await Bun.file('my.file').text()).toBe('foo_ar'); + }); + + test.each(['quiet', 'quietTotal'])('is not shown with --%s', async (flag) => { + expect((await piped({[flag]: true})).err).toBe(''); + }); + + test('does not stop the run with --halt', async () => { + expect(await piped({halt: true})).toEqual({ + out: '', + err: warning + 'my.file \n', + exit: null, + }); + expect(await Bun.file('my.file').text()).toBe('foo_ar'); + }); + + test('is not given when the piped data is used', async () => { + expect((await piped({replacementPipe: true})).err).toBe('my.file \n'); + expect((await piped({replacementJs: true}, 'pipe')).err).toBe('my.file \n'); + expect(await Bun.file('my.file').text()).toBe('fooxar'); + }); + + test.each([null, '', '\n'])('is not given when %j is piped in', async (pipedData) => { + expect((await piped({pipedData})).err).toBe('my.file \n'); + }); + + test('is not given when only data is piped in', async () => { + expect(await rr('b', '_', {pipedData: 'foobar'})).toEqual({ + out: 'foo_ar', + err: '', + exit: null, + }); + }); +}); diff --git a/test/js/fix/726.test.js b/test/js/fix/726.test.js new file mode 100644 index 00000000..3edf7f83 --- /dev/null +++ b/test/js/fix/726.test.js @@ -0,0 +1,100 @@ +// https://github.com/mathiasrw/rexreplace/issues/726 +// The working directory was put in front of the path of the file, also when the path was absolute, so the file could not be renamed to its backup and was not changed. +import {describe, test, expect} from 'bun:test'; +import fs from 'fs'; +import path from 'path'; +import {rr} from '../helper.js'; + +// Note that rr() from the helper sets voidAsync per default, so the async flow is `voidAsync: false` +const flows = [ + ['the async flow', {voidAsync: false}], + ['--void-async', {voidAsync: true}], +]; + +// Make `name` with the content `foobar` in a directory that is not the working directory and give the absolute path of it +async function outside(name) { + fs.mkdirSync('work'); + fs.mkdirSync('other'); + const file = path.resolve('other', name); + await Bun.write(file, 'foobar'); + process.chdir('work'); + return file; +} + +describe('#726 a file given with an absolute path is changed', () => { + test.each(flows)('as a path with %s', async (name, flags) => { + const file = await outside('my.file'); + const result = await rr('o', 'x', {includeGlob: [file], ...flags}); + expect(result).toEqual({out: '', err: file + ' \n', exit: null}); + expect(await Bun.file(file).text()).toBe('fxxbar'); + expect(fs.readdirSync(path.dirname(file))).toEqual(['my.file']); + }); + + test.each(flows)('as a glob with %s', async (name, flags) => { + const file = await outside('my.file'); + const glob = path.join(path.dirname(file), '*.file'); + const result = await rr('o', 'x', {includeGlob: [glob], ...flags}); + expect(result).toEqual({out: '', err: file + ' \n', exit: null}); + expect(await Bun.file(file).text()).toBe('fxxbar'); + expect(fs.readdirSync(path.dirname(file))).toEqual(['my.file']); + }); + + test('in the working directory', async () => { + await Bun.write('my.file', 'foobar'); + const file = path.resolve('my.file'); + const result = await rr('o', 'x', {includeGlob: [file]}); + expect(result).toEqual({out: '', err: file + ' \n', exit: null}); + expect(await Bun.file(file).text()).toBe('fxxbar'); + expect(fs.readdirSync('.')).toEqual(['my.file']); + }); + + test.each(flows)( + '--keep-backup puts the backup next to the file with %s', + async (name, flags) => { + const file = await outside('my.file'); + await rr('o', 'x', {includeGlob: [file], keepBackup: true, ...flags}); + const [backup, ...rest] = fs + .readdirSync(path.dirname(file)) + .filter((name) => name !== 'my.file'); + expect(rest).toEqual([]); + expect(backup).toMatch(/^my\.file\..+\.backup$/); + expect(await Bun.file(path.join(path.dirname(file), backup)).text()).toBe('foobar'); + expect(await Bun.file(file).text()).toBe('fxxbar'); + expect(fs.readdirSync('.')).toEqual([]); + } + ); + + test('--replacement-js gets the path of the file as it is', async () => { + const file = await outside('my.file'); + const result = await rr( + 'foobar', + '[file, dirpath, file_rel, dirpath_rel, typeof mtime].join("|")', + { + includeGlob: [file], + replacementJs: true, + output: true, + } + ); + expect(result.out).toBe( + [ + file, + path.dirname(file), + path.join('..', 'other', 'my.file'), + path.join('..', 'other'), + 'string', + ].join('|') + ); + }); + + test('a relative path still works from the working directory', async () => { + fs.mkdirSync('sub'); + await Bun.write('sub/my.file', 'foobar'); + const result = await rr('o', 'x', { + includeGlob: ['./sub/../sub/my.file'], + keepBackup: true, + }); + expect(result.exit).toBe(null); + expect(await Bun.file('sub/my.file').text()).toBe('fxxbar'); + expect(fs.readdirSync('sub')).toHaveLength(2); + }); +}); diff --git a/test/js/verbose.test.js b/test/js/verbose.test.js index 8a5ad54d..0c842dd8 100644 --- a/test/js/verbose.test.js +++ b/test/js/verbose.test.js @@ -29,13 +29,11 @@ describe('--verbose', () => { ); }); - test('tells when piped data is not used', async () => { + test('is not needed to be told that piped data is not used (a warning since #719)', async () => { fs.writeFileSync('my.file', 'foobar'); const config = {includeGlob: ['my.file'], pipedData: 'boo'}; - expect((await rr('o', 'x', {...config, verbose: true})).err).toContain( - 'Piped data never used.' - ); - expect((await rr('x', 'o', config)).err).not.toContain('Piped data'); + expect((await rr('o', 'x', {...config, verbose: true})).err).toContain('Data is piped in'); + expect((await rr('x', 'o', config)).err).toContain('Data is piped in'); }); test('never writes to stdout', async () => { @@ -84,7 +82,7 @@ describe('--verbose', () => { verbose: true, replacementJs: true, }; - expect((await rr('foo', 'pipe', config)).err).not.toContain('Piped data never used'); + expect((await rr('foo', 'pipe', config)).err).not.toContain('Data is piped in'); expect(fs.readFileSync('my.file', 'utf8')).toBe('boobar'); }); });