[atnak]

EmEditor Version: 16.0.2

How to reproduce:

1. Create a new External Tool with the following properties:
Input: “Document”
Output: “Replace Document”
Command: (Any command that reads stdin and outputs to stdout. Something like cat on Linux when testing.)

2. Create a new document that just contains the single letter “a”.

3. Call the External Tool created above.

4. The program is actually fed two bytes: 0x61 (‘a’) followed by 0x1A (what’s this??).

[Yutaka Emura]

Hello,

I believe 0x1A is an End-Of-File (EOF) character, and the cat command probably produces the EOF.

[atnak]

cat command probably produces the EOF

cat is really only an example. (Although what I said above is true for the cat.exe that I have.)

The outcome is the similar with various other commands:

SqlFormatter.exe regurgitates the 0x1A to the output.

python.exe -m json.tool complains about “Extra data” at the exact line and column where that 0x1A byte is expected to be.

This simple C# program will output the 0x1A on the end if specified through EmEditor’s “External Tools”. It will not do this if called through command prompt such as ‘Program.exe <a.txt >b.txt‘.

static class Program
{
    static void Main(string[] args)
    {
        for (int c; (c = System.Console.Read()) != -1;)
        {
            System.Console.Write((char)c);
        }
    }
}

[Yutaka Emura]

Hello atnak,

I used your sample program, and reproduce the issue. I found that EmEditor always outputs EOF (0x1a) when the “Selection” or “Document” is selected for the Input in the External Tool Properties.To work around this, you can select the “Custom” for the Input, and then enter “$(DocText)” for the “Custom” text box, and make sure the “Add EOF” check box is cleared.

On the next version, I will make the “Add EOF” check box enabled even when “Selection” or “Document” is selected.

Thanks!

[atnak]

To work around this, you can select the “Custom” for the Input, and then enter “$(DocText)” for the “Custom” text box, and make sure the “Add EOF” check box is cleared.

Thanks for the update and workaround. I can confirm it works correctly with Custom input of “$(DocText)”.

On the next version, I will make the “Add EOF” check box enabled even when “Selection” or “Document” is selected.

I’m not altogether certain what the purpose of this is, since a real EOF is received by the external program when EmEditor closes the pipe.

I tested with this program to confirm EmEditor is properly closing the pipe. I get the output “a$” to the input “a” when “Add EOF” is off, and “a<0x1a>$” when it’s checked.

static class Program
{
    static void Main(string[] args)
    {
        for (int c; (c = System.Console.Read()) != -1;)
        {
            System.Console.Write((char)c);
        }
        System.Console.Write("$");
    }
}

At the very least, I think the checkbox is ill-named as it can be confused with the real EOF (closing of an input stream) which that checkbox has nothing to do with. Maybe a name such as “Add &1AH (Ctrl-Z) byte” can avoid this confusion.

[Yutaka Emura]

Hello,

Some old programs need the EOF character (0x1a), and I believe some users asked for it. I will change the name of the check box and also explain it in the Help.

Thanks,

[atnak]

Some old programs need the EOF character (0x1a),

That’s something I never knew of! Thanks for the explanation.

Regards,

P.S.
I did some searching on this mysterious 0x1A character that I’ll share here in case someone else is interested.

• “Some early disk operating systems like CP/M actually did use a physical 0x1A (ASCII SUB character) to indicate EOF” — http://stackoverflow.com/a/3061174/1036728
• “There was a long long time ago an End Of File marker but it hasn’t been used in files for many years.” — http://stackoverflow.com/a/24992477/1036728
• “I encountered a couple of text files with a trailing ^Z back in the days of DOS 3” — http://www.nntp.perl.org/group/perl.perl5.porters/2013/05/msg202401.html

[Author]

Posts