Usando expressão regular para rm e ls

PERGUNTA:

O que está acontecendo aqui e como faço para corrigir isso?

PROBLEMA:

Quando eu faço:

$ls | grep -P "file[1-9]{3}.txt|file[1-9]{2}.txt"

Eu recebo:

file11.txt
file12.txt
file13.txt
file14.txt
file15.txt
file16.txt
file17.txt
file18.txt
file19.txt
file21.txt
file22.txt
file23.txt
file24.txt
file25.txt
file26.txt
file27.txt
file28.txt
file29.txt
etc.

Mas quando eu faço:

$ls | grep -P "file[1-9]{3}.txt|file[1-9]{2}.txt" | xargs -d"\n" rm

Eu recebo:

rm: cannot remove ''$'\033''[38;5;253mfile11.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile12.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile13.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile14.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile15.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile16.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile17.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile18.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile19.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile21.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile22.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile23.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile24.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile25.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile26.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile27.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile28.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile29.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile31.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile32.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile33.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile34.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile35.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile36.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile37.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile38.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile39.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile41.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile42.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile43.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile44.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile45.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile46.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile47.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile48.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile49.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile51.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile52.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile53.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile54.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile55.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile56.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile57.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile58.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile59.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile61.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile62.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile63.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile64.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile65.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile66.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile67.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile68.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile69.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile71.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile72.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile73.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile74.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile75.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile76.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile77.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile78.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile79.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile81.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile82.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile83.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile84.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile85.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile86.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile87.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile88.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile89.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile91.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile92.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile93.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile94.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile95.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile96.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile97.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile98.txt'$'\033''[0m': No such file or directory
rm: cannot remove ''$'\033''[38;5;253mfile99.txt'$'\033''[0m': No such file or directory
etc.

RESULTADO DESEJADO:

rm remove todos os arquivos que correspondem ao regex