Announcement

Collapse
No announcement yet.

GNU Parallel 20190322 Released - Wants To Help Speed Up Single-Threaded Linux Commands

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • GNU Parallel 20190322 Released - Wants To Help Speed Up Single-Threaded Linux Commands

    Phoronix: GNU Parallel 20190322 Released - Wants To Help Speed Up Single-Threaded Linux Commands

    GNU Parallel is a tool for carrying out multiple commands/jobs in parallel on one or more computers. Out today is the GNU Parallel 20190322 release with a few changes over last month's update...

    Phoronix, Linux Hardware Reviews, Linux hardware benchmarks, Linux server benchmarks, Linux benchmarking, Desktop Linux, Linux performance, Open Source graphics, Linux How To, Ubuntu benchmarks, Ubuntu hardware, Phoronix Test Suite

  • #2
    The cheat sheet is a good idea, GNU parallel is a complex tool with a bewildering number of options. Its main advantage over xargs is the progress bar. I wish it were rewritten in C or Go to get rid of the Perl dependency, though.

    Comment


    • #3
      Originally posted by fazalmajid View Post
      I wish it were rewritten in C or Go to get rid of the Perl dependency, though.
      I agree; although Perl is actually more portable than Go but both have largish runtimes. It just so happens I have the perl runtime already in base for my OS for various package managment tools. Whereas I do not have Go.

      So for this one, I vote C all the way!

      Comment


      • #4
        I've found this program very useful in the past for converting normally single threaded tasks and getting a bunch of them done faster than the simple manual way. But, every time I have to spend several minutes reading the man page for it and I swear they change the options around because my command lines never feel familiar. So, yeah, some good examples would go a long way to making it more useful.

        I appreciated the link in the article to the page about how to break apart a single task like a grep. The purist in me rebels at the idea of breaking a regex into several separate actions, but most regex aren't multiline, so it's probably no hard.

        One thing that isn't made clear until you read a lot of the manual page is that parallel does a lot of work to make the output (stdout) of the many parallel tasks end up in the same order it would have if the tasks had been done sequentially. That's one of the reasons the grep example works--and your output isn't all mashed together. That may or may not be a big issue, but when you need it, it's great to have it there. And, yes, there's an option to turn it off. There are options for everything......

        Comment


        • #5
          Originally posted by kpedersen View Post
          So for this one, I vote C all the way!
          I agree in principle, but it takes a lot more effort to make a reliable program in C than in Go(or Perl)...

          That's why I prefer GNU xargs and its -P flag to parallel, but there are cases when the latter's complexity is required.

          Comment


          • #6
            If we are suggesting languages why not C++, Rust or Swift? There are lots of options but I have to agree with the idea that Perl isn’t the best choice here.

            Comment


            • #7
              Michael any hope we'll get both sequential and also parallel (utilizing gnu parallel) benchmarks of lame/flac in the test suite? The single core performance is a nice number, but there are actually people trying to use all the 16+ cores of those high perf systems, and this is a simple but representative way of doing that. One could even provide multiple tests with different cpu affinities, with 1, 2, 4, ... 2^n threads, where 2^n is the number of hw cores.
              Last edited by caligula; 23 March 2019, 02:51 AM.

              Comment


              • #8
                Originally posted by wizard69 View Post
                If we are suggesting languages why not C++, Rust or Swift? There are lots of options but I have to agree with the idea that Perl isn’t the best choice here.
                Do you realize the utility is just coordinating execution. It takes very little time. It could even be implemented in Brainf*ck with a very tiny perf penalty.

                Comment


                • #9
                  Originally posted by caligula View Post
                  Michael any hope we'll get both sequential and also parallel (utilizing gnu parallel) benchmarks of lame/flac in the test suite? The single core performance is a nice number, but there are actually people trying to use all the 16+ cores of those high perf systems, and this is a simple but representative way of doing that. One could even provide multiple tests with different cpu affinities, with 1, 2, 4, ... 2^n threads, where 2^n is the number of hw cores.
                  I haven't looked into using GNU Parallel previously so not sure offhand how much of a pain (or not) it would be.
                  Michael Larabel
                  https://www.michaellarabel.com/

                  Comment


                  • #10
                    Originally posted by Michael View Post

                    I haven't looked into using GNU Parallel previously so not sure offhand how much of a pain (or not) it would be.
                    I love the work you do on Phoronix, so I joined the forum just to contribute a minuscule amount back. I actually use a parallel command to batch convert flac files.

                    Code:
                    [B][COLOR=#000000][FONT=Courier New]parallel ffmpeg -i {} -ab 320k {.}.mp3 ::: ./*.flac[/FONT][/COLOR][/B]
                    I can't remember where I stole it (and modified it to suit my needs) from, but it works much better than my old command.

                    Code:
                    for i in *.flac
                    do
                    ffmpeg -i "$i" -ab 320k "$(basename "${i/.flac}").mp3"
                    done
                    Examples, with time used, below.

                    First up is the folder and files I'll be converting.

                    Code:
                    dave_boo@linux-0xyx:~/Music/Sara Mansilla/2019 - Talking to the Wall> ls -la
                    total 233160
                    drwxr-xr-x 2 dave_boo users     4096 Mar 23 19:37 .
                    drwxr-xr-x 3 dave_boo users       40 Mar 23 19:22 ..
                    -rw-r--r-- 1 dave_boo users 21914824 Mar 23 18:48 01 - On Our Own.flac
                    -rw-r--r-- 1 dave_boo users 22825262 Mar 23 18:48 02 - Something Goin' On.flac
                    -rw-r--r-- 1 dave_boo users 23845813 Mar 23 18:48 03 - Talking to the Wall.flac
                    -rw-r--r-- 1 dave_boo users 16107396 Mar 23 18:48 04 - For You.flac
                    -rw-r--r-- 1 dave_boo users 30082426 Mar 23 18:48 05 - Better Place.flac
                    -rw-r--r-- 1 dave_boo users 19282366 Mar 23 18:48 06 - Back to You.flac
                    -rw-r--r-- 1 dave_boo users 20641853 Mar 23 18:48 07 - Nightfall.flac
                    -rw-r--r-- 1 dave_boo users 24211115 Mar 23 18:48 08 - Ezin Nahastu.flac
                    -rw-r--r-- 1 dave_boo users 25076310 Mar 23 18:48 09 - I Did Know.flac
                    -rw-r--r-- 1 dave_boo users 34691484 Mar 23 18:48 10 - Just Fine.flac
                    -rwxr-xr-x 1 dave_boo users       78 Mar 23 19:36 flac2mp3.sh
                    -rw-r--r-- 1 dave_boo users    45549 Mar 23 17:57 folder.jpg
                    Now the flac2mp3.sh script I was using.

                    Code:
                    dave_boo@linux-0xyx:~/Music/Sara Mansilla/2019 - Talking to the Wall> time ./flac2mp3.sh
                    ffmpeg version 3.4.4 Copyright (c) 2000-2018 the FFmpeg developers
                      built with gcc 7 (SUSE Linux)
                      configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --optflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --disable-openssl --enable-avresample --enable-libcdio --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libcelt --enable-libcdio --enable-libdc1394 --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libzimg --enable-libzvbi --enable-vaapi --enable-vdpau --enable-libfdk_aac --enable-nonfree --enable-libx264 --enable-libx265 --enable-libxvid
                      libavutil      55. 78.100 / 55. 78.100
                      libavcodec     57.107.100 / 57.107.100
                      libavformat    57. 83.100 / 57. 83.100
                      libavdevice    57. 10.100 / 57. 10.100
                      libavfilter     6.107.100 /  6.107.100
                      libavresample   3.  7.  0 /  3.  7.  0
                      libswscale      4.  8.100 /  4.  8.100
                      libswresample   2.  9.100 /  2.  9.100
                      libpostproc    54.  7.100 / 54.  7.100
                    Input #0, flac, from '01 - On Our Own.flac':
                      Metadata:
                        ALBUM           : Talking to the Wall
                        ARTIST          : Sara Mansilla
                        DATE            : 2019
                        album_artist    : Sara Mansilla
                        TITLE           : On Our Own
                        GENRE           : Jazz, rock, blues
                        track           : 01
                        disc            : 1
                        DISCTOTAL       : 1
                      Duration: 00:03:32.35, start: 0.000000, bitrate: 825 kb/s
                        Stream #0:0: Audio: flac, 44100 Hz, stereo, s16
                        Stream #0:1: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 500x500 [SAR 299:299 DAR 1:1], 90k tbr, 90k tbn, 90k tbc
                        Metadata:
                          comment         : Cover (front)
                    Stream mapping:
                      Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
                      Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
                    Press [q] to stop, [?] for help
                    [swscaler @ 0x55e81f9d6be0] deprecated pixel format used, make sure you did set range correctly
                    [mp3 @ 0x55e81f92ef20] Frame rate very high for a muxer not efficiently supporting it.
                    Please consider specifying a lower framerate, a different muxer or -vsync 2
                    Output #0, mp3, to '01 - On Our Own.mp3':
                      Metadata:
                        TALB            : Talking to the Wall
                        TPE1            : Sara Mansilla
                        TDRC            : 2019
                        TPE2            : Sara Mansilla
                        TIT2            : On Our Own
                        TCON            : Jazz, rock, blues
                        TRCK            : 01
                        TPOS            : 1
                        DISCTOTAL       : 1
                        TSSE            : Lavf57.83.100
                        Stream #0:0: Video: png, rgb24(progressive), 500x500 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 90k fps, 90k tbn, 90k tbc
                        Metadata:
                          comment         : Cover (front)
                          encoder         : Lavc57.107.100 png
                        Stream #0:1: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 320 kb/s
                        Metadata:
                          encoder         : Lavc57.107.100 libmp3lame
                    frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:20.82 bitrate=   0.1kbits/sframe=    1 fps=1.0 q=0.0 size=       0kB time=00:00:41.71 bitrate=   0.0kbits/sframe=    1 fps=0.7 q=0.0 size=       0kB time=00:01:02.61 bitrate=   0.0kbits/sframe=    1 fps=0.5 q=0.0 size=       0kB time=00:01:23.51 bitrate=   0.0kbits/sframe=    1 fps=0.4 q=0.0 size=       0kB time=00:01:44.85 bitrate=   0.0kbits/sframe=    1 fps=0.3 q=0.0 size=       0kB time=00:02:05.57 bitrate=   0.0kbits/sframe=    1 fps=0.3 q=0.0 size=       0kB time=00:02:26.83 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:02:47.36 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:03:07.42 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:03:28.69 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=-0.0 Lsize=    8502kB time=00:03:32.35 bitrate= 328.0kbits/s speed=41.7x
                    video:205kB audio:8296kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.015049%
                    ~~~~~~
                    real    0m56.026s
                    user    0m59.046s
                    sys     0m0.525s
                    And finally the parallel command.

                    Code:
                    dave_boo@linux-0xyx:~/Music/Sara Mansilla/2019 - Talking to the Wall> time parallel ffmpeg -i {} -ab 320k {.}.mp3 ::: ./*.flac
                    ffmpeg version 3.4.4 Copyright (c) 2000-2018 the FFmpeg developers
                      built with gcc 7 (SUSE Linux)
                      configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --optflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --disable-openssl --enable-avresample --enable-libcdio --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libcelt --enable-libcdio --enable-libdc1394 --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libzimg --enable-libzvbi --enable-vaapi --enable-vdpau --enable-libfdk_aac --enable-nonfree --enable-libx264 --enable-libx265 --enable-libxvid
                      libavutil      55. 78.100 / 55. 78.100
                      libavcodec     57.107.100 / 57.107.100
                      libavformat    57. 83.100 / 57. 83.100
                      libavdevice    57. 10.100 / 57. 10.100
                      libavfilter     6.107.100 /  6.107.100
                      libavresample   3.  7.  0 /  3.  7.  0
                      libswscale      4.  8.100 /  4.  8.100
                      libswresample   2.  9.100 /  2.  9.100
                      libpostproc    54.  7.100 / 54.  7.100
                    Input #0, flac, from './04 - For You.flac':
                      Metadata:
                        ALBUM           : Talking to the Wall
                        TITLE           : For You
                        DATE            : 2019
                        album_artist    : Sara Mansilla
                        ARTIST          : Sara Mansilla
                        GENRE           : Jazz, rock, blues
                        track           : 04
                        disc            : 1
                        DISCTOTAL       : 1
                      Duration: 00:02:41.44, start: 0.000000, bitrate: 798 kb/s
                        Stream #0:0: Audio: flac, 44100 Hz, stereo, s16
                        Stream #0:1: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 500x500 [SAR 299:299 DAR 1:1], 90k tbr, 90k tbn, 90k tbc
                        Metadata:
                          comment         : Cover (front)
                    Stream mapping:
                      Stream #0:1 -> #0:0 (mjpeg (native) -> png (native))
                      Stream #0:0 -> #0:1 (flac (native) -> mp3 (libmp3lame))
                    Press [q] to stop, [?] for help
                    [swscaler @ 0x55a3c4d85ea0] deprecated pixel format used, make sure you did set range correctly
                    [mp3 @ 0x55a3c4cdbd20] Frame rate very high for a muxer not efficiently supporting it.
                    Please consider specifying a lower framerate, a different muxer or -vsync 2
                    Output #0, mp3, to './04 - For You.mp3':
                      Metadata:
                        TALB            : Talking to the Wall
                        TIT2            : For You
                        TDRC            : 2019
                        TPE2            : Sara Mansilla
                        TPE1            : Sara Mansilla
                        TCON            : Jazz, rock, blues
                        TRCK            : 04
                        TPOS            : 1
                        DISCTOTAL       : 1
                        TSSE            : Lavf57.83.100
                        Stream #0:0: Video: png, rgb24(progressive), 500x500 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 90k fps, 90k tbn, 90k tbc
                        Metadata:
                          comment         : Cover (front)
                          encoder         : Lavc57.107.100 png
                        Stream #0:1: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 320 kb/s
                        Metadata:
                          encoder         : Lavc57.107.100 libmp3lame
                    frame=    1 fps=0.0 q=0.0 size=       0kB time=00:00:12.90 bitrate=   0.1kbits/sframe=    1 fps=1.0 q=0.0 size=       0kB time=00:00:25.07 bitrate=   0.1kbits/sframe=    1 fps=0.7 q=0.0 size=       0kB time=00:00:37.25 bitrate=   0.0kbits/sframe=    1 fps=0.5 q=0.0 size=       0kB time=00:00:49.42 bitrate=   0.0kbits/sframe=    1 fps=0.4 q=0.0 size=       0kB time=00:01:01.38 bitrate=   0.0kbits/sframe=    1 fps=0.3 q=0.0 size=       0kB time=00:01:12.43 bitrate=   0.0kbits/sframe=    1 fps=0.3 q=0.0 size=       0kB time=00:01:16.82 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:19.77 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:22.67 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:25.91 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:29.15 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:35.11 bitrate=   0.0kbits/sframe=    1 fps=0.2 q=0.0 size=       0kB time=00:01:47.10 bitrate=   0.0kbits/sframe=    1 fps=0.1 q=0.0 size=       0kB time=00:01:59.53 bitrate=   0.0kbits/sframe=    1 fps=0.1 q=0.0 size=       0kB time=00:02:11.60 bitrate=   0.0kbits/sframe=    1 fps=0.1 q=0.0 size=       0kB time=00:02:23.04 bitrate=   0.0kbits/sframe=    1 fps=0.1 q=0.0 size=       0kB time=00:02:35.29 bitrate=   0.0kbits/sframe=    1 fps=0.1 q=-0.0 Lsize=    6515kB time=00:02:41.46 bitrate= 330.5kbits/s speed=18.5x
                    video:205kB audio:6308kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.019597%
                    ~~~~~~~
                    real    0m22.863s
                    user    1m55.264s
                    sys     0m0.987s
                    Note that this is on an old Lenovo laptop rocking a 3630QM and 16GB of RAM. Using the parallel command all 8 threads were being utilized.

                    Falling from 56 seconds to 23 seconds is a nice time saver. As an aside using the parallel -j4 $your_command resulted in the below on MY machine. Not that big of an increase in time; I was monitoring the threads but not temperature so perhaps the laptop was able to boost higher?

                    Code:
                    real    0m25.137s
                    user    1m24.047s
                    sys     0m0.704s
                    Note that this is on an old Lenovo laptop rocking a 3630QM and 16GB of RAM. Using the parallel command all 8 threads were being utilized.

                    Comment

                    Working...
                    X