Technology

Here are some random tips for working on computers...

Detachable terminal sessions

You can use screen or dtach to make sure that if your connection is dropped from a linux server, that your session is preserved and you can link back to it. This is especially useful for keeping processes running and being able to come back and check on them later.

Detachable sessions with SCREEN

Screen is very powerful, it's like a text-terminal window manager. You can create a session with multiple terminal windows all detachable and reattachable in one screen session.

Steps:

1. Make sure you have screen installed on the target server. From a session on that server, run sudo yum install screen
2. Start a screen session by entering screen -S mysession at the prompt (replacing mysession with a name you'll remember when you re-attach)
3. screen is mostly run with ctl-a <something> key combos, for example, ctl-a h will bring up a help screen, and ctl-a ctl-c creates a new shell.
4. *A trick to remember is that in screen your normal vertical scrolling won't work. See the help to see how to activate scroll-mod
5. *Another trick is that ctl-a is co-opted by screen while you are in a screen session. To get an actual ctl-a, type ctl-a a
6. To detach a screen session so you can logout, type ctl-a d inside of screen, then you can log off (if you are disconnect, the detach happens automatically)
7. To re-attach a session, log back in to the machine, and then you can re-attach with screen -r mysession
8. *If you log in to a machine, you can check that there's a screen session with screen -list, and connect to a detached session with screen -r

Detachable sessions with dtach

Another, less powerful but lower-overhead tool to use that lets you detach like in screen but doesn't do the other screen stuff is dtach. It is not as powerful as screen, but has the advantage that you can see with the ps command what is running what socket.

Steps:

1. Make sure you have dtach installed on the target server. From a session on that server, run sudo yum install dtach
2. dtach uses named sockets, so you will be creating a socket in /usr/tmp, and running a bash session (type dtach --help or check the man page for details):
3. *dtach -c /tmp/checkbotrun bash creates a process run by dtach with the socket /tmp/checkbotrun
4. in the bash session, run whatever program you want
5. Detach your session with ctl-\ key if you want
6. Re-attach later (even after logging off) with dtach -a /temp/checkbotrun
7. *when you log back into the machine after detaching and logging, off, the job will appear as the initial dtach command, e.g. dtach -c /tmp/checkbotrun bash

How to Install Perl Modules on Cent OS?

Stuff that integrates with linux, like for example XML::Lib XML?, doesn't install right when using a cpan package manager like cpanm. Instead, you have to search first to see if there's a yum package for what you want, using "provides". For example:

 yum provides "perl(XML::LibXML)"

This will return:

 1:perl-XML-LibXML-1.70-5.el6.x86_64 : Perl interface to the libxml2 library
 Repo        : base
 Matched from:
 Other       : perl(XML::LibXML)

Then you can install it by:

 sudo yum install perl-XML-LibXML

How to Set Up a User-level Perl Library

If you can't figure out how to get elevated perms to install Perl modules and apps for everyone to use, you can set them to your local user account. Below where it says ".profile" you may need to change depending on your linux/unix environment to .bash_profile or .zprofile. This example also grabs and sets up cpanm on the first line, which you can skip if you already have it:

  curl http://cpanmin.us | perl - -l ~/perl5 App::cpanminus local::lib
  eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`
  echo 'eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`' >> ~/.profile
  echo 'export MANPATH=$HOME/perl5/man:$MANPATH' >> ~/.profile

You will now have a local perl library, ~/perl5 where all subsequent perl module installs you do from the command line will go.

How to read the system logs on RHEL Machines

system logs are not in /var/log anymore. App and service logs are, ex: Apache, cron. But if you want to see the system log, you need to run <b>jourtnalctl</b>, probably with sudo, a la sudo jourtnalctl. There are details on how to use this command here (also a man page, of course):

• https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-Using_the_Journal.html
• https://www.certdepot.net/rhel7-interpret-system-log-files/

How to see what services are running on an RHEL linux server

Two steps:

• systemctl services listed with shell command systemctl list-unit-files
• Sys V? services listed with shell command chkconfig --list

How to add a user to a linux server

To add a new user:

ex: sudo useradd <username> --groups webpub -p <password>

(the -e expires the account, if you have to put in an expire date, use YYYY-MM-DD format for expire date. You can expire the account *after* you create it, with usermod: ex: usermod <username> -e )

The password has certain rules (can't be a dictionary word or close), and this is not enforced with useradd and if it *is* too simple, the user won't be able to log in even after you create the account. So make it complicated OR you can set the password manually to make sure it passes after you create it as above:

 ex: sudo passwd <username>

 New password: 
 BAD PASSWORD: it is too short
 BAD PASSWORD: is too simple
 Retype new password: 
 etc

Also, if you want to add them to the sudoers group, an admin has to use sudo visudo to edit the sudo file in vi, add the new user like the other people in the list:

 ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
 #includedir /etc/sudoers.d
 trusteduser         ALL=(ALL)       ALL
 ...
 <username>          ALL=(ALL)       ALL

How to preview your Markdown document

.md files in git are written in Markdown. Even if you find a tutorial and figure it out, you may not be using a tool with preview to write or edit your markdown document. Here is one: https://daringfireball.net/projects/markdown/dingus

How to pretty-print JSON at the command line

Check out this page: http://www.skorks.com/2013/04/the-best-way-to-pretty-print-json-on-the-command-line/

For quick-and-dirty, if python is already installed, you can use a python tool mjson.tool, you just pipe the output to it, ex:

 curl http://stp-trt-otpapi.soundtransit.org/otp/routers/default/index/agencies | python -mjson.tool |less

There is a C tool called http://lloyd.github.io/yajl/ YAJL mentioned there that works great that can be grabbed to your machine with yum to a linux virtual:

 sudo yum install yajl

You can cat the file and pipe it into the YAJL tool like so, for example, if you wanted to save the file (first example) or don't care to (2nd example):

 wget --output-document=test.json http://oba-otp-prod.soundtransit.org/otp/routers/default/index/agencies/1
 cat test.json | json_reformat | less

OR

   curl http://oba-otp-prod.soundtransit.org/otp/routers/default/index/agencies/3 | json_reformat | less

The other advantage of YAJL is that its a library and it has bindings that allow it to be used in other languages like perland ruby

How to add up a column of numbers in a text file in Mac/Unix shell

You can use sed or perl to get your file to just a list of numbers, then this will give you a running total in the form of "{current added value}, {current total}":

 perl -anle '$x+=$_ for(@F); print "$_, $x"; END {print $x}' filename.txt

How to add a prefix to a set of files in Unix shell

There is a bash thing called http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Brace-Expansion Brace Expansion that just expands what's in the braces, so for example:

 for i in *.txt; do mv {,40_}$i; done;

If you had one file in the folder, "test.txt" the above would issue the command mv test.txt 40_test.txt. The comma at the beginning means no prefix, and if you wanted a bigger expansion for different purposes, you can add more {,40_,50_} etc

How to pretty-print CSV files at the command line

From https://www.stefaanlippens.net/pretty-csv.html (follow the Mac directions)

Make a script with the special less flags and perl described in that website: in your user ~/bin folder (create it if you haven't).

For example, create a zsh/bash script file called ~/bin/pretty-csv.sh, containing:

  #!/bin/bash
  perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | exec less  -F -S -X -K

Make it executable (chmod ug+x ~/bin/pretty-csv.sh) and create a bash alias for it (e.g in .zshrc or .zprofile)

  alias pretty_csv='~/bin/pretty-csv.sh'

Afterwords, you can invoke it on a file:

  pretty-csv stops.txt

or pipe an output to it:

 sed -ne '/Line/p' routes.txt | pretty-csv

ex results:

  % sed -ne '/Line/p' 40_gtfs-09-15-24-prod/routes.txt | pretty-csv
  40  100479   1 Line  Lynnwood - Angle Lake          0  1 Line runs between Lynnwood City Center Station and Angle Lake St>
  40  2LINE    2 Line  South Bellevue - Redmond Tech  0  2 Line runs between South Bellevue Station and Redmond Tech Statio>
  40  SNDR_EV  N Line  Everett - Seattle              2  N Line runs between Everett Station and King Street Station weekda>
  40  SNDR_TL  S Line  Seattle - Tacoma/Lakewood      2  S Line runs between King Street Station and Tacoma Dome or Lakewoo>
  40  TLINE    T Line  Tacoma Dome - St Joseph        0  T Line runs between Tacoma Dome Station and St Joseph Station seve>
  (END)

How to prepend text into stdout output in Unix shell

There are two ways to do this:

• You can use semi-colon in bash and bash-like shells to, say prepend the columns from a CSV file to a grep. You may need to use parethesis as well.
  • Example: (head -1 routes.txt ; grep "[A-Z] Line" routes.txt ) | pretty-csv
• You can use a parenthesis with a "(prepend text && cat)" construct, as follows. I guess this is better if you are building from left to right and remember you need to prepend after you already started typing your pipe command
  • Example: grep "[A-Z] Line" routes.txt | (head -1 routes.txt && cat) |pretty-csv

How to Disable the auto-complete bell on your Mac terminal

This is a linux thing, not a mac thing, but if you create or edit ~/.inputrc, and add the line (or alternately, find if there is a "set bell-style" and make it match below):

• For a visual flash bell:
  set bell-style visual
• For no indication at all:
  set bell-style none

To do it for all users, make the same change in /etc/inputrc

Then the next terminals you open should no longer beep on autocomplete (from https://linuxconfig.org/turn-off-beep-bell-on-linux-terminal)

How to Enable or Disable Experimental Features in Chrome

• In Chrome, navigate to chrome://flags
• search for enable-experimental-web-platform-features in the search box on that page
• Make sure it is set to “Enabled” (or the reverse) by switching it to that (assuming it is not)
• Chrome will warn you it needs to shut down and relaunch chrome, click “Relaunch”