Unix_shell

Downloading VirtualBox

Beginning with the Installation:

When you will open virtualbox it will look like as shown below:

Error 404

How to Install Debian 12 on VirtualBox

Pre-requisite

Step 1: Create a Debian 12 Virtual Machine

  1. Open VirtualBox and click on the New button. Error 404

  2. Enter VM Name and Location:

    • Name: Debian 12
    • Location: Choose your preferred folder
    • Type: Linux
    • Version: Debian (64-bit)

Error 404

  1. Allocate System Resources:
    • Memory (RAM): Recommended at least 2 GB
    • CPUs: Allocate at least 2 CPUs

Error 404

  1. Set Storage Space:
    • Recommended: 20 GB (Adjust if needed for more applications)

Error 404

  1. Verify and Create:
    • Review all the settings and click on Create.

Error 404

Step 2: Install Debian 12 on VirtualBox

  1. Start the VM:
    • Click on the Start button. Error 404
  2. Mount ISO File:
    • If not already linked, add the ISO file path and click on Mount and Retry Boot. Error 404
  3. Select Installation Type:
    • Choose Graphical install. Error 404
  4. Language Selection:
    • Choose the language for the installation process. Error 404
  5. Location Selection:
    • Set your location for time zone configuration. Error 404
  6. Keyboard Layout:
    • Choose the appropriate keyboard layout based on your language. Error 404
  7. Configure Network:
    • Hostname: Enter a name for your system.
    • Domain Name: Enter if applicable or leave it blank. Error 404 Error 404
  8. Additional Installation Files:
    • Select No if no additional files are needed. Error 404
  9. Mirror Location:
    • Select the nearest country for the Debian mirror list. Error 404
  10. Debian Archive Mirror:
    • Choose the default mirror if no preference. Error 404
  11. Proxy Information:
    • Enter if needed, otherwise leave blank. Error 404
  12. Begin Installation:
    • Wait for the installation process (can take up to 30 minutes). Errir 404
  13. Create Admin User:
    • Set up an admin user and password. Error 404
  14. Create Normal User:
    • Create another user without admin rights if needed. Error 404 Error 404 Error 404
  15. Set Time Zone:
    • Select the appropriate time zone based on your location. Error 404
  16. Partition Disks:
    • Choose to manually create partitions or use the entire disk. Error 404 Error 404 Error 404
  17. Format Disk:
    • Confirm the partition scheme and format the disk. Error 404 Error 404
  18. Popularity Contest:
    • Choose whether to send system statistics to the Debian community. Error 404
  19. Select Desktop Environment:
    • Choose a desktop environment (e.g., GNOME). Error 404
  20. Install GRUB Bootloader:
    • Install GRUB to manage the boot process. Error 404 Error 404
  21. Reboot System:
    • Complete the installation by rebooting the system. Error 404

Following these steps will help you successfully install Debian 12 on VirtualBox, allowing you to explore and utilize the Debian operating system in a virtual environment.

Chapter 1: Introducing the Shell

Overview

Questions

Objectives

Background

Humans and computers interact in various ways, such as through:

Graphical User Interface (GUI)

Command-Line Interface (CLI)

Example

Using a GUI to copy the third line of 1000 text files in different directories is time-consuming and error-prone. The Unix shell, a CLI, can automate and speed up this process.

The Shell

Learning the Shell

Advantages of the Shell

Getting Started with the Shell

The Prompt

Example

$ ls

Common Error

$ ks
ks: command not found

Nelle’s Pipeline: A Typical Problem

Nelle Nemo, a marine biologist, needs to process 1520 samples using an imaginary program goostats.sh.

Problem

Skills Nelle Needs

Key Points

Chapter 2: Navigating Files and Directories

Questions

  1. How can I move around on my computer?
  2. How can I see what files and directories I have?
  3. How can I specify the location of a file or directory on my computer?

Objectives

  1. Explain the similarities and differences between a file and a directory.
  2. Translate an absolute path into a relative path and vice versa.
  3. Construct absolute and relative paths that identify specific files and directories.
  4. Use options and arguments to change the behavior of a shell command.
  5. Demonstrate the use of tab completion and explain its advantages.

Key Concepts

Commands Overview

  1. pwd
    • Shows the current working directory.
    • Example: $ pwd might output /Users/nelle.
  2. ls
    • Lists the contents of the current directory.
    • Example: $ ls might output: 
      Applications Documents Library Music Public
Desktop Downloads Movies Pictures
    • Options:
      • -F: Classifies the output by adding markers to file and directory names.
        • / indicates a directory.
        • @ indicates a link.
        • * indicates an executable.
  3. cd
    • Changes the current working directory.
    • Example: $ cd Desktop moves to the Desktop directory.

The File System Structure

Examples

Moving Around the File System

  1. Finding Your Location: 
    $ pwd
/Users/nelle
  2. Listing Contents: 
    $ ls -F
Applications/ Documents/ Library/ Music/ Public/
Desktop/ Downloads/ Movies/ Pictures/
  3. Changing Directories: 
    $ cd Desktop
$ pwd
/Users/nelle/Desktop
  4. Moving to a Subdirectory: 
    $ cd shell-lesson-data/exercise-data
$ pwd
/Users/nelle/Desktop/shell-lesson-data/exercise-data
  5. Moving Up a Directory: 
    $ cd ..
$ pwd
/Users/nelle/Desktop/shell-lesson-data
  6. Returning to Home Directory: 
    $ cd
$ pwd
/Users/nelle

Viewing Hidden Files

Getting Help

  1. Using --help: 
    $ ls --help
  2. Using man: 
    $ man ls

Combining Options

Exploring Other Directories

Shortcuts

  1. Home Directory Shortcut (~): 
    $ cd ~
$ cd ~/Desktop
  2. Previous Directory Shortcut (-): 
    $ cd -

Absolute vs Relative Paths

Tab Completion

Key Points

This detailed note should help you understand the basics of navigating files and directories in the shell. Practice these commands and explore your file system to become more comfortable with these concepts.

Notes on “Working With Files and Directories”

Overview

Creating Directories

Good Naming Practices for Files and Directories

Creating Text Files

Naming Conventions

Moving Files and Directories

Copying Files and Directories

Removing Files and Directories

Operations with Multiple Files and Directories

Practical Exercises

These notes should give you a solid understanding of creating, copying, moving, and deleting files and directories, as well as using wildcards and organizing files efficiently in a Unix shell environment.

Chapter : 4 Pipes and Filters

Last updated on 2024-04-12 | Edit this page

Overview

Questions:

  1. How can I combine existing commands to produce a desired output?
  2. How can I show only part of the output?

Objectives:

  1. Explain the advantage of linking commands with pipes and filters.
  2. Combine sequences of commands to get new output.
  3. Redirect a command’s output to a file.
  4. Explain what usually happens if a program or pipeline isn’t given any input to process.

Introduction

Now that we are familiar with some basic commands, we can explore one of the shell’s most powerful features: combining existing programs in new ways. We will use the shell-lesson-data/exercise-data/alkanes directory, which contains files in Protein Data Bank format, describing simple organic molecules.

Example Files in Directory:

$ ls
cubane.pdb    methane.pdb    pentane.pdb
ethane.pdb    octane.pdb     propane.pdb

Word Count Command:

$ wc cubane.pdb

Output:

20  156 1158 cubane.pdb

wc (word count) counts lines, words, and characters in files.

Using Wildcards with wc:

$ wc *.pdb

Output:

  20  156  1158  cubane.pdb
  12  84   622   ethane.pdb
   9  57   422   methane.pdb
  30  246  1828  octane.pdb
  21  165  1226  pentane.pdb
  15  111  825   propane.pdb
 107  819  6081  total

wc *.pdb shows the total count of lines in the last line.

Displaying Only Line Counts:

$ wc -l *.pdb

Output:

  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total

The -l option shows only the number of lines per file.

Handling Commands Without Input

Issue with Missing Filename:

$ wc -l

If no filename is provided, wc waits for input from the command prompt. To exit this state, use Ctrl+C.

Capturing Output from Commands

Redirecting Output to a File:

$ wc -l *.pdb > lengths.txt

The > operator redirects the output to a file. If lengths.txt exists, it will be overwritten.

Displaying File Content:

$ cat lengths.txt

Output:

  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total

cat displays the content of files.

Viewing Content Page by Page:

$ less lengths.txt

less allows scrolling through the file page by page.

Filtering Output

Using sort Command: Example File (numbers.txt):

10
2
19
22
6

Sorting Numerically:

$ sort -n numbers.txt

Output:

2
6
10
19
22

The -n option sorts numerically rather than alphanumerically.

Sorting and Redirecting:

$ sort -n lengths.txt > sorted-lengths.txt
$ head -n 1 sorted-lengths.txt

Output:

  9  methane.pdb

head -n 1 shows the first line of the sorted file.

Important Note: Avoid redirecting output to the same file being processed to prevent data loss.

Appending Data:

$ echo hello >> testfile02.txt

>> appends output to a file.

Passing Output to Another Command

Using Pipes:

$ wc -l *.pdb | sort -n | head -n 1

The | operator pipes output from one command to another.

Combining Commands: Pipes can chain multiple commands together to avoid using intermediate files.

Example:

$ wc -l *.pdb | sort -n | head -n 3

This command finds the three files with the least number of lines.

Tools Designed to Work Together

Unix programs are designed to be simple and work well together using pipes and filters. Each program reads from standard input, processes the data, and writes to standard output.

Example Pipeline:

$ cat animals.csv | head -n 5 | tail -n 3 | sort -r > final.txt

Understanding what passes through each stage of the pipeline is crucial.

Using cut and uniq:

$ cut -d , -f 2 animals.csv | sort | uniq

The cut command extracts columns, and uniq removes duplicates.

Counting Animals:

$ cut -d, -f 2 animals.csv | sort | uniq -c

The -c option with uniq counts occurrences.

Summary

Use pipes and filters to link simple programs together, leveraging their ability to process and transform data streams.

Feel free to ask if you need further details or explanations!

Detailed Notes on Loops

**Last updated on 2023-11-10 Edit this page**

OVERVIEW

Loops in programming allow for the repetition of commands or sets of commands for each item in a list. This feature enhances productivity through automation by reducing the need for manual repetition and minimizing typing errors.

QUESTIONS

How can I perform the same actions on many different files?

You can use loops to apply one or more commands to each file in a set.

Objectives

Chapter : 5 LOOPS IN SHELL

Basic Structure

A loop in shell scripting is generally used to repeat commands for each item in a list. Here’s the basic structure of a for loop:

for variable in list_of_items
do
    command_using $variable
done

Example

Suppose we have several files: basilisk.dat, minotaur.dat, unicorn.dat. We want to print out the classification for each species (given on the second line of each file). Here’s how a loop can solve this:

for filename in basilisk.dat minotaur.dat unicorn.dat
do
    echo $filename
    head -n 2 $filename | tail -n 1
done

Output:

basilisk.dat
CLASSIFICATION: basiliscus vulgaris
minotaur.dat
CLASSIFICATION: bos hominus
unicorn.dat
CLASSIFICATION: equus monoceros

Prompt Behavior

Variable Expansion

Inside a loop, the $ symbol is used to refer to the value of a variable. For instance, $filename gets replaced with the current filename during each iteration.

Naming Variables

Looping Over Different Lists

Loops can handle different types of lists, including filenames, numbers, or other data subsets.

VARIABLES IN LOOPS

Loop Over File Extensions

Example: Listing files with .pdb extension:

for datafile in *.pdb
do
    ls $datafile
done

Limiting Sets of Files

To limit the files processed:

for filename in c*
do
    ls $filename
done

Using wildcards like *c* can match different patterns:

for filename in *c*
do
    ls $filename
done

SAVING TO A FILE IN A LOOP

Part One

If you save the content of each file into alkanes.pdb:

for alkanes in *.pdb
do
    echo $alkanes
    cat $alkanes > alkanes.pdb
done

Part Two

Appending content to all.pdb:

for datafile in *.pdb
do
    cat $datafile >> all.pdb
done

MORE COMPLEX EXAMPLES

Example Loop

To select lines 81-100 from each file:

for filename in *.dat
do
    echo $filename
    head -n 100 $filename | tail -n 20
done

Handling Spaces in Filenames

For filenames with spaces, use quotes:

for filename in "red dragon.dat" "purple unicorn.dat"
do
    head -n 100 "$filename" | tail -n 20
done

Backing Up Files

To copy files with a prefix:

for filename in *.dat
do
    cp $filename original-$filename
done

NELLE’S PIPELINE: PROCESSING FILES

Nelle builds up commands for processing data files with goostats.sh. She uses history commands and arrow keys to edit and re-run previous commands.

Using History Commands

Example with goostats.sh

for datafile in NENE*A.txt NENE*B.txt
do
    bash goostats.sh $datafile stats-$datafile
done

KEY POINTS

These notes cover the essentials of loops, their usage, and practical examples to enhance your shell scripting skills.

Chapter : 6 Shell Scripts

Last updated on 2023-08-05

Overview

Shell scripts are a powerful way to automate repetitive tasks by saving a series of commands in a file. This file can then be executed to run the commands as a program. Shell scripts enhance efficiency, accuracy, and reproducibility, making them a valuable tool for managing command-line operations.

Objectives

  1. Write a shell script to execute commands on a fixed set of files.
  2. Run a shell script from the command line.
  3. Create a script that operates on files specified by the user.
  4. Incorporate shell scripts into pipelines for more complex operations.

Creating and Running a Shell Script

  1. Creating a Basic Script

    Start by creating a new file called middle.sh in the alkanes/ directory:

    $ cd alkanes
$ nano middle.sh

    Insert the following line into middle.sh:

    head -n 15 octane.pdb | tail -n 5

    Save the file (Ctrl-O) and exit (Ctrl-X). Execute the script with:

    $ bash middle.sh

    This command will output lines 11 to 15 from octane.pdb.

  2. Using Variables in Scripts

    To make the script more flexible, edit middle.sh to use a variable for the filename:

    $ nano middle.sh
head -n 15 "$1" | tail -n 5

    Run the script with:

    $ bash middle.sh octane.pdb

    You can now use different filenames:

    $ bash middle.sh pentane.pdb

  3. Handling Multiple Arguments

    Modify middle.sh to handle additional arguments for line ranges:

    $ nano middle.sh
head -n "$2" "$1" | tail -n "$3"

    Run the script with:

    $ bash middle.sh pentane.pdb 15 5

    This will extract lines 15 through 19 from pentane.pdb.

  4. Adding Comments

    Improve readability by adding comments at the top of middle.sh:

    $ nano middle.sh
# Select lines from the middle of a file.
# Usage: bash middle.sh filename end_line num_lines
head -n "$2" "$1" | tail -n "$3"

Advanced Scripting Techniques

  1. Processing Multiple Files

    Create a script sorted.sh to sort files by their length:

    $ nano sorted.sh
# Sort files by their length.
# Usage: bash sorted.sh one_or_more_filenames
wc -l "$@" | sort -n

    Run the script with:

    $ bash sorted.sh *.pdb ../creatures/*.dat

  2. Listing Unique Species

    Write species.sh to list unique species from multiple files:

    $ nano species.sh
# List unique species from files.
# Usage: bash species.sh filename1 filename2 ...
cut -d , -f 2 "$@" | sort | uniq

    Run the script with:

    $ bash species.sh animals.csv

  3. Re-creating Commands from History

    Save recent commands to a script:

    $ history | tail -n 5 > redo-figure-3.sh

    Edit redo-figure-3.sh to remove history command numbers and extra lines.

Handling Script Errors

  1. Debugging Scripts

    If a script produces no output, use the -x option to debug:

    $ bash -x do-errors.sh NENE*A.txt NENE*B.txt

    The -x option will show the commands being executed, helping to identify errors.

Key Points

Shell scripts can greatly streamline workflows, making repetitive tasks more efficient and less prone to errors. By leveraging the power of scripting, you can automate complex operations and ensure consistency in your command-line tasks.

Chapter : 7 Finding Things

Overview

This guide explains how to find files and lines within files using Unix commands. It covers the use of grep to search for patterns in text files and find to locate files and directories. It also discusses combining command outputs and handling different types of files, including binary files.

Grep: Finding Lines in Files

Introduction

grep (Global Regular Expression Print) is a command-line utility used to search for lines in files that match a specified pattern. It is highly versatile and can handle a variety of search options.

Basic Usage

To search for a pattern in a file:

grep pattern filename

Examples

  1. Simple Pattern Search: 
    $ grep not haiku.txt
OUTPUT
Is not the true Tao, until
"My Thesis" not found
Today it is not working
  2. Word Boundary Search: To search for a whole word, use the -w option: 
    $ grep -w The haiku.txt
OUTPUT
The Tao that is seen
  3. Case Insensitive Search: To make the search case-insensitive, use the -i option: 
    $ grep -n -w -i "the" haiku.txt
OUTPUT
1:The Tao that is seen
2:Is not the true Tao, until
6:and the presence of absence:
  4. Inverting Search: To find lines that do not match the pattern, use the -v option: 
    $ grep -n -w -v "the" haiku.txt
OUTPUT
1:The Tao that is seen
3:You bring fresh toner.
5:With searching comes loss
7:"My Thesis" not found.
9:Yesterday it worked
10:Today it is not working
11:Software is like that.
  5. Recursive Search: To search for a pattern in files within subdirectories, use the -r option: 
    $ grep -r Yesterday .
OUTPUT
./LittleWomen.txt:"Yesterday, when Aunt was asleep and I was trying to be as still as a
./LittleWomen.txt:Yesterday at dinner, when an Austrian officer stared at us and then
./LittleWomen.txt:Yesterday was a quiet day spent in teaching, sewing, and writing in my
./haiku.txt:Yesterday it worked

Advanced Patterns

  1. Using Wildcards (Regular Expressions): To find lines with an ‘o’ in the second position: 
    $ grep -E "^.o" haiku.txt
OUTPUT
You bring fresh toner.
Today it is not working
Software is like that.
  2. Combining Commands: Using find with grep: 
    $ grep "searching" $(find . -name "*.txt")
OUTPUT
./writing/LittleWomen.txt:sitting on the top step, affected to be searching for her book, but was
./writing/haiku.txt:With searching comes loss

Find: Locating Files and Directories

Introduction

find is a command-line utility used to search for files and directories that match specified criteria.

Basic Usage

To search for files or directories:

find path criteria

Examples

  1. Find All Files and Directories: 
    $ find .
OUTPUT
.
./writing
./writing/LittleWomen.txt
./writing/haiku.txt
./creatures
./creatures/basilisk.dat
./creatures/unicorn.dat
./creatures/minotaur.dat
./animal-counts
./animal-counts/animals.csv
./numbers.txt
./alkanes
./alkanes/ethane.pdb
./alkanes/propane.pdb
./alkanes/octane.pdb
./alkanes/pentane.pdb
./alkanes/methane.pdb
./alkanes/cubane.pdb
  2. Find Only Directories: 
    $ find . -type d
OUTPUT
.
./writing
./creatures
./animal-counts
./alkanes
  3. Find Only Files: 
    $ find . -type f
OUTPUT
./writing/LittleWomen.txt
./writing/haiku.txt
./creatures/basilisk.dat
./creatures/unicorn.dat
./creatures/minotaur.dat
./animal-counts/animals.csv
./numbers.txt
./alkanes/ethane.pdb
./alkanes/propane.pdb
./alkanes/octane.pdb
./alkanes/pentane.pdb
./alkanes/methane.pdb
./alkanes/cubane.pdb
  4. Find Files by Name: 
    $ find . -name "*.txt"
OUTPUT
./writing/LittleWomen.txt
./writing/haiku.txt
./numbers.txt

Combining Find with Other Commands

  1. Count Lines in Files Found by Find: 
    $ wc -l $(find . -name "*.txt")
OUTPUT
21022 ./writing/LittleWomen.txt
11 ./writing/haiku.txt
5 ./numbers.txt
21038 total
  2. Find Files and Count Lines with grep and find: 
    $ grep "searching" $(find . -name "*.txt")
OUTPUT
./writing/LittleWomen.txt:sitting on the top step, affected to be searching for her book, but was
./writing/haiku.txt:With searching comes loss

Handling Binary Files

Introduction

Binary files contain data in formats other than text. grep is designed for text and may not work well with binary files. For binary files, it’s often necessary to convert or extract text-like elements.

Examples

  1. Convert Binary Data to Text: Tools like strings can be used to extract printable text from binary files: 
    strings binaryfile | grep pattern
  2. Limitations: Complex data extraction from formats like spreadsheets or images may require specialized tools or programming languages.

Key Points

Feel free to ask if you need more details on any of these topics!

This site is open source. Improve this page.