Distributed computing is a model in which components of a software system are shared among multiple computers to improve efficiency and performance. Linux, with its robust networking capabilities and open-source nature, is an excellent platform for setting up distributed computing environments. In this article, we will explore how to implement distributed computing on Linux using tools like MPI (Message Passing Interface) and Hadoop.

Examples:

Example 1: Setting Up a Simple MPI Cluster

MPI is a standardized and portable message-passing system designed to function on parallel computing architectures. Below is a step-by-step guide to setting up a simple MPI cluster on Linux.

1. Install MPI on all nodes:

   Use the following command to install OpenMPI, a popular MPI implementation:

   sudo apt-get update
   sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev

2. Configure SSH Access:

   Ensure that you can SSH into each node without a password. Generate SSH keys and copy them to each node:

   ssh-keygen -t rsa
   ssh-copy-id user@node_ip

3. Create a Host File:

   Create a file listing all the nodes in your cluster:

   echo "node1" > hosts
   echo "node2" >> hosts

4. Compile and Run an MPI Program:

   Create a simple MPI program:

   // hello.c
   #include <mpi.h>
   #include <stdio.h>
   
   int main(int argc, char** argv) {
      MPI_Init(NULL, NULL);
      int world_size;
      MPI_Comm_size(MPI_COMM_WORLD, &world_size);
      int world_rank;
      MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
      printf("Hello world from processor %d out of %d processors\n", world_rank, world_size);
      MPI_Finalize();
      return 0;
   }

   Compile and run the program:

   mpicc -o hello hello.c
   mpirun -np 4 --hostfile hosts ./hello

Example 2: Setting Up a Hadoop Cluster

Hadoop is another popular framework for distributed computing, particularly for processing large datasets.

1. Install Java:

   Hadoop requires Java. Install it using:

   sudo apt-get install openjdk-8-jdk

2. Download and Configure Hadoop:

   Download Hadoop from the Apache website and extract it:

   wget https://downloads.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
   tar -xzvf hadoop-3.3.1.tar.gz

   Configure Hadoop environment variables by editing ~/.bashrc:

   export HADOOP_HOME=/path/to/hadoop
   export PATH=$PATH:$HADOOP_HOME/bin

3. Configure Hadoop Files:

   Edit configuration files in the etc/hadoop directory, such as core-site.xml, hdfs-site.xml, and mapred-site.xml, to set up your cluster.

4. Start Hadoop Daemons:

   Format the HDFS filesystem and start the Hadoop daemons:

   hdfs namenode -format
   start-dfs.sh
   start-yarn.sh

5. Run a Sample Hadoop Job:

   Use the following command to run a sample Hadoop job:

   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar pi 16 1000