Search This Blog

Thursday, July 20, 2023

s3cmd to access Google Cloud Storage Bucket

Step1:

Installing s3cmd on Redhat Linux 7

 

$ yum install wget

$ wget https://sourceforge.net/projects/s3tools/files/s3cmd/2.2.0/s3cmd-2.2.0.tar.gz

$ yum install unzip

$ tar -xvf s3cmd-2.2.0.tar.gz

$ cd s3cmd-2.2.0

$ yum install python3

$ sudo python3 setup.py install

 

Step2:

Create a Storage Bucket

 

Step3:

Create the HMAC Key using URL: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#console

 

 

test-storage-connectivity@playground-s-11-c77a0584.iam.gserviceaccount.com

Access key: GOOG1EGAHS#############UYFGTXNJPEE4LKSFQ

Secret:        e5bDpZeUFb#########357bV0dssTOHpMTtZ

 

Make Sure the service account is having "Storage Bucket Creator" Permission

 

Step4:

Configure the s3cmd

 

$ s3cmd --configure

 

 

New settings:

  Access Key: GOOG1EGAHS#############UYFGTXNJPEE4LKSFQ

  Secret Key: e5bDpZeUFb#########357bV0dssTOHpMTtZ

  Default Region: US

  S3 Endpoint: https://storage.googleapis.com

  DNS-style bucket+hostname:port template for accessing a bucket: y

  Encryption password:

  Path to GPG program: /usr/bin/gpg

  Use HTTPS protocol: True

  HTTP Proxy server name:

  HTTP Proxy server port: 0

 

Do not test the connectivity, just save the configuration

 

Step5:

Upload the file.

[root@testvm tmp]# s3cmd put ./test.txt s3://xyzcbucket-celumpub/test.txt

upload: './test.txt' -> 's3://fanlocbucket-celumpub/test.txt'  [1 of 1]

0 of 0     0% in    0s     0.00 B/s  done

Monday, June 25, 2018

Installing Ansible


1. Install RHEL EPEL repository. The EPEL (Extra Packages for Enterprise Linux) repository is a package repository for Red Hat Enterprise Linux (RHEL) or CentOS, maintained by people from Fedora Project community, to provide add-on packages from Fedora, which are not included in the commercially supported Red Hat product line.

# sudo yum -y update
# sudo yum -y install wget
# sudo rpm -Uvh epel-release-6*.rpm

For Redhat 7

# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# rpm -ivh epel-release-latest-7.noarch.rpm
# yum install epel-release

2. Install “Development tools” group. The “Development tools” are a yum group, which is a predefined bundle of software that can be installed at once, instead of having to install each application separately. The Development tools will allow you to build and compile software from source code. Tools for building RPMs are also included, as well as source code management tools like Git, SVN, and CVS.

# sudo yum groupinstall -y 'development tools'

3. Install python-pip and python-devel

# sudo yum -y install python-pip python-devel
# sudo pip install --upgrade pip

4. Upgrade setup tools

# sudo pip install setuptools --upgrade

5. Install Ansible via pip

# sudo pip install ansible
  
After the installation has completed successfully, you will be able to run this command to show your Ansible’s version number:

# ansible --version
ansible 2.1.1.0

To upgrade ansible to the latest version available in pip repository:

# sudo pip install ansible --upgrade

Thank you for reading.

Reading another article, visit “https://sites.google.com/site/unixwikis/

For Training, Drop an email to ibhagat@outlook.com 

Saturday, February 10, 2018

Complete Docker Installation on Multiple Platforms (CentOS/Red Hat)

Before enabling any repositories, be sure that you have installed the necessary required packages to support Docker but are not automatically installed as dependencies.


[root@vodafone-openstack1 ~]# yum install -y yum-utils device-mallerpper-persistent-data lvm2

Using the appropriate yum utility, add the Docker CE repository at https://download.docker.com/linux/centos/docker-ce.repo; update the local yum cache once added.


[root@vodafone-openstack1 ~]# yum-config-manager -add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo



[root@vodafone-openstack1 ~]# yum update
Loaded plugins: fastestmirror
docker-ce-stable                                                                                      | 2.9 kB  00:00:00
docker-ce-stable/x86_64/primary_db                                                                    |  11 kB  00:00:00
Loading mirror speeds from cached hostfile
 * base: mirror.ash.fastserv.com
 * epel: s3-mirror-us-east-1.fedoraproject.org
 * extras: mirror.net.cen.ct.gov
 * nux-dextop: mirror.li.nux.ro
 * updates: mirrors.advancedhosters.com

Execute the appropriate command to install the Docker CE application from the newly-configured repository.

[root@vodafone-openstack1 ~]# yum install docker-ce

Once installed, verify that the 'docker' group exists on your system. If not, create one. Once verified or created, add the 'user' account to that group so that sudo rights are not necessary to run Docker commands.

[user@tcox4 ~]$ cat /etc/group | grep docker
docker:x:988:


Running Command # docker images by non root user

root@vodafone-openstack1 ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

[root@vodafone-openstack1 ~]# exit
logout
[user@vodafone-openstack1 ~]$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/images/json: dial unix /var/run/docker.sock: connect: permission denied

[user@vodafone-openstack1 ~]$ sudo su -
[sudo] password for user:
Last login: Sat Feb  3 11:57:17 UTC 2018 on pts/1

[root@vodafone-openstack1 ~]# cd /var/run

[root@vodafone-openstack1 run]# ls -al docker.sock
srw-rw----. 1 root docker 0 Feb  3 12:20 docker.sock

[root@vodafone-openstack1 run]# usermod -aG docker user
[root@vodafone-openstack1 run]# exit
logout
[user@vodafone-openstack1 ~]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
[user@vodafone-openstack1 ~]$


Using the appropriate service management commands, enable the Docker CE service so that it starts automatically on boot, and then start the Docker CE service. Verify that it is running after. NOTE: You will need to log out and then back in for the new group setting above to work.


[root@vodafone-openstack1 ~]# systemctl enable docker && systemctl start docker && systemctl status docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2018-02-03 12:20:32 UTC; 64ms ago
     Docs: https://docs.docker.com
 Main PID: 18867 (dockerd)
   Memory: 24.3M
   CGroup: /system.slice/docker.service
           ─18867 /usr/bin/dockerd
           └─18870 docker-containerd --config /var/run/docker/containerd/containerd.toml

Feb 03 12:20:30 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:30.097970477Z" level=info ms...e]"
Feb 03 12:20:30 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:30.921110598Z" level=info ms...se"
Feb 03 12:20:31 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:31.176534012Z" level=info ms...ds"
Feb 03 12:20:31 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:31.177436342Z" level=info ms...t."
Feb 03 12:20:31 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:31.922590822Z" level=info ms...ss"
Feb 03 12:20:32 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:32.110700926Z" level=info ms...e."
Feb 03 12:20:32 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:32.174015008Z" level=info ms...-ce
Feb 03 12:20:32 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:32.174191177Z" level=info ms...on"
Feb 03 12:20:32 vodafone-openstack1.mylabserver.com dockerd[18867]: time="2018-02-03T12:20:32.254348813Z" level=info ms...ck"
Feb 03 12:20:32 vodafone-openstack1.mylabserver.com systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.



To test that the service is listening AND Step #4 was correctly completed, pull the default repository image called 'httpd' to your server.

[user@tcox4 ~]$ docker pull httpd
Using default tag: latest
latest: Pulling from library/httpd
aa18ad1a0d33: Pull complete
2b28e4afdec2: Pull complete
802b6cd5ed3b: Pull complete
6f2336b7c318: Pull complete
d7c441746c9e: Pull complete
a36c7f15867a: Pull complete
a0d42b9fc107: Pull complete
Digest: sha256:cf774f082e92e582d02acdb76dc84e61dcf5394a90f99119d1ae39bcecbff075
Status: Downloaded newer image for httpd:latest

Run the appropriate Docker CE command that will display a list of all images that are locally installed.


[user@tcox4 ~]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
httpd               latest              cf6b6d2e8463        42 hours ago        182MB

Sunday, December 25, 2016

Part-13-AWS-Network-VPC

VPC (Virtual Private Cloud)








·        Logically isolated AWS resource
·        Can be securely integrated with internal systems via a VPN
·        AWS is now defaulting to using VPC for EC2 instances
o   Older accounts can still launch into "EC2 classic"
Ø  Ability to simulate a private cloud within the public AWS infrastructure
o   Utilize AWS resources in an isolated virtual network
o   Provides complete control of the virtual network
§  IP address range, subnets, route tables, gateways, access control, etc.
§  Can control both inbound and outbound network traffic
Ø  No costs for using a VPC
o   Just pay for resources used
o   There is additional hourly charge for a VPN connection
Ø  As discussed earlier, EC2 is now requiring all instances in a VPC
Ø  Additionally, you can create a Hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate datacenter.

Hardware Tenancy

Ø  Can specify hardware tenancy
o   Default is multitenant
§  Hardware is potentially shared with many customers
o   Dedicated
§  Hardware is dedicated for instances running in your VPC
§  Additional fees apply
o   Host
§  Your instance runs on a Dedicated Host, which is an isolated server with configurations that you can control.
Ø  You cannot change the tenancy of a default instance after you've launched it. You can change the tenancy of an instance from dedicated to host after you've launched it, and vice versa. 

What can you do with a VPC?

·        Launch instances into a subnet of your choosing
·        Assign custom IP address ranges in each subnet
·        Configure route tables between subnets
·        Create internet gateways and attach them to subnets (or not)
·        Much better security control over your AWS resources
·        Instance security groups
·        Subnet network access control lists (ACLS)

Default VPC vs Custom VPC

·        Default VPC is user friendly, allowing you to immediately deploy instances.
·        All subnet in default VPC have an internet gateway attached.
·        Each EC2 instance has both a public and private IP address.
·        If you delete the default VPC the only way to get it back is to contact AWS.

VPC Peering

·        Allows you to connect one VPC with another via a direct network route using private IP addresses.
·        Instances behave as if they were on the same private network
·        You can peer PVC’s with other AWS accounts as well as with other VPCs in the same account.
·        Peering is in a star configuration, i.e. 1 Central VPC peers with 4 other. No Transitive Peering!!!





VPC Peering is simply a connection between two VPCs that enables you to route traffic between them using private IP addresses. Instances in either VPC can communicate with each other as if they within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account within a single region.
AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on a separated piece of physical hardware. There is no single point of failure for communication or a bandwidth bottleneck.



·        You cannot create a VPC peering connection between VPCs that have matching or overlapping CIDR blocks.
·        You cannot create a VPC peering connection between VPCs in different regions.
·        VPC peering does not support transitive peering relationship.

Multiple VPC Configurations


VPC with a Single Public Subnet Only

Ø  Used for servers that need to be accessed over the public Internet
o   Least private option
o   We have already used this during the class



VPC with Public and Private Subnets

Ø  Used when some servers need to be accessed over the Internet while other servers need to be kept private
o   Servers on the public subnet can be accessed from the Internet
o   Servers on the private subnet cannot be accessed from Internet
§  If needed, servers on private subnet can connect outbound to the Internet on the public subnet using Network Address Translation (NAT)



VPC with Public and Private Subnets, Hardware VPN Access

Ø  Used when some servers need to be accessed over the Internet while other servers need to be kept private
o   Servers on the public subnet can be accessed from the Internet
o   Servers on the private subnet can only be accessed over a hardware VPN connection
o   Used to extend your private data center into the cloud
o   Hardware VPN connection will make the VPC servers appear to be on your local private network


VPC with Private Subnet Only and Hardware VPN Access

Ø  Used to extend your private data center into the cloud
o   Can only access VPC resources from behind your private VPN firewall
o   Most private option
§  A hosted private cloud


Common Uses

Ø  VPC with a single public subnet only
o   Public-facing website
Ø  VPC with public and private subnets
o   Multitiered public-facing website
§  Web server in the publicly accessible subnet
§  Database and application servers in private subnet
Ø  VPC with public and private subnets and hardware VPN access
o   Web applications that require access to your internal data center
§  Public-facing web server running in AWS requires ability to access resources within your data center
Ø  VPC with a private subnet only and hardware VPN access
o   Seamlessly extend your data center to gain capacity required
o   Can use for disaster recovery
§  Utilize VPC to host redundant systems and data
§  A fraction of the cost of traditional disaster-recovery options

Configuring VPC

1. Create VPC


Log in to the AWS console.
Navigate to Services->VPC->Your VPCs.
Click “Create VPC”.
When you create a VPC, you specify a set of IP addresses in the form of a Classless Inter-Domain Routing (CIDR) block (for example, 10.0.0.0/16). For more information about CIDR notation and what "/16" means, see Classless Inter-Domain Routing.
You can assign a single CIDR block to a VPC. The allowed block size is between a /28 netmask and /16 netmask. In other words, the VPC can contain from 16 to 65,536 IP addresses.
You cannot change a VPC’s size after creating it. If your VPC is too small for your needs, you’ll need to terminate all of the instances in the VPC, delete it, and then create a new, larger VPC.
To create your VPC, go to the Create VPC dialog box, specify the following VPC details and then click “Yes, Create”.
CIDR Block: Specify the CIDR block for your VPC. I prefer 10.0.0.0/16.
Tenancy: Default tenancy: This is for running instances on shared hardware and is free of charge.
Dedicated Tenancy: This is for running your instances on single-tenant hardware. A $2 fee applies for each hour in which any dedicated instance is running in a region.



2. Create Subnets

In the navigation pane click on “Subnets”.
Click “Create Subnet”.
Before we create a subnet, let’s understand the best practices for creating them.
You should create subnets across multiple availability zones, with each subnet residing within a single zone. Creating subnets in and launching instances across multiple availability zones will ensure a high-availability environment.
For this example, we created subnets using zones us-east1a, user-east1b and us-east-1c. These subnets are called “private subnets” because the instances we launch are not accessible from the Internet. In other words, these instances don’t have a public IP unless you assign an EIP.
EC2: 10.0.1.0/24 (us-east1a), 10.0.2.0/24 (us-east1b), 10.0.3.0/24 (us-east1c)





3. Create Internet Gateway

By default, instances that are launched into a VPC can't communicate with the Internet. However, you can enable Internet access by attaching an Internet gateway to the VPC.
Go to Internet Gateways in the navigation pane and click “Create Internet Gateway”.





Now attach the gateway to a VPC by right clicking on “VPC” and selecting “Attach to VPC”. We should have one Internet Gateway per VPC.


4. Create Route Tables

A route table contains a set of rules, called routes that determine where network traffic is directed.
Each subnet in your VPC must be associated with a route table that will control that subnet’s routing. You can associate multiple subnets with a single route table; however, you can only associate a subnet with one route table.
Creating a VPC automatically creates a main route table which, by default, enables the instances in your VPC to communicate with one other.
Go to Route Tables in the navigation pane and click on “Create Route Table”.



Do not associate any subnets with the main route table.
Now navigate to the main route table to add a route to allow Internet traffic to the VPC.
Go to Routes and specify the following values:
Destination: 0.0.0.0/0
Target: Select “Internet Gateway” from the dropdown menu.


As a best practice create separate route tables for each tier. This will provide more control in maintaining the security of each subnet.
Now associate the subnets to the route tables.
Click on one route table and go to the Associations tab.
Select the subnet and click “Associate”.



 Here we had associate Subnet 10.0.1.0/24 as internet facing.

 5. Create an EC2 instance now.

So Subnet 10.0.1.0/24 will be our webserver which will have internet facing.
Subnet 10.0.2.0.24 will be our Database server which will not have internet connectivity.
We will create an EC2 instance in each subnet.
Creating EC2 instance for webserver in subnet 10.0.1.0/24
Step1: Choose an Amazon Machine Image(AMI) (Amazon Linux AMI 2016.03.3 (HVM), SSD Volume Type)
Step2: Choose an Instance Type (General Purpose t2 micro)
Step3: Configure Instance Details


Step4: Add Storage
Step5: Tag Instance (MyWebServer)
Step6: Configure Security Group
Create a new Security Group.



Step7: Review Instance Launch
Step8: Create an new key pair & Launch Instance.
Creating EC2 instance for Database Server in subnet 10.0.2.0/24
Step1: Choose an Amazon Machine Image(AMI) (Amazon Linux AMI 2016.03.3 (HVM), SSD Volume Type)
Step2: Choose an Instance Type (General Purpose t2 micro)
Step3: Configure Instance Details



Step4: Add Storage
Step5: Tag Instance (MyDBServer)
Step6: Configure Security Group
Select an existing SG MyVPCSG



Step7: Review Instance Launch
Step8: Create an new key pair & Launch Instance.
Now we could see that MyWebServer got PublicIP & MyDBserver do not have Public IP address. Now to test our setup, we will connect with putty to both the host, and try to check whether internet is working on which server.
I tried to connect to the host MyWebserver using Putty, and found that Internet is working on the host.
[root@ip-10-0-1-13 ~]# ping google.com
PING google.com (172.217.5.14) 56(84) bytes of data.
64 bytes from lga15s49-in-f14.1e100.net (172.217.5.14): icmp_seq=1 ttl=47 time=7.79 ms
64 bytes from lga15s49-in-f14.1e100.net (172.217.5.14): icmp_seq=2 ttl=47 time=7.89 ms
64 bytes from lga15s49-in-f14.1e100.net (172.217.5.14): icmp_seq=3 ttl=47 time=7.91 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 7.794/7.865/7.911/0.088 ms
[root@ip-10-0-1-13 ~]#
Now we will try to connect MyDB Server.
We are connecting to MyDBServer with Private IP address, as it has not having Public IP address
[root@ip-10-0-1-13 ~]# vi mynewkeypair.pem
[root@ip-10-0-1-13 ~]# chmod 600 mynewkeypair.pem
[root@ip-10-0-1-13 ~]# ssh ec2-user@10.0.2.181 -i mynewkeypair.pem
We could see that internet is not working on this server.
[ec2-user@ip-10-0-2-181 ~]$ ping google.com
PING google.com (172.217.1.78) 56(84) bytes of data.
^C
--- google.com ping statistics ---

8 packets transmitted, 0 received, 100% packet loss, time 7055ms